With the advancement and development of Internet of Things (IoT) applications, the need for securing infrastructure in this domain has gained particular importance due to the limitations of computational and storage resources. Botnets are among IoT security challenges in which, by infecting computational nodes of this technology, they are capble of turning the network into a collection of compromised machines under the control of attackers. This paper proposes an anomaly detection system based on ensemble learning to prevent and identify IoT botnet attacks at the initial scanning stage and DDoS attacks. This system uses feature selection and optimal hyperparameter tuning for each classifier to increase model accuracy and prevent overfitting. The data used in this paper is taken from the BoT-IoT dataset, which covers activities related to different stages of the botnet lifecycle. For feature selection and classification, two ensemble learning algorithms, LightGBM and Random Forest, are used, and hyperparameter optimization is performed using the TPE method. Results demonstrated that the LightGBM algorithm achieved an error rate of 0.98% and an accuracy of 99.02%, while the Random Forest algorithm exhibited an error rate of 0.01% and an accuracy of 99.99%, indicating highly satisfactory performance in attack detection. The proposed models, with increased training and prediction time, have offered significantly higher accuracy compared to previous models.