Abstarct: The increasing growth of Internet usage, has considerably motivated attackers to develop cybercrime. In recent decade, network security and network traffic monitoring have been significantly important due to these expanded new threats. In general, traffic flows can occur due to two main reason; malicious purpose and attacks, or benign purpose. Traffic flows are classified regarding their purposes. Botnets have been recently recognized as the most formidable threats on the Internet. Different approaches have been proposed for detecting these types of attacks; the most effective approaches are baxsed on machine learning. One of the main reasons for the trend towards these methods is their strength of generalization to identify new types of botnets. Because of the importance of botnets in the recent decade, in this research, a self-learning botnet detection system consisting of incremental learning, has been proposed baxsed on traffic classification. This system classifies traffic flows according to their application; botnet or benign. An incremental training is conducted and the system updates its classifier continuously, regarding the new samples to obtain more capacity of generalization. In addition to pursue the learning process, like the other online methods, this system is capable of using the new incoming samples in its classifier, without knowing their real label; this is because it can predict the labels fairly precisely. Moreover, in order to achieve a valid evaluation of the system performance, which is rarely found in previous studies, the system has been evaluated with a comprehensive data set that has a wide variety of botnets. The experiments results and comparisons demonstrate that the proposed system can perform properly in a dynamic environment. The Maximum improvement rate which is provided by this system is 13% in botnet detection rate.