On Detection of Current and Next-Generation Botnets

Botnets are one of the most serious security threats to the Internet and its end users. A botnet consists of compromised computers that are remotely coordinated by a botmaster under a Command and Control (C&C) infrastructure. Driven by financial incentives, botmasters leverage botnets to conduct various cybercrimes such as spamming, phishing, identity theft and Distributed-Denial-of-Service (DDoS) attacks.
There are three main challenges facing botnet detection. First, code polymorphism and obfuscation is a technique widely employed by current botnets, so a signature-based detection alone could be hampered in the presence of a large number of bot variants. Second, the C&C infrastructure of botnets has evolved recently. Traditional botnets utilize centralized IRC-based or HTTP-based C&C mechanisms, which are prone to disruption if the central servers are identified and neutralized. To be more resilient, attackers have recently shifted towards the decentralized C&C such as using P2P infrastructures. Evidently, any detection solution designed specifically for one particular botnet can hardly keep up with the advancement of botnets. Third, current botnets are built primarily upon personal computers, and the proliferation of intelligent and powerful smartphones presents a new platform for future botnet construction. Defense techniques targeting existing botnets are likely to be outsmarted when botnets invade smartphones.
Recognizing these challenges, this dissertation proposes behavior-based botnet detection solutions from a small scale to a large scale at three different levels — the host, the edge network and the Internet infrastructure, and investigates the next-generation botnet targeting smartphones. First, it addresses the problem of botnet seeding by devising a per-process containment scheme for host systems. Second, it proposes a botnet detection framework for edge networks utilizing combined host- and network-level information. The framework is able to detect multiple botnets with different C&C structures. Third, to address the scalability issue, the dissertation explores the structural properties of botnet topologies from a graph perspective and measures different network components' capabilities of large-scale botnet detection at the Internet infrastructure level. Finally, the dissertation presents a proof-of-concept mobile botnet employing SMS messages as the C&C and P2P as the topology to facilitate future research on countermeasures against next-generation botnets.