The Internet has become a core component of our lives and businesses. Its reliability and availability are of paramount importance. There are many types of malware that impact the availability of the Internet, including network worms, bot-nets, viruses, etc. Detecting such attacks is a critical component of defending against them. This dissertation focuses on detecting and understanding self-propagating network worms, a type of malware with a proven record of disrupting the Internet. According to Computer Economics, the Code-Red worm caused more than 2.5 billion dollars in damages, and it was an unsophisticated worm that hit nearly 10 years ago when the Internet was less important than it is now. The recent StuxNet worm is a tremendously more sophisticated worm than Code-Red, and had it been targeted at disrupting the Internet it seems a near certainty that it could have caused significantly more damage than Code-Red. For this reason it is supremely important that we focus on detecting and stopping worms. Many worm detectors have been proposed and are being deployed, but the literature does not clearly indicate which one is best. New worms such as IKEE.B (also known as the iPhone worm) present new challenges to worm detection, raising the question of how effective our worm defenses are.

This dissertation studies the detection of self-propagating network worms with the goal of improving our ability to detect slowly propagating “stealthy” worms. We make the following contributions to the field: (i) we introduce a worm-detector evaluation framework that allows us to easily evaluate a detector’s performance across a variety of environments and worm types; (ii) we use this evaluation environment to compare existing worm detectors to determine their strengths and weaknesses; (iii) we examine evasive worms that attempt to avoid detection, measuring how effective they are at remaining undetected and the propagation rate they are able to achieve while doing so; and (iv) we introduce a new worm detector, SWORD2, which provides superior performance at detecting stealthy or evasive worms.

This dissertation includes previously published co-authored material.