Two Sides of Intrusion Detection: Strengthening and Attacking Model-Based Detectors

Dr. Giffin is from the University of Wisconsin, Madison
Model-based anomaly detectors discover computer system attacks that cause malicious process execution. The detectors verify system calls invoked by a process against a model of expected behavior. Execution that deviates from the model indicates that the process is under an attacker's control. Existing model-based detectors produce false alarms, require manual effort, cause significant performance degradation, and miss attacks masked as normal execution. I will present a strong, usable intrusion detection system that addresses many of these deficiencies.

I eliminate false positives and the need for manual work by automatically extracting models using static binary program analysis. Statically-constructed models historically traded accuracy for detection speed. I will show that my Dyck model, a new stack-deterministic push-down automaton, eliminates the trade-off by reducing the complexity of accurate model enforcement from cubic time to linear time. The Dyck model pushes model-based detection into the realm of real-world feasibility.

I then evaluate the ability of a program model to detect intrusions. I find undetected attacks: malicious system call sequences erroneously allowed by a model as valid execution. Using model-checking, I automatically discover attacks previously found only with manual inspection of a program model. These undetected attacks demonstrate deficiencies of model-based detection that future research will need to address.