Iago Attacks: Why The System Call API Is a Bad Untrusted RPC Interface

In recent years, researchers have proposed systems for running
trusted code on an untrusted operating system. Protection
mechanisms deployed by such systems keep a malicious kernel from
directly manipulating a trusted application's state. Under such
systems, the application and kernel are, conceptually, peers, and
the system call API defines an RPC interface between them.

We introduce Iago attacks, attacks that a malicious kernel can
mount in this model. We show how a carefully chosen sequence of
integer return values to Linux system calls can lead a supposedly
protected process to act against its interests, and even to
undertake arbitrary computation at the malicious kernel's behest.

Iago attacks are evidence that protecting applications from
malicious kernels is more difficult than previously realized.

Joint work with Stephen Checkoway.
Hovav Shacham joined UC San Diego's Department of Computer Science and Engineering in Fall 2007. Shacham received his Ph.D. in computer science in 2005 from Stanford University, where he had also earned, in 2000, an A.B. in English. His Ph.D. advisor was Dan Boneh. His thesis, "New Paradigms in Signature Schemes," was runner up for the Stanford Department of Computer Science's Arthur L. Samuel Thesis Award, and was nominated for the ACM Doctoral Dissertation Competition. In 2006 and 2007, he was a Koshland Scholars Program postdoctoral fellow at the Weizmann Institute of Science, hosted by Moni Naor. At the Weizmann, Shacham taught a survey on pairings in cryptography, one of the first such courses to be offered. In 2007, Shacham participated in California Secretary of State Debra Bowen's "Top-to-Bottom" of the voting machines certified for use in California. He was a member of the team reviewing Hart InterCivic source code; the report he co-authored was cited by the Secretary in her decision to withdraw approval from Hart voting machines.