Heartbleed: behind the scenes at CSE

The story of Heartbleed – one of the most significant security bugs found since the advent of the commercial Internet – is fairly well recognized, especially in tech circles. Publicly disclosed on April 7, 2014, Heartbleed is a memory handling error in the popular OpenSSL cryptographic software library that allows for the stealing of information this is normally protected in secure communications with Internet servers and other nodes. Attackers can leverage Heartbleed to steal passwords and the keys intended to secure all types of sensitive personal, business, and government information.

What’s less known is how a team of computer science researchers at Michigan was able to rapidly pinpoint vulnerable servers on the Internet, quantifying Heartbleed’s scope and providing data on when and where servers were patched to repair the flaw.

This work, and related work that documented the nature of attacks on vulnerable test servers, is the immediate inside story of how the efforts to mend Heartbleed were focused and reported upon. And the story continues: the same researchers are now identifying the extent of vulnerabilities found in other Internet-connected devices, from home routers, firewalls, and printers to your connected television and refrigerator; to business equipment such as cash registers, industrial controls, and gas pumps; and infrastructure such as traffic lights.

When the news of Heartbleed broke, it was immediately seen for what it truly was: a catastrophic, widespread bug with serious security implications. According to Zakir Durumeric, a graduate student researcher in CSE, “There have been some other flaws of this nature detected previously, but they were all more difficult to exploit. This one essentially let anyone on the Internet go out and pull the most recent contents of memory from any affected server, with very few barriers to doing so. It’s much more dangerous.”

Large services, such as Gmail and Yahoo! mail, were quickly recognized to be vulnerable. Speculation about the scope of the problem caused some to estimate that as many as 66% of Internet servers might be compromised.

Durumeric realized that it was actually possible to quickly measure and quantify the extent of the Heartbleed flaw. He, along with fellow CSE graduate student Eric Wustrow and Prof. J. Alex Halderman, had in August 2013 released ZMap, a tool that can perform a scan of the entire public IPv4 address space on the Internet in less than 45 minutes, which is up to 1000 times faster than with traditional techniques.

On the advice of Halderman, Durumeric immediately dropped everything to track Heartbleed. “I basically lived here for a week. David Adrian, an undergraduate CS student, worked with me and was also here a lot.” The two raced to customize ZMap so that it would look for characteristics of vulnerable servers without actually probing or exploiting them.

On April 8 – just one day after the Heartbleed bug was made known – Durumeric, under advisement from Research Prof. Michael Bailey and Prof. Halderman, was able to perform an Internet-wide scan for Heartbleed vulnerabilities using ZMap. The research team chose to focus on the status of the one million most trafficked sites as tracked by Alexa, scanning them every eight hours to see which ones were patched and how quickly. They found that about 15% of Internet servers were actually compromised, not 66%. Durumeric posted the initial results of these scans in a blog posting.

Word of the blog posting and their work spread after Durumeric tweeted about it, and organizations began to contact the researchers. “They’d ask us which of their servers were vulnerable,” said Durumeric. “Others asked which of their customers were vulnerable so that they could start working with them. It wasn’t long before I had to remove Twitter from my phone because the onslaught of inquiries was so fierce.”

The researchers decided to post a list of vulnerable servers in a Heartbleed Bug Health Report, which provided a large portion of the data that was used by the media in reporting on vulnerable sites and when to change passwords that took place around Heartbleed.

At the same time, Durumeric, Wustrow, Bailey, and Halderman decided to answer the question of who if anyone was actually trying to take advantage of vulnerable servers. They deployed a number of “honey pots” – servers intentionally left vulnerable – on the Internet and watched to see what happened. Their work, still underway, has shown that hackers – including from China – very quickly sought out even these relatively obscure servers to exploit.

With the initial Heartbleed work behind them, the research team is now digging deeper with ZMap to find what other types of devices are vulnerable – connected devices such as routers, printers, and cameras. It appears the Internet still needs a patch.