Heartbleed: the flaw that let attackers read server memory

On April 7, 2014, the OpenSSL project fixed a flaw that let anyone on the internet read pieces of a server’s memory, including passwords and sometimes the server’s secret keys. OpenSSL is a software building block that many web servers, mail servers and devices use to encrypt their connections, for example for HTTPS, the secure form of the web. A measurement study estimates that 24–55 % of the HTTPS servers of the million most visited websites were vulnerable at first.

How it worked

Encrypted connections on the web use a protocol called TLS. It has a “heartbeat” feature that keeps a connection open: one side sends a small message together with a number that says how long the message is, and the other side sends the message back.

OpenSSL trusted that number and never compared it with the real length. An attacker could send a message of one byte but claim that it was 65,535 bytes long. The server then copied up to 64 KB of whatever lay next to the message in its memory into the reply. That could include session cookies, which keep users logged in, passwords, and sometimes the server’s private key, the secret that protects its encrypted connections.

The request left no trace in normal logs and could be repeated again and again, each time for a new piece of memory. The same bug also worked the other way round, against programs that connected to a malicious server. The faulty code was added to OpenSSL on December 31, 2011, and released in version 1.0.1 on March 14, 2012, so it was in use for about two years before it was found.

Reach and damage

The best measurements come from “The Matter of Heartbleed”, a 2014 study by researchers at the University of Michigan, the University of California Berkeley, the research institute ICSI and Purdue University. It uses the Alexa Top 1 Million, a ranking of the million most visited websites. The authors estimate that 24–55 % of the HTTPS servers on that list were vulnerable when the flaw became public, including at least 44 of the top 100 sites. One of the finders had first spoken of 66 %, based on the market share of the web servers Apache and nginx; the study calls this too high.

Large sites patched fast: within 24 hours, all but 5 of the top 100 had installed the fix. Two days after disclosure, 11.5 % of the HTTPS sites in the Alexa Top 1 Million were still vulnerable, as were 6 % of all HTTPS servers on the internet, about 2 million machines. After about two weeks, the decline almost stopped.

Websites were not the only systems at risk. In a sample of 1 % of the internet, 7.5 % of the mail servers with encryption were vulnerable. Three days after disclosure, 48 % of the relays of Tor, a network for anonymous browsing, were still vulnerable. A full scan of the internet on April 24, 2014, found 588,686 vulnerable machines; 56 % of them were embedded devices, small devices with built-in software.

The researchers also searched recorded traffic from four monitored networks for attacks. They found no attempt before the flaw was made public. The first came 21 hours and 29 minutes after disclosure, and in total they counted 5,948 attempts from 692 sources against 217 targets. How much data was stolen worldwide is unknown. The best-known case comes from the press: the Canada Revenue Agency, Canada’s tax authority, said that 900 social insurance numbers were taken through Heartbleed. No reliable total of the damage exists.

The fix and the response

Google’s security team found the flaw on March 21, 2014, and patched its own servers the same day; it told the OpenSSL team on April 1. The Finnish security firm Codenomicon found the flaw independently in early April and reported it through NCSC-FI, Finland’s national cyber security center. On April 7, OpenSSL released version 1.0.1g. The fix is simple: OpenSSL now drops heartbeat messages that claim to be longer than the data they contain.

Updating alone was not enough, because an attacker could have copied a server’s private key without leaving a trace. The US-CERT, the US government’s computer emergency team, warned: “Any keys generated with a vulnerable version of OpenSSL should be considered compromised.” Operators had to create new keys, get new certificates, the digital ID cards of websites, and revoke the old ones, so that browsers would stop trusting them. Between April 9 and April 30, 23 % of all HTTPS sites in the Alexa Top 1 Million replaced their certificates and 4 % revoked them.

Sites that were slow to patch were also slow to replace their keys:

Of the vulnerable sites that did replace their certificate, 14 % reused the same private key, which gave no protection, and only 19 % revoked the old certificate. Warnings helped: in an experiment with about 150,000 vulnerable machines, operators whom the researchers notified patched 47 % more than those who were not notified.

The National Vulnerability Database rates Heartbleed 7.5 out of 10 on CVSS 3.1, “high” rather than “critical”, because the flaw lets attackers read data but not change it or shut down a server.