NTP monlist: how time servers amplified DDoS attacks

In December 2013, attackers began to misuse ordinary time servers for distributed denial-of-service (DDoS) attacks, which knock a target offline by flooding it with more traffic than its connection can carry. They used monlist, a diagnostic command in ntpd, the server software of the NTP project. NTP, the Network Time Protocol, keeps the clocks of computers and network devices in sync. A measurement study found vulnerable servers at 2.2 million internet addresses and counted 437,000 attacked addresses in 15 weeks.

How it worked

Time servers answer short requests from devices that want to set their clocks. For diagnostics, ntpd also kept a list of the clients that had contacted it most recently. The monlist command returned this list, up to 600 entries, to anyone who asked. According to US-CERT, the US government’s computer emergency team, older devices with NTP had the command switched on by default.

NTP uses UDP, a way of sending single data packets without first setting up a connection, so the server cannot check who really sent a request. An attacker writes the victim’s address into the request as the sender, and the server sends its answer there. Because a monlist answer can be far larger than the request, the attack is amplified: the attacker sends a little data, and the victim receives a lot.

US-CERT gives NTP an amplification factor of 556.9: the answer carries 556.9 times the data of the request, counting only the content of the packets. For DNS, the internet’s address system, it gives 28 to 54. The study described below counted whole packets and found that a typical vulnerable server returned about 4 times the data of the request, and a quarter at least 15 times. A few “mega amplifiers” answered a single packet with gigabytes, the largest with more than 136 gigabytes in one day, probably because of a network loop.

Because the floods came from thousands of legitimate servers, they were hard to block. The servers were not broken into; they were used as tools against others.

Reach and damage

The most detailed measurements come from “Taming the 800 Pound Gorilla”, a 2014 study by researchers at the University of Michigan, Merit Network, Colorado State University and the University of Illinois. It combines traffic data from the security firm Arbor Networks, which covers a third to a half of all internet traffic, with weekly scans of the internet and data from two regional networks.

In November 2013, NTP made up about 0.001 % of the traffic in Arbor’s data. By mid-February 2014 it had reached about 1 %, more than DNS, and it peaked on February 11. In February and March, most attacks of more than 2 Gbps (gigabits per second) used NTP; three months earlier, only 0.07 % of all attacks had involved it.

The Open NTP Project began to scan the whole internet every week on January 10, 2014. Its first scan found 1,405,186 servers that answered monlist requests. Over 15 weeks, the scans saw 2,166,097 different addresses, partly because 13–35 % of the servers were end-user computers, such as home PCs, whose addresses can change.

The monlist lists also record the forged sender addresses, that is, the victims. In 15 weeks, the researchers found 437,000 attacked addresses in 184 countries, hit with 2.92 trillion packets, about 1.2 petabytes of data. Because a list typically covered only the last two days before a scan, they estimate the real numbers at about 3.8 times higher. About half of the victims were end-user computers, and many attacks targeted the network ports, a kind of numbered door, that online games use; the authors conclude that a large share of the attacks was aimed at gamers.

On February 10, 2014, an attack on a target in Cloudflare’s network set a record for NTP at the time. Cloudflare, a company that protects websites against such attacks, reported that it peaked just under 400 Gbps and came from 4,529 NTP servers in 1,298 networks.

No total of the damage exists, and the sources document no outages of critical services such as hospitals or power supply. Networks that carried the floods paid too: at Merit, a regional network in Michigan, the attacks added more than 2 % to its total traffic, which raised the fees it paid for its own internet connection.

The fix and the response

The NTP developers had removed monlist from ntpd long before, in development version 4.2.7p26 of April 24, 2010, and replaced it with a newer command, mrulist. But the stable series, version 4.2.6, kept monlist; the first stable version without it, ntp 4.2.8, appeared only on December 19, 2014. On December 30, 2013, a request on a public security mailing list pointed to “a recent rash of DDoS involving the monlist functionality”, and the flaw received the number CVE-2013-5211 the same day.

In January 2014, CERT/CC at Carnegie Mellon University published a vulnerability note, and US-CERT an alert. Their advice: update ntpd, or switch off the queries with one setting in the configuration file. ICS-CERT, the US team for industrial control systems, warned as well.

The vulnerable servers disappeared quickly. Two weeks after the first scan, fewer than half were left. By March 21, about 110,000 remained, and the number hardly moved after that: on April 18, it stood at 106,445, 92 % fewer than in January. The study praises the “community response”: the Open NTP Project published its counts every week, researchers notified operators through CERTs (computer emergency response teams) and directly, and many networks felt the attacks in their own traffic. The authors consider it likely that the notifications helped, but say that cause and effect cannot be proven.

The cleanup was far faster than for open DNS resolvers, servers that look up addresses for anyone and can be abused in the same way. But it was not complete: about 4 million NTP servers still answered another query, version, which can also be abused, though rarely with such large replies. That group shrank by only 19 %.

The National Vulnerability Database rates the flaw 5.0 out of 10 on the older CVSS version 2, “medium”, and gives no score on version 3. The score is not higher because an attacker can neither read nor change data on the server: the flaw only affects availability, mostly that of others.