Citrix ADC: the gateway flaw that opened internal networks

On December 17, 2019, Citrix warned of a flaw in Citrix ADC and Citrix Gateway that let anyone on the internet run commands on these devices without a password. The devices stand at the edge of many company and government networks: they let staff reach internal applications from outside and spread incoming traffic across servers. At first Citrix had no fix, only a workaround. The first fixes came 33 days later, on January 19, 2020. By then, attack code was public and attacks were under way around the world.

How it worked

Citrix Gateway has a built-in web server that shows users the login page. Behind it lies a folder named /vpns/ with small programs, Perl scripts, that should be reachable only after logging in. The flaw, CVE-2019-19781, is a path traversal: an attacker who puts “../” into the address of a request, as in /vpn/../vpns/, climbs out of the public area and reaches these scripts without logging in.

The CERT Coordination Center at Carnegie Mellon University described one way to use this. A first request makes one of the scripts write a file with hidden commands onto the device. A second request makes the device’s template engine, the part that builds web pages from templates, read that file, and the commands run. The attacker then controls a device that sits between the internet and the internal network.

Citrix’s workaround was a rule that rejects requests that contain “/../” or aim at /vpns/. On some builds of version 12.1, a separate bug kept the rule from working. The flaw affected versions 10.5 to 13.0, and in January 2020 Citrix found that its SD-WAN WANOP appliances were affected too, because they contain Citrix ADC. Citrix ADC was formerly called NetScaler ADC, and today it is sold under that name again.

Reach and damage

The Dutch volunteer group DIVD, which searches the internet for vulnerable systems and warns their owners, reports that a worldwide scan in December 2019 found an estimated 128,777 vulnerable servers. The security company Positive Technologies, which had found the flaw, put the number of companies at risk at 80,000 in 158 countries. In Germany, the federal cyber security agency BSI had reported almost 5,000 vulnerable systems to network operators since January 7, 2020. On January 16, about 1,500 of them were still open, and the BSI had numerous reports of successful attacks.

On the night of January 10 to 11, 2020, several groups published working attack code. The SANS Internet Storm Center, an early warning service of the SANS Institute, saw “heavy exploitation” the next day. The security company FireEye found successful attacks by several groups before January 12. One of them tidied up behind itself: it deleted other attackers’ tools and blocked further attacks, but kept a backdoor, a hidden way back in, that only it could open with a secret key.

The update did not remove such backdoors. The Dutch security company Fox-IT found known signs of compromise on 1,030 servers in January 2020 and on 3,332 by June 2020, including servers of government organizations and Fortune 500 companies. Of the compromised servers in June, 23 % had been patched but still had a backdoor. In July 2021, agencies of the United States, the United Kingdom and Australia called CVE-2019-19781 the most exploited flaw of 2020.

The worst documented damage came months later. In September 2020, attackers encrypted about 30 servers of Düsseldorf University Hospital with ransomware, malware that locks data and demands money for its release. According to the BSI, they had entered through the hospital’s Citrix gateway, which they had compromised before the update was installed; the hospital had installed the fixes from January 27, 2020. The hospital could not admit emergency patients and was off emergency care for 13 days in a row; its central emergency department reopened on September 23, 2020. The ransom note was addressed to the university to which the hospital belongs. When the police told the attackers that they had hit a hospital, they handed over a key to restore the data.

On the night of September 11 to 12, a critically ill woman who was to be brought to the hospital by ambulance was taken to a hospital further away in Wuppertal instead, with a delay of about 20 to 25 minutes, and died. Prosecutors concluded in January 2021 that she would very likely have died even with earlier treatment, so a link between the attack and her death could not be proven.

The fix and the response

Citrix released the fixes version by version:

Before the fixes, governments went further than usual. The Dutch National Cyber Security Centre (NCSC) raised its rating of the flaw to the highest level on December 24, 2019. When it became unclear whether Citrix’s workaround was effective in all cases, the NCSC urgently advised the government and operators of vital services on January 17, 2020, to switch off their Citrix servers until a fix existed. The chief information officer of the Dutch government asked ministries to do the same unless strict conditions were met. According to the government, the first fixes covered about half of the vulnerable systems in the Netherlands.

In the United States, CISA, the federal agency for cyber security, released a free tool on January 13, 2020, that checks whether a device is vulnerable. On January 22, Citrix and FireEye released a scanner that looks for signs of attack. On January 31, CISA warned that unknown attackers had compromised numerous organizations and that a patch does not clean up a device that was already compromised. The BSI repeated this warning after the attack on the hospital in September 2020. On November 3, 2021, CISA added the flaw to its catalog of exploited vulnerabilities.

The National Vulnerability Database rates CVE-2019-19781 9.8 out of 10, “critical”: an attacker needs neither a password nor help from a user and can take full control of the device.