Citrix ADC: the gateway flaw that opened internal networks

On December 17, 2019, Citrix warned of a flaw in Citrix ADC and Citrix Gateway. It let anyone on the internet run commands on these devices without a password. The devices stand at the edge of many company and government networks. They let staff reach internal applications from outside and spread incoming traffic across servers. At first Citrix had no fix, only a workaround. The first fixes came 33 days later, on January 19, 2020. By then, attack code was public and attacks were under way around the world.

How it worked

Citrix Gateway has a built-in web server that shows users the login page. Behind it lies a folder named /vpns/ with small programs, Perl scripts, that should be reachable only after logging in. The flaw, CVE-2019-19781, is a path traversal. An attacker puts “../” into the address of a request, as in /vpn/../vpns/. This lets the attacker climb out of the public area and reach these scripts without logging in.

The CERT Coordination Center at Carnegie Mellon University described one way to use this. A first request makes one of the scripts write a file with hidden commands onto the device. A second request makes the device read that file as a template for a web page, and the commands run. The attacker then controls a device that sits between the internet and the internal network.

Citrix’s workaround was a rule that rejects requests containing “/../” or aimed at /vpns/. On some builds of version 12.1, a separate bug kept the rule from working. The flaw affected versions 10.5 to 13.0. In January 2020, Citrix found that its SD-WAN WANOP appliances were affected too. They contain Citrix ADC. Citrix ADC was formerly called NetScaler ADC, and today it is sold under that name again.

Reach and damage

The Dutch volunteer group DIVD searches the internet for vulnerable systems and warns their owners. According to DIVD, a worldwide scan in December 2019 found an estimated 128,777 vulnerable servers. The security company Positive Technologies, which had found the flaw, put the number of companies at risk at 80,000 in 158 countries. In Germany, the federal cyber security agency BSI had reported almost 5,000 vulnerable systems to network operators since January 7, 2020. On January 16, about 1,500 of them were still open, and the BSI had numerous reports of successful attacks.

On the night of January 10 to 11, 2020, several groups published working attack code. The SANS Internet Storm Center, an early warning service of the SANS Institute, saw “heavy exploitation” the next day. The security company FireEye found successful attacks by several groups before January 12. One of them deleted other attackers’ tools and blocked further attacks. But it kept a backdoor, a hidden way back in that only it could open with a secret key.

The update did not remove such backdoors. The Dutch security company Fox-IT found known signs of compromise on 1,030 servers in January 2020. By June 2020, it found them on 3,332, including servers of government organizations and Fortune 500 companies. Of the compromised servers in June, 23 % had been patched but still had a backdoor. In July 2021, agencies of the United States, the United Kingdom and Australia called CVE-2019-19781 the most exploited flaw of 2020.

The worst documented damage came months later. In September 2020, attackers encrypted about 30 servers of Düsseldorf University Hospital with ransomware. This malware locks data and demands money for its release. According to the BSI, they had entered through the hospital’s Citrix gateway. They had compromised it before the update was installed. The fixes were installed there from January 27, 2020. The hospital could not admit emergency patients for 13 days in a row. Its central emergency department reopened on September 23, 2020. The ransom note was addressed to the university to which the hospital belongs. The police told the attackers that they had hit a hospital. The attackers then handed over a key to restore the data.

On the night of September 11 to 12, a critically ill woman was to be brought to the hospital by ambulance. She was taken to a hospital further away in Wuppertal instead, with a delay of about 20 to 25 minutes, and died. Prosecutors concluded in January 2021 that she would very likely have died even with earlier treatment. So a link between the attack and her death could not be proven.

The fix and the response

Citrix released the fixes version by version:

Before the fixes, governments went further than usual. The Dutch National Cyber Security Centre (NCSC) raised its rating of the flaw to the highest level on December 24, 2019. It became unclear whether Citrix’s workaround was effective in all cases. On January 17, 2020, the NCSC urgently advised the government and operators of vital services to switch off their Citrix servers until a fix existed. The chief information officer of the Dutch government asked ministries to do the same unless strict conditions were met. According to the government, the first fixes covered about half of the vulnerable systems in the Netherlands.

In the United States, CISA, the federal agency for cyber security, released a free tool on January 13, 2020. It checks whether a device is vulnerable. On January 22, Citrix and FireEye released a scanner that looks for signs of attack. On January 31, CISA warned that unknown attackers had compromised numerous organizations. It also warned that a patch does not clean up an already compromised device. The BSI repeated this warning after the attack on the hospital in September 2020. On November 3, 2021, CISA added the flaw to its catalog of exploited vulnerabilities.

The National Vulnerability Database rates CVE-2019-19781 9.8 out of 10, “critical”. An attacker needs neither a password nor help from a user and can take full control of the device.