In this post I will explain how the Microsoft Azure Virtual Machine (VM) extension works and how we found a fatal vulnerability in the extension mechanism affecting Azure VM Linux systems. As part of the responsible disclosure policy, we reported the vulnerability to Microsoft Security Response Center (MSRC). They soon patched and assigned it CVE-2021-27075.
This vulnerability would have allowed an unprivileged user to leak any Azure VM extension’s private data. Paired with the design of the VMAccess extension, an official Azure extension built for assisting system admins, we will demonstrate how this could have been used to achieve privilege escalation and possibly lateral movement.
Proof of Concept
Azure VM Extensions
Azure VM offers developers and admins an integrated plugin system to install additional components onto their machines. Both first party (e.g., Microsoft Azure Diagnostics and Microsoft Azure Network Watcher) and third party apps (like Datadog Agent) are offered through this mechanism.
To manage extension installations and keep them up to date, Microsoft Azure Guest Agent is installed on the system. This component is open-source and hosted on GitHub. This agent, along with Azure extensions, is installed at /var/lib/waagent. This directory is inaccessible for non-root users as it contains extension-related secrets.
Extensions are installed in this directory in their own sub directories, e.g., /var/lib/waagent/Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux-1.4.1587.1 Many shared configurations are laid out in the root extension directory /var/lib/waagent.
Here is an example of the directory layout:
/var/lib/waagent directory structure
When an extension is added to the VM, the extension’s configuration file is updated behind the scenes in the Azure VM manager, also known as the Fabric Controller. The WAAgent constantly polls the Fabric Controller for this file, and once updated, the extension is downloaded and deployed. In Microsoft Azure Cloud, the WAAgent communicates with the ‘Wire Server,’ an HTTP service belonging to the Fabric Controller.
WAAgent communicates with the Fabric Controller by accessing a special IP address: 18.104.22.168. Through some testing, we discovered that this endpoint is the same as 169.254.169.254, better known as the Azure Instance Metadata Service IP address.
The WAAgent receives the extension configuration URL by parsing the ‘GoalState.’
GoalState endpoint request
A response looks like this:
GoalState endpoint response
The GoalState contains URLs to all relevant configurations available to the Wire Server. Using the GoalState we can query the ExtensionsConfig file ourselves:
ExtensionsConfig endpoint request
Here is an example of how the LinuxDiagnostic extension configuration might look:
ExtensionsConfig endpoint response
The protectedSettings field holds sensitive extension configurations such as private keys and is additionally protected with an encryption scheme. The protectedSettingsCertThumbprint field holds the filename of the key used to decrypt the protectedSettings. This key is stored in /var/lib/waagent/F54265F38F8D16C35C0E1FD3190882831A6C4384.prv with its certificate stored in /var/lib/waagent/F54265F38F8D16C35C0E1FD3190882831A6C4384.crt.
The private key and certificate pair are not supplied by the ExtensionConfig file. So, how do they get deployed to the server?
For this deployment, the Certificates endpoint is used. After reverse engineering the WAAgent communication with the Wire Server, we observed that the Certificates endpoint requires a ‘transport’ certificate which is used to supply the F542… extension key:
Certificates endpoint request
The Wire Server returns an encrypted form of the extension key which can be decrypted via the Transport Certificate’s private key.
Certificates endpoint response
Leaking Azure VM Extensions’ Private Data
Flaw #1: Certificates Endpoint Does Not Validate Transport Certificate
An attacker can create their own Transport Private Key and its corresponding Transport Certificate. Using the Certificate endpoint, the attacker supplies their own Transport Certificate and receives an encrypted form of the keys from the Wire Server (in our example this is the F54265F38F8D16C35C0E1FD3190882831A6C4384 key).
Finally, the encrypted keys are decrypted via the Transport Key and the attacker can proceed to decrypt the protectedSettings:
To our surprise, after developing the PoC with the root user, it would not work with an unprivileged user. It seems that packets destined to the Wire Server endpoint at 22.214.171.124 were not sent out by the server. This is because of an iptables rule that drops packets which aren’t from user ID 0 (root) to the endpoint:
Azure VM IPTables
Flaw #2: Bypassing Wire Server Unprivileged Access Defense
As mentioned earlier, we discovered that 126.96.36.199 is the same machine as 188.8.131.52. We replaced every request to 184.108.40.206 with 220.127.116.11 and this allowed us to communicate with the Wire Server without a privileged user.
In addition, this iptables defense did not apply to processes that run in Docker containers (even when the PoC ran as an unprivileged user), allowing containers to leak information about their host through the Wire Server. This issue was fixed by MSRC as well.
Combining the Flaws
Utilizing both flaws, an unprivileged user can leak any Azure VM extension’s private settings. This is especially dangerous when paired with extensions that handle sensitive data.
A particularly severe example is the VMAccess extension, an official Microsoft Azure extension used for changing passwords conveniently on controlled machines. Guardicore previously documented that VMAccess persists passwords in the protectedSettings field even after it’s done changing the user’s password and no longer needs to keep it on disk.
Combined with the vulnerability we found, an attacker would be able to elevate themself to a higher privileged user by leaking the VMAccess admin password. Also, in the event that the VMAccess password is shared with other Azure VMs (as is often the case), the attacker could perform lateral movement across the system. This is what the flow looks like:
And the proof of concept:
Leaking VMAccess extension data
The CVE issued by Microsoft also applies to other Azure products such as Azure Spring Cloud, as discovered by researcher Wouter ter Maat independently at a later date, who was also credited in the CVE.
Microsoft fixed the issue after revamping the whole Linux extension mechanism and no user interaction is needed to update VMs.
This research is meant to further the discussion around the relationship between cloud service providers (CSPs) and their customers. Ultimately, the customer is responsible for any data breach that occurs. For a more complete cloud security strategy, Intezer recommends adopting a two-pronged approach. Do the basics, like fixing known vulnerabilities and hardening your systems to reduce the likelihood of getting attacked. You also need runtime threat detection to identify and respond to attacks as they occur following unknown vulnerability exploitation or a backdoor in the supply chain.
Intezer Protect can help reduce the attack surface while detecting and responding to attacks in runtime. Try Intezer Protect for free on up to 10 hosts.