Today Microsoft is everywhere. Active Directory was the enterprise infrastructure backbone once, and still is. However, our dependence on Microsoft has grown ever more as the company has released new product offerings, including hardware and software, covering almost all areas of enterprise IT. They are now the de facto standard for enterprise email, communication, and collaboration for organizations of all sizes. And now they are becoming the staple for enterprise security, device management and business intelligence too. All this, besides being the number two CSP.
I personally like the Microsoft 365 set of products too. The convenience of one suite to bundle your M365 email, Office 365, Defender for endpoint, device, and cloud, etc. is very appealing. You can consolidate the necessities you need to run your business in one product suite and call it a day. Long story short, we have enjoyed the convenience and user-friendliness of the M365 suite.
However, because Microsoft is everywhere, they have become the target of choice for everyone. And Microsoft is not setting a great example here. With that in mind, I am writing this blog.
The CISA CSRB Report on the Microsoft Attack
We are a cybersecurity provider. And personally, given a choice between security and convenience, I will always pick security, but that’s just me. However, the landscape is changing, and the consequences are significant. We are not just talking about petty thieves anymore. There are nation state actors actively conducting cyber espionage and warfare. There’s a “Cyber Cold War” going on, and by the looks of it, we are not doing that great. So yes, a little inconvenience is a small price to pay in the larger scheme of things.
In April 2024, CISA released the CSRB review of the successful attack on Microsoft Exchange Online that was carried out in the summer of 2023 by Storm-0558, a Chinese APT. It is a short 34-page read, and I would recommend every cybersecurity professional read it.
To start with, I commend the CSRB for its work and CISA for releasing the report. And a big thank you to all contributors, and even to Microsoft for cooperating completely and openly with the CSRB.
And that is where my commendation for Microsoft stops. Here are a few excerpts from the CISA CSRB report (I am quoting the report verbatim):
- “The Board concludes that this intrusion should never have happened. Storm-0558 was able to succeed because of a cascade of security failures at Microsoft, as outlined in this report.”
- “In May and June 2023, a threat actor compromised the Microsoft Exchange Online mailboxes of 22 organizations and over 500 individuals around the world.”
- “In fact, when combined with another flaw in Microsoft’s authentication system, the key permitted Storm-0558 to gain full access to essentially any Exchange Online account anywhere in the world.”
- Observation: Microsoft’s failure to detect the compromise of its cryptographic crown jewels on its own, relying instead on a customer to reach out to identify anomalies the customer had observed.
- “Once the State Department alerted Microsoft to the intrusion, Microsoft did not have the logs or other forensic data to determine how or when Storm-0558 had stolen the key.”
- “As of the date of this report, Microsoft does not know how or when Storm-0558 obtained the signing key.”
- “The Board identified a series of Microsoft operational and strategic decisions that collectively point to a corporate culture that de-prioritized both enterprise security investments and rigorous risk management.”
- “The loss of a signing key is a serious problem, but the loss of a signing key through unknown means is far more significant because it means that the victim company does not know how its systems were infiltrated and whether the relevant vulnerabilities have been closed off.”
- “The Board is convinced that Microsoft should address its security culture.”
In fact, when combined with another flaw in Microsoft’s authentication system, the key permitted Storm-0558 to gain full access to essentially any Exchange Online account anywhere in the world.
And this is just a snapshot from the report. I will go into the details below.
How it Happened
Storm-0558 Token Abuse Attack on Microsoft Summer 2023 (Courtesy: CISA CSRB)
From the CSRB:
“On June 26, 2023, Microsoft discovered that the threat actor had used OWA to access emails directly using tokens that authenticated Storm-0558 as valid users. Such tokens should only come from Microsoft’s identity system, yet these had not. Moreover, tokens used by the threat actor had been digitally signed with a Microsoft Services Account (MSA) cryptographic key that Microsoft had issued in 2016. This particular MSA key should only have been able to sign tokens that worked in consumer OWA, not Enterprise Exchange Online. Finally, this 2016 MSA key was originally intended to be retired in March 2021, but its removal was delayed due to unforeseen challenges associated with hardening the consumer key systems. This was the moment that Microsoft realized it had major, overlapping problems: first, someone was using a Microsoft signing key to issue their own tokens; second, the 2016 MSA key in question was no longer supposed to be signing new tokens; and third, someone was using these consumer key-signed tokens to gain access to enterprise email accounts.”
Those are 3 strikes if you ask me!
How It Could Have Been Prevented
Again, this is from the CSRB:
If Microsoft:
had not paused manual rotation of keys;
had completed the migration of its MSA environment to rotate keys automatically;
had put in place a technical or other control to generate alerts for aging keys,
the 2016 MSA key would not have been valid in 2023.
Further, if Microsoft had not made the error that allowed consumer keys to authenticate to enterprise customer data (or, alternatively, if it had detected and addressed this flaw), the scope of the intrusion would have been far narrower and would not have impacted the State Department, Commerce Department, or any other enterprise customers.
If Microsoft had deployed alerting or prevention to detect forged tokens that do not conform to Microsoft’s own token generation algorithms, this incident likely could also have been stopped or detected by Microsoft all on its own.
Even after all this, if Microsoft had other security controls in place for its digital identity system—as the Board finds other CSPs had in place at the time—this intrusion vector could have been blocked or detected.
What Makes This Worse
This quote sums up the problem at Microsoft:
The decision to completely stop manual rotation of signing keys in 2021 after a large cloud outage, along with failing to prioritize the development of an automated key rotation solution, are troubling examples of decision-making processes within the company that did not prioritize security risk management at a level commensurate with the threat and with Microsoft technology’s vital importance to more than one billion of its customers worldwide.
And I always wondered why I need to pay extra for security logging. I am already trusting you with my IP and confidential data, and paying a decent amount for it. Why should I pay extra to check whether that trust is misplaced? The CSRB has similar thoughts:
“The Board notes that some CSPs, including Microsoft until recently, offer granular logging, which can be invaluable in security incident detection, investigation, and response—as a part of a paid package offering to their core services. This course of business should stop. Security-related logging should be a core element of cloud offerings and CSPs should provide customers the foundational tools that provide them with the information necessary to detect, prevent, or quantify an intrusion, recognizing that many customers will still require additional or third-party analytic capabilities to build a fully mature security program”.
Compared To AWS, GCP and OCI
We are not heavy AWS or GCP users internally, and we haven't used OCI yet. So the CSRB's comments on how these providers do things differently are a welcome insight. In short:
- Google Cloud: reworked its identity system around stateful tokens, each recorded at issuance as proof Google issued it. It added automatic key rotation, shortened the validity of stateless tokens, and protects these systems with Zero Trust networking and FIDO hardware-backed 2FA.
- AWS: IAM SigV4 gives each user or role its own keys, which are not bearer tokens. Each request is signed with derived, compartmentalized keys that authorize only that specific action, so there is no token to steal and replay.
- Oracle Cloud Infrastructure: requires each customer tenancy to have its own key pair that signs every request over TLS, in a way that resists token spoofing.
CSRB’s Recommendations for Microsoft
These are the CSRB's recommendations for Microsoft, summarized. I would have expected Microsoft to have them in place already:
- Per-customer keys: Microsoft used one key to sign tokens for all consumers, and, because of the validation flaw, enterprise customers too. Tying keys to each customer tenancy limits what a single stolen key can reach.
- Bound tokens: Microsoft's tokens were bearer tokens with no proof of possession. Binding tokens to a specific request or session stops token theft and replay.
- Common authentication libraries: different services validated tokens with different libraries, which made correct validation, and the cleanup, much harder.
- Secure key storage: keep key material in isolated systems, ideally dedicated HSMs.
- Linkable tokens: show customers, in their logs, which tokens came from the same sign-in, so identity attacks can be traced.
- Proprietary token markers: include data that helps a provider tell its own tokens from forged ones.
The CSRB also urges cloud providers to adopt emerging standards such as OAuth 2.0 Demonstrating Proof-of-Possession (DPoP) and OpenID Shared Signals and Events (SSE).
But Wait There's More
After the cyberattack carried out by Storm-0558 in May-June 2023, Microsoft ended up being a victim of another state-sponsored group. This time it was Midnight Blizzard, the Russian state-sponsored actor also known as NOBELIUM. Here's Microsoft's update on the attack.
This time the attackers used password spraying to get access to a legacy OAuth application in a test tenant which had excessive privileges to the Microsoft corporate environment. That gave the attackers access to the email accounts of senior Microsoft leaders. The target was email correspondence between Federal Civilian Executive Branch (FCEB) agencies and Microsoft. Beyond email, the attackers also accessed the company’s source code repositories and internal systems.
Thank you, Microsoft, for being transparent here.
According to CISA Emergency Directive 24-02 and Microsoft's own blog, Midnight Blizzard is still using information initially exfiltrated from the corporate email systems, including authentication details shared between Microsoft customers and Microsoft by email, to gain, or attempt to gain, additional access to Microsoft customer systems.
Midnight Blizzard’s successful compromise of Microsoft corporate email accounts and the exfiltration of correspondence between agencies and Microsoft presents a grave and unacceptable risk to agencies. - CISA Emergency Directive (ED 24-02: Mitigating the Significant Risk from Nation-State Compromise of Microsoft Corporate Email System)
Reevaluating Our Dependence on Microsoft
We are a Microsoft shop and so is most of the corporate world. And therein lies the problem.
Because Microsoft is so ubiquitous, so entrenched in our organizational IT infrastructure, it has become the biggest target for these state-sponsored attackers.
We, on the other hand, have become too complacent. M365 is everywhere. There's so much reliance on it that we have become borderline apathetic to such breaches. We say "it is not a question of if, but when". "Another day, another breach".
The question we must ask is: what are we doing, as customers, to make a difference? Maybe I am being naive, overreacting even, but I was hoping Microsoft, with its billions of dollars and thousands of engineers, would have been able to better stand up to this challenge.
Having said all that, I am now thinking that consolidating on a single vendor may not be a good strategy anymore. This isn't about Microsoft's products, which are good. It's about concentration risk: when one provider holds your identity, email, and security tooling, one compromise inside that provider reaches all of it. Maybe defense in depth means we don't put all our eggs in the same basket. Maybe, and I know it is going to be painful for almost all organizations, it’s time we rethink our software stack and diversify. Use different tools and different providers for different purposes. And not use a single identity stack if that stack can be compromised from within.
And maybe it is time to reevaluate our dependence on Microsoft.
I don't have the definite answers to these questions. But I know that during this Memorial Day 2024 weekend, we will be moving from M365 to Google Workspace. I am not saying Google Workspace will never be successfully attacked. But going by the recent record, I like our odds better.
The migration will be a pain. We will miss the ease of collaborating with our partners, clients and vendors who mostly use the M365 suite. And we will still need to buy Office 365 application licenses. In fact, I am writing this article on Microsoft Word.
However, I believe it’s time to take the path less taken.
We are a technology company. We always figure it out, and this time too we will find a way to make it work.
M365, we will miss you and wish you the best. Maybe our paths will cross again.