Cloudflare, which operates a widely used web content delivery network, announced a security bug on February 23 that caused sensitive data to leak from its customers’ websites. The exact number of websites potentially affected is unknown but some estimates place the total in excess of 5 million. The Google security researcher who discovered the bug – nicknaming it “Cloudbleed” after the 2014 Heartbleed bug – reported it to Cloudflare on February 18, 2017. Cloudflare disabled the compromised software and stopped the leak later the same day.
The leaked data reportedly included passwords, private messages, encryption keys, session cookies that would let an attacker log into an account without a password, IP addresses and other data. Leaked data was exposed to search engine crawlers, which began to automatically cache the data, thus complicating remediation.
As of this writing there have been no publicized reports that leaked data has been exploited and Cloudflare has published analysis concluding that the vast majority of its customers probably were not affected. However, operators of millions of websites and their users are left to wonder whether they were affected and what they should do next.
Below is a summary of what we know now and our thoughts on next steps.
What is Cloudflare?
Cloudflare makes a web content delivery product used by 6 million customers to enhance website performance and security. When you visit a website in Cloudflare’s network, your request for the site is automatically routed to Cloudflare, which uses routing techniques and its own copy of the site’s static content to load the site faster than it would conventionally.
Cloudflare also offers features designed to enhance the security of web content, such as rewriting unencrypted http content to encrypted https, using “server-side exclude” technology to ensure data is seen only by its intended audience, and obfuscating email addresses.
What does the Cloudbleed bug do?
The bug was found in a parser used to power three security features – https rewrites, server-side excludes, and email obfuscation. To execute these features, Cloudflare saves website content and data to memory for parsing. The bug caused this data to leak – at random – into code of web pages in the Cloudflare network such that when you visited a web page, that page would include leaked data from an entirely different Cloudflare-supported website.
What type of information was leaked?
The Google researcher who discovered the bug gave this report:
I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.
Cloudflare’s CTO initially reported that end-user passwords, authentication cookies, OAuth tokens used to log into multiple website accounts, and encryption keys were at risk of exposure. In its most recent blog post, Cloudflare reports that it has not yet found any instances of passwords, credit cards or health records among leaked data but that leakage of this and other sensitive data cannot be ruled out. In addition, Cloudflare has emphasized that leakage occurred randomly and leaks would include a mixed bag of both potentially sensitive data and useless non-sensitive “noise”.
Where did leaked data go?
Leaked data could be stored in the browser caches of users who unwittingly downloaded leaked data, was cached by search engines like Google, Bing and Yahoo, and may have been saved by other bots that roam the Internet. While Cloudflare worked with search engines to remove 770 cached instances of leaked data from 161 different domains before announcing the bug and was unable to find leaked data on sites like Pastebin, researchers subsequently reported that leaked sensitive data is still discoverable in search engines.
When did the leak become active?
As early as Sept. 22, 2016, when the https rewrite feature was first enabled.
The period of greatest exposure was between February 13, 2017 (when the email obfuscation feature was migrated to the compromised parser) and February 18, 2017 (when the compromised features were disabled).
How many Cloudflare customers were affected?
Cloudflare has not provided an official estimate but its latest blog post reports that it found data of approximately 150 Cloudflare customers among the more than 80,000 cached pages that have been purged by the search engines. The post provides some useful analysis of the probability of a leak based on a website’s level of traffic. For example, a site that made 250-500 million requests to the Cloudflare network per month is expected to have leaked 25-56 times. Cloudflare estimates that the 99% of its customers sending fewer than 10 million requests per month probably had no leak at all. The post also reports that a maximum of 6,457 websites could have triggered the bug. However, because the websites that triggered the bug pulled information from other websites in Cloudflare’s network, the number of affected Cloudflare customers is unknown and could be much higher.
Has any data been exploited?
Cloudlfare reports that it has found no evidence of the bug being exploited before it was patched.
What is the risk to individuals?
The risk to specific individuals is difficult to evaluate at this stage. The good news is that Cloudflare acted quickly to remediate the bug and purge known instances of leaked data from search engine caches—all before any reported instances of the bug being discovered or exploited by malicious actors. In addition, the random distribution of leaked data across the Internet may limit the kind of accumulation of data in one place that would make it easier for a malicious actor to exploit it at scale. Finally, according to Cloudflare’s analysis, the risk of a leak appears to be low for 99% of its customers.
The bad news is that some of the leaked data – including passwords, encryption keys, authentication tokens and conversations – is clearly sensitive and potentially exploitable to the extent it is still discoverable in search engine caches or elsewhere.
What should companies do now?
Like the Heartbleed bug before it, Cloudbleed is the latest internet security bug to expose a wide swath of the Internet to potential data leaks while in most cases leaving no way to conclusively confirm whether or not a particular company’s or individual’s data was leaked or exploited. While Cloudbleed may pose low risk to most websites according to Cloudflare, its customers should take this news seriously given the sensitivity of the data exposed and the media attention that the bug has attracted. Following some basic incident response best practices can help companies mitigate risk and assure customers and partners that appropriate steps are being taken.
- Evaluate the impact. Companies that use Cloudflare should evaluate the impact to their own websites and any potentially sensitive data that those sites process, while continuing to follow new developments. A careful reading of Cloudflare’s initial and follow-up posts, as well as the post by the Google security researcher who discovered the bug, followed by inquiries with Cloudflare, are a good start.
- Take mitigating action. Because the Cloudbleed bug has been patched, efforts should focus on mitigating risk to potentially impacted individuals. Cloudflare has recommended that concerned customers invalidate and reissue persistent secrets, such as long lived session identifiers, tokens or keys (the company says that customer SSL keys were not exposed and do not need to be rotated). Other options include:
- Informing customers about Cloudbleed and its potential impact.
- Recommending that customers change passwords and use two-factor authentication to protect accounts
- Forcing a change of administrator credentials for potentially impacted sites
- Forcing a change of customer passwords
- Requiring customers to log back into websites without changing passwords (if not already required by invalidating session identifiers)
The right approach will vary for each company based on its own business, the operational costs of making these changes, the sensitivity of the data it handles and the probability of data leakage based on the volume of traffic it sends through Cloudflare. Companies should perform their own risk assessments in determining the appropriate mitigating steps, weighing both the probability that leaked data could be exploited and the potential impact to the company and individuals if it is. It is a good idea to document this analysis as part of your incident response process (discussed below).
- Search for your data. Although Cloudflare took steps to purge leaked data from search engine caches, there have been reports on social media that leaked data remained discoverable after Cloudlfare’s purge. Thus, potentially affected companies should make a reasonable effort to discover whether their own data is still searchable as part of their incident response efforts. Whether these searches are performed with the assistance of security incident response vendors or in-house, it is advisable to document the methodology used for the search, why it is believed to be sound and the results of the searches.
- Develop a communications strategy. Cloudbleed has attracted significant media attention and it is only a matter of time before companies are asked whether they are affected. Proactively communicating your company’s response to Cloudbleed and efforts to investigate can help alleviate concerns and demonstrate that your company takes security seriously. This message can be relayed through support emails, customer notices or talking points tailored to customers or other external parties who may inquire. However, like any external communication relating to an information security incident, these messages should be carefully crafted with the assistance of legal counsel and other relevant internal stakeholders before distribution.
- Consider security incident notice obligations. Companies should consult legal counsel to assess whether even potential leakage of data triggers breach notification obligations under legal or contractual obligations. While most legal breach notification requirements would not be triggered by unconfirmed potential data leakage, the question is worth closer examination if health data or other particularly sensitive data is at issue or if the company is subject to stringent contractual security incident notification requirements.
- Initiate an incident under your incident response plan. Companies are increasingly required by law, contractual obligations or internal policy to follow a security incident response plan that addresses how to detect, respond to and mitigate security incidents affecting sensitive data. If you have an incident response plan and handle sensitive data with potential exposure to Cloudbleed, this is a great reason to formally initiate in incident and respond according to your plan. Doing this, and documenting the effort, can help you ensure a sound response and demonstrate that you responded responsibly in the event auditors, customers or governmental investigators inquire. You will also learn something about how well your incident response plan works and what can be done to improve it.
______________________________________________________________________
*Admitted only in Maryland. Practice supervised by principals of the firm admitted in the District of Columbia.
To subscribe for updates from our Data Protection Report blog, visit the email sign-up page.
