Cloudflare service outage June 12, 2025
On June 12, 2025, Cloudflare suffered a major service outage that affected a big set of our important providers, together with Staff KV, WARP, Entry, Gateway, Photographs, Stream, Staff AI, Turnstile and Challenges, AutoRAG, Zaraz, and elements of the Cloudflare Dashboard.
This outage lasted 2 hours and 28 minutes, and globally impacted all Cloudflare clients utilizing the affected providers. The reason for this outage was resulting from a failure within the underlying storage infrastructure utilized by our Staff KV service, which is a important dependency for a lot of Cloudflare merchandise and relied upon for configuration, authentication and asset supply throughout the affected providers. A part of this infrastructure is backed by a third-party cloud supplier, which skilled an outage right this moment and straight impacted availability of our KV service.
We’re deeply sorry for this outage: this was a failure on our half, and whereas the proximate trigger (or set off) for this outage was a third-party vendor failure, we’re in the end liable for our chosen dependencies and the way we select to architect round them.
This was not the results of an assault or different safety occasion. No information was misplaced on account of this incident. Cloudflare Magic Transit and Magic WAN, DNS, cache, proxy, WAF and associated providers weren’t straight impacted by this incident.
As a rule, Cloudflare designs and builds our providers on our personal platform constructing blocks, and as such lots of Cloudflare’s merchandise are constructed to depend on the Staff KV service.
The next desk particulars the impacted providers, together with the user-facing influence, operation failures, and will increase in error charges noticed:
|
Product/Service |
Affect |
|---|---|
|
Staff KV |
Staff KV noticed 90.22% of requests failing: any key-value pair not cached and that required to retrieve the worth from Staff KV’s origin storage backends resulted in failed requests with response code 503 or 500. The remaining requests had been efficiently served from Staff KV’s cache (standing code 200 and 404) or returned errors inside our anticipated limits and/or error finances. This didn’t influence information saved in Staff KV. |
|
Entry |
Entry makes use of Staff KV to retailer software and coverage configuration together with consumer id info. Through the incident Entry failed 100% of id primarily based logins for all software sorts together with Self-Hosted, SaaS and Infrastructure. Consumer Identification info was unavailable to different providers like WARP and Gateway throughout this incident. Entry is designed to fail closed when it can’t efficiently fetch coverage configuration or a consumer’s id. Lively Infrastructure Software SSH classes with command logging enabled failed to save lots of logs resulting from a Staff KV dependency. Entry’ System for Cross Area Identification (SCIM) service was additionally impacted resulting from its reliance on Staff KV and Sturdy Objects (which relied on KV) to retailer consumer info. Throughout this incident, consumer identities weren’t up to date resulting from Staff KV updates failures. These failures would lead to a 500 returned to id suppliers. Some suppliers might require a guide re-synchronization however most clients would have seen instant service restoration as soon as Entry’ SCIM service was restored resulting from retry logic by the id supplier. Service authentication primarily based logins (e.g. service token, Mutual TLS, and IP-based insurance policies) and Bypass insurance policies had been unaffected. No Entry coverage edits or adjustments had been misplaced throughout this time. |
|
Gateway |
This incident didn’t have an effect on most Gateway DNS queries, together with these over IPv4, IPv6, DNS over TLS (DoT), and DNS over HTTPS (DoH). Nonetheless, there have been two exceptions: DoH queries with identity-based guidelines failed. This occurred as a result of Gateway could not retrieve the required consumer’s id info. Authenticated DoH was disrupted for some customers. Customers with lively classes with legitimate authentication tokens had been unaffected, however these needing to start out new classes or refresh authentication tokens couldn’t. Customers of Gateway proxy, egress, and TLS decryption had been unable to attach, register, proxy, or log site visitors. This was resulting from our reliance on Staff KV to retrieve up-to-date id and system posture info. Every of those actions requires a name to Staff KV, and when unavailable, Gateway is designed to fail closed to forestall site visitors from bypassing customer-configured guidelines. |
|
WARP |
The WARP shopper was impacted resulting from core dependencies on Entry and Staff KV, which is required for system registration and authentication. Consequently, no new shoppers had been capable of join or join throughout the incident. Current WARP shopper customers classes that had been routed by way of the Gateway proxy skilled disruptions, as Gateway was unable to carry out its required coverage evaluations. Moreover, the WARP emergency disconnect override was rendered unavailable due to a failure in its underlying dependency, Staff KV. Shopper WARP noticed the same sporadic influence because the Zero Belief model. |
|
Dashboard |
Dashboard consumer logins and a lot of the present dashboard classes had been unavailable. This was resulting from an outage affecting Turnstile, DO, KV, and Entry. The precise causes for login failures had been: Customary Logins (Consumer/Password): Failed resulting from Turnstile unavailability. Signal-in with Google (OIDC) Logins: Failed resulting from a KV dependency situation. SSO Logins: Failed resulting from a full dependency on Entry. The Cloudflare v4 API was not impacted throughout this incident. |
|
Challenges and Turnstile |
The Problem platform that powers Cloudflare Challenges and Turnstile noticed a excessive charge of failure and timeout for siteverify API requests throughout the incident window resulting from its dependencies on Staff KV and Sturdy Objects. Now we have kill switches in place to disable these calls in case of incidents and outages akin to this. We activated these kill switches as a mitigation in order that eyeballs are usually not blocked from continuing. Notably, whereas these kill switches had been lively, Turnstile’s siteverify API (the API that validates issued tokens) might redeem legitimate tokens a number of occasions, doubtlessly permitting for assaults the place a foul actor would possibly attempt to use a beforehand legitimate token to bypass. There was no influence to Turnstile’s capacity to detect bots. A bot trying to unravel a problem would nonetheless have failed the problem and thus, not obtain a token. |
|
Browser Isolation |
Current Browser Isolation classes by way of Hyperlink-based isolation had been impacted resulting from a reliance on Gateway for coverage analysis. New link-based Browser Isolation classes couldn’t be initiated resulting from a dependency on Cloudflare Entry. All Gateway-initiated isolation classes failed due its Gateway dependency. |
|
Photographs |
Batch uploads to Cloudflare Photographs had been impacted throughout the incident window, with a 100% failure charge on the peak of the incident. Different uploads weren’t impacted. Total picture supply dipped to round 97% success charge. Picture Transformations weren’t considerably impacted, and Polish was not impacted. |
|
Stream |
Stream’s error charge exceeded 90% throughout the incident window as video playlists had been unable to be served. Stream Stay noticed a 100% error charge. Video uploads weren’t impacted. |
|
Realtime |
The Realtime TURN (Traversal Utilizing Relays round NAT) service makes use of KV and was closely impacted. Error charges had been close to 100% at some stage in the incident window. The Realtime SFU service (Selective Forwarding Unit) was unable to create new classes, though present connections had been maintained. This brought about a discount to twenty% of regular site visitors throughout the influence window. |
|
Staff AI |
All inference requests to Staff AI failed at some stage in the incident. Staff AI is dependent upon Staff KV for distributing configuration and routing info for AI requests globally. |
|
Pages & Staff Property |
Static property served by Cloudflare Pages and Staff Property (akin to HTML, JavaScript, CSS, photographs, and many others) are saved in Staff KV, cached, and retrieved at request time. Staff Property noticed a mean error charge improve of round 0.06% of complete requests throughout this time. Through the incident window, Pages error charge peaked to ~100% and all Pages builds couldn’t full. |
|
AutoRAG |
AutoRAG depends on Staff AI fashions for each doc conversion and producing vector embeddings throughout indexing, in addition to LLM fashions for querying and search. AutoRAG was unavailable throughout the incident window due to the Staff AI dependency. |
|
Sturdy Objects |
SQLite-backed Sturdy Objects share the identical underlying storage infrastructure as Staff KV. The typical error charge throughout the incident window peaked at 22%, and dropped to 2% as providers began to get better. Sturdy Object namespaces utilizing the legacy key-value storage weren’t impacted. |
|
D1 |
D1 databases share the identical underlying storage infrastructure as Staff KV and Sturdy Objects. Just like Sturdy Objects, the typical error charge throughout the incident window peaked at 22%, and dropped to 2% as providers began to get better. |
|
Queues & Occasion Notifications |
Queues message operations together with–pushing and consuming–had been unavailable throughout the incident window. Queues makes use of KV to map every Queue to underlying Sturdy Objects that comprise queued messages. Occasion Notifications use Queues as their underlying supply mechanism. |
|
AI Gateway |
AI Gateway is constructed on high of Staff and depends on Staff KV for shopper and inside configurations. Through the incident window, AI Gateway noticed error charges peak at 97% of requests till dependencies recovered. |
|
CDN |
Automated site visitors administration infrastructure was operational however acted with decreased efficacy throughout the influence interval. Particularly, registration requests from Zero Belief shoppers elevated considerably on account of the outage. The rise in requests imposed extra load in a number of Cloudflare places, triggering response from automated site visitors administration. In response to those situations, programs rerouted incoming CDN site visitors to close by places, decreasing influence to clients. There was a portion of site visitors that was not rerouted as anticipated and is beneath investigation. CDN requests impacted by this situation would expertise elevated latency, HTTP 499 errors, and / or HTTP 503 errors. Impacted Cloudflare service areas included São Paulo, Philadelphia, Atlanta, and Raleigh. |
|
Staff / Staff for Platforms |
Staff and Staff for Platforms depend on a 3rd occasion service for uploads. Through the incident window, Staff noticed an total error charge peak to ~2% of complete requests. Staff for Platforms noticed an total error charge peak to ~10% of complete requests throughout the identical time interval. |
|
Staff Builds (CI/CD) |
Beginning at 18:03 UTC Staff builds couldn’t obtain new supply code administration push occasions resulting from Entry being down. 100% of latest Staff Builds failed throughout the incident window. |
|
Browser Rendering |
Browser Rendering is dependent upon Browser Isolation for browser occasion infrastructure. Requests to each the REST API and by way of the Staff Browser Binding had been 100% impacted throughout the incident window. |
|
Zaraz |
100% of requests had been impacted throughout the incident window. Zaraz depends on Staff KV configs for web sites when dealing with eyeball site visitors. Because of the identical dependency, makes an attempt to save lots of updates to Zaraz configs had been unsuccessful throughout this era, however our monitoring exhibits that solely a single consumer was affected. |
Staff KV is constructed as what we name a “coreless” service which suggests there must be no single level of failure because the service runs independently in every of our places worldwide. Nonetheless, Staff KV right this moment depends on a central information retailer to supply a supply of reality for information. A failure of that retailer brought about a whole outage for chilly reads and writes to the KV namespaces utilized by providers throughout Cloudflare.
Staff KV is within the technique of being transitioned to considerably extra resilient infrastructure for its central retailer: regrettably, we had a niche in protection which was uncovered throughout this incident. Staff KV eliminated a storage supplier as we labored to re-architect KV’s backend, together with migrating it to Cloudflare R2, to forestall information consistency points (attributable to the unique information syncing structure), and to enhance assist for information residency necessities.
One in all our rules is to construct Cloudflare providers on our personal platform as a lot as doable, and Staff KV is not any exception. A lot of our inside and exterior providers rely closely on Staff KV, which beneath regular circumstances helps us ship probably the most sturdy providers doable, as a substitute of service groups trying to construct their very own storage providers. On this case, the cascading influence from the failure from Staff KV exacerbated the difficulty and considerably broadened the blast radius.
Incident timeline and influence
The incident timeline, together with the preliminary influence, investigation, root trigger, and remediation, are detailed beneath.
Staff KV error charges to storage infrastructure. 91% of requests to KV failed throughout the incident window.
Cloudflare Entry proportion of profitable requests. Cloudflare Entry depends straight on Staff KV and serves as a great proxy to measure Staff KV availability over time.
All timestamps referenced are in Coordinated Common Time (UTC).
|
Time |
Occasion |
|---|---|
|
2025-06-12 17:52 |
INCIDENT START |
|
2025-06-12 18:05 |
Cloudflare Entry staff acquired an alert resulting from a fast improve in error charges. Service Stage Goals for a number of providers drop beneath targets and set off alerts throughout these groups. |
|
2025-06-12 18:06 |
A number of service-specific incidents are mixed right into a single incident as we determine a shared trigger (Staff KV unavailability). Incident precedence upgraded to P1. |
|
2025-06-12 18:21 |
Incident precedence upgraded to P0 from P1 as severity of influence turns into clear. |
|
2025-06-12 18:43 |
Cloudflare Entry begins exploring choices to take away Staff KV dependency by migrating to a unique backing datastore with the Staff KV engineering staff. This was proactive within the occasion the storage infrastructure continued to be down. |
|
2025-06-12 19:09 |
Zero Belief Gateway started working to take away dependencies on Staff KV by gracefully degrading guidelines that referenced Identification or Machine Posture state. |
|
2025-06-12 19:32 |
Entry and Machine Posture drive drop id and system posture requests to shed load on Staff KV till third-party service comes again on-line. |
|
2025-06-12 19:45 |
Cloudflare groups proceed to work on a path to deploying a Staff KV launch in opposition to another backing datastore and having important providers write configuration information to that retailer. |
|
2025-06-12 20:23 |
Providers start to get better as storage infrastructure begins to get better. We proceed to see a non-negligible error charge and infrastructure charge limits because of the inflow of providers repopulating caches. |
|
2025-06-12 20:25 |
Entry and Machine Posture restore calling Staff KV as third-party service is restored. |
|
2025-06-12 20:28 |
IMPACT END |
|
INCIDENT END |
We’re taking instant steps to enhance the resiliency of providers that rely upon Staff KV and our storage infrastructure. This contains present deliberate work that we’re accelerating on account of this incident.
This encompasses a number of workstreams, together with efforts to keep away from singular dependencies on storage infrastructure we don’t personal, bettering the flexibility for us to get better important providers (together with Entry, Gateway and WARP)
Particularly:
-
(Actively in-flight): Bringing ahead our work to enhance the redundancy inside Staff KV’s storage infrastructure, eradicating the dependency on any single supplier. Through the incident window we started work to chop over and backfill important KV namespaces to our personal infrastructure, within the occasion the incident continued.
-
(Actively in-flight): Brief-term blast radius remediations for particular person merchandise that had been impacted by this incident so that every product turns into resilient to any lack of service attributable to any single level of failure, together with third occasion dependencies.
-
(Actively in-flight): Implementing tooling that permits us to progressively re-enable namespaces throughout storage infrastructure incidents. This may enable us to make sure that key dependencies, together with Entry and WARP, are capable of come up with out risking a denial-of-service in opposition to our personal infrastructure as caches are repopulated.
This listing just isn’t exhaustive: our groups proceed to revisit design selections and assess the infrastructure adjustments we have to make in each the close to (instant) time period and long run to mitigate the incidents like this going ahead.
This was a critical outage, and we perceive that organizations and establishments which might be massive and small rely upon us to guard and/or run their web sites, purposes, zero belief and community infrastructure. Once more we’re deeply sorry for the influence and are working diligently to enhance our service resiliency.