Just after 7am one morning in October, one of Google Cloud’s London data centres lost power. Google’s incident report lists almost forty of their services affected, from servers and storage to databases and load balancing, and the disruption ran on until mid-afternoon. Every site we host runs in Google’s London region. None of them went down.
That wasn’t luck, or not mostly. Here’s what happened and why it didn’t touch you.
What Actually Failed
Google’s London region is split into three separate zones: in effect three separate data centres, each with its own power, cooling and network. They’re built so that trouble in one doesn’t spread to the others. The power failure took out one of the three.
Anything that lived only in that zone stopped. Websites, shops and apps hosted on a single server there went offline, many of them for most of the working day.
Why Your Site Kept Going
We built the platform on the assumption that a zone will fail one day, so no single zone can take a site down.
- Your site runs twice, in two places. Every live WordPress site runs as at least two copies, and we require those copies to sit in different zones. When one zone goes dark, the other copy is already running and already serving your visitors. Nothing has to start up or be switched over.
- Google rebuilds what’s lost elsewhere. The servers your site runs on are managed by Google across all three zones. When the failed zone dropped out, replacement capacity came up in the other two automatically.
- Pages, images and static sites come from the edge. Most of what a visitor downloads (images, styles, whole pages on static sites) is served from Google’s content network, close to the visitor, not from London at all. It keeps working even while things are being shuffled about behind it.
- App data is stored three times over. The data behind our custom apps, such as bookings, payments and pupil records, is held in Google’s database service, which keeps copies in several locations at once. Losing one changes nothing.
- The front door isn’t in London. Traffic reaches your site through Google’s global load balancer, which simply stops sending visitors to anything that isn’t healthy.
The Honest Bit
Two things are worth saying plainly.
First, it wasn’t completely silent. In the first half hour, while Google’s systems moved work out of the failed zone, a handful of visitor requests got an error. Later in the morning, with a lot of Google’s customers rushing into the remaining two zones at once, spare capacity there ran short for a while, and a couple of our booking apps blipped for a few minutes before Google found them room.
Second, one part of our platform does live in a single zone: the server that holds your WordPress database and files. That day it was in one of the zones that stayed up. If the power had failed in its zone instead, most visitors would still have seen your main pages, served from Google’s cache, but anything that needs the database (logging in, forms, checkout, less-visited pages) would have stopped working until Google brought that building back. Your data would have been safe, because the disk survives a power cut and we keep daily backups in separate storage, but it would have been a partial outage, and we’d rather tell you that than have you read it somewhere else.
That’s a deliberate choice for where we are now. We’re a young platform, and a database that runs in two zones at once roughly doubles the cost of the most expensive part of the setup, a cost that would land on your bill. A single well-protected server is the right trade at our current size. As we grow, we’ll move to a database that keeps a live copy in a second zone and switches over by itself, so a power cut like this one wouldn’t touch any part of your site. We’ll make that move as soon as our scale makes it sensible, and we’ll tell you here when it’s done.
What This Means for You
- No action needed. Your site and your data were never at risk.
- Nothing to restore. Nothing was lost, so there’s nothing to recover.
- Built for the next one. Data centres do fail. The question is whether your site notices, and this time it didn’t.
If you’d like to know more about how your own site is set up, or you’re hosted somewhere that went down in this outage and want to talk about moving, get in touch.