The Internet died two short deaths this weekend. On Friday, a thunderstorm near Washington, D.C. knocked out Amazon's servers (which also hosts Instagram and Netflix). Last night, a leap second knocked out a good bit of the Internet.
The thunderstorm story is pretty obvious. A storm knocked out a bunch of servers and left 1.5 million in the D.C. area without power.
The leap second story is more interesting.
Once in a while, all the atomic clocks across the world pause for one second. This time it was at 12:00a Greenwich Mean Time that all the atomic clocks paused, something that has happened 24 times since 1972. Not exactly common.
A lot of electronic devices link to the atomic clock, and these devices aren't used to seeing the same second twice. And last night, when they did, things went to hell. Many sites, including Yelp, Reddit and the Gawker family were down.
One notable one wasn't, though. This one inevitably prepared for months for a leap second no one else even knew about. This one is obviously Google.
Their interestingly complex solution is below, straight from Google.
The solution we came up with came to be known as the "leap smear." We modified our internal NTP servers to gradually add a couple of milliseconds to every update, varying over a time window before the moment when the leap second actually happens. This meant that when it became time to add an extra second at midnight, our clocks had already taken this into account, by skewing the time over the course of the day. All of our servers were then able to continue as normal with the new year, blissfully unaware that a leap second had just occurred. We plan to use this "leap smear" technique again in the future, when new leap seconds are announced by the IERS.
Almost all of the sites are back up and running, and this problem won't present itself for a long while. So rest easy, this morbid weekend is over.