Wikipedia Down

March 24, 2010 18:00 GMT: http://www.Wikipedia.org has been inaccessible for about 30 minutes now.

dig A http://www.wikipedia.org
http://www.wikipedia.org.      514     IN      CNAME   rr.wikimedia.org.

dig ANY http://www.wikimedia.org
http://www.wikimedia.org.      3600    IN      CNAME   rr.wikimedia.org.

dig ANY rr.wikimedia.org
there’s no entry for rr.wikimedia.org being published..

Also looks like ns2.wikimedia.org has fallen of the face of the earth and isn’t answering DNS queries.

ns2.wikimedia.org.      3600    IN      A       91.198.174.4

#nslookup http://www.wikimedia.org ns0.wikimedia.org

Server:         ns0.wikimedia.org
Address:        208.80.152.130#53

http://www.wikimedia.org       canonical name = rr.wikimedia.org.
rr.wikimedia.org        canonical name = rr.pmtpa.wikimedia.org.
Name:   rr.pmtpa.wikimedia.org
Address: 208.80.152.2

Update 18:30 GMT: In a tweet, Wikipedia confirmed and apologized for the site’s outage:

@Wikipedia; Sorry, everyone! English Wikipedia should be back in about an hour. Our tech team explains what happened: http://techblog.wikimedia.org/2010/03/global-outage-cooling-failure-and-dns/

Due to an overheating problem in our European data center many of our servers turned off to protect themselves. As this impacted all Wikipedia and other projects access from European users, we were forced to move all user traffic to our Florida cluster, for which we have a standard quick failover procedure in place, that changes our DNS entries.

However, shortly after we did this failover switch, it turned out that this failover mechanism was now broken, causing the DNS resolution of Wikimedia sites to stop working globally. This problem was quickly resolved, but unfortunately it may take up to an hour before access is restored for everyone, due to caching effects.

We apologize for the inconvenience this has caused.