Starting yesterday at 8am EST a failure with the Comcast, Cogent, and Level3 networks operating in the midwest caused some traffic routing issues to the LiquidWeb datacenters in Lansing, Michigan.
"In a situation like this, LiquidWeb said it would typically reroute traffic, but it is being hindered by the amount of providers experiencing outages. LiquidWeb public relations specialist Cale Sauter tells the WHIR in an email that its network team is seeing if it can shift traffic to other providers, but since its most prevalent providers are affected, it is making it difficult." - TheWhir
As of 4pm EST the network connectivity had been restored, however a possibly related global networking issue emerged causing further downtime.
"Last night the global routing table exceeded 512,000 entries. This caused problems for many popular router models, including the Cisco routers that Liquid Web uses. The default memory allocation for the BGP table size is not large enough to hold the global table after surpassing the 512,000 entry mark. This caused many routers across the globe to experience issues, including the routers at Liquid Web. Unfortunately at this time we do not have an exact diagnosis of what has definitively caused this issue. We are still investigating and troubleshooting all possible solutions." - LiquidWeb
We can expect intermittent latency and routing issues until the hardware running these major networks is updated to accommodate the growth of these routing tables.
More information:
http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers
http://www.zdnet.com/internet-hiccups-today-youre-not-alone-heres-why-7000032566/
http://bgp.potaroo.net/as1221/bgp-active.html
