Something is amiss with the Interwebs! BGP is a flapping.

[Update] See http://www.bgpmon.net/what-caused-todays-internet-hiccup/ for a good summary of what happened.

 

Tuesday Morning, various networks experienced outages from 4-6am EDT (8-10am UTC) [1]. I appears the outage was the result of a somewhat anticipated problem with older routers and their inability to deal with the ever increasing size of the Internet's routing table.

These BGP routers need to store a map of the internet defining which IP address range belongs to which network. Due to the increasing scarcity of IPv4 space, registrars and ISPs assign smaller and smaller netblocks to customers, leading to a more and more fragmented topology. Many older routers are limited to store 512k entries, and the Internet's routing table has become large enough to reach this limit. Tuesday morning, it appears to have exceeded this limit for a short time [2][3].

The large number of route announcements, and immediate removals shown in [2] could indicate a malicious intend behind this events (or a simple configuration error), but either way likely point to one entity "pushing" the size of the routing table beyond the 512k limit briefly. At around this time, one larger ISP (Windstream, AS7029) recovered from an unrelated outage and routing changes due to the recovery are one suspect that may have triggered the event.

Vendors published guidance for users of older routers how to avoid this issue [5]. This guidance has been available for a while. Please contact your vendor if you are affected. You may also want to consider upgrading your router. The routing table is likely going to get larger over the next few years until networks rely less on IPv4 and take advantage of IPv6.

 

[1] https://puck.nether.net/pipermail/outages/2014-August/007090.html
[2] http://www.cymru.com/BGP/prefix_delta.html (see the spike in deltas around that time)
[3] http://www.cidr-report.org/2.0/#General_Status  (note how close it is to 512k and rising)
[4] http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-major-outage-across-us-network-providers
[5] http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/117712-problemsolution-cat6500-00.html