An Internet outage Monday November 6, 2017 was caused, according to several media sources, by a misconfiguration of manually typed characters at Level 3, an internet backbone company headquartered in Broomfield, Colorado. The misconfiguration caused ASes (Autonomous Systems) to forward incorrect Internet Protocol (IP) Level 3 is an enterprise level ISP that serves other big networks. Internet network analysts reported Monday that the misconfiguration caused a routing issue known as a “route leak” that created a cascade of problems for companies such as Comcast, Spectrum, Verizon, Cox, and RCN nationwide.
Level 3 said in a statement that it resolved the issue in about 90 minutes.
“Our network experienced a service disruption affecting some customers with IP-based services. The disruption was caused by a configuration error.”
Comcast users started reporting internet outages about the time as the Level 3 outages. Comcast said that it was monitoring “an external network issue” and not a problem with its own infrastructure. End user account status pages showed outages to Internet, TV, and Phone services.
RCN, which serves residents and businesses in the City of Chicago, confirmed that its network problems on Monday were caused by Level 3. RCN said it had restored RCN service by rerouting traffic to a different backbone.
Large outages caused by accidental route leaks have occurred before. Route leaks can be malicious or accidental. Malicious “route leaks” also known as “route hijacks” or “BGP hijacks” are caused by illegitimate takeover of groups of IP addresses by corrupting Internet routing tables maintained by the Border Gateway Protocol (BGP).
ISPs use “Autonomous Systems,” also known as ASes, to keep track of IP addresses and route packets on interacting networks. ISPs use the Border Gateway Protocol (BGP) to establish and communicate routes. Packets can route between several networks so that the end user can reach all available IP addresses.
In a “route leak,” an AS, or multiple ASes, end(s) up forwarding incorrect information about the IP addresses on their network, which causes delayed or blocked routing and failures for both the originating ISP and other ISPs trying to route traffic through various networks.
The Level 3 outage occurred just days after CenturyLink officially confirmed it had completed the acquisition of Level 3 on October 31, 2017 with a $34bn stock and cash merger deal. The misconfiguration problem could have occurred as CenturyLink was working to integrate the Level 3 network, or could have coincidentally occurred during routine traffic engineering and efficiency work.
CenturyLink, headquartered in Monroe, Louisiana, is the third largest telecommunications company in the United States, rated by lines served, after AT&T and Verizon. CenturyLink was originally Century Telephone (CenturyTel). Sprint purchased Centel in 1993, which was headquartered in Chicagoland until 1993, and Centel Cellular Company became Sprint Cellular Company. The cellular operations of Centel were spun off in 1996, and became 360 Communications Company. Alltel acquired 360 Communications Company in 1998 (Verizon Wireless acquired Alltel in 2008). Also in 1998, Centel acquired some Wisconsin Bell lines from Ameritech. In 2006, Sprint spun off Centel local telephone subsidiaries. Next Centel became CenturyLink in 2009 when it acquired Embarq, which was the name of the former local operations of Sprint Nextel (including the former Centel operations. Centel at one time served Las Vegas (NV), Des Plaines, Park Ridge (IL), Tallahassee (FL), and Charlottesville (VA).
CenturyLink purchased Qwest for US$10.6 billion in April 2011. In August 2011, the Qwest branding was retired and replaced as CenturyLink. Qwest was a Denver-based fiber optics long-distance company, that had taken over Regional Bell Operating Company (baby bell) US West in 2000.
CenturyLink maintains three large Chicagoland data center facilities (one at Busse Road and Devon Avenue in Elk Grove Village) that provide hosting and cloud assets. However, in May 2017, Medina Capital, a Miami-based private equity firm led by Manny Medina, and global private equity firm BC Partners formed a new venture in a $2.8 billion transaction combining a worldwide network of data centers (some sold by CenturyLink) with other cybersecurity and data analytics companies. The consortium of companies is known as Cyxtera Technologies, which runs over 60 data enters. CenturyLink also retains a minority stake in Cyxtera Technologies.
Monday’s outages put a light on the fragmented companies and service involved in the reliability vs vulnerability of Internet connectivity and vital services available via the Internet.
facebook …
GET ALERTS on Facebook.com/ArlingtonCardinal
GET ALERTS on Facebook.com/CardinalEmergencies
GET ALERTS on Facebook.com/ArlingtonHeightsCrime
Get updates from The Cardinal ALL NEWS FEEDS on Facebook. Just ‘LIKE’ the ‘Arlington Cardinal Page (become a fan of our page). The updates cover all posts and sub-category posts from The Cardinal — Arlingtoncardinal.com. You can also limit feeds to specific categories. See all of The Cardinal Facebook fan pages at Arlingtoncardinal.com/about/facebook …
Help fund The Cardinal Arlingtoncardinal.com/sponsor