An apparent technical glitch from an errant router misconfiguration caused Google to lose control of several million of its IP addresses for more than an hour last month.
The mishap made Google’s search and other services unavailable to many users intermittently. It caused problems for Spotify, Google cloud customers, G-Suite users and Youtube viewers, among others.
The problem started when the MainOne Cable Company in Lagos, Nigeria, improperly updated tables in the internet’s global routing system to declare that its autonomous system was the proper path to reach 212 IP prefixes belonging to Google. China Telecom shortly thereafter again improperly accepted the route and announced it worldwide. That move, in turn, caused Russia-based Transtelecom and other large service providers to follow the route.
The fact that during the event, internet traffic was misrouted to China and Russia from Nigeria, appears suspicious and sparked fears of a malicious hijacking attempt. This was especially so, given that China Telecom, the Chinese government-owned internet service provider (ISP), was recently found to have improperly routed Western carriers’ traffic through mainland China.
However, Google adamantly denied any such impropriety, saying that the mishap resulted from a prefix leak in configuring the Border Gateway Protocol (BGP), the internet’s main routing protocol. Each ISP advertises to all others a list of internet protocols (IPs) it owns. A prefix leak occurs when an ISP advertises a range of IPs it does not own. BGP is a technology that is several decades old and is not cryptographically secure thereby leading to these types of mistakes by third parties.
A spokesperson for Google said, “We’re aware that a portion of internet traffic was affected by incorrect routing of IP addresses, and access to some Google services was impacted. The root cause of the issue was external to Google, and there was no compromise of Google services.” He added that since nearly all internet traffic to Google services is encrypted, there was no increased risk of data exposure as a result of this leak.
However, this reassurance from the search giant has not placated many. Experts point out that there have certainly been nefarious BGP hijackings in the past, and they are likely to happen again, because there is no quick fix to the problem. The mechanisms for managing the routing of traffic across the internet have been an area of concern for some time, as there is no real authentication for the information. It is entirely a trust-based approach.
Security authorities have also drawn attention to the ‘coincidence’ that the traffic was rerouted to countries already known for their ‘big-brother’ use of technology to spy on citizens.
Regardless of an intentional attack or mistake, the implications from such disruptions can range from denial of service and slow response of service to the compromise of data in transit. Viewed as an accident, this incident will drive attention and activity toward a more robust solution, and hopefully ISPs will implement more stringent processes to avoid such an event happening again.
Viewed as a malicious action, it highlights the inherent insecurity of routing protocols. While core providers are likely to have significant controls around the manipulation of protocols and tables within their organization, that does not eliminate the possibility of malfeasance by internal and external parties. Either way, we can expect to see renewed activity in this space.
