Categories
Improving Technical Security Mutually Agreed Norms for Routing Security (MANRS) Securing Border Gateway Protocol (BGP)

Route Leak Causes Major Google Outage

Google recently faced a major outage in many parts of the world thanks to a BGP leak. This incident that was caused by a Nigerian ISP – Mainone – occurred on 12 November 2018 between 21.10 and 22.35 UTC, and was identified in tweets from the BGP monitoring service BGPMon, as well as the network monitoring provider Thousand Eyes.

Google also announced the problem through their status page:

We’ve received a report of an issue with Google Cloud Networking as of Monday, 2018-11-12 14:16 US/Pacific. We have reports of Google Cloud IP addresses being erroneously advertised by internet service providers other than Google. We will provide more information by Monday, 2018-11-12 15:00 US/Pacific.

In order to understand this issue, MainOne Inc (AS37282) is peering at IXPN (Internet Exchange Point of Nigeria) in Lagos where Google (AS151169) and China Telecom (AS4809) are also members.

Google (AS15169) advertise their prefixes (more than 500) through the IXPN Route Server, where PCH (Packet Clearing House) collects a daily snapshot of BGP announcements of IXPN. Unfortunately, 212 prefixes (aggregates of those 500+ announcements) from Google were leaked, which was recorded by BGPMon and RIPEstat.

Looking at the RIPE stats it is evident that the first announcement via MainOne Inc (AS37282) was recorded at 21:12 UTC and the issue lasted for more than an hour:

As per the tweet from BGPMon, the issues lasted for 74 minutes:

Looking at the circumstances around this incident, it’s likely this was an inadvertent leak from MainOne caused by a configuration mistake. A Google representative is quoted in ArsTechnical as saying “officials suspect the leak was accidental and not a malicious hijack”, and also added that the affected traffic was encrypted which limited the harm that could result from malicious hijackings.

Later in the same day, the MainOne Twitter account posted on the BGPMon analysis thread, accepting the mistake and assuring the world that corrective measures are now in place:

So this was a configuration mistake that was quickly rectified and didn’t cause any reported financial damage (even though service outages do cause financial and reputational damage to the service provider and its users), but it does demonstrate the problems that can be caused by accidental mistakes, and especially how an actor with bad intent could do a great deal of damage  as with the Amazon Route 53 hijack. It therefore illustrates why greater efforts need to be made towards improving the security and resilience of the Internet.

This BGP leak could have been easily avoided if proper prefix filtering had been undertaken by MainOne (AS37282) or China Telecom (AS4809). It is very difficult for the networks in the middle to block such leaks, because the prefixes are still legitimately originating from the correct AS number (in this scenario AS15169 – Google).

As mentioned in many previous blogs, Mutually Agreed Norms for Routing Security (MANRS) can be part of the solution here. It calls for four simple but concrete actions that ALL network operators should implement to reduce the most common routing threats, including filtering which prevents the propagation of incorrect routing information (the other three are anti-spoofing, address validation, and global coordination).

Network operators have a responsibility to ensure a globally robust and secure routing infrastructure, and your network’s safety depends on a routing infrastructure that weeds out accidental misconfigurations and bad actors. The more network operators who work together, the fewer incidents there will be, and the less damage they can do. It’s time to implement the MANRS actions now!

Categories
Deploy360 Improving Technical Security Securing Border Gateway Protocol (BGP)

BGPmon: Using BGP Data To Fight Spam

BGPmon logoCan we use BGP data to find email spammers? And could securing BGP provide a mechanism to help reduce spam?

In a fascinating article on BGPmon’s site, Andree Toonk explores how they found that “IP squatting” is used by spammers.  Essentially the attack seems to work like this:

  1. The spammers identify a block of IP addresses (IPv4) that are not currently being used on the actual Internet.
  2. The spammers send out BGP announcements routing that block of IP addresses to their servers.
  3. The spammers send out their spam email messages.
  4. When done (or when the IP address block is blocked by anti-spam tools), the spammers stop announcing the BGP routes for those IP address blocks.

They then can move on to announcing other IP address blocks to send more spam.

The article provides a very compelling and very readable description of two case studies where they found this to happen. In one case the spammers also used an Internet Route Registry (IRR) to attempt to give their BGP route announcement more legitimacy.

The BGPmon article doesn’t get into solutions… but preventing these kind of attacks is precisely why we set up the Securing BGP topic area of this site.

A general area of “source address validation” is critical here – the idea being to have some way to know that the router announcing the BGP routes has the actual authority to do so. New tools such as RPKI are emerging that let us securely validate the origin of route announcements to prevent spammers from performing the attacks like this.  With such tools a router would reject BGP announcements that came from the spammers’ systems because the spammers would not be able to securely assert that they had the right to announce those IP address blocks.  The challenge, of course, is to get more routers start signing route announcements – and more routers start validating route announcements.  (Read about how Jan set up RPKI for his lab.)  There are other tools and methods being explored, too.  The point is to not allow “spoofed” IP address blocks to get into the global routing tables.

This idea of securing BGP route announcements is also part of the “Routing Resilience Manifesto” that continues to be developed as (voluntary) guidelines for network operators.

If we are collectively able to implement some of these mechanisms for securing BGP we can potentially make a significant reduction in the ability of spammers to send their email – and make the Internet more secure and working better in the process.  Please do check out our Securing BPG section and consider what you can do in your network today!