Categories
Improving Technical Security Mutually Agreed Norms for Routing Security (MANRS) Securing Border Gateway Protocol (BGP)

Route Leak Causes Major Google Outage

Google recently faced a major outage in many parts of the world thanks to a BGP leak. This incident that was caused by a Nigerian ISP – Mainone – occurred on 12 November 2018 between 21.10 and 22.35 UTC, and was identified in tweets from the BGP monitoring service BGPMon, as well as the network monitoring provider Thousand Eyes.

Google also announced the problem through their status page:

We’ve received a report of an issue with Google Cloud Networking as of Monday, 2018-11-12 14:16 US/Pacific. We have reports of Google Cloud IP addresses being erroneously advertised by internet service providers other than Google. We will provide more information by Monday, 2018-11-12 15:00 US/Pacific.

In order to understand this issue, MainOne Inc (AS37282) is peering at IXPN (Internet Exchange Point of Nigeria) in Lagos where Google (AS151169) and China Telecom (AS4809) are also members.

Google (AS15169) advertise their prefixes (more than 500) through the IXPN Route Server, where PCH (Packet Clearing House) collects a daily snapshot of BGP announcements of IXPN. Unfortunately, 212 prefixes (aggregates of those 500+ announcements) from Google were leaked, which was recorded by BGPMon and RIPEstat.

Looking at the RIPE stats it is evident that the first announcement via MainOne Inc (AS37282) was recorded at 21:12 UTC and the issue lasted for more than an hour:

As per the tweet from BGPMon, the issues lasted for 74 minutes:

Looking at the circumstances around this incident, it’s likely this was an inadvertent leak from MainOne caused by a configuration mistake. A Google representative is quoted in ArsTechnical as saying “officials suspect the leak was accidental and not a malicious hijack”, and also added that the affected traffic was encrypted which limited the harm that could result from malicious hijackings.

Later in the same day, the MainOne Twitter account posted on the BGPMon analysis thread, accepting the mistake and assuring the world that corrective measures are now in place:

So this was a configuration mistake that was quickly rectified and didn’t cause any reported financial damage (even though service outages do cause financial and reputational damage to the service provider and its users), but it does demonstrate the problems that can be caused by accidental mistakes, and especially how an actor with bad intent could do a great deal of damage  as with the Amazon Route 53 hijack. It therefore illustrates why greater efforts need to be made towards improving the security and resilience of the Internet.

This BGP leak could have been easily avoided if proper prefix filtering had been undertaken by MainOne (AS37282) or China Telecom (AS4809). It is very difficult for the networks in the middle to block such leaks, because the prefixes are still legitimately originating from the correct AS number (in this scenario AS15169 – Google).

As mentioned in many previous blogs, Mutually Agreed Norms for Routing Security (MANRS) can be part of the solution here. It calls for four simple but concrete actions that ALL network operators should implement to reduce the most common routing threats, including filtering which prevents the propagation of incorrect routing information (the other three are anti-spoofing, address validation, and global coordination).

Network operators have a responsibility to ensure a globally robust and secure routing infrastructure, and your network’s safety depends on a routing infrastructure that weeds out accidental misconfigurations and bad actors. The more network operators who work together, the fewer incidents there will be, and the less damage they can do. It’s time to implement the MANRS actions now!

Categories
Deploy360 IPv6

Google buys a /12 IPv4 Address Block

As per the RIPE Stat – BGPlay, Merit Network Inc (AS237) withdrew its advertisement of 35.192.0.0/11 on 18 October 2016. It didn’t ring any bells because they have plenty of IPv4 address space, but on 21 March 2017, ARIN announced that 35.192.0.0/12 has been added to the transferred list.

As no-one was advertising this block on the Internet, it was unclear who’d bought such a big block and at what price. On 29 April 2017, Andree Toonk (founder of BGPMon) tweeted about this announcement and surprisingly enough it was announced by Google.

More digging in RIPE Stat – BGPlay suggests that Google started announcing 35.192.0.0/13, 35.200.0.0/14, 35.204.0.0/15 from AS15169 on 12 April 2017. As per the Whois information, Google has allocated this block for Google Cloud customers *** The IP addresses under this Org-ID are in use by Google Cloud customers ***

This transaction of more than a million IPv4 addresses started a debate why Google had to make this move when their IPv6 stats suggest that IPv6 deployment is increasing worldwide and most of their services are already available through IPv6.

The above graph from the Google IPv6 Statistics shows a growth of almost 7% in the last 12 months. This looks great but is that enough? The answer is of course not. Other statistics from the APNIC IPv6 Measurement Map show another side of IPv6 deployment status around the world. It is much better than previous years and it’s improving every month but is still not close to satisfactory.

Google has to serve its customers all around the world and if those customers don’t have IPv6 then they need to give them the option of IPv4. As rightly commented by Mark Smith on the AusNOG mailing list,

“The reason why I think Google buying the /12 is significant, despite Google services being thoroughly IPv6 enabled for quite a while, they’re not buying those IPv4 addresses to solve their own lack of IPv6 deployment. They’re trying to overcome others lack of IPv6 deployment, and paying a large amount of money to mostly solve somebody else’s problem rather than their own. I can only see them and others in a similar situation tolerating those costs for a limited time. They have a financial motivation to actively minimise or avoid those costs sooner rather than later.”

To summarise the discussion, whether it’s Google or any other major cloud or content provider, they can’t serve you IPv4 forever and may give up on you sooner or later. If you are an ISP not providing IPv6 to their customers then your customer will move to another ISP, and this is only a matter of time.

If you want to find out more about how to deploy IPv6 in your network, you can check out our IPv6 resources, attend any of our upcoming IPv6 training (workshop/tutorial) around the world, or reach out to us and let us know how can we help.

Categories
IPv6

Google buys a /12 IPv4 Address Block

As per the RIPE Stat – BGPlay, Merit Network Inc (AS237) withdrew its advertisement of 35.192.0.0/11 on 18 October 2016. It didn’t ring any bells because they have plenty of IPv4 address space, but on 21 March 2017, ARIN announced that 35.192.0.0/12 has been added to the transferred list.

As no-one was advertising this block on the Internet, it was unclear who’d bought such a big block and at what price. On 29 April 2017, Andree Toonk (founder of BGPMon) tweeted about this announcement and surprisingly enough it was announced by Google.

More digging in RIPE Stat – BGPlay suggests that Google started announcing 35.192.0.0/13, 35.200.0.0/14, 35.204.0.0/15 from AS15169 on 12 April 2017. As per the Whois information, Google has allocated this block for Google Cloud customers *** The IP addresses under this Org-ID are in use by Google Cloud customers ***

This transaction of more than a million IPv4 addresses started a debate why Google had to make this move when their IPv6 stats suggest that IPv6 deployment is increasing worldwide and most of their services are already available through IPv6.

The above graph from the Google IPv6 Statistics shows a growth of almost 7% in the last 12 months. This looks great but is that enough? The answer is of course not. Other statistics from the APNIC IPv6 Measurement Map show another side of IPv6 deployment status around the world. It is much better than previous years and it’s improving every month but is still not close to satisfactory.

Google has to serve its customers all around the world and if those customers don’t have IPv6 then they need to give them the option of IPv4. As rightly commented by Mark Smith on the AusNOG mailing list,

“The reason why I think Google buying the /12 is significant, despite Google services being thoroughly IPv6 enabled for quite a while, they’re not buying those IPv4 addresses to solve their own lack of IPv6 deployment. They’re trying to overcome others lack of IPv6 deployment, and paying a large amount of money to mostly solve somebody else’s problem rather than their own. I can only see them and others in a similar situation tolerating those costs for a limited time. They have a financial motivation to actively minimise or avoid those costs sooner rather than later.”

To summarise the discussion, whether it’s Google or any other major cloud or content provider, they can’t serve you IPv4 forever and may give up on you sooner or later. If you are an ISP not providing IPv6 to their customers then your customer will move to another ISP, and this is only a matter of time.

If you want to find out more about how to deploy IPv6 in your network, you can check out our IPv6 resources, attend any of our upcoming IPv6 training (workshop/tutorial) around the world, or reach out to us and let us know how can we help.

Categories
Deploy360 IPv6 Transport Layer Security (TLS)

Google Cloud Platform gets IPv6 support

The Google Cloud Platform (GCP) is now able to support IPv6 clients using HTTP(S), SSL proxy and TCP proxy load balancing. The load balancer will accept IPv6 connections from users, and proxy those over  IPv4 to virtual machines (i.e. instances). This allows instances to appear as IPv6 services to IPv6 clients.

At the moment, this functionality is an alpha release and is not currently recommended for production use but it demonstrates a commitment to support IPv6 services. GCP allocates a /64 address range for forwarding purposes.

Google Cloud Platform is a cloud computing service offering website and application hosting, data storage and compute facilities on Google’s infrastructure.

More information on how to set-up IPv6 support is available on the GCP website.

 

 

Categories
Deploy360 IPv6

Google Stats Now Showing Over 8% IPv6

A nice way to end a Friday afternoon… I happened to look at Google’s IPv6 statistics for the first time in a while and see that they’ve climbed up over 8%!

Google IPv6 statsWe’ve kind of stopped celebrating each individual percentage point because the reality is the graph just keeps on going up and to the right in the direction we want!

Still, this was a nice way to end the work week.  Looking forward to celebrating a bit more when they hit 10%!

P.S. As the graph shows, IPv6 at this point is on the trajectory where it is just going to happen … if you haven’t already started figuring out what to do, please do look at our Start Here page to understand how you can get started transitioning your networks, devices and applications to make sure they’re ready!

Categories
Deploy360 IPv6

Google's IPv6 Traffic Hits 5% Globally, 28% in Belgium, 12% in USA and Germany

Google's IPv6 statistics at 5%

Outstanding news!  Today marked another milestone in the continued evolution of the Internet from the development version based on IPv4 to the production version of the Internet based on IPv6 – Google’s IPv6 traffic statistics showed that global traffic over IPv6 has passed the 5% mark!   Even better, if you go into the per-country IPv6 statistics, you can see the increased growth in IPv6 traffic in countries such as Belgium (28.45%) and the USA (11.85%), Germany (11.88%), Luxembourg (11.38%), Switzerland (9.94%) and a number of others.

As our colleague Phil Roberts writes in an Internet Technology Matters post today, these numbers compare well to what Akamai is showing in their per-country IPv6 traffic statistics.  Phil also mentioned the World IPv6 Launch measurements, which break down the measurements on a per-network basis and show even higher levels of IPv6 deployment such as the 59.4% measured on Verizon Wireless’ networks (because of their IPv6-based LTE).  I would add that APNIC’s IPv6 statistics tell a similar story (and use a different measurement technique) – if you scroll down APNIC’s page you’ll see the list of the top countries and the IPv6 connectivity in those regions.

As far as the global 5% measurement, we definitely agree with Phil:

While 5% might not seem like a large percentage, it’s a big step on the path to IPv6 becoming the prominent Internet Protocol on the Internet, and billions more people and devices being able to connect to an Internet that works like the one we’ve enjoyed and benefitted from so far. And that’s worth celebrating.

Anyone who still doubts that IPv6 will ever happen in their lifetime clearly isn’t reading the statistics!  That graph is going up and to the right… and if you look at the fact just two years ago the % was under 1%… the deployment IS happening!

What about you?  Are your networks, services and applications ready for IPv6?  If you haven’t started yet, definitely check out our Start Here page to find resources for your type of role or organization.  And please let us know if you need more information – the time to make the move is TODAY!

Categories
IPv6

Google IPv6 Traffic Passes 5% – IPv6 Internet Growing Faster than IPv4!

We’ve been watching IPv6 traffic numbers climb ever since World IPv6 Day, and today we’re excited to see that the Google IPv6 traffic graph (pictured above) has crossed a nice round number – 5%. This means that 5% of all traffic globally reaching Google’s servers uses IPv6 as the Internet Protocol of choice. This is great news because IPv6 deployment and use means the Internet can continue to grow!

Even better, there are several countries around the world where more than 10% of the traffic from networks in those countries use IPv6 – Belgium, Luxembourg, Germany, and the US – with several others close to that mark. Akamai reports similar numbers in its global per-country measurements. This is all very exciting progress as we march toward full deployment of IPv6 across the globe.

Earlier this year we predicted double digit IPv6 traffic by the end of 2014. That was a very aggressive and optimistic prediction meant to grab headlines and prompt action, and we didn’t quite make it to that target. In reality, traffic growth this year has been slower than in 2013. I’m happy to see IPv6 still increasing steadily, but of course I wish it had continued along a growth path that looked something like an S-curve. Still, it is growth, and most importantly, it means that the IPv6 Internet is growing faster than the IPv4 Internet.

When I look at the new networks that have been joining the global deployment of IPv6, as reported in our World IPv6 Launch measurements, I’m hopeful that the growth in IPv6 will continue and the rate of growth will increase. We’re beginning to see multiple mobile networks rolling out IPv6 and growth in that realm can accelerate the footprint of IPv6 rapidly.

While 5% might not seem like a large percentage, it’s a big step on the path to IPv6 becoming the prominent Internet Protocol on the Internet, and billions more people and devices being able to connect to an Internet that works like the one we’ve enjoyed and benefitted from so far. And that’s worth celebrating.

What is your network planning to do with IPv6 deployment in 2015? Our Deploy360 Programme can help you get started!

Categories
Deploy360 Domain Name System Security Extensions (DNSSEC) IPv6 Transport Layer Security (TLS)

Awesome News About HTTPS As A Ranking Signal, Google! Now Can We Please Get IPv6 And DNSSEC, Too?

Google logoThe big news hitting the online marketing world today is that Google has indicated that the use of HTTPS in your web site will potentially help your site rank better in Google’s search results. In other words, the use of a TLS (formerly “SSL”) certificate to encrypt the connection to your website will be one of the signals Google uses to rank results.  To be precise, here is the key part of the post:

For these reasons, over the past few months we’ve been running tests taking into account whether sites use secure, encrypted connections as a signal in our search ranking algorithms. We’ve seen positive results, so we’re starting to use HTTPS as a ranking signal. For now it’s only a very lightweight signal — affecting fewer than 1% of global queries, and carrying less weight than other signals such as high-quality content — while we give webmasters time to switch to HTTPS. But over time, we may decide to strengthen it, because we’d like to encourage all website owners to switch from HTTP to HTTPS to keep everyone safe on the web.

Because you almost never get SEO advice directly from Google this was big news today.  And even though the post says that fewer than 1% of search engine queries will be helped today by enabling HTTPS, I’ve already seen a ton of associated articles from SEO consultants and others saying that you need to go enable TLS for your site today.  (Well, okay, to be honest the ones I’ve seen are all saying to go enable “SSL” but maybe some day we can get everyone to use “TLS”! On that note, kudos to Google for NOT using “SSL” in their article!)

I’m sure that many web hosting providers are similarly getting inquiries from customers today about how TLS can be enabled on their websites.

Naturally we’re pleased to see this news out of Google because the goal of our TLS for Applications area here on Deploy360 is to help people get TLS happening across their sites and services.  So to the degree that Google can help drive that deployment of TLS – and wind up getting the whole ecosystem of SEO consultants and marketing/PR people to help drive that deployment – we all win with a more secure Internet!

Of course, our thinking immediately jumps to the next step – what if Google were to say that having a site available over IPv6 would count as a ranking signal?  Several people on Twitter suggested exactly that today. Here’s one:

Can you imagine how many website owners might suddenly be asking their ISPs and hosting providers how to get IPv6?  (Tip to website owners/operators: check our our IPv6 resources targeted to you!)

Or… what if the fact that a web site’s domain was signed with DNSSEC counted as a ranking signal?

Can you imagine how many website owners might suddenly be trying to get their domains signed?  (Again, we’ve got you covered with some steps you can take.)

How about it, Google?  Please?   🙂

P.S. If you do want to get your site or network moved to IPv6 or DNSSEC, please check out our “Start Here” page to find resources focused on your type of organization or role.

 

 

Categories
Deploy360 IPv6

Google IPv6 Stats Pass 4% Globally, 20% in Belgium, 8% in USA, 11% in Switzerland

It seems that Belgium is beating the USA in more than just World Cup Soccer (What an amazing game!)… they are also doing it in IPv6 deployment! In looking at Google’s latest IPv6 statistics, my colleague Phil Roberts noted that IPv6 deployment globally has doubled in 9 months and is now over 4%.  It’s great to see a chart that goes up and to the right like this one:

google-4percent

But while Google is measuring 4% IPv6 availability globally, I like looking at Google’s “Per-Country IPv6 adoption” page that gives a view into how diverse the IPv6 deployment really is.  Here’s the global map (click/tap the map to go to the site) as of today:

ipv6-google-global-4percent

And if you hover over the green spots you can easily see some of the various numbers:

  • 8.09% – United States
  • 4.59% – Peru
  • 5.14% – France
  • 8.69% – Germany
  • 5.67% – Romania

If you click on the “Europe” link at the bottom of the image you can zoom in a bit more and see these great stats (you could get this from the global map, too, if your hand is good enough to hover over the smaller countries):

  • 20.07% – Belgium
  • 10.68% – Switzerland
  • 7.99% – Luxembourg

VERY cool to see this high level of IPv6 availability in Belgium and indeed in all these countries!

Obviously these are just one set of measurements from Google, but there are a range of other IPv6 statistics sites, including the World IPv6 Launch measurements site, that also continue to show the increased global growth of IPv6 – great to see!

P.S. if you want to get started with IPv6, please visit our “Start Here” page to find resources targeted at your type of organization so that you can get started today!

Categories
IPv6

GigaOm: Cloud Providers Need To Get IPv6!

GigaOm article about IPv6Over on GigaOm today we were delighted to see the article “With billions of devices coming online, cloud providers better get with IPv6 program“.  In that article, author Barb Darrow writes:

As we enter the internet of things era, with millions; check that, billions of devices coming online, we’re going to need a lot more unique IP addresses. That means the big cloud providers need to get on the stick to support IPv6, the internet protocol that opens up billions of new addresses for just that purpose.

EXACTLY!

This is a key point we’ve been making in our events and presentations – with all these many devices coming online, and also with 3-4 billion more people to come online, we need to move to using IPv6!

In the article, she goes on to note that IPv6 is NOT supported by Microsoft Azure, Google Computer Engine and most of Amazon Web Services.  She does point out that IBM Softlayer does support IPv6 as will a new “Verizon Cloud” service apparently coming out later this year.  (All of which has made me note that we need a page on this Deploy360 site about “cloud services that support IPv6”.)

A few weeks back I asked a friend of mine who has an Internet of Things (IoT) startup whether his new service supported IPv6.  He runs his system, not surprisingly, on a cloud platform – in his case Amazon’s Elastic Compute Cloud (EC2) – and because EC2 doesn’t have IPv6, he can’t run his apps over IPv6.

We need to get there.  We need all the cloud providers to be enabled for IPv6, because they will then enable all the companies, large and small and everything in between, to make the move to the “production” version of the Internet.

Barb Darrow mentions in the GigaOm article that “the device population explosion pose to cloud providers and the very architecture of data centers will be a hot topic next week at Structure“, where Structure is GigaOm’s conference on the whole “cloud” topic.  That sounds great… although in looking at the agenda I don’t see anything specifically mentioning IPv6.  Hopefully that is a topic that gets covered and maybe we’ll be able to write about some of the IPv6-related news next week.

UPDATE: In a comment to this post, Barb Darrow indicates that IPv6 will be a topic in the Structure panel “What has to happen to enable the infrastructure to support IOT?”  And indeed, to support the Internet of Things (IoT) we very definitely need to move to IPv6!

Meanwhile, if you are a cloud provider – or anyone else – do check out our “Start Here” page or just browse through some of our IPv6 resources to get started with the move to IPv6!

Categories
Deploy360 IPv6

Playing Google's Interactive Rubik's Cube Over IPv6

Today Google’s front page is an interactive “doodle” celebrating the 40th anniversary of the Rubik’s Cube.  Apart from feeling old since I remember when the Rubik’s Cube first came out, I was also of course thrilled to note that being on a Google website, this interactive Rubik’s Cube is available over IPv6:

Google-rubikscube

Using the IPvFoo add-on for Google Chrome I can verify that almost all of my communication is indeed going over IPv6:

Google Rubik's Cube Doodle

I am guessing from the URLs that the IPv4 connections are to some type of measurement site.  Regardless, the main communication is all happening over IPv6.

And this doodle reminded me, too, that it’s been a looooonnnnnggg time since I worked with a Rubik’s Cube! 🙂

If you play with the interactive version today, I hope you have fun… and if you aren’t seeing it over IPv6, why not check out our “Start Here!” pages to see how you can join the rest of us?

Categories
Deploy360 Transport Layer Security (TLS)

Google Is Now Always Using TLS/SSL for Gmail Connections

Gmail logoWe were pleased today to read that Google is now changing their Gmail service to always use TLS-encrypted connections. As they note in their announcement blog post:

Starting today, Gmail will always use an encrypted HTTPS connection when you check or send email. Gmail has supported HTTPS since the day it launched, and in 2010 we made HTTPS the default. Today’s change means that no one can listen in on your messages as they go back and forth between you and Gmail’s servers—no matter if you’re using public WiFi or logging in from your computer, phone or tablet. 

The key point is the one I emphasized in bold in the text: attackers cannot listen in on your messages as they go between your mail client (which could be your web browser) and Gmail’s servers.   Obviously the messages could still be potentially viewed either on your client device or on Gmail’s servers… but this step is removing the ability for the messages to be viewed “on the wire”.

This is a great example of the kind of action we’d like to see to make communication over the Internet more secure- and why we launched our new “TLS for Applications” section of this site.  We want to encourage more application providers and developers to make the steps that Google has done here.

Kudos to the Google/Gmail team for taking this step!