[LINK] World IPv6 Day mostly smooth

[stephen at melbpc.org.au]

World IPv6 Day went mostly smoothly, with a few surprises

By Iljitsch van Beijnum | Pub June 9, 2011
<http://arstechnica.com/web/news/2011/06/world-ipv6-day-went-mostly-
smoothly-with-a-few-surprises.ars>


Tuesday evening, hundreds of Web destinations, including the four most 
popular ones worldwide, showed up on the IPv6 Internet.

IPv6 traffic went up a lot in relative terms, but still barely registered 
as a blip on the radar.

The 0.05 percent of users who were expected to encounter delays or errors 
didn't — or decided to call their Moms, watch TV, or otherwise refrain 
from complaining. Twenty-four hours later things went back to the way 
they were before—for the most part.

But not everything went according to plan: a small but significant number 
of websites did participate, but experienced some IPv6-related problems. 

Bumps in the road

The most salient is the story of a mobile operator in a small European 
country. I'm not sure how much of this is true, but what I heard was that 
the CEO learned about World IPv6 Day the day before, and instructed his 
engineers to participate. When WIPv6D rolled around, the frantic efforts 
to enable IPv6 reachability for the company's website had completely 
failed and they were unreachable for the better part of the day. 

When Facebook gained its IPv6 address about half an hour before midnight 
Zulu time, it remained unreachable for several minutes and then 
disappeared from the DNS again. About half an hour later the address came 
back and everything worked as expected. In many cases, the DNS updates 
propagated really quickly, but depending on how much (and how standards-
compliant) caching is present in various DNS servers, the operating 
system, and even applications, in a few cases it would take a while for 
newly injected IPv6 addresses to be recognized by applications. It 
probably took even more time for the IPv6 addresses to disappear once 
they were removed when the experiment ended. 

Internet backbone operator Level3 had a lot of issues with its IPv6-
enabled website: when consulted over IPv4, the site would display fine, 
but when loaded over IPv6, the exact same URL produced a 404 error 
message. In hindsight, it makes sense that such problems would occur for 
a fraction of the participants. The page shown for a HTTP request can 
depend on both the address that the client connects to and the "host" 
header sent by the client, which contains the domain name from the 
requested URL. Web servers themselves, as well as load balancers, look at 
both these variables. You need to explicitly test that asking for the 
right host over a connection to the right address makes the server send 
the right page. Testing this requires some trickery to execute before the 
IPv6 address is added to the DNS. 

Not long after the start of the experiment, someone reported that 
www.nist.gov didn't work. However, NIST's IPv6 address did appear to be 
reachable. When I asked for a webpage, it just didn't produce any 
results. This changed when I set the maximum packet size on my network 
connection to 1280, which is the "minimum maximum" specified for IPv6—in 
other words, every IPv6 system must be able to handle packets of at least 
1280 bytes. With the new setting, my computer would tell the NIST Web 
server that I can only handle 1280-byte packets. And then everything 
worked. 

So NIST was suffering from a path MTU discovery problem, probably the 
result of a filter that filters IPv6 ICMP packets that routers send to 
make IPv6 systems adjust their packet sizes where necessary. This is a 
problem that's hard to debug, because it only shows its ugly face when 
communicating over a path that has a packet size limitation. Most of the 
Internet uses the standard 1500-byte Ethernet (or larger) packets so the 
problem doesn't occur. 

This problem disappeared and reappeared at least once during the day. But 
once the NIST site had loaded successfully, clicking on any link produced 
a 404 error. I've been told that a message was later added to the page 
explaining that this was the case and that the page was a copy made to 
participate in World IPv6 Day; viewing the NIST website required visiting 
over IPv4. 

An IPv6 all-nighter in Amsterdam

But how does an average user do that? In fact, the group of researchers, 
students and IPv6 enthusiasts of all walks of life that had assembled at 
the University of Amsterdam all-night lab had some trouble with that. 
There are some browser extensions that claim to show whether a site is 
IPv4 or IPv6. Unfortunately, these extensions don't have access to the 
browser's internals, so they can't tell whether a page was really loaded 
over IPv4 or IPv6. My personal solution for this was to connect my Mac to 
the dual-stack (IPv4 + IPv6) LAN and use Safari as my dual-stack browser. 
Firefox lets you turn off IPv6 within the application by going into the 
about:config and then setting "network.dns.disableIPv6" to "true." So 
that was my IPv4-only browser. I then connected my iPhone to the IPv6-
only Wi-Fi network and turned off cellular data to avoid getting IPv4 
over 3G, so Safari on the iPhone was my IPv6-only browser. 

An interesting side effect of using the iPhone as my IPv6 canary in the 
coal mine was even after Bing, Yahoo, and CNN added their IPv6 addresses, 
I still couldn't visit those sites, because each of them redirected me 
from "www" to "m," and their mobile sites were all IPv4-only. It's also 
common for sites that are reachable over IPv6 to load additional page 
elements such as javascripts, images, or CSS files from domains that only 
have an IPv4 address. When visiting those using an IPv6-only system, the 
pages don't exactly look as intended. None of this is a big deal for the 
purposes of this experiment, as it was designed to flush out clients with 
broken IPv6 setups. But in the future, we'll have to have an IPv6-only 
day, to see what will have to happen before people who only have IPv6 and 
no IPv4 can visit participating sites. 

Once the experiment was in full swing, it was time to check up on IPv6 
traffic levels. Being in Amsterdam, we of course first checked the 
Amsterdam Internet Exchange IPv6 statistics. Normally, about 0.3 percent 
of the AMS-IX traffic is IPv6, which is higher than found elsewhere. But 
if anything, WIPv6D reduced AMS-IX IPv6 traffic a little. Strange. Not so 
at the Deutsche Commercial Internet eXchange (DE-CIX) in Frankfurt: 
there, IPv6 traffic was almost double the normal amount, reaching about 
0.1 percent of IPv4 traffic. 


Hurricane Electric, which is very active in the IPv6 world and provides 
free IPv6-over-IPv4 tunneling services to the public, saw its IPv6 
traffic go up by more than a factor of five. And Sandvine, which 
builds "traffic optimization" devices, observed that even though native 
(untunneled) IPv6 traffic remained stable, 6to4 traffic, which is 
automatically tunneled over the IPv4 internet to the closest gateway to 
the IPv6 internet, increased by a factor of ten. They also say that 
6over4 traffic increased by a factor of five, but 6over4 is a very rarely 
used mechanism. They probably mean packets tunneled through manually 
created IPv6-in-IPv4 tunnels, such as the ones that Hurricane Electric 
offers. 

Akamai, one of the founding participants of the Day, saw its IPv6 
requests quickly skyrocket to more than ten times the normal level.
In a blog post, Arbor Networks explains that normally, the little IPv6 
traffic that they see is encrypted file transfers, peer-to-peer traffic, 
and experimental protocols. But for the duration of the experiment, HTTP 
pretty much became the only game in town. 

Facebook had a million visitors over IPv6, but no increase in the number 
of users seeking help from their help center. Hopefully one or more World 
IPv6 Day participants (such as Google) have measured the number of users 
which experienced problems, because discovering those problems was the 
whole point of the exercise. So far, no thwarted users have come out of 
the woodwork. 

But it seems safe to say that adding a few IPv6 addresses to the DNS is 
not enough kill the Internet. As such, a few participants, such as 
xbox.com, have decided to keep their IPv6 addresses in the DNS. Based on 
the very high amounts of tunneled IPv6 traffic observed, an obvious next 
step is to do whatever's needed to get native, untunneled IPv6 off the 
ground for consumers. That's not going to be easy, but then again, 
staying with IPv4 ones after all the addresses have been given out isn't 
going to be easy, either. 

--

Cheers,
Stephen