L2 devices can break PMTUD
I've come across two service providers in the last couple of weeks that have had issues with L2 devices eating IPv6 PMTUD packets. I am allowed to share some of the information from one of those service providers here. $ISP contacted me to ask more about why PMTUD was being reported as broken on Android, Linux, Mac - but not being reported on Windows. After some back and forth I was able to get $ISP to prove that ICMPv6 Packet Too Big messages were not making it to the client. Windows just happens to work around this issue. Ultimately, they narrowed it down to be the access switch. They set one up in a lab, and sure enough, they could reproduce the problem and actually capture packets upstream and downstream of it. Device in question: Calix E7-2 and E7-20. To the vendor's credit, Calix started investigating immediately. Within a business week they were able to confirm it was a bug and told the $ISP that the next maintenance release should have the fix. Last comment from $ISP: I’m not sure if I shared with you that the issue did not occur if the VLAN was configured as a “TLAN” (transparent LAN). Of course, in the VLAN per service model (1:N) that isn’t set because you don’t’ want everyone flooding their broadcast and multicast traffic to everyone else.
RIPE-631: IPv6 Troubleshooting for Residential ISP Helpdesks
https://www.ripe.net/ripe/docs/ripe-631 We hope anyone deploying IPv6, and consequently staffing a help desk, to find this document useful. Please feel free to borrow and adapt it for your organization's needs. I'm sharing it here on NANOG because this document is not RIPE region specific. Disclaimer: this documents the use of http://test-ipv6.com/helpdesk - which I'm perhaps a bit biased about. That said, the bulk of the body of this work is coming from the community at large. -jason
Re: First ISP-hosted transparent test-IPv6.com mirror
Love the service that you guys have. I use it as part of training helpdesk agents as well as field techs. My ISP wants to set up a transparent mirror, and I encourage other to do so as well. Awesome. If you're not familiar with it already, be sure to try helpdesk.test-ipv6.com or test-ipv6.com/helpdesk. There's also a document floating around that we're encouraging people to contribute to, specifically to be used by help desks, if you're interested. https://git.steffann.nl/go6/ipv6-troubleshooting-for-helpdesks/blob/master/IPv6-troubleshooting-for-helpdesks.md for the document as it is today. Do you support us adding a hosted by logo, or a link to our IPv6 speedtest server? Within limits, yes. Text only; keeping it classy. You can see what I've done with http://beta.test-ipv6.com/ ; you'll see what Comcast users see when they visit the site. I'm not hung up on the exact wording, but do want to keep things minimal/classy. Mirrors operating on other domains, are welcome to put in footers at the bottom, larger logos, etc. They get loaded and displayed one the test is done running. You can see an example of this at http://test-ipv6.co.za (thanks Graham!).
First ISP-hosted transparent test-IPv6.com mirror
TL:DR? “Thanks, Comcast!” and “Who’s Next?” The test-ipv6.com site started out 4 years ago, at a table in Seattle, after an IPv6 round table meeting hosted by Internet Society. John Brzozowski and myself were each trying to come up with a way to help end users figure out that their IPv6 internet was good or bad. Ultimately I kept plugging away at it, as John was distracted with some kind of broadband IPv6 rollout for his employer (Comcast). And the test-ipv6.com site went live about a month later, with solicitation to a few operations lists for feedback. All in all, pretty successful. I’ve had two concerns since deploying test-ipv6.com: one, how to scale; and two, how to ensure the user’s connectivity back to the service is awesome (or at least, not bad). John was thinking the same thing - worried about sending too many of his customers to my site, and crushing it in the process. Not good for either of us. Both of those are relatively easy to solve. Simply deploy tons of mirrors around the world, problem solved - if you have the cash and/or smart business plan to back it. I don’t monetize the site with advertising; nor do I charge fees. Nor do I have a crack CFO who can help me IPO, and make me rich in the process. I don’t really have the time or energy to solicit for corporate handouts. As it turns out, it appears that I’m bad when it comes to making money on this project. So any solution has to be cheap. Asking folks to run regional mirrors (such as “test-ipv6.cz” or “test-ipv6.co.za”) is great; it offers a community local resources that are more immune to global connectivity issues. However, people must explicitly decide to visit these mirrors; to chose the location they want to test from. Those regional mirrors are mostly light duty as a result. They are still invaluable - they provide the back end that the global connectivity test uses, for any IPv6-validated customer visiting any of the mirrors. With this global test, we effectively crowd source getting IPv6 peering problems fixed. John and I decided to take things a step further; something I’m happy to see finally make it across the finish line after a fair bit of upfront dev work. Comcast is now running two mirrors and preparing a third - which directly act as “test-ipv6.com”. Nothing changes for the user. John has to worry less about transient (and transit!) connectivity back to test-ipv6.com. This is done with a poor-man’s GSLB (Global Server Load Balancer). We’re using an in-house built DNS server that looks at the internet routing table to see what ISP the DNS queries come from. Based on the source BGP ASN, we can decide which ISP mirror gets the traffic. (PS: thanks to routeviews.org and everyone who feeds data to it; that stuff is great!) In the end: we both get to worry less about Comcast traffic volume to test-ipv6.com; as well as ensure a good user experience for the customers visiting. What’s next? That’s where you come in :-). If you’re ... * working at a large ISP * doing real IPv6 deployment * or considering using “helpdesk.test-ipv6.com” with customers I’d love to help you set up a transparent mirror (acting as “test-ipv6.com”). For you, it means controlling the user experience using this site; as well as removing any capacity concerns. For me, it means the same thing. Win, win. More info at http://github.com/falling-sky/source/wiki/TransparentMirrors (http://tinyurl.com/m7nnhfn). If you want to help, or have questions, don’t hesitate to ask. -jason (link for sharing, if you're inclined: http://test-ipv6.com/comcast.html)
Wiki for people doing IPv6-only testing
On a recent IPv6 providers call, there was a desire for participants to share information with each other on what works and what breaks in an IPv6-only environment. I offered to set that up. It was further suggested I should share this with more than just that small community; to anyone who might be doing work to test out IPv6-only scenarios. http://wiki.test-ipv6.com This is distinct from ARIN's wiki in so far that this is less about being a general IPv6 resource and more about the IPv6-only scenario resource. Contributions are welcome, but we're requiring folks to sign up before contributing to keep the spam down. -jfes...@gigo.com / jfes...@test-ipv6.com
Re: Verizon IPv6 LTE
Safari on the iPad seems to be preferring A over if a hostname has both, though. I can browse to a bracketed IPv6 address so it is working. I think perhaps it is time to update test-ipv6.com a bit, and have it penalize the first number when IPv4 is used in preference. IPv4 CGN will make me a sad panda - especially when IPv6 is available but not being used.
test-ipv6.com / omgipv6day.com down
I know a lot of people are using / pointing to test-ipv6.com . The hardware picked a bad week to quit sniffing glue. Ill be working on trying to get it back up today, I need to source hardware. Also looking at borrowing a VM for short term. (speaking only for @test-ipv6.com, not for $employer - my personal mail address is down too).
Re: test-ipv6.com / omgipv6day.com down
On Jun 4, 2012, at 7:09 AM, Jeroen Massar wrote: You got a bunch of mirrors for it right? Should not be to tricky to get someone to let their act as the real thing for a bit. I've got redirects up now to spread the load across VMs. For the next couple of days, I don't expect a single VM to handle the load. Thanks to all who've sent me a response; and thanks to Host Virtual and to Network Design GmbH, for taking the immediate load. Once we're stable, and I get my *official* day job requirements met for World IPV6 Launch, Ill come back to getting the original gear replaced. I've got a couple hardware offers in (Alex, Mark, thank you), and this might just be the reason to flat out refresh the hardware if ixSystems has something suitable already built. -jason
Re: Microsoft's participation in World IPv6 day
In that case can anyone explain why the number of IPv4 *only* systems is increasing rather than decreasing: http://server8.test-ipv6.com/stats.html Increased traffic from less-geeky people = more sane numbers overall. The problem with the graphs on that site is that the audience is self selecting; so only when some major site says go here! do we get a more random(ish) audience, versus people setting up tunnelbrokers and the like. I would have expected the green+azure areas in those graphs to have increased in the past half year but counter-intutitively, it appears that IPv4 only usage is increasing. You're assuming there's significant rollout of IPv6. Everything I've seen so far says that *starts* nowish, and more laterish this year, in any impacting way. Really, we're just just before the start of getting end user adoption to start rising.
Re: Why no IPv6-only day (Was: Protocol-41 is not the only tunneling protocol)
But anyway, just consider it: a portion of the major websites go IPv6-only for 24 hours. What happens is that well, 99% of the populace can't reach them anymore, as the known ones are down, they start calling and thus overloading the helpdesks of their ISPs. Won't happen this year or next. Too much money at stake for the web sites. Only when IPv4 is single digits or less could this be even remotely considered. Even the 0.05% hit for a day was controverial at $dayjob.
Re: Yahoo and IPv6
Of course I'm assuming individual participants will do stuff, but that doesn't change that this IPv6 day as it stands now is a one-off event, not the first step towards the Ultimate Goal. The intent is to get folks together after we digest the data, to talk about next steps. Date is not yet picked. I'm hoping we collectively prove there is no broken user problem. I realistically expect we'll have another v6d - either as 24h, or as a roll-on-and-stick. But, until we get through the day, and analyze the data, any decisions on what to do next are premature. The NANOG following v6d should be interesting; I'm hoping a number of folks from both access and content are willing to share any early stats they have.
Re: Yahoo and IPv6
Actually, I have just noticed a slightly more disturbing thing on the Yahoo IPv6 help page... Not speaking in any official capacity, but .. thanks. The location that's affecting the results is pending removal from DNS; and ASAP we hope to have the name moved to the geo-LB that suppors v6, instead of the round robin it is today.
Re: IPv6 mistakes, was: Re: Looking for an IPv6 naysayer...
In my recent probe of route servers, I found 22 legacy /8's that were partly or completely unused. I'm a little surprised ARIN/ICANN thinks it's a waste of time to even try to reclaim them. How long would that be tied up in legal issues before they were freed?
test-ipv6.com
Several people have suggested I (re)post information about test-ipv6.com here. http://test-ipv6.com .. tests ipv4 and ipv6 by dns name tests dual stack (will the client break on World IPv6 Day?) tests ipv6 by IP literal (teredo can pass this) gives advice to end user about current status and (depending on circumstances) more information broken users (can't connect to dual stack) are solicited for info Caution: does depend on javascript. http://test-ipv6.com/simple_test.html Eyeball test only for user, with instructions; no javascript required. Please direct any comments, flames, etc directly to me instead of the list. I've added enough noise already :-)
Re: test-ipv6.com
Note you can have totally broken IPv6 connectivity and still be fine on World IPv6 day. You just need applications with good multi-homing support. Agreed so far. No web site can check this for you. Hmm. What's wrong with asking the browser to try a dual-stack url today, as a proxy for what will happen to said web browser on June 8? The concern with World IPv6 day is with the users who have IPv6 enable, and have a default route - yet have broken IPv6 connectivity. This specific population will see timeouts on June 8. If you are a application developer and want TCP example code that will work well with a broken IPv6 connection have a look at my blog. Hopefully browsers will adopt your idea (or Happy Eyeballs). It may be the only remedy available, short of content providers collectively moving forward with dual stack, 0.05% broken users be damned. http://tools.ietf.org/html/draft-wing-v6ops-happy-eyeballs-ipv6-01 I'm personally not a big fan of either method, as that's going to increase the amount of tcp sessions to my web servers. It is merely less bad than the alternative.
Re: Connectivity to an IPv6-only site
Its a shame there is not a pair of images on this site - one originated from a v4 only box, one a v6 only box. The img src= could point to the I've been working on something in this direction this past week, that is primarilly for user facing debugging purposes (versus for a content provider). http://test-ipv6.com will tell the user what to expect, after having them try a combination of image fetches (ipv4, ipv6, dual stack, ipv4 literal, ipv6 literal). It does each set of images 2-3 times (minimum is 2; a third pass is done if they go quick enough) and gets the best time of each type of fetch. Based on the successes and failures, and the times, it tries to give a straight-English explanation to the end user on what the future internet might look for them, based on their *current* internet service / OS / browser. Lastly, it posts the results back to my server, along with the user agent string, in case there are any trends that can be learned. On my todo list is to have it detect the case where the user timed out trying to reach the IPv6 and dual stack names; and ask the user for more details (ie, netstat -nr and ifconfig/ipconfig). Feedback welcome, preferably off-list. If there's a desire for me to summarize, or anything earth shattering, I'll followup on-list. I'm especially interested in people who've allowed utorrent to enable ipv6 to send me their results. :)
Re: Fiber cut - response in seconds?
The fact that they are so closely monitoring the construction and wanting to fix it that fast seems a bit over the top for redundant systems. Even despite what we saw recently in the SF bay area? If black helicopters are involved, I suspect this is about par on the paranoia scale.
Re: ingress SMTP
I agree, it's not the right way to do things. Running a mail server used to be much easier. Volunteers to help set things up the right way are always welcome. :-) Supporting those clients who can't connect is cheaper or more accessible for you?
