Why is IPv6 broken?
It's broken, first and foremost, because not all network providers who claim to
be tier 1 are tier 1.
Even worse, some of these providers run 6to4 relays or providers to home users.
A user has no choice which provider is running their 6to4 relay...so, they
might end up using a relay that is run by a provider who doesn't peer with
their intended destination. I don't think the IETF saw that one coming. But
the result is to make 6to4 even more broken. Now, I know some people want 6to4
to die, but while it still exists in some form, user experience is worse than
it could be. The temporary fix is for any provider to run their own 6to4 relay
for their own customers (assuming that they themselves have full connectivity).
Right now, unless you buy transit from multiple tier 1s, and do so with
carefully chosen tier ones, you have only part of the IPv6 internet. Many tier
1s are unsuitable even as backup connections, since you still want your backup
connection to have access to the whole internet! Good tier 2 providers might
be an excellent choice, sine good providers have already done this leg work and
can monitor their providers for compliance.
A few myths...
Routing table size has nothing to do with completeness of routes. Google may
be one route, through aggregation. And SmallCo may advertise a large route
through one provider, and, due to traffic engineering, a smaller route through
a second one - in many cases, anyone that had the large route would be able to
contact SmallCo, even without the smaller route being present. So routing
table size doesn't work. In addition, some providers aggregate their routing
tables to reduce routing load and such. Others intentionally don't or
deaggregate it intentionally so that they can brag about having bigger routing
tables. What you need to ask is: "How many /64s can you get to from your
network, and how many of these /64s are reachable from at least one other major
provider (you don't care about internal-only networks, after all)?" They can
give you that information, but many won't want to.
It's also not about technical people not getting along. It's about business
players trying to make money, but not just that either. It's also about
ensuring that providers don't end up assuming more than their share of costs
for a link. Just because you have a common peering point doesn't mean that
turning peering on would reduce your costs. In some cases it may increase
costs tremendously, particularly on your long haul backbone links, because the
other party would like to take advantage of an attitude of trust on the
internet. That's why we end up with peering policies and contracts.
What is the issue?
Let's take Hurricane. This is no different than other providers...basically,
they want to say, "We shouldn't need to pay for IPv6 transit from anyone."
This is what Cogent said on IPv4 a few years ago. Google used to say this too
for IPv6, not sure if they are still saying it. Basically, "We know we're big
enough that you won't want to screw your users by not peering with us."
A small network couldn't do this tactic - a 100 node network who said to the
IPv4 tier 1s: "Hey, I'm in the Podunk Internet Exchange, so are you, so I'm
going to peer from you so I don't have to buy any bandwidth for my web server
(placed in the Podunk exchange). Sure, they would like to - it would save a
ton of money if their site got lots of hits. I mean, who wouldn't want free
connectivity?
In IPv6, we're going through what we settled years ago in IPv4 - who has to pay
who to connect. After all, even free peering connections have a cost in
manpower, debugging, traffic engineering, documentation, etc.
Some players who aren't getting free interconnection to tier 1s in IPv4 want to
get it in IPv6. So they've worked to attract lots of users, and done so under
the guise of "We like IPv6 and want to promote it." Others have not bothered
with trying to attract the users, but have said, "We're too big for you to not
want to give us connectivity for free, since it would piss off your users if
you don't" (Google did this at one point in the past, may still be doing it).
The Google example is basically trying to use a monopoly position to force
business decisions.
Now, HE, Google, and others would want you to think, "Hey, IPv6 is all new, and
these $#@! other providers just want to make a buck on something they have no
right to." Well, perhaps. But what they aren't saying is, "We can turn on BGP
for IPv6 on our existing connections to other providers, with no cost to us,
and actually have full connectivity." The issue isn't about cost today -
nobody is charging extra for IPv6 in addition to IPv4 on a pipe where you
already buy IPv4 bandwidth. And Google and HE already buy IPv4 bandwidth.
What they are thinking of is the future, 15 years from now, when there is no
IPv4 - in that future, IPv6 isn't insignificant bandwidth, it's everything.
Wouldn't it be nice to be a tier 1 and not pay for that? Of course! And
certainly one can argue for or against the current tier 1 club's exclusivity.
But it's the way the internet works right now, for better or worse. In the
meantime, in pursuit of this future, today's customers are screwed by these
providers trying to position themselves to make more profit margin down the
road.
Which is better for the customer? A system where they are screwed today so
that their provider can have a better negotiating position in business
discussions OR a position where they do whatever they have to take to provide
the customer with full connectivity? (To HE's credit, they are giving away
transit today on IPv6, so it's not like you are losing anything of value by not
having the full internet routing tables, but it's a huge reason to not pay HE
anything in other services, such as data center colocation - go with a provider
that you pay and which gives you what you pay for - full transit).
A bit about peering...
Lots of people who aren't running big networks don't understand peering. They
think, "Doesn't this benefit everyone if everyone exchanges traffic?" Maybe,
on a pure level, but the business doesn't work that way.
I'll give you an example. Let's say you are a little ISP, and located in
Virginia, near a major peering point. You say, "All the tier 1s are there, I
can pull fiber to that peering point, which is only a block away, and have free
internet, other than the cost of the line." So, let's say you run the line,
and, let's say that all the tier 1s agree to let you peer for free, since they
want your traffic too. Now, let's say your user downloads 1,000 TB from a
server in California, on Qwest's network.
You paid, let's say, $15,000 for your piece of fiber going a block. You needed
to hire contractors and buy permits and such, after all. So you shared in the
costs of letting the user get to the server. What did Qwest pay? Well, they
dug trenches, pulled fiber, negotiated with cities, counties, and states, paid
taxes on their work, lit this fiber, etc. It cost a lot because they went a
lot further than your one block. And a lot more than $15,000.
You say, "So what! Their customer benefits too!" That's true, but let's go a
bit further. Let's say you have a network that extends to California - you by
DS3s from Sprint to do it. There's some cost in that, but your user in
Virginia would need more bandwidth than your DS3s. So you decide NOT to peer
in California, just in Virginia. That way you don't have to upgrade your lines
for your Virginia user. Maybe you even legally break your company into two
entities, so that you can peer in California and Virginia both, but you can say
with a straight face, "We only have Virginia offices for this user - the other
company is a separate entity, and not the entity that owns either the server or
the end user."
In other words, you found a way to shift most of the traffic burden and
infrastructure costs to Qwest, away from your user.
This is why Qwest has some sort of peering policy. Among other things, it will
require multiple exchange points, and Qwest will probably say they will send
traffic to the closest peering point, to minimize their costs. You get to do
the same (more on that later).
Let's say that you currently buy bandwidth from NTT - you're not big enough to
get free peering from everyone, but Qwest agrees to peer with you. Of course
Qwest and NTT also have a business relationship, to give each other free
peering. If Qwest gives you and many other customers free peering, however,
you'll send less traffic across NTT's network. That might be good from a
technical standpoint, but NTT now is selling you a smaller pipe - and making
less money. In effect, Qwest undercut NTT's business and lowered NTT's profits
on the connection. How will NTT respond to that, when they were also giving
free peering to and from Qwest? Well, they might decide that Qwest isn't a
very nice partner and tell Qwest, "Pay us for transit or get lost." That could
be ugly - both NTT and Qwest could lose, but Qwest, if they actually care about
stable service, won't want to risk it. So generally you don't give peering to
anyone who is a customer of one of your free peers. You don't hurt their
business. In fact, it's often a requirement in the peering connection,
legally. (that said, you could argue whether or not there is an abuse of
monopoly here...that's a different issue)
Going one further, let's say you have the server, and Qwest has the end-user.
That doesn't change anything - the economics are still such that Qwest has the
cost, you don't. That said, it's convention that the person receiving the
traffic pays for most of the backhaul.
Asymmetry in the Internet:
What's the path between your host and a remote server? How do you find it? If
you said "traceroute", you might be right, but are probably wrong. You need to
trace route both sides.
Every provider on the internet is trying to minimize costs. This means that
you want traffic to leave your network and go to the destination network with
as little distance traveled as possible, because costs go up with distance.
It's cheaper to increase the size of pipes within a city to get to a peering
point than to increase your backbone pipe size. So, peering contracts
typically specify that you dump traffic to the peer as soon as possible. That
means the person receiving the traffic generally pays more. It also means that
any traffic that crosses an AS boundary almost certainly travels a completely
different path each way. In many cases, one third party provider may be used
in one direction, another in the other direction. So seeing packet loss on
your traceroute at some random tinet router doesn't mean that this router is
the cause of any problem, since the return path for that packet from that
provider's router might actually cross yet another network that is never
transited in either direction for your network connection. (I'm ignoring that
most large providers also don't always send ICMP reliably BECAUSE they limit
this intentionally to spare the router CPU from overload - it takes router CPU
to generate an ICMP TTL exceeded, but it doesn't take router CPU to forward a
packet - so traceroute or ping indicating loss at a router doesn't mean
anything in itself - the path itself likely has zero percent loss).
So, here's the scenerio.
Let's say a user and a server are on two seperate networks, U (user) and S
(server).
Let's say they both utilize transit provider T. So the path could be: U -- T
-- S. S buys an OC12 from T, while U buys a T1.
But let's say that the user has a second transit provider, BIG, who is a free
peer of T. He bought an OC3 from BIG. So there's another path between U and
S: U -- BIG -- T -- S. Likely this path is much faster than U -- T -- S.
So, the path for the traffic to S goes U -- T -- S.
Now, what path does the traffic from T's router go, when T's router generates
an ICMP TTL exceeded in response from a traceroute from a user? Does it go
straight over the T1 line, or does it go over the peering connection to BIG and
then to the customer? The answer, it turns out, depends on network
configuration and policy. Let's say it goes out over the T1, but the T1 is
congested. It will look like the congestion is at the connection between BIG
and T, because this is the first hop that will show packet loss. BUT...the
congestion is actually at the U's connection to T, which is irrelevant to the
actual traffic path between U and S. So the user, at this point, calls up BIG
and T and bitches about "Your peering congestion is congested" when the real
problem is that traffic completely unrelated to the user's problem is going via
a congested path that is never used for connectivity between U and S.
If you add several providers into this loop, you can end up with a situation
where traffic uses Sprint in one direction, but never hits a Sprint router in
the other. This is actually very common. A user with slow downloads might be
experiencing packet loss on the path from server to user, but not the other way
around. In other words, the problem is a provider that never shows up on the
user's traceroute!
Remember that the providers hand off the traffic as soon as possible to
their peer. So, whoever receives the larger amount of traffic needs the
bigger cross-country (or trans-oceanic) links. If one side transmits a
T1's worth of data, the other side transmits an OC48's worth of data,
only one needs the OC48s across the country - the one receiving the
traffic. That's why you hear about "traffic ratios". If the traffic is even
both ways, both sides have to pay for the same amount of cross-country
infrastructure to carry that traffic. So most providers won't peer with
someone for free that sends, say, 10 times the amount of traffic that they will
receive. It would end up costing a lot of money
Back to IPv6...that's interesting, but what does it have to do with IPv6?
Some providers want to do away with traffic ratio policies, mutliple location
peering, not providing free services to the other's customers, etc.
THAT is why you can't ping some sites from your HE tunnel. It's not just that
providers won't peer. It's also that providers have rules to keep themselves
from getting screwed.
Certainly, there's ways around some of this (for example, traffic ratios - if I
make sure my network is used for the cross-country traffic I send, not yours,
then I've addressed that concern at a bit of increased expense for myself).
But it's generally not worth doing until the size of the providers is
sufficiently large. Other things don't have a good technical fix, like not
peering with your peer's customer - that's a business rule.
