Re: [IETF] Not Listening to the Ops Customer
On Jun 1, 2013, at 12:35 AM, Brian E Carpenter wrote: > On 01/06/2013 15:00, John C Klensin wrote: >> >> --On Friday, May 31, 2013 17:23 -0700 Randy Bush >> wrote: >> >>> < rant > >>> >>> the sad fact is that the ietf culture is often not very good at >>> listening to the (ops) customer. look at the cf we have made >>> out of ipv6. the end user, and the op, want the absolute >>> minimal change and cost, let me get an ipv6 allocation from >>> the integer rental monopoly, flip a switch or two, and get 96 >>> more bits no magic. 15 years later, dhcp is still a cf, i >>> have to run a second server (why the hell does isc not merge >>> them?), i can not use it for finding my gateway or vrrp exit, >>> ... at least we got rid of the tla/nla classful insanity. but >>> u/g? puhleeze. >> >> Randy, >> >> I suggest that characterizing that set of issues as IETF versus >> Ops is probably not completely right either. > It was more complicated. I was actually running a team that ran site > network ops at the time, and (since DHCP was not yet deployable), > IPv4 was then a serious nuisance to manage, compared say to Novell > Netware and, even, Appletalk. There were good reasons we wanted > stateless auto-configuration and unlimited subnet size, to mention > two IPv6 bells and whistles. If DHCP had already been deployed, > our opinion might have been different. > Yes, it was more complicated, but the sad part is that DHCP was widely deployed in V4 long before V6 was a viable transport for real work on the Internet. Operators kept asking for DHCP and kept being slapped down by zealots who refused to listen to / consider why the Ops were asking for this. I *really* want to make sure that my CEO always gets the same address, and want him to be assigned specific DNS servers and use a certain gateway. The folk who manage the DHCP are the "Internal Services Infrastructure Group" (aka "The windows monkeys") and I'm sure as heck not giving them access to my router to twiddle knobs on it. >> For example, with >> IPv6, the IETF has proposed multiple transition solutions with >> no roadmap as to which one to apply under what circumstances and >> growing evidence that some of those solutions are unworkable in >> practice and others are incompatible with what are claimed to be >> fundamental and important features of the Internet. > > It turns out that as soon as you envisage a network in which some > nodes only support 32 bit addresses and other nodes can't get > a globally unique 32 bit address, you are forced into a world > of hurt that requires some combination of NAT, tunnels and > dual-stack. That is completely independent of the design of > IPng, and we knew it 1994. > Yes, hindsight always makes things *much* clearer. It also provides a nice sandbox to stand on…. Anyway, I think that going down the path of "There are warts in V6, I want a refund!" is not helpful. What I hope we can try and focus on is how we can improve understanding of what the customer (operators, users) actually wants, how and where the protocols of the future will be used, how we can incorporate changes as we get feedback and understand new requirements, etc. I was not intending to point fingers (although I realize I did), but rather to try show places where the protocol could have improved with a better understanding of what folk were looking for / with more operator input... > So while your criticism is valid that we collectively came up > with too many such combinations, that collective mistake was > (IMHO) not a result of the design of IPv6 as such. It was more > caused by the design of human beings. I'd go further -- some of the issue is (IMO) caused by the consensus driven nature of how we do things. We all have different sets of interests and different priorities. Because of a need to achieve consensus, we end in something kind of like horse-trading. We add features to appease some set of folk, and compromise on other bits to appease others. I suspect that if one or two folk had designed this, soup-to-nuts, we'd have ended up with something cleaner. > >> It doesn't >> take a skilled operations person to understand that is a >> problem; someone with a pointy head and barely enough clue to >> figure out what a "bit" is much less how many of them are in >> various addresses can derive a "don't be the first person on the >> block" or "don't migrate if you can figure out an alternative" >> lesson from that. >> >> Similarly, various applications folks within the IETF have >> pointed out repeatedly that any approach that assigns multiple >> addresses, associated with different networks and different >> policies and properties, either requires the applications to >> understand those policies, properties, and associated routing >> (and blows up all of the historical application-layer APIs in >> the process) or requires request/response negotiation that TCP >> doesn't allow for (and blows up most of
Re: [IETF] Not Listening to the Ops Customer
--On Saturday, June 01, 2013 11:28 -0400 Warren Kumari wrote: >... > I *really* want to make sure that my CEO always gets the same > address, and want him to be assigned specific DNS servers and > use a certain gateway. The folk who manage the DHCP are the > "Internal Services Infrastructure Group" (aka "The windows > monkeys") and I'm sure as heck not giving them access to my > router to twiddle knobs on it. Warren, I sympathize. But I also note that your comment above can be read, in more general form, as "the folks in IT are turkeys and therefore we need to export the issues they can't be trusted to handle well to the Internet". That isn't a good model, if only because it involves moving the intelligence to the middle of the net. Assuming that you still work where I think you work, you've got a significant advantage over most of the people in similar circumstances: you have an Executive Chairman and a bunch of VPs who actually do understand the problem with incompetent internal IT departments and maybe even why they come out that way and many of them are pretty accessible. Identify the problem and suggest a workable solution. >... >> It turns out that as soon as you envisage a network in which >> some nodes only support 32 bit addresses and other nodes >> can't get a globally unique 32 bit address, you are forced >> into a world of hurt that requires some combination of NAT, >> tunnels and dual-stack. That is completely independent of the >> design of IPng, and we knew it 1994. > > Yes, hindsight always makes things *much* clearer. It also > provides a nice sandbox to stand on…. But the "knew it in 1994" part means that this is _not_ hindsight. Several of the other comments we have made aren't either -- they were well-understood in the community in 1994 and then rejected or ignored for one reason or another. One can complain about the rejection in hindsight, but not about lack of information. >... >> So while your criticism is valid that we collectively came up >> with too many such combinations, that collective mistake was >> (IMHO) not a result of the design of IPv6 as such. It was more >> caused by the design of human beings. > > > I'd go further -- some of the issue is (IMO) caused by the > consensus driven nature of how we do things. We all have > different sets of interests and different priorities. Because > of a need to achieve consensus, we end in something kind of > like horse-trading. We add features to appease some set of > folk, and compromise on other bits to appease others. I > suspect that if one or two folk had designed this, > soup-to-nuts, we'd have ended up with something cleaner. See my note about the nature of standards. >... john
Re: [IETF] Not Listening to the Ops Customer (more)
On Jun 2, 2013, at 10:15 AM, John C Klensin wrote:
> --On Saturday, June 01, 2013 11:28 -0400 Warren Kumari
> wrote:
>> ...
>>> It turns out that as soon as you envisage a network in which
>>> some nodes only support 32 bit addresses and other nodes
>>> can't get a globally unique 32 bit address, you are forced
>>> into a world of hurt that requires some combination of NAT,
>>> tunnels and dual-stack. That is completely independent of the
>>> design of IPng, and we knew it 1994.
>>
>> Yes, hindsight always makes things *much* clearer. It also
>> provides a nice sandbox to stand on….
>
> But the "knew it in 1994" part means that this is _not_
> hindsight. Several of the other comments we have made aren't
> either -- they were well-understood in the community in 1994 and
> then rejected or ignored for one reason or another. One can
> complain about the rejection in hindsight, but not about lack of
> information.
Indeed, it was well-understood that there needed to be some solution
involving NAT to provide for basic interoperability (even if no one
liked standardizing that aspect); specifically, "The IPng transition
plan must define the procedures required to successfully implement
the functions which vendors will be likely to include in their devices.
This is the case even if there are good arguments to recommend against
a particular function, header translation for example. If products
will exist it is better to have them interoperate than not." [RFC 1752,
Section 16.1]
The problem was that the IETF felt enormous pressure _for its own
efficiency_ to come to a decision about IPng; i.e. we did not have
operators clamoring that they needed IPng to be decided ASAP (I'd
wager that most commercial service providers were far too busy trying
to keep up with the epic customer demand in the early 90's to pay
much attention to the IPv4 depletion problem which was many, many
years away...) As you know, there were real concerns about transition
expressed by operators at the IPdecide BoF ('93 Amsterdam IETF) -
"It was stated frequently and forcibly that the transition costs
should be a significant factor in the selection criteria. Concerns
were expressed by several service providers that the developers
had little appreciation of the real-world networking complexities
that transition would force people to cope with. If the cost of
transition outweighed the pain of other solutions (application
gateways or address translators) customers would not deploy IPng."
[ftp://ftp.ietf.org/ietf/93jul/ipdecide-minutes-93jul.txt]
(Highly recommended reading for folks who are interested in
the history of IPv6)
In the end, the IETF decided it should take active steps toward making
the technical decision on IPng (rather than waiting for the "marketplace"
to decide) and that IESG would form the IPng Directorate [RFC 1719] which
was to evaluate how long we had to IPv4 runout and develop the technical
requirements and criteria for IPng. Once this was completed at the end
of 1994, there was enormous pressure to select a single IPng candidate
protocol to work on (despite the unresolved issues regarding transition)
so that the work of the IETF could be focused in a single effort going
forward. This approach might have worked out fine, but once the actual
IPng candidate decision was made, all of the prioritization on transition
requirements dissipated as the work was shuffled off to various working
groups in the years that followed, with the result being that the once
mandatory IPng requirement for a straightforward transition plan from
IPv4 was never fulfilled.
I was asked to be more specific on how to avoid such issues the future,
so here goes:
If multiple solutions can readily coexist, let the market decide.
If not, define the problem from the perspective of those who must
deploy the solution, including their inherent need for a clear and
functional transition mechanism.
In retrospect, it's fairly obvious that there's a huge difference between
problem spaces which allow for multiple solutions (e.g. interior routing
protocols) versus problem spaces which the IETF is aiming for only _one_
protocol in the final solution (e.g. IP, DNS); the process making that
type of architectural decision should probably be quite visible and rather
explicit because it comes with a huge responsibility for addressing all
necessary transition aspects. This responsibility for the transition
aspects should not be set aside by the IETF for convenience or due to the
level of effort that results, as it is the price to be paid when making
an architectural decision to have a single protocol solution for a given
problem space.
FYI,
/John
p.s. John - you made a comparison to electrical power standards with
respect to operator & end-user participation: I guess the parallel
would be that the electrical equipment manufacturers (along with
engineers from electrical production an
Re: [IETF] Not Listening to the Ops Customer (was Re: Issues in wider geographic participation)
On May 31, 2013, at 8:23 PM, Randy Bush wrote:
> < rant >
>
> the sad fact is that the ietf culture is often not very good at
> listening to the (ops) customer. look at the cf we have made out of
> ipv6. the end user, and the op, want the absolute minimal change and
> cost, let me get an ipv6 allocation from the integer rental monopoly,
> flip a switch or two, and get 96 more bits no magic.
Unfortunatly
Yup, the "issue" with v4 was simply a lack of addresses -- more address bits
was what ops wanted, this probably needed a new protocol number, but that's
about all.
Unfortunately the was a bad case of creeping featuritis and we got:
A new, and unfortunately very complex way of resolving L2 addresses.
Magic IPSec, which then went away again.
Extension headers that make it so you cannot actually forward packets in modern
hardware ( http://tools.ietf.org/html/draft-wkumari-long-headers-00)
SLAAC, which then required privacy addressing and then fun that that entails /
the DHCP debacle.
RA instead of VRRP
Countless transition strategies.
The list goes on and on, but gets depressing really fast.
Operators learnt a number of lessons (the hard way) while operating IPv4
networks that reoccured in new ways in IPv6.
For example, operators learnt that having a large subnet behind a router makes
the router fall over (doing all the ARP processing) during an IP scan. This is
almost identical to the issue described in RFC 6583 ("Operational Neighbor
Discovery Problems")
Operators learnt (a long time back) that allowing source-routing is a really
bad idea, and so a: devices disable it by default and / or b: the standard
templates all have "no ip source-route" (or equivalent). This is almost
identical to the Routing Header issues in V6 (see RFC 5095 - "Deprecation of
Type 0 Routing Headers in IPv6" )
Most operators address ptp Ethernet links with a /30 or /31 in V4. Took a long
time to get this in V6 (RFC 6164 - "Using 127-Bit IPv6 Prefixes on Inter-Router
Link") and it is still controversial.
We ended up in a space where perceived elegance and shiny features overshadowed
what folk actually wanted -- 96 more bits, no magic.
> 15 years later,
> dhcp is still a cf, i have to run a second server (why the hell does
> isc not merge them?), i can not use it for finding my gateway or vrrp
> exit, ... at least we got rid of the tla/nla classful insanity. but
> u/g? puhleeze.
Yup
>
> at ripe/dublin, olaf gave a really nice but somewhat glib talk about
> technology adoption, using dnssec and ipv6 as the positive examples.
Yes, this was a great talk -- I think that it was somewhat glib for the
audience, but having a longer, more in depth version of it at an IETF would
(IMO) be valuable. I think that Olaf has a few versions of that talk…. For folk
who'd like to see the RIPE version - https://ripe66.ripe.net/archives/video/7/
- Innovation at the Waist
> as
> some curmudgeonly schmuck pointed out at the mic, dnssec is forward
> compatible and there are no alternatives, so it is being adopted despite
> its complexity and warts. ipv6 is not forward compatible, we put
> unnecessary obstacles in the deployment path, and there are
> alternatives. d o o m.
>
> if we had wanted ipng deployed, we would have done everything we could
> to make it simple and easy for net-ops and end users to turn it on.
> instead, we made it complex and hard and then blame everyone else for
> not instantly adopting it en masse.
> the ietf did not listen to or
> consider the customer. this is fatal. and the arrogance is taking what
> is left of the e2e internet down the drain with it.
>
+1.
w
> randy
>
--
Don't be impressed with unintelligible stuff said condescendingly.
-- Radia Perlman.
Re: [IETF] Not Listening to the Ops Customer (was Re: Issues in wider geographic participation)
Warren Kumari wrote: Unfortunately the was a bad case of creeping featuritis and we got: A new, and unfortunately very complex way of resolving L2 addresses. You may use ARP (and DHCP) with IPv6. Extension headers that make it so you cannot actually forward > packets in modern hardware > ( http://tools.ietf.org/html/draft-wkumari-long-headers-00) True. SLAAC, which then required privacy addressing and then fun that > that entails / the DHCP debacle. The problem of SLAAC is that it is stateful in fully distributed manner. That is all the nodes have their own states on address assignment Most operators address ptp Ethernet links with a /30 or /31 in V4. Took a long time to get this in V6 (RFC 6164 - "Using 127-Bit IPv6 Prefixes on Inter-Router Link") and it is still controversial. We ended up in a space where perceived elegance and shiny > features overshadowed what folk actually wanted > -- 96 more bits, no magic. Maybe. But the folk actually needed 8 (or 16 at most) more bits. >> 15 years later, >> dhcp is still a cf, i have to run a second server (why the hell does >> isc not merge them?), i can not use it for finding my gateway or vrrp >> exit, ... at least we got rid of the tla/nla classful insanity. but >> u/g? puhleeze. > > Yup TLA/NLA is a good thing, *IF* multiple addresses of a node and automatic renumbering including routers and DNS were properly supported. It is not very difficult to have done so. Masataka Ohta
