Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Shumon Huque]

On Tue, Jun 19, 2018 at 2:51 PM 神明達哉  wrote:

> At Mon, 18 Jun 2018 17:51:26 -0400,
> Shumon Huque  wrote:
>
> > Client applications delegate address sorting to name resolution functions
> > like getaddrinfo() - correct?
> >
> > Doing some quick tests of getaddrinfo() just now on a recent *NIX
> machine,
> > it appears to return addresses sorted roughly in accordance with RFC
> > 6724, but rule 9 (longest prefix match) seems to be only applied to
> > IPv6 addresses. For IPv4 addresses (using an upstream resolver
> > that randomizes the response RRset), the order returned by getaddrinfo()
> > is ever changing - I assume presented in the order received.
>
> The very original implementation of BSD getaddrinfo() conformed to the
> RFC, but FreeBSD seems to have changed the behavior a few years ago so
> rule 9 wouldn't apply to IPv4 addresses:
>
> https://github.com/freebsd/freebsd/commit/1390d13ae69089142f6c6465dbb24438f295edee
> The commit log suggests the rrset round-robin was exactly the
> motivation for the change (whether it was good or bad).
>

I took a quick look at glibc upstream source (from which I presume most
Linux distros take code) and see the following:

https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/posix/getaddrinfo.c;h=553833d1f22d6ea4ae24c68fe14e2140aa059d50;hb=HEAD#l1559

/* Rule 9: Use longest matching prefix.  */
[...]
  /* Outside of subnets, as defined by the network masks,
 common address prefixes for IPv4 addresses make no sense.
 So, define a non-zero value only if source and
 destination address are on the same subnet.  */

So again, no Rule 9 implementation for IPv4, unless there is a local subnet
destination involved, because it makes no sense otherwise (with which I
agree, as I previously remarked).

Shumon.
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Shumon Huque]

On Tue, Jun 19, 2018 at 2:51 PM 神明達哉  wrote:

> At Mon, 18 Jun 2018 17:51:26 -0400,
> Shumon Huque  wrote:
>
> > Client applications delegate address sorting to name resolution functions
> > like getaddrinfo() - correct?
> >
> > Doing some quick tests of getaddrinfo() just now on a recent *NIX
> machine,
> > it appears to return addresses sorted roughly in accordance with RFC
> > 6724, but rule 9 (longest prefix match) seems to be only applied to
> > IPv6 addresses. For IPv4 addresses (using an upstream resolver
> > that randomizes the response RRset), the order returned by getaddrinfo()
> > is ever changing - I assume presented in the order received.
>
> The very original implementation of BSD getaddrinfo() conformed to the
> RFC, but FreeBSD seems to have changed the behavior a few years ago so
> rule 9 wouldn't apply to IPv4 addresses:
>
> https://github.com/freebsd/freebsd/commit/1390d13ae69089142f6c6465dbb24438f295edee
> The commit log suggests the rrset round-robin was exactly the
> motivation for the change (whether it was good or bad).
>

Very useful information. Thanks for confirming!

My tests were on a Linux machine, so apparently they did the same ..

Shumon.
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[神明達哉]

At Mon, 18 Jun 2018 17:51:26 -0400,
Shumon Huque  wrote:

> Client applications delegate address sorting to name resolution functions
> like getaddrinfo() - correct?
>
> Doing some quick tests of getaddrinfo() just now on a recent *NIX machine,
> it appears to return addresses sorted roughly in accordance with RFC
> 6724, but rule 9 (longest prefix match) seems to be only applied to
> IPv6 addresses. For IPv4 addresses (using an upstream resolver
> that randomizes the response RRset), the order returned by getaddrinfo()
> is ever changing - I assume presented in the order received.

The very original implementation of BSD getaddrinfo() conformed to the
RFC, but FreeBSD seems to have changed the behavior a few years ago so
rule 9 wouldn't apply to IPv4 addresses:
https://github.com/freebsd/freebsd/commit/1390d13ae69089142f6c6465dbb24438f295edee
The commit log suggests the rrset round-robin was exactly the
motivation for the change (whether it was good or bad).

--
JINMEI, Tatuya

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Paul Vixie]

Florian Weimer wrote:

* Paul Vixie:


in other words we should re-order rrsets by default, so that very few
people or agents are ever prone to think their order is stable. the spec
says they are unordered, but human nature says, expect more of what
you're seeing.


But the client has to sort them again based on shared prefix length
with its own address.


happy eyeballs says so, but some clients won't implement anything like 
that, or will have local knowledge (access to a routing table via an API 
of some kind) that gives them even better information. the question of 
what a client should do, and whether a given client can or should, is 
clearly independent of what order the response comes to them in.



I think we should fix that as well, otherwise the overall protocol
disadvantages new entrants who cannot get a contiguous prefix in
which they can place all their load balancer endoints, so that they
are immune from that mandatory client sorting.


i have no strongly held view on whether clients should sort. but i'm 
sure a client will have better information to base such sorting on than 
the dns servers will have. therefore i'd like to pursue a simple rule 
whereby the server perturbs the order somehow. simplistic clients, and 
clients who aren't nearby any shard of a load balanced service, should 
see some kind of chaos in the ordering, not just know it can be there.


--
P Vixie

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Shumon Huque]

On Mon, Jun 18, 2018 at 7:05 PM Darcy Kevin (FCA) 
wrote:

> RFC 6724 specifically says: "Rules 9 and 10 MAY be superseded if the
> implementation has other
> means of sorting destination addresses. For example, if the
> implementation somehow knows which destination addresses will result
> in the 'best' communications performance."
>
> So, technically, if an implementation chooses a method of "the exact order
> in which I received the address records from my upstream resolver" as a way
> to produce the "'best' communication performance", given the circumstances,
> then that is technically not a violation of the standard. The local
> optimization is to trust the upstream resolver to Do The Right Thing. It's
> not always a wise choice, but most of the time it's better than sorting
> based on prefix-length matching (right?)
>
> RFC 6724 is, after all, about *default* address selection (that word is
> even in the title of the RFC). Defaults are made to be superseded -- that's
> kind of the definition of what a default *is*.
>

This whole thread is about "defaults"  though!

Application deployers attempting to rely (rightly or wrongly) on load
balancing of addresses presented by name resolution APIs will have
preferences for a particular default that allows them to achieve that goal.

(I'm well aware that the spec allows and that OSes provide knobs to tweak
the address selection algorithm - I've lost track of how many times I've
edited my gai.conf file to do various kinds of tests!)

Shumon.
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Darcy Kevin (FCA)]

RFC 6724 specifically says: "Rules 9 and 10 MAY be superseded if the 
implementation has other
means of sorting destination addresses. For example, if the
implementation somehow knows which destination addresses will result
in the 'best' communications performance."

So, technically, if an implementation chooses a method of "the exact order in 
which I received the address records from my upstream resolver" as a way to 
produce the "'best' communication performance", given the circumstances, then 
that is technically not a violation of the standard. The local optimization is 
to trust the upstream resolver to Do The Right Thing. It's not always a wise 
choice, but most of the time it's better than sorting based on prefix-length 
matching (right?)

RFC 6724 is, after all, about *default* address selection (that word is even in 
the title of the RFC). Defaults are made to be superseded -- that's kind of the 
definition of what a default *is*.


- Kevin



-Original Message-
From: DNSOP  On Behalf Of Florian Weimer
Sent: Monday, June 18, 2018 4:23 AM
To: Erik Nygren ; dnsop WG 
Subject: Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on 
bind 9.12 bug (sorting rrsets by default)

On 06/15/2018 05:45 PM, Erik Nygren wrote:
> I suspect starting assumptions are roughly in the range of:
> 
> * Recursive (and stub?) resolvers (SHOULD/MUST?) do some form of 
> round-robin in RRset responses.
> 
> * There are a variety of ways to implement round-robin (randomize, 
> permute, etc).
> 
> * Server operators need to be aware that round-robin may be a part of 
> a load balancing scheme (especially if capacity is far greater than 
> average demand) but should not be relied on exclusively.  (Perhaps 
> with examples of why...)
> 
> Am I missing something in-terms of an existing BCP to this effect?

Unless all addresses happen to have identical shared prefix length with the 
client address (that is, count-leading-zeros(client-address XOR
server-address) is the same), RFC 6724 Rule 9 requires that clients do
*not* perform random server selection.

I think this was a mistake in RFC 3484, and it is still wrong.  But I think 
procedurally, you cannot change this in a BCP.

Thanks,
Florian

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Shumon Huque]

On Mon, Jun 18, 2018 at 5:14 PM Florian Weimer  wrote:

> * Paul Vixie:
>
> > in other words we should re-order rrsets by default, so that very few
> > people or agents are ever prone to think their order is stable. the spec
> > says they are unordered, but human nature says, expect more of what
> > you're seeing.
>
> But the client has to sort them again based on shared prefix length
> with its own address.


I assume you're talking about the default destination address selection
rules specified in RFC 6724 (Section 6, Rule 9). Yeah, I agree this doesn't
seem wise to me. I'm not sure what the goal here was - this doesn't get
the client the topologically closest destination (in the general case; even
for IPv6 addresses unless the Internet was using solely provider aggregated
hierarchical routing, which isn't the case).


> I think we should fix that as well, otherwise
> the overall protocol disadvantages new entrants who cannot get a
> contiguous prefix in which they can place all their load balancer
> endoints, so that they are immune from that mandatory client sorting.
>

Client applications delegate address sorting to name resolution functions
like getaddrinfo() - correct?

Doing some quick tests of getaddrinfo() just now on a recent *NIX machine,
it appears to return addresses sorted roughly in accordance with RFC
6724, but rule 9 (longest prefix match) seems to be only applied to
IPv6 addresses. For IPv4 addresses (using an upstream resolver
that randomizes the response RRset), the order returned by getaddrinfo()
is ever changing - I assume presented in the order received.

I haven't actually read the code, so perhaps someone can confirm this, or
share what portions of the default address selection algorithm common
name resolution libraries actually implement.

Shumon.
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Florian Weimer]

* Paul Vixie:

> in other words we should re-order rrsets by default, so that very few 
> people or agents are ever prone to think their order is stable. the spec 
> says they are unordered, but human nature says, expect more of what 
> you're seeing.

But the client has to sort them again based on shared prefix length
with its own address.  I think we should fix that as well, otherwise
the overall protocol disadvantages new entrants who cannot get a
contiguous prefix in which they can place all their load balancer
endoints, so that they are immune from that mandatory client sorting.

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Paul Vixie]

to the extent that the dns technical community has a choice about 
default behaviour, we should consider the costs to the rest of the 
internet community of each default.


in my prior e-mail to this thread i gave examples of assumptions of 
ordering that were violated by the first round-robin implementation. in 
both cases, the assumption was dangerous -- led to fragility. (leaving a 
bad NS in an RRset, and putting verses of poetry in TXT RRsets.)


assumptions of non-ordering are less dangerous. (expect load balancing 
and don't get it.)


we should, if we can engage on the topic of defaults at all, recommend a 
default that is compatible with less-dangerous assumptions.


in other words we should re-order rrsets by default, so that very few 
people or agents are ever prone to think their order is stable. the spec 
says they are unordered, but human nature says, expect more of what 
you're seeing.


vixie

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Andrew Sullivan]

On Fri, Jun 15, 2018 at 04:17:00PM -0400, Erik Nygren wrote:
> We have many years of software that relies on emergent behaviors from the
> current default.

Well, I think more accurately we have years of software that relies on
emergent behaviours of the prior default of certain implemnetations.

> While pedantically it may be true that these should be treated as unordered
> sets

I don't think that's being pedantic.  That's literally the definition
of an RRset, and (as I guess has been shown by others in this thread)
relying on an existing order behaviour to which one has become
accustomed is going to be a problem even if people just switch out
their software or change the configuration of their systems.  This is
a bug in the relying software, because the network _never_ made the
promise that software was relying upon.

> Software should have safe defaults that matches common expectations.

I think this is true only if the common expectations are reasonable
ones, and given what RRsets are the expectation in this case is not a
reasonable one.  What you're really saying is that, if there was ever
one dominant system on the Internet (in this case, BIND), then the
standards all need to be rewritten to conform not only to what that
software implemented (a position with which I have some sympathy) but
also to conform with the default settings of that software (a position
I think needs rather more support than you've offered).  

> is that the order of results is NOT consistent.

Sure.  "Unordered sets."

>  In many environments, this
> lack
> of consistency is relied upon for systems to work properly.

To me, this is like saying that, in many environments, the order of
TCP packets (which very frequently do come in order) is relied upon
for systems to work properly.  This is true, but still broken.

> This ambiguity in the current specifications

What is the ambiguity?  There is only an ambiguity if you think that
people's expectation of something nobody ever promised them is part of
the specification, and it isn't.

Best regards,

A
-- 
Andrew Sullivan
a...@anvilwalrusden.com

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Florian Weimer]

On 06/15/2018 05:45 PM, Erik Nygren wrote:

I suspect starting assumptions are roughly in the range of:

* Recursive (and stub?) resolvers (SHOULD/MUST?) do some form of 
round-robin in RRset responses.


* There are a variety of ways to implement round-robin (randomize, 
permute, etc).


* Server operators need to be aware that round-robin may be a part of a 
load balancing scheme (especially if capacity is far greater than 
average demand) but should not be relied on exclusively.  (Perhaps with 
examples of why...)


Am I missing something in-terms of an existing BCP to this effect?


Unless all addresses happen to have identical shared prefix length with 
the client address (that is, count-leading-zeros(client-address XOR 
server-address) is the same), RFC 6724 Rule 9 requires that clients do 
*not* perform random server selection.


I think this was a mistake in RFC 3484, and it is still wrong.  But I 
think procedurally, you cannot change this in a BCP.


Thanks,
Florian

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Peter van Dijk]

On 16 Jun 2018, at 2:14, Shumon Huque wrote:


Yeah, good point about side channels. Let's stick to recommending
randomization!



Unbound has interesting middle ground here:

   rrset-roundrobin: 
  If yes, Unbound rotates RRSet order in response (the 
random number is taken from the query ID,

  for speed and thread safety).  Default is no.

It rotates, but you cannot predict (easily) by how much. It keeps the 
implementation simple but mostly avoids (as far as I can judge) the side 
channel.


I do want to point out that the default is ‘no’, suggesting it is 
getting away with no ‘round robin’ at all in many deployments.


Kind regards,
--
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Erik Nygren]

On Fri, Jun 15, 2018 at 8:12 PM, Shumon Huque  wrote:

> On Fri, Jun 15, 2018 at 6:40 PM Mark Andrews  wrote:
>
>> What we should be asking is why a service that needs multiple machines
>> isn’t using SRV. It has randomisation between servers explicitly defined
>> along with proportional load balancing.
>>
>
> That would be nice, but sadly major applications like the web, have (in my
> assessment) never seriously shown any interest in using SRV records. To
> quote some of the discussion of HTTP2 on this subject:
>

Agreed that it would be nice (and a good recommendation for protocols to
support SRV records),
but for all of the other protocols we're stuck operating today today with
the implementations we have,
not just the ones we wish we had.

(Side-note, I do recommend folks provide feedback on the ALTSVC draft:
https://tools.ietf.org/html/draft-schwartz-httpbis-dns-alt-svc-01
It adds enough value and extensibility beyond just SRV that
browsers show more interest in implementing it.)

> Yeah, good point about side channels. Let's stick to recommending
randomization!

Good point!  Agreed.  I hadn't thought about that and had otherwise been
thinking of shuffling and randomization as roughly equivalent as long as you
pick a random initial offset for the shuffling.
(More reasons to write current best practices down in a draft.)
Shuffling may also have some weird patterns showing up depending on the
relation between
TTLs and number of clients querying, but I don't know if this is ever
problematic
beyond the side-channel issue.

Erik
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Shumon Huque]

Yeah, good point about side channels. Let's stick to recommending
randomization!

Shumon.

On Fri, Jun 15, 2018 at 8:01 PM Colm MacCárthaigh  wrote:

>
> I think so too; and I wouldn't be so strict on backwards compatibility
> there.
>
> That behavior is a side-channel that defeats DNS privacy in some cases.
> E.g. I can query a record, watch you send an encrypted query, then query
> the record again, and tell what you queried. Within some probability at
> least.
>
> For that reason, It'd be worth experimenting with an implementation that
> does shuffle the results each time.
>
> On Fri, Jun 15, 2018 at 4:54 PM, Shumon Huque  wrote:
>
>> On Fri, Jun 15, 2018 at 5:55 PM Colm MacCárthaigh 
>> wrote:
>>
>>>
>>> Just a question on this: was the old/classic behavior really
>>> random/shuffled? Or was it that bind would "rotate" through iterations
>>> where the order was the same each time if you think of the rrset list as a
>>> ring, but with a different start and end point within that ring? (That's
>>> what's described here:
>>> https://docstore..mik.ua/orelly/networking_2ndEd/dns/ch10_07.htm
>>> )
>>>
>>
>> ISC veterans can confirm, but my recollection is that the earliest
>> implementations were indeed as described above - the response RRset was
>> cycled/rotated, rather than randomized.
>>
>> Shumon.
>>
>>
>
>
> --
> Colm
>
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Shumon Huque]

On Fri, Jun 15, 2018 at 6:40 PM Mark Andrews  wrote:

> What we should be asking is why a service that needs multiple machines
> isn’t using SRV. It has randomisation between servers explicitly defined
> along with proportional load balancing.
>

That would be nice, but sadly major applications like the web, have (in my
assessment) never seriously shown any interest in using SRV records. To
quote some of the discussion of HTTP2 on this subject:

from https://lists.w3.org/Archives/Public/ietf-http-wg/2015AprJun/0674.html:

--- excerpt

"We discussed this, but decided not to support SRV in HTTP/2 - mostly
because HTTP/2 needs to be backwards-compatible with the existing Web,
which doesn't use SRV. In discussion, it also became pretty clear that if
we were to use an additional record type for HTTP, SRV may not suit,
because we need to do things like protocol version negotiation.

There were also concerns about latency and interoperability (especially
considering how many DNS requests a page load can make, and the unfortunate
limitations of some DNS gateways/proxies)."

--- end excerpt

The latter point about additional latency, due to the need for additional
follow-on queries to resolve the address records for the SRV targets (if
the resolver hasn't provided them), has been a pretty potent blocker for
SRV adoption in the web.

Also, in my experience, many implementations of client applications that do
employ SRV don't even bother to implement the load balancing aspect, just
the priority aspect. This was particularly true in Kerberos from my
recollection.


> It also addresses CNAME at the apex which quite frankly is only used to
> find the name of the HTTP server for the zone which SRV  is quite capable
> of doing.
>

Yup ..

Shumon.
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Colm MacCárthaigh]

I think so too; and I wouldn't be so strict on backwards compatibility
there.

That behavior is a side-channel that defeats DNS privacy in some cases.
E.g. I can query a record, watch you send an encrypted query, then query
the record again, and tell what you queried. Within some probability at
least.

For that reason, It'd be worth experimenting with an implementation that
does shuffle the results each time.

On Fri, Jun 15, 2018 at 4:54 PM, Shumon Huque  wrote:

> On Fri, Jun 15, 2018 at 5:55 PM Colm MacCárthaigh 
> wrote:
>
>>
>> Just a question on this: was the old/classic behavior really
>> random/shuffled? Or was it that bind would "rotate" through iterations
>> where the order was the same each time if you think of the rrset list as a
>> ring, but with a different start and end point within that ring? (That's
>> what's described here: https://docstore..mik.ua/
>> orelly/networking_2ndEd/dns/ch10_07.htm
>> )
>>
>
> ISC veterans can confirm, but my recollection is that the earliest
> implementations were indeed as described above - the response RRset was
> cycled/rotated, rather than randomized.
>
> Shumon.
>
>


-- 
Colm
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Shumon Huque]

On Fri, Jun 15, 2018 at 5:55 PM Colm MacCárthaigh  wrote:

>
> Just a question on this: was the old/classic behavior really
> random/shuffled? Or was it that bind would "rotate" through iterations
> where the order was the same each time if you think of the rrset list as a
> ring, but with a different start and end point within that ring? (That's
> what's described here:
> https://docstore..mik.ua/orelly/networking_2ndEd/dns/ch10_07.htm
> )
>

ISC veterans can confirm, but my recollection is that the earliest
implementations were indeed as described above - the response RRset was
cycled/rotated, rather than randomized.

Shumon.
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Mark Andrews]

What we should be asking is why a service that needs multiple machines isn’t 
using SRV. It has randomisation between servers explicitly defined along with 
proportional load balancing. 

It also addresses CNAME at the apex which quite frankly is only used to find 
the name of the HTTP server for the zone which SRV  is quite capable of doing.

-- 
Mark Andrews

> On 16 Jun 2018, at 01:45, Erik Nygren  wrote:
> 
> A number of folks have been bitten by a bug in bind 9..12 where it silently 
> changes the default sorting of rrsets to always be sorted (even if the 
> authoritative response wasn't sorted).  This causes problems for services 
> assuming at least some degree of round-robin behavior by clients as now many 
> clients would sent all traffic to only the lowest IP.  Bug details:  
> https://gitlab.isc.org/isc-projects/bind9/issues/336   If you are upgrading 
> to or have upgraded to bind 9.12 you likely want to take a fix or override in 
> config.
> 
> 
> This raises the question of whether there would be value in a more modern BCP 
> covering round-robin expectations for recursive resolvers?  I suspect many 
> (most?) service operators take at least some degree of DNS round-robin 
> behavior by recursive resolvers as a default.
> 
> I suspect starting assumptions are roughly in the range of:
> 
> * Recursive (and stub?) resolvers (SHOULD/MUST?) do some form of round-robin 
> in RRset responses.
> 
> * There are a variety of ways to implement round-robin (randomize, permute, 
> etc).
> 
> * Server operators need to be aware that round-robin may be a part of a load 
> balancing scheme (especially if capacity is far greater than average demand) 
> but should not be relied on exclusively.  (Perhaps with examples of why)
> 
> Am I missing something in-terms of an existing BCP to this effect?  There's 
> RFC 1794, but I couldn't find much else (but given the sheer number of DNS 
> RFCs it's very likely I missed one).
> 
> Best, Erik
> 
> ___
> DNSOP mailing list
> DNSOP@ietf.org
> https://www.ietf.org/mailman/listinfo/dnsop
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Colm MacCárthaigh]

Just a question on this: was the old/classic behavior really
random/shuffled? Or was it that bind would "rotate" through iterations
where the order was the same each time if you think of the rrset list as a
ring, but with a different start and end point within that ring? (That's
what's described here:
https://docstore.mik.ua/orelly/networking_2ndEd/dns/ch10_07.htm)


On Fri, Jun 15, 2018 at 1:17 PM, Erik Nygren  wrote:

> On Fri, Jun 15, 2018 at 3:52 PM, Mukund Sivaraman  wrote:
>
>> On Fri, Jun 15, 2018 at 02:38:00PM -0400, Bob Harold wrote:
>> > Round-robin is a documented feature that many applications use.
>> Removing
>> > it from DNS resolvers, and then having to add it to a much larger
>> number of
>> > applications, does not seem like a good trade-off.
>>
>> The _default_ in BIND 9.12 was changed from order random to order
>> none. It seems to be missing from the release notes by mistake, but the
>> administrator manual mentions what the default is
>>
>
> We have many years of software that relies on emergent behaviors from the
> current default.
> While pedantically it may be true that these should be treated as
> unordered sets and that
> applications or stub resolver libraries should do some permutations or
> randomized selection,
> that doesn't match the current reality for widely used software (eg, curl
> and ssh, which I'm
> sure is just the tip of the iceberg).
>
> Software should have safe defaults that matches common expectations.
> Those common expectations, as demonstrated by the configuration of all
> of the large public resolvers I've tested, as well as by how common
> software behaves,
> is that the order of results is NOT consistent.  In many environments,
> this lack
> of consistency is relied upon for systems to work properly..  Switching to
> consistent
> order is no big deal on a small scale, but a widespread shift (eg, as
> would happen
> due to a change in default in popular software) would almost certainly
> have
> significant operational impact and is something that warrants significant
> discussion
> about the practical implications.
>
> This ambiguity in the current specifications results in this mismatch
> between the pedantic (rrsets are explicitly unordered, and a consistent
> order is a subset of that) and the current reality (applications and
> services
> rely on resolvers-at-scale to be explicitly inconsistent in the ordering
> of rrsets)
> is why I started off by proposing that we may need a BCP or informational
> RFC
> that describes the currently assumed defaults and best-practices
> (ie, round-robin is assumed in many places so don't consistently order
> at-scale by default).
>
> Erik
>
>
>
>
>
>
>
>
> ___
> DNSOP mailing list
> DNSOP@ietf.org
> https://www.ietf.org/mailman/listinfo/dnsop
>
>


-- 
Colm
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Paul Vixie]

Erik Nygren wrote:

...

This ambiguity in the current specifications results in this
mismatch between the pedantic (rrsets are explicitly unordered, and a
consistent order is a subset of that) and the current reality
(applications and services rely on resolvers-at-scale to be
explicitly inconsistent in the ordering of rrsets) is why I started
off by proposing that we may need a BCP or informational RFC that
describes the currently assumed defaults and best-practices (ie,
round-robin is assumed in many places so don't consistently order
at-scale by default).


i am +1 to this view, which is why i didn't suggest you move it to 
bind-workers@.


--
P Vixie

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Mukund Sivaraman]

On Fri, Jun 15, 2018 at 04:17:00PM -0400, Erik Nygren wrote:
> Software should have safe defaults that matches common expectations.
> Those common expectations, as demonstrated by the configuration of all
> of the large public resolvers I've tested, as well as by how common
> software behaves,

I agree with this.

Perhaps this should be done by default when recursion is on.

Mukund

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Erik Nygren]

On Fri, Jun 15, 2018 at 3:52 PM, Mukund Sivaraman  wrote:

> On Fri, Jun 15, 2018 at 02:38:00PM -0400, Bob Harold wrote:
> > Round-robin is a documented feature that many applications use.  Removing
> > it from DNS resolvers, and then having to add it to a much larger number
> of
> > applications, does not seem like a good trade-off.
>
> The _default_ in BIND 9.12 was changed from order random to order
> none. It seems to be missing from the release notes by mistake, but the
> administrator manual mentions what the default is
>

We have many years of software that relies on emergent behaviors from the
current default.
While pedantically it may be true that these should be treated as unordered
sets and that
applications or stub resolver libraries should do some permutations or
randomized selection,
that doesn't match the current reality for widely used software (eg, curl
and ssh, which I'm
sure is just the tip of the iceberg).

Software should have safe defaults that matches common expectations.
Those common expectations, as demonstrated by the configuration of all
of the large public resolvers I've tested, as well as by how common
software behaves,
is that the order of results is NOT consistent.  In many environments, this
lack
of consistency is relied upon for systems to work properly.  Switching to
consistent
order is no big deal on a small scale, but a widespread shift (eg, as would
happen
due to a change in default in popular software) would almost certainly have
significant operational impact and is something that warrants significant
discussion
about the practical implications.

This ambiguity in the current specifications results in this mismatch
between the pedantic (rrsets are explicitly unordered, and a consistent
order is a subset of that) and the current reality (applications and
services
rely on resolvers-at-scale to be explicitly inconsistent in the ordering of
rrsets)
is why I started off by proposing that we may need a BCP or informational
RFC
that describes the currently assumed defaults and best-practices
(ie, round-robin is assumed in many places so don't consistently order
at-scale by default).

Erik
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Mukund Sivaraman]

On Fri, Jun 15, 2018 at 02:38:00PM -0400, Bob Harold wrote:
> Round-robin is a documented feature that many applications use.  Removing
> it from DNS resolvers, and then having to add it to a much larger number of
> applications, does not seem like a good trade-off.

The _default_ in BIND 9.12 was changed from order random to order
none. It seems to be missing from the release notes by mistake, but the
administrator manual mentions what the default is:

 
By default, records are returned in indeterminate but
consistent order (see none above).
  

As before, an admin can configure it to any of {fixed, random, cyclic,
none} per class/type/domain, in named.conf. See the section "RRset
Ordering" in the administror manual.

No sorting functionality was removed. The default was changed.

Mukund

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Jared Mauch]

> On Jun 15, 2018, at 2:01 PM, Peter van Dijk  
> wrote:
> 
> Hello Andrew,
> 
> On 15 Jun 2018, at 19:12, Andrew Sullivan wrote:
> 
>> I believe that RRsets are unordered sets by definition.  So I supect
>> that if people are relying on the order in which they come off the
>> wire, they're making a mistake.
> 
> This. One hundred million times, this.

Yes, so because they’re unordered, they could also be ordered, and that should 
be seen as a version of unordered vs strictly ordered.  Just because the last 
10m times I asked it came in the same order doesn’t mean that for 10m+1 it will 
be or should be.

I’m reminded of some large content providers telling me how they routed their 
ICMP bits differently than their non-ICMP because they had dedicated hosts to 
just respond to the ICMP requests of www.example.com folks used as a 
measurement point to see if their connectivity was working :-)

- Jared
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Bob Harold]

On Fri, Jun 15, 2018 at 2:28 PM Shumon Huque  wrote:

> On Fri, Jun 15, 2018 at 1:13 PM Andrew Sullivan 
> wrote:
>
>> On Fri, Jun 15, 2018 at 11:45:19AM -0400, Erik Nygren wrote:
>> > A number of folks have been bitten by a bug in bind 9..12 where it
>> silently
>> > changes the default sorting of rrsets to always be sorted (even if the
>> > authoritative response wasn't sorted).
>>
>> I believe that RRsets are unordered sets by definition.  So I supect
>> that if people are relying on the order in which they come off the
>> wire, they're making a mistake.
>>
>
> This is all true.
>
> Yet, large numbers of application services on the Internet already rely on
> the assumption that address RRsets are returned from resolvers in a
> shuffled or randomized order for load balancing reasons. So changing this
> behavior will elicit many fierce complaints (I already observed it).
>
> --
> Shumon.
>

Round-robin is a documented feature that many applications use.  Removing
it from DNS resolvers, and then having to add it to a much larger number of
applications, does not seem like a good trade-off.

-- 
Bob Harold
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Shumon Huque]

On Fri, Jun 15, 2018 at 1:13 PM Andrew Sullivan 
wrote:

> On Fri, Jun 15, 2018 at 11:45:19AM -0400, Erik Nygren wrote:
> > A number of folks have been bitten by a bug in bind 9..12 where it
> silently
> > changes the default sorting of rrsets to always be sorted (even if the
> > authoritative response wasn't sorted).
>
> I believe that RRsets are unordered sets by definition.  So I supect
> that if people are relying on the order in which they come off the
> wire, they're making a mistake.
>

This is all true.

Yet, large numbers of application services on the Internet already rely on
the assumption that address RRsets are returned from resolvers in a
shuffled or randomized order for load balancing reasons. So changing this
behavior will elicit many fierce complaints (I already observed it).

-- 
Shumon.
___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Peter van Dijk]

Hello Andrew,

On 15 Jun 2018, at 19:12, Andrew Sullivan wrote:

> I believe that RRsets are unordered sets by definition.  So I supect
> that if people are relying on the order in which they come off the
> wire, they're making a mistake.

This. One hundred million times, this.

Kind regards,
-- 
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Paul Vixie]

Andrew Sullivan wrote:

On Fri, Jun 15, 2018 at 11:45:19AM -0400, Erik Nygren wrote:

A number of folks have been bitten by a bug in bind 9..12 where it silently
changes the default sorting of rrsets to always be sorted (even if the
authoritative response wasn't sorted).


I believe that RRsets are unordered sets by definition.  So I supect
that if people are relying on the order in which they come off the
wire, they're making a mistake.


that's what i've always said. especially after adding round-robin and 
getting all the complaints about it. famously one complaint came from 
TGV, who had a terminal server product that used TXT RRsets to describe 
possible "hosts" that could be reached through their product. for their 
internal corporate network, they had encoded carroll's _Jabberwocky_, 
one verse per internal server. the bug report that came to me as the 
BIND4 maintainer was, "you're scrambling jabberwocky" or words to that 
effect. my response when closing the report was, "how can you tell?"



This is a common mistake with unordered data sets; it results in lots
of complaints about SQL data responses, too, for instance.  (Of
course, in that case you always have the option of using an ORDER BY
clause, which we don't have in the DNS.)


i think round-robin does more good than harm. another famous bug report 
from the BIND4 era when i first added round-robin was from UC Berkeley, 
who had two NS RRs for berkeley.edu, and the second one had never 
worked, and after i added round-robin, it started to receive queries for 
the first time ever. to that complaint i said, "fix your NS RRset."


so while there's no way to please everybody, and while depending on the 
order (or in today's case, on the lack of order) of data whose ordering 
is not guaranteed is certainly an error, it's very implicit one, and we 
need sensible defaults rather than interminable whichness-of-what arguments.



--
P Vixie

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Andrew Sullivan]

On Fri, Jun 15, 2018 at 11:45:19AM -0400, Erik Nygren wrote:
> A number of folks have been bitten by a bug in bind 9..12 where it silently
> changes the default sorting of rrsets to always be sorted (even if the
> authoritative response wasn't sorted).

I believe that RRsets are unordered sets by definition.  So I supect
that if people are relying on the order in which they come off the
wire, they're making a mistake.

This is a common mistake with unordered data sets; it results in lots
of complaints about SQL data responses, too, for instance.  (Of
course, in that case you always have the option of using an ORDER BY
clause, which we don't have in the DNS.)

Best regards,

A

-- 
Andrew Sullivan
a...@anvilwalrusden.com

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Tony Finch]

Erik Nygren  wrote:

> A number of folks have been bitten by a bug in bind 9.12 where it silently
> changes the default sorting of rrsets to always be sorted (even if the
> authoritative response wasn't sorted).

Huh, I noticed this and put the workaround in my config but I didn't
realise it counted as a bug.

Anyway, there's a related issue, RFC 3484 address sorting. I believe glibc
used to derandomize addresses in getaddrinfo(), but it seems to have
stopped doing so. (However I can't find a changelog item saying so...)

Windows had a similar issue (I think it was fixed in Windows 7?):

https://support.microsoft.com/en-us/help/968920/windows-vista-and-windows-server-2008-dns-clients-do-not-honor-dns-rou

I never understood how RFC 3484 sorting could possibly make sense when
there's no way for an edge device (or even an expert sysadmin) to know
anything meaningful about how IP addresses relate to network topology.

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/
Sole, Lundy, Fastnet: West or southwest 3 or 4, increasing 5 or 6. Moderate or
rough. Occasional rain or showers. Good, occasionally poor.

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop

Re: [DNSOP] BCP on rrset ordering for round-robin? Also head's up on bind 9.12 bug (sorting rrsets by default)

[Jared Mauch]

> On Jun 15, 2018, at 11:45 AM, Erik Nygren  wrote:
> 
> A number of folks have been bitten by a bug in bind 9..12 where it silently 
> changes the default sorting of rrsets to always be sorted (even if the 
> authoritative response wasn't sorted).  This causes problems for services 
> assuming at least some degree of round-robin behavior by clients as now many 
> clients would sent all traffic to only the lowest IP.  Bug details:  
> https://gitlab.isc.org/isc-projects/bind9/issues/336   If you are upgrading 
> to or have upgraded to bind 9.12 you likely want to take a fix or override in 
> config.
> 
> 
> This raises the question of whether there would be value in a more modern BCP 
> covering round-robin expectations for recursive resolvers?  I suspect many 
> (most?) service operators take at least some degree of DNS round-robin 
> behavior by recursive resolvers as a default.
> 
> I suspect starting assumptions are roughly in the range of:
> 
> * Recursive (and stub?) resolvers (SHOULD/MUST?) do some form of round-robin 
> in RRset responses.

Stub is in your desktop (and possible application, but may vary depending on 
OS, etc).

> * There are a variety of ways to implement round-robin (randomize, permute, 
> etc).

Sure, but services should also be sufficiently robust should there be some 
amount of polarization in the hash algorithm.  I recall when the entropy went 
into bind4 to assist with balancing of some web services as well as mail/ftp 
stuff.

Perhaps this is something that could be done better with a flavor of 
happy-eyeballs in the client application, whereby you race not just v4 vs v6 
but also some limited subset of answers the application gets back as part of 
getaddrinfo()

> * Server operators need to be aware that round-robin may be a part of a load 
> balancing scheme (especially if capacity is far greater than average demand) 
> but should not be relied on exclusively.  (Perhaps with examples of why...)
> 
> Am I missing something in-terms of an existing BCP to this effect?  There's 
> RFC 1794, but I couldn't find much else (but given the sheer number of DNS 
> RFCs it's very likely I missed one).

Have you checked the DNS Camel :-)

https://powerdns.org/dns-camel/

I didn’t find anything else looking real quick, and it gave me a reminder of 
some of the folks that I used to interact with at ANS back in the day, so at 
least for a Friday it’s a fun and quick re-read to hit up 1794.

- Jared

> 
> Best, Erik
> 
> ___
> DNSOP mailing list
> DNSOP@ietf.org
> https://www.ietf.org/mailman/listinfo/dnsop

___
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop