Re: DNS resolver and mixed case responses

[Baptiste]

Hi all,

Thanks a lot for your various investigations!
As a conclusion, HAProxy's behavior is "as expected".

Baptiste

Re: DNS resolver and mixed case responses

[Ben Draut]

At the moment, AWS's provided DNS servers and Route53 appear to always
match the question case in the answer. (As do Google's DNS servers)

Bind seems to be the odd man out in not doing that by default.

On Thu, Apr 12, 2018 at 7:08 AM, Jim Freeman  wrote:

> It will be important to know which behavior AWS's Route53/DNS servers use ?
>
> Using stock Debian/Stretch BIND9 (1:9.10.3.dfsg.P4-12.3+deb9u4), we see
> haproxy downing backend servers with
> "Server is going DOWN for maintenance (unspecified DNS error)."
> https://github.com/haproxy/haproxy/search?q=unspecified+dns+error
>
> We're expecting/testing to see if bind9's "no-case-compress { any; }"
> directive
> addresses this, but many folks do not control their DNS services (and as
> requisite
> AWS/Route53 capabilities mature, neither will we).
>
>
> On Tue, Apr 10, 2018 at 3:11 PM, Ben Draut  wrote:
>
>> It's interesting that the default behavior of HAProxy resolvers can
>> conflict with the default behavior of bind. (If you're unlucky with
>> whatever bind has cached)
>>
>> By default, bind uses case-insensitive compression, which can cause it to
>> use a different case in the ANSWER than in the QUESTION. (See
>> 'no-case-compress': https://ftp.isc.org/isc/bind9/cur/9.9/do
>> c/arm/Bv9ARM.ch06.html) We were impacted by this recently.
>>
>> Also interesting: https://indico.dns-oarc.net/event/20/session/2/
>> contribution/12/material/slides/0.pdf
>>
>>
>> On Mon, Apr 9, 2018 at 2:12 AM, Baptiste  wrote:
>>
>>> So, it seems that responses that does not match the case should be
>>> dropped:
>>> https://twitter.com/PowerDNS_Bert/status/983254222694240257
>>>
>>> Baptiste
>>>
>>
>>
>

Re: DNS resolver and mixed case responses

[Jim Freeman]

It will be important to know which behavior AWS's Route53/DNS servers use ?

Using stock Debian/Stretch BIND9 (1:9.10.3.dfsg.P4-12.3+deb9u4), we see
haproxy downing backend servers with
"Server is going DOWN for maintenance (unspecified DNS error)."
https://github.com/haproxy/haproxy/search?q=unspecified+dns+error

We're expecting/testing to see if bind9's "no-case-compress { any; }"
directive
addresses this, but many folks do not control their DNS services (and as
requisite
AWS/Route53 capabilities mature, neither will we).


On Tue, Apr 10, 2018 at 3:11 PM, Ben Draut  wrote:

> It's interesting that the default behavior of HAProxy resolvers can
> conflict with the default behavior of bind. (If you're unlucky with
> whatever bind has cached)
>
> By default, bind uses case-insensitive compression, which can cause it to
> use a different case in the ANSWER than in the QUESTION. (See
> 'no-case-compress': https://ftp.isc.org/isc/bind9/cur/9.9/
> doc/arm/Bv9ARM.ch06.html) We were impacted by this recently.
>
> Also interesting: https://indico.dns-oarc.net/event/20/session/
> 2/contribution/12/material/slides/0.pdf
>
>
> On Mon, Apr 9, 2018 at 2:12 AM, Baptiste  wrote:
>
>> So, it seems that responses that does not match the case should be
>> dropped:
>> https://twitter.com/PowerDNS_Bert/status/983254222694240257
>>
>> Baptiste
>>
>
>

Re: DNS resolver and mixed case responses

[Ben Draut]

It's interesting that the default behavior of HAProxy resolvers can
conflict with the default behavior of bind. (If you're unlucky with
whatever bind has cached)

By default, bind uses case-insensitive compression, which can cause it to
use a different case in the ANSWER than in the QUESTION. (See
'no-case-compress':
https://ftp.isc.org/isc/bind9/cur/9.9/doc/arm/Bv9ARM.ch06.html) We were
impacted by this recently.

Also interesting:
https://indico.dns-oarc.net/event/20/session/2/contribution/12/material/slides/0.pdf


On Mon, Apr 9, 2018 at 2:12 AM, Baptiste  wrote:

> So, it seems that responses that does not match the case should be dropped:
> https://twitter.com/PowerDNS_Bert/status/983254222694240257
>
> Baptiste
>

Re: DNS resolver and mixed case responses

[Baptiste]

So, it seems that responses that does not match the case should be dropped:
https://twitter.com/PowerDNS_Bert/status/983254222694240257

Baptiste

Re: DNS resolver and mixed case responses

[Baptiste]

On Fri, Apr 6, 2018 at 11:10 AM, Dennis Jacobfeuerborn <
denni...@conversis.de> wrote:

> On 04.04.2018 16:30, Tim Düsterhus wrote:
> > Dale,
> >
> > Am 03.04.2018 um 16:17 schrieb Dale Smith:
> >> I'm trying to understand what system is at fault here; the DNS server
> for
> >> not responding with the same case as the query, or HAProxy which
> >> should be
> >> performing a case insensitive match.
> >
> > This is left unspecified in the standards, but on the other hand there
> > is this Internet Draft:
> > https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00 which wants to
> > mandate case preserval to make DNS spoofing harder by introducing more
> > entropy in the DNS request.
> >
> > I recommend to fix your internal DNS server, because case preserving
> > behaviour seems to be somewhat expected according to a quick Google
> search.
>
> There is this:
>
> Domain Name System (DNS) Case Insensitivity Clarification:
> https://tools.ietf.org/html/rfc4343#section-3.1
>
> In section 3 it says this:
>
> 3.  Name Lookup, Label Types, and CLASS
>
>According to the original DNS design decision, comparisons on name
>lookup for DNS queries should be case insensitive [STD13].  That is
>to say, a lookup string octet with a value in the inclusive range
>from 0x41 to 0x5A, the uppercase ASCII letters, MUST match the
>identical value and also match the corresponding value in the
>inclusive range from 0x61 to 0x7A, the lowercase ASCII letters.  A
>lookup string octet with a lowercase ASCII letter value MUST
>similarly match the identical value and also match the corresponding
>value in the uppercase ASCII letter range.
>
>(Historical note: The terms "uppercase" and "lowercase" were invented
>after movable type.  The terms originally referred to the two font
>trays for storing, in partitioned areas, the different physical type
>elements.  Before movable type, the nearest equivalent terms were
>"majuscule" and "minuscule".)
>
> This reads to me like HAProxy should match characters in the ranges 0x41
> to 0x5A and 0x61 to 0x7A insensitively as long as the label type is ASCII.
>
> Section 4.1 "DNS Output Case Preservation" mentions this: "No "case
> conversion" or "case folding" is done during such output operations,
> thus "preserving" case."
>
> Regrads,
>   Dennis
>
>
Hi All,

Let me ask some advices to our friends of PowerDNS :)

Baptiste

Re: DNS resolver and mixed case responses

[Dennis Jacobfeuerborn]

On 04.04.2018 16:30, Tim Düsterhus wrote:
> Dale,
> 
> Am 03.04.2018 um 16:17 schrieb Dale Smith:
>> I'm trying to understand what system is at fault here; the DNS server for
>> not responding with the same case as the query, or HAProxy which
>> should be
>> performing a case insensitive match.
> 
> This is left unspecified in the standards, but on the other hand there
> is this Internet Draft:
> https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00 which wants to
> mandate case preserval to make DNS spoofing harder by introducing more
> entropy in the DNS request.
> 
> I recommend to fix your internal DNS server, because case preserving
> behaviour seems to be somewhat expected according to a quick Google search.

There is this:

Domain Name System (DNS) Case Insensitivity Clarification:
https://tools.ietf.org/html/rfc4343#section-3.1

In section 3 it says this:

3.  Name Lookup, Label Types, and CLASS

   According to the original DNS design decision, comparisons on name
   lookup for DNS queries should be case insensitive [STD13].  That is
   to say, a lookup string octet with a value in the inclusive range
   from 0x41 to 0x5A, the uppercase ASCII letters, MUST match the
   identical value and also match the corresponding value in the
   inclusive range from 0x61 to 0x7A, the lowercase ASCII letters.  A
   lookup string octet with a lowercase ASCII letter value MUST
   similarly match the identical value and also match the corresponding
   value in the uppercase ASCII letter range.

   (Historical note: The terms "uppercase" and "lowercase" were invented
   after movable type.  The terms originally referred to the two font
   trays for storing, in partitioned areas, the different physical type
   elements.  Before movable type, the nearest equivalent terms were
   "majuscule" and "minuscule".)

This reads to me like HAProxy should match characters in the ranges 0x41
to 0x5A and 0x61 to 0x7A insensitively as long as the label type is ASCII.

Section 4.1 "DNS Output Case Preservation" mentions this: "No "case
conversion" or "case folding" is done during such output operations,
thus "preserving" case."

Regrads,
  Dennis

Re: DNS resolver and mixed case responses

[Tim Düsterhus]

Dale,

Am 03.04.2018 um 16:17 schrieb Dale Smith:

I'm trying to understand what system is at fault here; the DNS server for
not responding with the same case as the query, or HAProxy which should be
performing a case insensitive match.


This is left unspecified in the standards, but on the other hand there 
is this Internet Draft: 
https://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00 which wants to 
mandate case preserval to make DNS spoofing harder by introducing more 
entropy in the DNS request.


I recommend to fix your internal DNS server, because case preserving 
behaviour seems to be somewhat expected according to a quick Google search.


Best regards
Tim Düsterhus

Re: DNS resolver and mixed case responses

[Dale Smith]

On Tue, 3 Apr 2018 at 17:35 Baptiste  wrote:

>
> Hi Dale,
>
> Thanks for the report!
> Please share your patch here and I'll have a look, so we could merge it.
>
> Baptiste
>

Hi Baptiste,

I've attached a patch file for haproxy-1.7 that performs the case
insensitive match instead of a direct memcmp. This resolves the issue in my
testing, I'd be interested in your thoughts on this; and happy to rework or
re-test any suggestions you have for the patch.

I previously mentioned an additional dns_toupper within dns_read_name, but
this does not appear to be necessary to resolve this particular issue so I
have removed it.

Regards,
Dale Smith


dns_strnicmp.patch
Description: Binary data

Re: DNS resolver and mixed case responses

[Baptiste]

On Tue, Apr 3, 2018 at 4:17 PM, Dale Smith  wrote:

> Hello All,
>
> I have noticed a potential issue in HAProxy 1.7 where a backend server has
> reported in logs an 'unspecified DNS error' and been marked down for
> maintenance.
>
> After some investigation, I have found that our internal DNS server is
> responding with non-lowercase domain names in the Answer section. When
> HAProxy 'check' performs the lookup in dns.c the validation on the Answer
> domain with memcmp fails.
>
> I'm trying to understand what system is at fault here; the DNS server for
> not responding with the same case as the query, or HAProxy which should be
> performing a case insensitive match.
> I have made some test changes to src/dns.c to implement case insensitive
> matching in line 260 (in dns_resolve_recv) and unifying case before
> returning in dns_read_name. After these changes I can no longer reproduce
> the problem. I'm happy to share my patch file, but expect that your own
> libs and implementation will be cleaner.
> Relevant RFC: https://tools.ietf.org/html/rfc4343#section-3
>
> The current viable workaround appears to be simply matching the case that
> the DNS server is configured to reply with. For us this means specifying
> something like 'Server.db.example.com' instead of 'server.db.example.com'
> in haproxy.cfg.
>
>
> Reproduction steps:
> * Use 'check' and 'resolvers' in server section to enable
> post-initialisation dns lookups.
> * Have the DNS resolver reply in upper/lower/mixed case that does not
> match config file. (I can share a small python script that can be used to
> test this)
> The initial resolution during config read is fine, as this is done via
> libc. The subsequent DNS resolutions during checks fail and eventually set
> the server as offline.
>
> ---
> Sample configuration to reproduce:
> defaults
> log global
> modetcp
> timeout connect 10s
> timeout client  1m
> timeout server  1m
> timeout check   10s
>
> resolvers dns-resolver
> nameserver ns1 127.0.0.1:1153
> resolve_retries  3
> timeout retry1s
> hold valid   2s # small, so we can reproduce quickly
> hold other   10s
>
> frontend fe15000
> bind :15000
> default_backend be15000
>
> backend be15000
> server dest15000 example.com:5000 check resolvers dns-resolver
> init-addr libc,none
> ---
>
> Sample DNS response that triggers the issue:
> (Note the reply CaSe does not match query. I can also provide a simple
> Python server that performs uppercase in its reply, for replication of
> this.)
> ---
> $ dig @127.0.0.1 -p 1153 example.com
> 
> ;; QUESTION SECTION:
> ;EXAMPLE.COM. IN A
>
> ;; ANSWER SECTION:
> EXAMPLE.COM. 60 IN A 127.0.0.1
> 
>
>
> Thanks for any assistance,
> Dale Smith
>
>

Hi Dale,

Thanks for the report!
Please share your patch here and I'll have a look, so we could merge it.

Baptiste

DNS resolver and mixed case responses

[Dale Smith]

Hello All,

I have noticed a potential issue in HAProxy 1.7 where a backend server has
reported in logs an 'unspecified DNS error' and been marked down for
maintenance.

After some investigation, I have found that our internal DNS server is
responding with non-lowercase domain names in the Answer section. When
HAProxy 'check' performs the lookup in dns.c the validation on the Answer
domain with memcmp fails.

I'm trying to understand what system is at fault here; the DNS server for
not responding with the same case as the query, or HAProxy which should be
performing a case insensitive match.
I have made some test changes to src/dns.c to implement case insensitive
matching in line 260 (in dns_resolve_recv) and unifying case before
returning in dns_read_name. After these changes I can no longer reproduce
the problem. I'm happy to share my patch file, but expect that your own
libs and implementation will be cleaner.
Relevant RFC: https://tools.ietf.org/html/rfc4343#section-3

The current viable workaround appears to be simply matching the case that
the DNS server is configured to reply with. For us this means specifying
something like 'Server.db.example.com' instead of 'server.db.example.com'
in haproxy.cfg.


Reproduction steps:
* Use 'check' and 'resolvers' in server section to enable
post-initialisation dns lookups.
* Have the DNS resolver reply in upper/lower/mixed case that does not match
config file. (I can share a small python script that can be used to test
this)
The initial resolution during config read is fine, as this is done via
libc. The subsequent DNS resolutions during checks fail and eventually set
the server as offline.

---
Sample configuration to reproduce:
defaults
log global
modetcp
timeout connect 10s
timeout client  1m
timeout server  1m
timeout check   10s

resolvers dns-resolver
nameserver ns1 127.0.0.1:1153
resolve_retries  3
timeout retry1s
hold valid   2s # small, so we can reproduce quickly
hold other   10s

frontend fe15000
bind :15000
default_backend be15000

backend be15000
server dest15000 example.com:5000 check resolvers dns-resolver
init-addr libc,none
---

Sample DNS response that triggers the issue:
(Note the reply CaSe does not match query. I can also provide a simple
Python server that performs uppercase in its reply, for replication of
this.)
---
$ dig @127.0.0.1 -p 1153 example.com

;; QUESTION SECTION:
;EXAMPLE.COM. IN A

;; ANSWER SECTION:
EXAMPLE.COM. 60 IN A 127.0.0.1



Thanks for any assistance,
Dale Smith