Re: debian11 + bind-9.16.15 + dnssec-policy = lost zonefiles + crashes
On Sun, Aug 15, 2021 at 10:35:27PM +1000, raf wrote:
> But the real problem is that bind crashed, and dumped
> core, and couldn't start at all. There were a hectic
> few minutes there. :-) I deleted the coredump and the
> key files, and the .jnl files, restored backup
> zonefiles, updated the serials to be greater than that
> of the new DNSSEC signed zones, and then bind was able
> to start again.
>
> Does anyone have any idea why bind-6.19.15 would have
> crashed repeatedly?
I've had a sleep, looked in the logs, and found this:
general: notice: all zones loaded
general: notice: running
general: critical: rbtdb.c:6780: REQUIRE(((rbtnode->nsec ==
DNS_RBT_NSEC_NSEC3 && (rdataset->type ==
((dns_rdatatype_t)dns_rdatatype_nsec3) || rdataset->covers ==
((dns_rdatatype_t)dns_rdatatype_nsec3)))
|| (rbtnode->nsec != DNS_RBT_NSEC_NSEC3 && rdataset->type !=
((dns_rdatatype_t)dns_rdatatype_nsec3) &&
rdataset->covers != ((dns_rdatatype_t)dns_rdatatype_nsec3 failed, back
trace
general: critical: #0 0x558ce49ffeed in ??
general: critical: #1 0x7fd079be6d9a in ??
general: critical: #2 0x7fd079d7f73c in ??
general: critical: #3 0x7fd079e45680 in ??
general: critical: #4 0x7fd079c1b720 in ??
general: critical: #5 0x7fd079c20f52 in ??
general: critical: #6 0x7fd07995cea7 in ??
general: critical: #7 0x7fd079590def in ??
general: critical: exiting (due to assertion failure)
That assertion failed 13 times before I cleaned up.
Perhaps this is an old bug that's been fixed by now.
The only problems logged in the lead up to these
assertion failures were permission errors trying to
create jnl files in /etc/bind for the zones that
shouldn't have been signed anyway, e.g.:
general: error: /etc/bind/db.empty.jnl: create: permission denied
general: error: /etc/bind/db.255.jnl: create: permission denied
AppArmor prevented it, but the directory permissions
would have also prevented it (drwxr-sr-x root bind).
I'm convinced that the dnssec-policy usage directive
doesn't belong in the options {} stanza, and should
only appear in zone {} stanzas.
As for testing that approach on a separate VM, the
behaviour is very different, and completely wonderful.
Instead of overwriting my source zone files and then
crashing, it has created ZONE.jbk, ZONE.signed, and
ZONE.signed.jnl files, all of which are binary. But
last night, I definitely saw the overwritten ZONE files
as a text version of the signed zone. Wierd. Never
mind.
So it's looking good and I'm happy now. But how long
after the zone has been signed can I expect to see
CDS/CDNSKEY RRs appear? Why aren't they created at
the same time as the DNSKEY RRs? I assume there's
a good reason but I can't think what it is.
Also, please document the dangers of putting a
dnssec-policy usage directive in the options {} stanza
(unless something signficant has changed since version
9.16.15, and bind now knows not to sign zones that
really shouldn't be signed locally - but even if that's
the case, you could document what version that changed in).
Thanks again for making DNSSEC so easy to implement
(as long as you avoid classic rookie errors). :-)
cheers,
raf
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
ISC funds the development of this software with paid support subscriptions.
Contact us at https://www.isc.org/contact/ for more information.
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: Tracking Down Odd bind Behavior
On 8/15/21 9:07 AM, G.W. Haywood via bind-users wrote: > Hi there, > > On Sun, 15 Aug 2021, Tim Daneliuk wrote: > >> I have a bind slave instance running on FreeBSD 13-STABLE. Periodically >> (after >> a few days of perfect operation), it loses its ability to resolve at >> least some names - in this case, git.freebsd.org. ... >> ... >> Aug 14 17:07:03 ozzie named[32292]: running as: named -4 -u bind -c >> /usr/local/etc/namedb/named.conf >> ... > > Wild guess: try running without '-4'? > > Otherwise, see "Troubleshooting" in the ARM. Then, assuming that > you've set up the logging as per the ARM to be sufficiently verbose, > wait until the resolution failures start happening again and post > relevant extracts. > I did have extensive logging setup but only had info level recorded. I've since updated this to debug 3 per the ARM. Let's see what they provides. -- Tim Daneliuk tun...@tundraware.com PGP Key: http://www.tundraware.com/PGP/ ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: Tracking Down Odd bind Behavior
Hi there, On Sun, 15 Aug 2021, Tim Daneliuk wrote: I have a bind slave instance running on FreeBSD 13-STABLE. Periodically (after a few days of perfect operation), it loses its ability to resolve at least some names - in this case, git.freebsd.org. ... ... Aug 14 17:07:03 ozzie named[32292]: running as: named -4 -u bind -c /usr/local/etc/namedb/named.conf ... Wild guess: try running without '-4'? Otherwise, see "Troubleshooting" in the ARM. Then, assuming that you've set up the logging as per the ARM to be sufficiently verbose, wait until the resolution failures start happening again and post relevant extracts. -- 73, Ged. ___ Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: debian11 + bind-9.16.15 + dnssec-policy = lost zonefiles + crashes
raf via bind-users wrote:
>
> But that means that it applies to all of the zones in
> /etc/bind/named.conf.default-zones which is not helpful. It also applies
> to the zones in /etc/bind/zones.rfc1918 if that is included in
> /etc/bind/named.conf.local (which a comment there suggested). That's not
> helpful either.
A tangential point, but I think this kind of setup (with lots of empty
zones) isn't necessary, if you are doing DNSSEC validation and you turn on
synth-from-dnssec - much less configuration clutter.
> Can you sign 127.in-addr.arpa and 168.192.in-addr.arpa?
You can: it's harmless but pointless and wasteful.
> I've come to the conclusion that putting dnssec-policy in the options {}
> stanza, so that it applies to all zones, is a terrible idea. It only
> makes sense (to me) to add dnssec-policy to each individual (real)
> authoritative zone {} stanza.
It makes sense to configure dnssec-policy in one place if you have an
authoritative-only primary server, which is configured with just the zones
that it is signing. (The extra zones in the Debian config are only
suitable for a recursive server.)
> What I found more upsetting, was that my carefully crafted, well
> documented zonefiles in /var/cache/bind had been overwritten by bind so
> as to include the DNSSEC records. It might seem obvious to anyone who
> uses DNS updates that that was going to happen, but I wasn't expecting
> it. It would be great if the DNSSEC guide could be updated to mention
> that this happens, and include advice to keep your zonefile sources
> somewhere other than where bind looks for them.
Yes, this is a relatively common gotcha. You can avoid overwritten zone
files by turning on inline-signing mode, so that your source zone files
are separate from the signed zone files maintained by named. If your
configuration uses dynamic updates or DNSSEC then the zone files are
normally owned by named and will be rewritten, and you are right that
there isn't much warning of this in the documentation.
> But the real problem is that bind crashed, and dumped core, and couldn't
> start at all.
There should be something in the logs to indicate why this might have
happened; failing that a stack trace from the core dump would have been
helpful.
> I have a new question about the process of updating zonefiles when
> dnssec-policy is in use. From now on, I'll have my zonefile sources
> somewhere other than /var/cache/bind (e.g. /etc/bind/zone). When I make
> changes, I'll install them to /var/cache/bind and reload bind9. Bind9
> will replace them with the signed versions. Is it OK for me to modify my
> unsigned sources, install them over the top of bind's signed versions,
> and will bind happily recreate all of the DNSSEC records each time?
No, I'm afraid it's more complicated than that.
You can do what you suggest if the server uses inline-signing mode. If
not, your process will go wrong horribly: you need to use `rndc freeze`
and `rndc thaw` if you are manually editing a zone file that is owned by
named, BUT if you do that with a signed zone, you will lose all the
signatures and it will have to be re-signed from scratch. Not good.
Another alternative is to enable `update-policy local`, and use nsdiff and
nsupdate to make the live zone match your source files.
http://dotat.at/prog/nsdiff/
Tony.
--
f.anthony.n.finchhttps://dotat.at/
North Fitzroy, Sole: Westerly or northwesterly 4 to 6 veering
northerly or northwesterly 3 to 5. Rough becoming moderate. Showers
then fair. Good, occasionally moderate.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
ISC funds the development of this software with paid support subscriptions.
Contact us at https://www.isc.org/contact/ for more information.
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
debian11 + bind-9.16.15 + dnssec-policy = lost zonefiles + crashes
Hi,
I've just upgraded my bind9 server to debian-11 which
has bind-9.16.15. I've been looking forward to this. I
had my local dnssec-policy ("annual") all ready to go.
But it didn't go well at all.
For the first few seconds, I thought it was great. I
uncommented my new config to enable DNSSEC signing, and
the keys directory was full of new files, but there
were too many. There were key files for all of the
RFC1918 zones. I thought that can't be right.
I had put dnssec-policy "annual" in the options {}
stanza, so that it would apply to all zones. This is
suggested as a possibility in the DNSSEC guide and/or
the configuration reference.
But that means that it applies to all of the zones in
/etc/bind/named.conf.default-zones which is not
helpful. It also applies to the zones in
/etc/bind/zones.rfc1918 if that is included in
/etc/bind/named.conf.local (which a comment there
suggested). That's not helpful either. At least, I
can't see how. Can you sign 127.in-addr.arpa and
168.192.in-addr.arpa? Maybe this is a Debian package
issue. Maybe named.conf.default-zones isn't an
upstream thing. But it's good to be aware of when
writing the documentation.
I've come to the conclusion that putting dnssec-policy
in the options {} stanza, so that it applies to all
zones, is a terrible idea. It only makes sense (to me)
to add dnssec-policy to each individual (real)
authoritative zone {} stanza.
It would be great if the DNSSEC guide and the
configuration reference were updated to include a
warning about this.
But that was just a surprise, maybe wasteful, but
possibly harmless (although possibly not).
What I found more upsetting, was that my carefully
crafted, well documented zonefiles in /var/cache/bind
had been overwritten by bind so as to include the
DNSSEC records. It might seem obvious to anyone who
uses DNS updates that that was going to happen, but I
wasn't expecting it. It would be great if the DNSSEC
guide could be updated to mention that this happens,
and include advice to keep your zonefile sources
somewhere other than where bind looks for them.
I never received any indication when reading the
documentation about DNSSEC that this would happen. I
knew that bind would read in my zonefiles and construct
a signed version of it, but the documentation never
mentioned that that signed version would be written to
disk over my original source zonefiles. At least, I
never got that impression. Thank goodness for backups. :-)
I'll know for next time, and I'll restructure things,
but it would have been nice to be warned in advance
by the documentation.
But the real problem is that bind crashed, and dumped
core, and couldn't start at all. There were a hectic
few minutes there. :-) I deleted the coredump and the
key files, and the .jnl files, restored backup
zonefiles, updated the serials to be greater than that
of the new DNSSEC signed zones, and then bind was able
to start again.
Does anyone have any idea why bind-6.19.15 would have
crashed repeatedly? Here's the subset of the
config that I think could be relevant:
options
{
allow-recursion { localhost; };
dnssec-validation auto;
directory "/var/cache/bind";
key-directory "/var/cache/bind/keys";
dnssec-policy "annual"; # I'm moving this away!
};
dnssec-policy "annual"
{
keys
{
ksk key-directory lifetime 366d algorithm ecdsap256sha256;
zsk key-directory lifetime 61d algorithm ecdsap256sha256;
};
nsec3param iterations 5 optout no salt-length 16;
publish-safety 7d;
retire-safety 7d;
purge-keys 90d;
signatures-validity-dnskey 14d;
signatures-validity 14d;
signatures-refresh 7d;
dnskey-ttl 1d;
max-zone-ttl 1d;
parent-ds-ttl 1d;
zone-propagation-delay 300;
parent-propagation-delay 1d;
};
Might it just be because the RFC1918 zones were signed
and therefore invalid?
I'll try a more gradual approach on a separate test VM
and see how things go. But any advice would be greatly
appreciated. I want to be able to keep using whatever
version of bind9 is on debian stable. So workarounds
that don't involve an upgrade would be appreciated.
Hmm, on a separate VM just adding dnssec-policy "annual"
to one zone {} statement, and reloading, nothing happens.
I should come back to this tomorrow.
I have a new question about the process of updating
zonefiles when dnssec-policy is in use. From now on,
I'll have my zonefile sources somewhere other than
/var/cache/bind (e.g. /etc/bind/zone). When I make
changes, I'll install them to /var/cache/bind and
reload bind9. Bind9 will replace them with the signed
versions. Is it OK for me to modify my unsigned
sources, install them over the top of bind's signed
versions, and will bind happily recreate all of the
DNSSEC records each time? Is that what's expected? That
bind and I keep overwriting each other's zone files?
Thanks for your time, and apologies for the length of
this.
cheers,
raf
___
