On 6/29/25 3:57 AM, [email protected] wrote:
What package would a bug even be written against here? systemd?
netctl? Any/all thoughts welcome. I've been through 1000s lines of
dmesg, the journal, the network configs, and my eyes are starting to
cross. I'm done for the night and will check back tomorrow. Frustrating...
netctl itself? openresolv?
What dhcp package are you using?
Thank you for the follow-up:
I'm using ISC dhcp:
dhcp 4.4.3.P1-4
along with bind for name resolution:
bind 9.20.10-1
That provides the cleanest way for dhcp to provide dynamic updates to
DNS and have a secondary server configured for backup DNS that takes
zone transfers from the primary.
I don't they dhcp is at fault. It is a symptom of the netctl unit
failing rather than the cause.
This is a good Sherlock Holmes mystery as far as "Who dunnit?"'. If
this were a new setup, then I could see having to work though
everything, but with this box running in the same configuration for
almost a decade - something changed... but what?
The only other piece that may somehow now be causing issues with
resolv.conf that never did before is that the file has the immutable
attribute set to keep it from changing. There were significant problems
with it being overwritten, and taking from the 2022 date it would have
been sometime late November 2022 (maybe early December) when that
started happening. To prevent it, the immutable attribute was set on
resolv.conf
(we can probably find that thread in the archives)
The latest change to anything network related on this box was Dec 9
2022 and it has run perfectly since with updates and reboots for every
kernel. Literally that is the only time the server is rebooted (except
for hurricanes and ice-storms where power-outages extend beyond the UPS
battery life)
So I'm completely perplexed and at a lost as the WTF happened to
start causing the enp5s0 interface to start normally but then be taken
done and the unit to be marked as FAILED. That's just not something that
is possible to overlook on a reboot :)
I summarized the journal picking though it line by line (as well as
dmesg -- which is basically reflected in the journal) and the relevant
entries seem to be:
journal:
Jun 29 00:44:52 valkyrie kernel: r8169 0000:05:00.0: can't disable ASPM;
OS doesn't have ASPM control
Jun 29 00:44:52 valkyrie kernel: r8169 0000:05:00.0 eth0:
RTL8168evl/8111evl, fc:aa:14:9e:33:19, XID 2c9, IRQ 36
Jun 29 00:44:52 valkyrie kernel: r8169 0000:05:00.0 eth0: jumbo features
[frames: 9194 bytes, tx checksumming: ko]
Jun 29 00:44:52 valkyrie kernel: r8169 0000:05:00.0 enp5s0: renamed from
eth0
Jun 29 00:44:47 valkyrie systemd[1]: Started Network Time Synchronization.
Jun 29 00:44:47 valkyrie systemd[1]: Reached target System Time Set.
Jun 29 00:44:48 valkyrie systemd[1]: Found device RTL8111/8168/8211/8411
PCI Express Gigabit Ethernet Controller (Onboard Ethernet).
Jun 29 00:44:51 valkyrie systemd-timesyncd[383]: Network configuration
changed, trying to establish connection.
Jun 29 00:44:51 valkyrie systemd[1]: Listening on System Extension Image
Management.
Jun 29 00:44:51 valkyrie systemd-timesyncd[383]: Network configuration
changed, trying to establish connection.
Jun 29 00:44:53 valkyrie systemd-timesyncd[383]: Network configuration
changed, trying to establish connection.
Jun 29 00:44:53 valkyrie systemd-timesyncd[383]: Network configuration
changed, trying to establish connection.
Jun 29 00:44:53 valkyrie systemd-timesyncd[383]: Network configuration
changed, trying to establish connection.
Jun 29 00:44:53 valkyrie systemd-timesyncd[383]: Network configuration
changed, trying to establish connection.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on Avahi mDNS/DNS-SD
Stack Activation Socket.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on D-Bus System Message
Bus Socket.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG network
certificate management daemon for /etc/pacman.d/gnupg.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG cryptographic
agent and passphrase cache (access for web browsers) for
/etc/pacman.d/gnupg.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG cryptographic
agent and passphrase cache (restricted) for /etc/pacman.d/gnupg.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG cryptographic
agent (ssh-agent emulation) for /etc/pacman.d/gnupg.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG cryptographic
agent and passphrase cache for /etc/pacman.d/gnupg.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on GnuPG public key
management service for /etc/pacman.d/gnupg.
Jun 29 00:44:54 valkyrie systemd[1]: Listening on OpenSSH Server Socket
(systemd-ssh-generator, AF_UNIX Local).
Jun 29 00:44:54 valkyrie systemd[1]: Listening on Hostname Service Socket.
Jun 29 00:44:54 valkyrie systemd[1]: Reached target Socket Units.
Jun 29 00:44:54 valkyrie systemd[1]: Starting Avahi mDNS/DNS-SD Stack...
Jun 29 00:44:54 valkyrie systemd[1]: Starting Loading IP Sets...
Jun 29 00:44:54 valkyrie systemd[1]: Starting IPv4 Packet Filtering
Framework...
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Found user 'avahi' (UID 84)
and group 'avahi' (GID 84).
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Successfully dropped root
privileges.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: avahi-daemon 0.8 starting up.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: WARNING: No NSS support for
mDNS detected, consider installing nss-mdns!
Jun 29 00:44:54 valkyrie systemd[1]: Started Avahi mDNS/DNS-SD Stack.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Successfully called chroot().
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Successfully dropped
remaining capabilities.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Loading service file
/services/LaserJet.service.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Loading service file
/services/dcr4100n.service.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Joining mDNS multicast group
on interface lo.IPv6 with address ::1.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: New relevant interface
lo.IPv6 for mDNS.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Joining mDNS multicast group
on interface lo.IPv4 with address 127.0.0.1.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: New relevant interface
lo.IPv4 for mDNS.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Network interface
enumeration completed.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Registering new address
record for ::1 on lo.*.
Jun 29 00:44:54 valkyrie avahi-daemon[724]: Registering new address
record for 127.0.0.1 on lo.IPv4.
Jun 29 00:44:55 valkyrie systemd[1]: Finished IPv4 Packet Filtering
Framework.
Jun 29 00:44:55 valkyrie systemd[1]: Reached target Preparation for Network.
Jun 29 00:44:55 valkyrie systemd[1]: Starting A basic static ethernet
connection...
Jun 29 00:44:55 valkyrie network[779]: Starting network profile
'ethernet-statis-valkyrie'...
Jun 29 00:44:55 valkyrie kernel: RTL8211E Gigabit Ethernet
r8169-0-500:00: attached PHY driver (mii_bus:phy_addr=r8169-0-500:00,
irq=MAC)
Jun 29 00:44:55 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down
Jun 29 00:44:55 valkyrie avahi-daemon[724]: Server startup complete.
Host name is valkyrie.local. Local service cookie is 357913226.
Jun 29 00:44:56 valkyrie avahi-daemon[724]: Service "dcr4100n"
(/services/dcr4100n.service) successfully established.
Jun 29 00:44:56 valkyrie avahi-daemon[724]: Service "LaserJet"
(/services/LaserJet.service) successfully established.
Jun 29 00:44:57 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Up -
1Gbps/Full - flow control rx/tx
Jun 29 00:44:57 valkyrie systemd[1]: Started A basic static ethernet
connection.
Jun 29 00:44:57 valkyrie systemd[1]: Reached target Network.
Jun 29 00:44:57 valkyrie systemd[1]: Reached target Network is Online.
Jun 29 00:44:57 valkyrie avahi-daemon[724]: Joining mDNS multicast group
on interface enp5s0.IPv4 with address 192.168.6.14.
Jun 29 00:44:57 valkyrie avahi-daemon[724]: New relevant interface
enp5s0.IPv4 for mDNS.
Jun 29 00:44:57 valkyrie avahi-daemon[724]: Registering new address
record for 192.168.6.14 on enp5s0.IPv4.
Jun 29 00:44:57 valkyrie systemd[1]: Started ClamAV virus database updater.
Jun 29 00:44:57 valkyrie systemd[1]: Starting IPv4 DHCP server...
Jun 29 00:44:57 valkyrie systemd[1]: Starting Dovecot IMAP/POP3 email
server...
Jun 29 00:44:57 valkyrie systemd[1]: Starting Fail2Ban Service...
Jun 29 00:44:57 valkyrie systemd[1]: Started Apache Web Server.
Jun 29 00:44:57 valkyrie systemd[1]: Starting MariaDB 11.8.2 database
server...
Jun 29 00:44:57 valkyrie systemd[1]: Started Internet domain name server.
Jun 29 00:44:57 valkyrie systemd[1]: Starting Network UPS Tools - power
device driver controller...
Jun 29 00:44:57 valkyrie systemd[1]: Starting Postfix Mail Transport
Agent...
Jun 29 00:44:57 valkyrie systemd[1]: Starting PostgreSQL database server...
Jun 29 00:44:58 valkyrie systemd[1]: Starting Samba SMB Daemon...
Jun 29 00:44:58 valkyrie systemd[1]: Started Spamassassin daemon.
Jun 29 00:44:58 valkyrie systemd[1]: Starting OpenSSH Daemon...
Jun 29 00:44:58 valkyrie systemd[1]: Starting Permit User Sessions...
Jun 29 00:44:58 valkyrie systemd[1]: Started Fail2Ban Service.
Jun 29 00:44:58 valkyrie systemd[1]: Finished Permit User Sessions.
Jun 29 00:44:58 valkyrie systemd[1]: Started Command Scheduler.
Jun 29 00:44:58 valkyrie systemd[1]: Starting Light Display Manager...
Jun 29 00:44:58 valkyrie network[847]: resolvconf: signature mismatch:
/etc/resolv.conf
Jun 29 00:44:58 valkyrie network[847]: resolvconf: run `resolvconf -u`
to update
Jun 29 00:44:59 valkyrie kernel: r8169 0000:05:00.0 enp5s0: Link is Down
Jun 29 00:44:59 valkyrie avahi-daemon[724]: Interface enp5s0.IPv4 no
longer relevant for mDNS.
Jun 29 00:44:59 valkyrie systemd[1]:
netctl@ethernet\x2dstatis\x2dvalkyrie.service: Main process exited,
code=exited, status=1/FAILURE
Jun 29 00:44:59 valkyrie network[779]: Failed to bring the network up
for profile 'ethernet-statis-valkyrie'
Jun 29 00:44:59 valkyrie avahi-daemon[724]: Leaving mDNS multicast group
on interface enp5s0.IPv4 with address 192.168.6.14.
Jun 29 00:44:59 valkyrie systemd[1]:
netctl@ethernet\x2dstatis\x2dvalkyrie.service: Failed with result
'exit-code'.
Jun 29 00:44:59 valkyrie avahi-daemon[724]: Withdrawing address record
for 192.168.6.14 on enp5s0.
So it very much looks like either systemd or netctl are now taking
the inability to update resolv.conf as a FAILURE of the
netctl@ethernet\x2dstatis\x2dvalkyrie.service which shouldn't be
happening unless something changed. It has been in the exact same
configuration since 2015 and has used the exact same files since
November 2022 when the immutable attribute was set on resolv.conf.
And again, that unit failing isn't something you could ever possibly
overlook on boot. Bad situation for a server. No network, no server.
Ironically (and also hiding the problem) when the interface is
brought up manually with "ip set enp5s0 up" there must be no check of
resolv.conf. Otherwise, if that is what is causing the issue, I would
expect bringing the interface up to result in the same error and it to
go back down. But it doesn't. After being brought up, the server is
running perfectly, just as it did before the Jun 28 update.
Hah! we may have just found it. openresolv was updated in the Jun 28
update. That may be the culprit. I'll downgrade reboot and report.
--
David C. Rankin, J.D.,P.E.
