Package: chrony
Version: 4.8-4
Severity: normal
chrony's networkd-dispatcher integration can leave NTP sources
permanently offline after a transient network outage. debian/links
symlinks 20-chrony-onoffline into networkd-dispatcher's routable.d and
off.d only. On a link flap the dispatched states are "no-carrier" and
then "routable" - "off" is never dispatched at all, so routable.d is
the only hook that ever runs, and it runs while the interface is still
being configured. "chronyc onoffline" consults the routing table,
correctly finds no route for the affected address family, and marks
those sources offline. Nothing dispatches afterwards to re-evaluate,
so they are never brought back and chronyd free-runs indefinitely
logging "Can't synchronise: no selectable sources". On my host that
persisted for 16.5 hours, until I ran "chronyc online" by hand.
I observed this on Ubuntu 26.04 with chrony 4.8-2ubuntu1, but I am
filing here because the packaging is Debian's: I checked debian/links
in the current Debian source package and it carries the same two
symlinks with no configured.d entry. Everything below was captured on
that host - an instrumented reproduction with per-state dispatcher
logging, and a verification that adding the missing configured.d
symlink resolves it. I have also raised this upstream on chrony-users,
reviving a June 2023 thread that reached the same diagnosis.
STEPS TO REPRODUCE
==================
On a networkd host running chrony, cause carrier loss followed by a
slow DHCP re-acquisition - e.g. power-cycle the upstream router. In my
case a router firmware update caused ~36 s of carrier flapping, and
DHCP did not complete for a further 2m15s.
OBSERVED
========
systemd-networkd (times UTC; LAN addresses redacted):
06:19:10 eth0: Lost carrier
06:19:10 eth0: DHCP lease lost
06:19:13 eth0: Gained carrier
06:19:19 eth0: Lost carrier
06:19:42 eth0: Gained carrier
06:19:44 eth0: Lost carrier
06:19:46 eth0: Gained carrier
06:22:01 eth0: DHCPv4 address 192.168.x.x/22, gateway 192.168.x.1
acquired from 192.168.x.1
chronyd:
06:21:36 Source 185.125.190.123 offline
06:21:36 Source 91.189.91.112 offline
06:21:36 Source 2620:2d:4000:1::1123 offline
06:21:36 Source 91.189.91.113 offline
06:21:36 Can't synchronise: no selectable sources (5 unreachable sources)
06:21:36 Source 2620:2d:4000:1::2123 offline
Note the ordering: chronyd marked all five sources offline at
06:21:36, twenty-five seconds BEFORE the interface acquired an address
and gateway at 06:22:01. Nothing ran the hook again afterwards.
State 16.5 hours later, before any manual intervention:
$ chronyc tracking
Reference ID : 45AE48D5 (ntp-nts-2.ps5.canonical.com)
Ref time (UTC) : Fri Jul 24 06:15:09 2026
System time : 0.000144547 seconds slow of NTP time
Leap status : Normal
$ timedatectl show -p NTPSynchronized --value
no
"chronyc -n sources" at that point showed all five sources as '^?'
(unusable), Reach 0, LastRx 16h. "sudo chronyc online" restored them
immediately, and NTPSynchronized returned to "yes" within one poll.
DELIBERATE REPRODUCTION
=======================
I reproduced this by physically disconnecting and reconnecting the
Ethernet cable, with networkd-dispatcher at increased verbosity (so
invoked hooks are named) and a logging script installed in every state
directory:
#!/bin/sh
logger -t dispatch-debug "STATE=$STATE IFACE=$IFACE
Admin=$AdministrativeState Oper=$OperationalState"
Full sequence (trimmed to the relevant units; addresses redacted):
04:06:46 kernel: eth0: Link is Down
04:06:46 systemd-networkd: eth0: Lost carrier
04:06:46 systemd-networkd: eth0: DHCP lease lost
04:06:46 networkd-dispatcher: Invoking '.../no-carrier.d/00-debug'
for interface eth0
04:06:46 dispatch-debug: STATE=no-carrier Admin=configured
Oper=no-carrier
04:07:41 kernel: eth0: Link is Up - 100Mbps/Full
04:07:41 systemd-networkd: eth0: Gained carrier
04:07:41 networkd-dispatcher: Invoking '.../degraded.d/00-debug' for
interface eth0
04:07:41 dispatch-debug: STATE=degraded Admin=configured
Oper=degraded
04:07:41 networkd-dispatcher: Invoking '.../configuring.d/00-debug'
for interface eth0
04:07:41 dispatch-debug: STATE=configuring Admin=configuring
Oper=degraded
04:07:42 networkd-dispatcher: Invoking '.../routable.d/00-debug' for
interface eth0
04:07:42 dispatch-debug: STATE=routable Admin=configuring
Oper=routable
04:07:42 networkd-dispatcher: Invoking
'.../routable.d/chrony-onoffline' for interface eth0
04:07:43 networkd-dispatcher: Invoking '.../configured.d/00-debug'
for interface eth0
04:07:43 dispatch-debug: STATE=configured Admin=configured
Oper=routable
04:07:45 systemd-networkd: eth0: DHCPv4 address 192.168.x.x/22,
gateway 192.168.x.1 acquired
04:07:45 networkd-dispatcher: Invoking '.../configuring.d/00-debug'
for interface eth0
04:07:45 dispatch-debug: STATE=configuring Admin=configuring
Oper=routable
04:07:45 networkd-dispatcher: Invoking '.../configured.d/00-debug'
for interface eth0
04:07:45 dispatch-debug: STATE=configured Admin=configured
Oper=routable
This establishes four things:
1. On carrier loss the dispatched state is "no-carrier", not "off".
chrony's off.d symlink is never invoked by a link flap. The only
chrony symlink that executes in this entire sequence is
routable.d/chrony-onoffline.
2. "routable" is dispatched with AdministrativeState=configuring, at
04:07:42 - three seconds before DHCPv4 supplies an address and gateway
at 04:07:45. Note that OperationalState reads "routable" from 04:07:42
onward, including at both "configured" dispatches; what changes across
the sequence is AdministrativeState (configuring -> configured ->
configuring -> configured). The claim is therefore specific: the
dispatch chrony's hook is attached to fires while networkd's
ADMINISTRATIVE state is still "configuring". The interface is
operationally routable at that point via IPv6 - as the IPv4-only
offline set confirms - but IPv4 is not yet configured.
3. chrony's hook runs at exactly that moment, and nothing runs it
again. Because the only hook is on "routable", and "routable" precedes
IPv4 configuration, a link flap does not merely risk leaving IPv4
sources offline - it does so deterministically.
4. "configured" is dispatched at 04:07:45, immediately after the
DHCPv4 lease. A configured.d symlink would therefore be invoked with a
working default route present.
Point 2 answers the question raised upstream in 2023 ("Doesn't the
routable state follow configured?" - msg03269): the "routable"
dispatch precedes the post-DHCP "configured" dispatch.
Note there are no chronyd log lines in this window: the IPv4 sources
were already offline from an identical earlier test, so "chronyc
onoffline" produced no state change and logged nothing. The
verification capture further below shows the same
routable.d/chrony-onoffline invocation immediately followed by "Source
... offline" for all three IPv4 sources, with the two IPv6 sources
untouched - onoffline correctly reading the routing table as
IPv6-routable and IPv4-not-yet.
The defect is not in onoffline's logic. It is that the only dispatch
wired to it fires before IPv4 configuration completes, and nothing
re-runs it afterwards.
Caveat on ordering: the sequence is not monotonic - "configuring" is
dispatched again at 04:07:45 after a "configured" at 04:07:43. On a
flapping link states can re-enter. In this capture "configured" is
dispatched last and after DHCP, so the proposed symlink would have
restored the sources; but it does not by itself guarantee the hook
always runs last.
ANALYSIS
========
/usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline guards its
action with "[ $# -ge 2 ]". networkd-dispatcher passes interface state
through environment variables (IFACE, STATE) and not positional
arguments, so that guard is always false and the script
unconditionally runs "chronyc onoffline". That command determines
online/offline from the routing table, so the outcome depends entirely
on WHEN it runs relative to address and route configuration.
debian/links, from the 4.8-2ubuntu1 source package, installs the
script into two networkd-dispatcher state directories only:
# Update sources in response to systemd-networkd events (LP: #1718227).
# This is reusing the NetworkManager dispatch script which has no hard
# dependency to NetworkManager (not using any of its arguments)
usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline
usr/lib/networkd-dispatcher/routable.d/chrony-onoffline
usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline
usr/lib/networkd-dispatcher/off.d/chrony-onoffline
(Identical in current Debian packaging. "dpkg -L chrony" on the
running system confirms these are the only two symlinks shipped.)
The comment makes clear the argument-less fall-through is deliberate
and understood. The gap is only that "configured" is not in the list,
so nothing runs the hook at the point where networkd reports the link
fully configured - i.e. after DHCP has supplied an address and
gateway.
VERIFICATION OF THE PROPOSED FIX
================================
I added the missing symlink locally:
/usr/lib/networkd-dispatcher/configured.d/50-chrony-onoffline-local
-> /usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline
and repeated the cable disconnect/reconnect:
04:25:59 systemd-networkd: eth0: Lost carrier
04:25:59 systemd-networkd: eth0: DHCP lease lost
04:25:59 networkd-dispatcher: Invoking '.../no-carrier.d/00-debug'
for interface eth0
04:25:59 dispatch-debug: STATE=no-carrier Admin=configured
Oper=no-carrier
04:26:39 systemd-networkd: eth0: Gained carrier
04:26:39 dispatch-debug: STATE=degraded Admin=configured
Oper=degraded
04:26:40 dispatch-debug: STATE=configuring Admin=configuring
Oper=degraded
04:26:42 dispatch-debug: STATE=routable Admin=configuring
Oper=routable
04:26:42 networkd-dispatcher: Invoking
'.../routable.d/chrony-onoffline' for interface eth0
04:26:42 chronyd: Source 185.125.190.123 offline
04:26:42 chronyd: Source 91.189.91.112 offline
04:26:42 chronyd: Source 91.189.91.113 offline
04:26:42 dispatch-debug: STATE=configured Admin=configured
Oper=routable
04:26:42 networkd-dispatcher: Invoking
'.../configured.d/50-chrony-onoffline-local' for interface eth0
04:26:44 systemd-networkd: eth0: DHCPv4 address 192.168.x.x/22,
gateway 192.168.x.1 acquired
04:26:44 dispatch-debug: STATE=configuring Admin=configuring
Oper=routable
04:26:44 dispatch-debug: STATE=configured Admin=configured
Oper=routable
04:26:44 networkd-dispatcher: Invoking
'.../configured.d/50-chrony-onoffline-local' for interface eth0
04:26:44 chronyd: Source 185.125.190.123 online
04:26:44 chronyd: Source 91.189.91.112 online
04:26:44 chronyd: Source 91.189.91.113 online
(The local symlink is named 50-chrony-onoffline-local only to avoid
colliding with any future package-shipped file; it is the same
script.)
The defect still occurs: routable.d/chrony-onoffline runs at 04:26:42
and parks the three IPv4 sources. The added hook then runs twice -
once at 04:26:42, still before DHCP, where it correctly makes no
change; and again at 04:26:44 immediately after the DHCPv4 lease,
where it restores all three sources.
Exposure drops from indefinite to two seconds, unattended. The
harmless pre-DHCP invocation also demonstrates that running onoffline
on an additional state introduces no churn - it is self-determining
and idempotent.
This run again confirms that carrier loss dispatches "no-carrier", not
"off": chrony's off.d symlink is not invoked at any point.
Separately, "systemctl restart networkd-dispatcher" silently repairs
the condition via its startup triggers (observed on this host at
04:18:12, where the restart's replayed "routable" dispatch brought the
sources back online with a route present). That is likely one reason
the defect is rarely reported - any routine restart of the dispatcher,
or of chrony, clears it.
PRIOR REPORTS, AND WHY THIS IS NOT A FEATURE REQUEST
====================================================
This defect has been independently reported before. LP #2043967 (Paul
Gear, November 2023) describes the same mechanism with a different
trigger: after suspend/resume the dispatcher ran "chronyc onoffline"
before IPv6 interfaces had completed Duplicate Address Detection,
leaving those sources ignored "until chronyc onoffline is run again
after the interfaces are usable". That report was marked Incomplete
pending the Ubuntu release, chrony version and reproduction steps, and
expired in January 2024 without them. All three are supplied above.
Taken together, the two reports generalise the defect: DAD and DHCP
are simply two different reasons an interface can reach "routable"
before it is actually usable. The problem is specific to neither - it
is that "routable" does not imply usable, and it is the only state
with a chrony hook attached.
It is also worth addressing the framing of the earlier report, which
was characterised as a feature request. The shipped integration exists
specifically to keep chrony's sources in step with network
availability. On this host it reliably takes sources offline on a link
flap and provides no path to bring them back, producing a 16.5 hour
loss of time synchronisation that required manual intervention to
clear. The one-line addition below restores the integration's stated
purpose rather than extending it.
SUGGESTED FIX
=============
Add a configured.d symlink alongside the existing two, so the hook
also runs once networkd reports the link fully configured:
usr/lib/NetworkManager/dispatcher.d/20-chrony-onoffline
usr/lib/networkd-dispatcher/configured.d/chrony-onoffline
Since "chronyc onoffline" is self-determining and idempotent, running
it on an additional state is harmless - demonstrated above, where the
pre-DHCP invocation correctly made no change.
A more targeted fix - guarding the script on $STATE /
$AdministrativeState - was proposed upstream in 2023 (see references).
It has not been applied: examples/chrony.nm-dispatcher.onoffline
carries only the positional-argument guard both in the 4.8 source
tarball and in upstream git master, and is byte-identical to the
script shipped on this system.
A caveat on the symlink as a general fix: the 2023 thread did not
stall for want of evidence - Jongkyu Seok supplied four event-sequence
logs (msg03289). Those logs showed that the order in which
AdministrativeState and OperationalState change VARIES BETWEEN
ENVIRONMENTS, and that linking into configured.d could misfire on a
host where AdministrativeState becomes "configured" while
OperationalState is not yet routable. That is a legitimate objection
to the bare symlink as a universal solution, and no maintainer
response to those logs appears in the archive.
The evidence in this report should therefore be read as a demonstrated
fix FOR THIS CONFIGURATION and an additional data point on ordering,
not as a claim that the symlink is universally correct. Note that
adding it ALONGSIDE the existing routable.d link rather than replacing
it means both orderings receive a dispatch - though the resulting
state is still determined by whichever fires last. If the variability
Seok documented makes a placement-only fix untenable, the
guarded-script approach looks like the more robust route, and that is
a decision for upstream.
REFERENCES
==========
- chrony-users, Jongkyu Seok, June 2023 - same diagnosis, proposes the
configured.d script:
https://www.mail-archive.com/[email protected]/msg03268.html
- Miroslav Lichvar's reply asking for event-sequence examples:
https://www.mail-archive.com/[email protected]/msg03269.html
- Jongkyu Seok's reply supplying four event-sequence logs, and noting
that the ordering of AdministrativeState and OperationalState varies
between environments - the thread ends here with no maintainer
response in the archive:
https://www.mail-archive.com/[email protected]/msg03289.html
- Upstream examples/chrony.nm-dispatcher.onoffline in git master still
has only the "[ $# -ge 2 ]" guard.
- Debian #1093763 - a different defect in the same integration
("chronyc onoffline" returning non-zero aborting if-up.d, killing the
DHCP client), fixed in chrony 4.8-4 by checking chronyd.sock rather
than the PID file and ensuring hook scripts always exit successfully.
Note 4.8-4 postdates the 4.8-2ubuntu1 in Ubuntu 26.04.
- LP #2043967 - same defect, different trigger (IPv6 sources not
restored after suspend/resume, onoffline running before DAD
completed). Filed Nov 2023, marked Incomplete pending
release/version/reproduction, expired Jan 2024.
- LP #2155244 - related but distinct: 20-chrony-dhcp has no
networkd-dispatcher equivalent at all.
- LP #1718227 - origin of the current networkd-dispatcher symlinks,
cited in debian/links.
- LP #2132159 - recent: chrony now declares "Recommends:
networkd-dispatcher" directly, since systemd no longer does. Note this
means networkd-dispatcher is only a Recommends, so on installs without
it the hook never runs in either direction.
