This issue was likely introduced by me with https://gerrit.fd.io/r/c/vpp/+/36770 One of the goal was to avoid the time going backward, because it confused timers, but by doing so it means that if a worker runs ahead, it can never resynchronize, and the error accumulate. Maybe a better design would be to actually adjust the pace of time based on how much a worker is late or in advance: if the worker is in advance, we slow down its clock a bit whereas if it is late, we accelerate its clock a bit. A bit similar to NTP. I can come up with a tentative patch if you want to try.
Best ben ________________________________________ From: [email protected] <[email protected]> on behalf of Pieter Meyer via lists.fd.io <[email protected]> Sent: Wednesday, September 30, 2026 17:15 To: [email protected] Subject: Re: [vpp-dev] 26.06: worker thread clocks drift ahead of main, LACP stops transmitting (regression from I4cdc85fe / change 36770) Hi Baldur, We see the same per-thread clock drift on 26.06. On one router after 33 days of uptime, the workers were -3.0 s and -1.6 s against main. That's enough to leave the LACP periodic timer armed in the future on the main thread, because lacp-input sets it using worker time. We'd advise against reading vlib_get_first_main()'s time from a worker as the fix: - vlib_time_now() asserts that the vm belongs to the calling thread. - clib_time_now() mutates the clib_time_t it reads. So a worker calling it on main's structure is a data race. What we run instead is a small lacp_time_now() helper, based on clock_gettime(CLOCK_MONOTONIC). Every LACP timestamp and timer uses it on every thread, so they all share one timebase. It has been running on 3 production routers since 2026-09-30. Happy to post it to gerrit if that's the preferred approach. The drift itself looks like it comes from the clock-correction clamps added in change 36770 (time.c ~255/308, threads.h ~404-408). That probably deserves its own look, since LACP is unlikely to be the only cross-thread timer consumer affected. Regards, Pieter Meyer
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#27222): https://lists.fd.io/g/vpp-dev/message/27222 Mute This Topic: https://lists.fd.io/mt/121360743/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
