[vpp-dev] Rejecting large frequency change
Hello, We have started seeing the below warning message coming continuously. Traffic is running on several vhost-ports and physical ports. These messages are coming even after a restart with a slow rate. VPP clib_warning: clib_time_verify_frequency:248: Rejecting large frequency change of 1.01% Does this mean something ? Thanks, Leela sankar -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19924): https://lists.fd.io/g/vpp-dev/message/19924 Mute This Topic: https://lists.fd.io/mt/84701132/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Rejecting large frequency change
Which version of vpp are you running? Which Linux kernel is involved? What kind of hardware? Lastly, did you just start running NTP on the system? Yes, of course it means something: every N seconds, vpp calculates its idea of the current clock frequency by asking the kernel [via the clock_gettime ( CLOCK_REALTIME) syscall] what time it is. It asks the CPU for the current clock tick [on x86_64, via "RDTSC"]. Some arithmetic later, we have a new clock rate for the processor. We can't simply believe the new clock rate, because e.g. NTP synchronization can change the kernel reference clock by minutes or hours. Every timer in the system breaks catastrophically if vpp's idea of the CPU clock rate is seriously wrong. After a suitable number of experiments, we decided to reject frequency changes of more than one percent. Vpp has to continuously adjust its idea of the current clock rate, otherwise long-running timers won't be accurate enough. See .../src/vppinfra/time.c:clib_time_verify_frequency() and .../src/vppinfra/time.h:clib_time_now_internal() for details. From: vpp-dev@lists.fd.io On Behalf Of Gudimetla, Leela Sankar via lists.fd.io Sent: Thursday, August 5, 2021 11:31 PM To: vpp-dev@lists.fd.io Subject: [vpp-dev] Rejecting large frequency change Hello, We have started seeing the below warning message coming continuously. Traffic is running on several vhost-ports and physical ports. These messages are coming even after a restart with a slow rate. VPP clib_warning: clib_time_verify_frequency:248: Rejecting large frequency change of 1.01% Does this mean something ? Thanks, Leela sankar -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19925): https://lists.fd.io/g/vpp-dev/message/19925 Mute This Topic: https://lists.fd.io/mt/84701132/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Rejecting large frequency change
Hi Dave, I am seeing this with VPP 21.01. Yes NTP is on and the print starts filling up the log. I see fixes made in this area in time.h and time.c addressing the algorithm. Questions: 1. "Vpp has to continuously adjust its idea of the current clock rate" -- How long could this be? Seconds, minutes or hours? 2. Is the warning suggesting a problem (with few expected failures around this event that rely on timers) or information displayed to the user that the clock rate is being adjusted? What is the outcome expected with entities of the system relying on timers whenever this happens? Thanks, Prashant On Fri, Aug 6, 2021 at 5:08 AM Dave Barach wrote: > Which version of vpp are you running? Which Linux kernel is involved? What > kind of hardware? Lastly, did you just start running NTP on the system? > > > > Yes, of course it means something: every N seconds, vpp calculates its > idea of the current clock frequency by asking the kernel [via the > clock_gettime ( CLOCK_REALTIME) syscall] what time it is. It asks the CPU > for the current clock tick [on x86_64, via “RDTSC”]. Some arithmetic later, > we have a new clock rate for the processor. > > > > We can’t simply believe the new clock rate, because e.g. NTP > synchronization can change the kernel reference clock by *minutes* or > *hours*. Every timer in the system breaks catastrophically if vpp’s idea > of the CPU clock rate is seriously wrong. After a suitable number of > experiments, we decided to reject frequency changes of more than one > percent. > > > > Vpp has to continuously adjust its idea of the current clock rate, > otherwise long-running timers won’t be accurate enough. > > > > > > See .../src/vppinfra/time.c:clib_time_verify_frequency() and > .../src/vppinfra/time.h:clib_time_now_internal() for details. > > > > *From:* vpp-dev@lists.fd.io *On Behalf Of *Gudimetla, > Leela Sankar via lists.fd.io > *Sent:* Thursday, August 5, 2021 11:31 PM > *To:* vpp-dev@lists.fd.io > *Subject:* [vpp-dev] Rejecting large frequency change > > > > Hello, > > > > We have started seeing the below warning message coming continuously. > Traffic is running on several vhost-ports and physical ports. These > messages are coming even after a restart with a slow rate. > > VPP clib_warning: clib_time_verify_frequency:248: Rejecting large > frequency change of 1.01% > > > > Does this mean something ? > > > > Thanks, > > Leela sankar > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20153): https://lists.fd.io/g/vpp-dev/message/20153 Mute This Topic: https://lists.fd.io/mt/84701132/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Rejecting large frequency change
Discontinuous time jumps are bad, mkay? NTP doesn’t want to jump the clock, but will if you tell it to, or rather, if you allow it to, and it finds itself in a situation where that might be better than the alternative. NTP *wants* to adjust your clock very, very gradually. This is called “slewing” the clock. But if your clock is WAAAY OFF, then adjusting it slowly could take a long time. Effectively “forever”. By default, if your clock is more than 600 sec ‘off’, NTP will jump the clock. You can disable this with ‘-x 0’ on the command line. Otherwise, the maximum possible slew rate for the clock is 500ppm Ref: http://doc.ntp.org/archives/4.2.8-series/clock/ "As a result, the clock can take 33 minutes to amortize each second the clock is outside the acceptable range. During this interval the clock will not be consistent with any other network clock and the system cannot be used for distributed applications that require correctly synchronized network time.” See also: http://doc.ntp.org/reflib/memos/memo96b.pdf http://doc.ntp.org/archives/4.2.8-series/debug/#large-delay-variations > On Sep 17, 2021, at 1:02 AM, Prashant Gajarampalli > wrote: > > Hi Dave, > > I am seeing this with VPP 21.01. Yes NTP is on and the print starts filling > up the log. I see fixes made in this area in time.h and time.c addressing the > algorithm. > Questions: > 1. "Vpp has to continuously adjust its idea of the current clock rate" -- How > long could this be? Seconds, minutes or hours? > 2. Is the warning suggesting a problem (with few expected failures around > this event that rely on timers) or information displayed to the user that the > clock rate is being adjusted? What is the outcome expected with entities of > the system relying on timers whenever this happens? > > Thanks, > Prashant > > On Fri, Aug 6, 2021 at 5:08 AM Dave Barach wrote: > Which version of vpp are you running? Which Linux kernel is involved? What > kind of hardware? Lastly, did you just start running NTP on the system? > > > > Yes, of course it means something: every N seconds, vpp calculates its idea > of the current clock frequency by asking the kernel [via the clock_gettime ( > CLOCK_REALTIME) syscall] what time it is. It asks the CPU for the current > clock tick [on x86_64, via “RDTSC”]. Some arithmetic later, we have a new > clock rate for the processor. > > > > We can’t simply believe the new clock rate, because e.g. NTP synchronization > can change the kernel reference clock by minutes or hours. Every timer in the > system breaks catastrophically if vpp’s idea of the CPU clock rate is > seriously wrong. After a suitable number of experiments, we decided to reject > frequency changes of more than one percent. > > > > Vpp has to continuously adjust its idea of the current clock rate, otherwise > long-running timers won’t be accurate enough. > > > > > > See .../src/vppinfra/time.c:clib_time_verify_frequency() and > .../src/vppinfra/time.h:clib_time_now_internal() for details. > > > > From: vpp-dev@lists.fd.io On Behalf Of Gudimetla, Leela > Sankar via lists.fd.io > Sent: Thursday, August 5, 2021 11:31 PM > To: vpp-dev@lists.fd.io > Subject: [vpp-dev] Rejecting large frequency change > > > > Hello, > > > > We have started seeing the below warning message coming continuously. Traffic > is running on several vhost-ports and physical ports. These messages are > coming even after a restart with a slow rate. > > VPP clib_warning: clib_time_verify_frequency:248: Rejecting large frequency > change of 1.01% > > > > Does this mean something ? > > > > Thanks, > > Leela sankar > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20154): https://lists.fd.io/g/vpp-dev/message/20154 Mute This Topic: https://lists.fd.io/mt/84701132/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Rejecting large frequency change
First, why are we adjusting the clock rate? The CPU clock read via RDTSC (x86_64) derives from a fairly cheap oscillator which drifts (changes frequency) over time. Integrated over a respectable period of time, the resulting frequency errors lead to errors in policy enforcement. “6am to midnight” eventually means 6:15 am to 12:15 am if a putative clock rate of 4gGz is actually 3.995 gHz and we don’t correct the clock rate. See also “show clock verbose” The algorithm in time.c uses the system’s idea of “now” to derive the current CPU frequency, uses exponential smoothing to slew vpp’s idea of the CPU frequency. If a particular candidate rate sample jumps by too much, we ignore it and print a message. It wouldn’t be completely unreasonable to suppress the messages since they end up in syslog. FWIW... Dave From: Prasanth G Sent: Friday, September 17, 2021 2:02 AM To: Dave Barach Cc: lgudi...@ciena.com; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Rejecting large frequency change Hi Dave, I am seeing this with VPP 21.01. Yes NTP is on and the print starts filling up the log. I see fixes made in this area in time.h and time.c addressing the algorithm. Questions: 1. "Vpp has to continuously adjust its idea of the current clock rate" -- How long could this be? Seconds, minutes or hours? 2. Is the warning suggesting a problem (with few expected failures around this event that rely on timers) or information displayed to the user that the clock rate is being adjusted? What is the outcome expected with entities of the system relying on timers whenever this happens? Thanks, Prashant On Fri, Aug 6, 2021 at 5:08 AM Dave Barach mailto:v...@barachs.net> > wrote: Which version of vpp are you running? Which Linux kernel is involved? What kind of hardware? Lastly, did you just start running NTP on the system? Yes, of course it means something: every N seconds, vpp calculates its idea of the current clock frequency by asking the kernel [via the clock_gettime ( CLOCK_REALTIME) syscall] what time it is. It asks the CPU for the current clock tick [on x86_64, via “RDTSC”]. Some arithmetic later, we have a new clock rate for the processor. We can’t simply believe the new clock rate, because e.g. NTP synchronization can change the kernel reference clock by minutes or hours. Every timer in the system breaks catastrophically if vpp’s idea of the CPU clock rate is seriously wrong. After a suitable number of experiments, we decided to reject frequency changes of more than one percent. Vpp has to continuously adjust its idea of the current clock rate, otherwise long-running timers won’t be accurate enough. See .../src/vppinfra/time.c:clib_time_verify_frequency() and .../src/vppinfra/time.h:clib_time_now_internal() for details. From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io> > On Behalf Of Gudimetla, Leela Sankar via lists.fd.io <http://lists.fd.io> Sent: Thursday, August 5, 2021 11:31 PM To: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] Rejecting large frequency change Hello, We have started seeing the below warning message coming continuously. Traffic is running on several vhost-ports and physical ports. These messages are coming even after a restart with a slow rate. VPP clib_warning: clib_time_verify_frequency:248: Rejecting large frequency change of 1.01% Does this mean something ? Thanks, Leela sankar -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20157): https://lists.fd.io/g/vpp-dev/message/20157 Mute This Topic: https://lists.fd.io/mt/84701132/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Rejecting large frequency change of +infinity%
Hello
I am running vpp (latest stable/1908) on kubernetes pod. Sometimes I get a
warning
> clib_time_verify_frequency:249: Rejecting large frequency change of
> +infinity%
WIth the warning vpp stops sending and receiving any packets of all
interfaces. This warning logged by vpp a lot of times by each run.
Removing next block of code helps, but I think it is not correct way to
resolve this issue.
> /*
>* Reject large frequency changes, another consequence of
>* system clock changes particularly with old kernels.
>*/
> new_clocks_per_second =
> flt_round_nearest ((f64) dtc / (dtr * c->round_to_units))
> * c->round_to_units;
> delta = new_clocks_per_second - c->clocks_per_second;
> if (delta < 0.0)
> delta = -delta;
> if (PREDICT_FALSE ((delta / c->clocks_per_second) > .01))
> {
> clib_warning ("Rejecting large frequency change of %.2f%%",
>(delta / c->clocks_per_second) * 100.0);
> c->log2_clocks_per_frequency_verify = c->log2_clocks_per_second;
> return;
> }
Could you please help to find a root cause of an issue and resolve it.
Thanks
--
*Artem Belov*
Software Engineer
Email: artem.be...@xored.com
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15160): https://lists.fd.io/g/vpp-dev/message/15160
Mute This Topic: https://lists.fd.io/mt/69693279/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
