Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-08-02 Thread Brad Spencer
Taylor R Campbell writes: [snip] > Can you try the attached patch on a dom0 and see if you still observe > drift? I had a little time to mess with trying 10.99.6 on the DOM0.. I applied your patch to 10.99.6 and built a XEN3_DOM0 kernel (XEN3_DOM0 + bigger IFQ_MAXLEN). The result was, while

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-08-01 Thread Brad Spencer
Taylor R Campbell writes: > [1:text/plain Hide] > >> Date: Tue, 01 Aug 2023 16:02:17 -0400 >> From: Brad Spencer >> >> Taylor R Campbell writes: >> >> > So I just added a printf to the kernel in case this jump happens. Can >> > you update to xen_clock.c 1.15 (and sys/arch/x86/include/cpu.h 1

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-08-01 Thread Taylor R Campbell
> Date: Tue, 01 Aug 2023 16:02:17 -0400 > From: Brad Spencer > > Taylor R Campbell writes: > > > So I just added a printf to the kernel in case this jump happens. Can > > you update to xen_clock.c 1.15 (and sys/arch/x86/include/cpu.h 1.135) > > and try again? > > Sure... Correction: xen_cloc

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-08-01 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Mon, 31 Jul 2023 12:47:20 -0400 >> >> # dtrace -x nolibs -n 'sdt:xen:hardclock:jump { @ = quantize(arg1 - arg0) } >> sdt:xen:hardclock:jump /arg2 >= 430/ { printf("hardclock jump violated >> timecounter contract") }' >> dtrace: description 'sdt:xen:hardclock:

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-08-01 Thread Taylor R Campbell
> Date: Mon, 31 Jul 2023 12:47:20 -0400 > > # dtrace -x nolibs -n 'sdt:xen:hardclock:jump { @ = quantize(arg1 - arg0) } > sdt:xen:hardclock:jump /arg2 >= 430/ { printf("hardclock jump violated > timecounter contract") }' > dtrace: description 'sdt:xen:hardclock:jump ' matched 2 probes > dtrace:

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-31 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Sun, 30 Jul 2023 14:56:53 -0400 >> From: Brad Spencer >> >> Taylor R Campbell writes: >> >> > Can you please try running with the attached patch and share the >> > warnings it produces? Should give slightly more information. >> >> Caught another one. As f

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-30 Thread Taylor R Campbell
> Date: Sun, 30 Jul 2023 14:56:53 -0400 > From: Brad Spencer > > Taylor R Campbell writes: > > > Can you please try running with the attached patch and share the > > warnings it produces? Should give slightly more information. > > Caught another one. As far as I know the system is up to date

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-30 Thread Brad Spencer
Taylor R Campbell writes: > Can you please try running with the attached patch and share the > warnings it produces? Should give slightly more information. Caught another one. As far as I know the system is up to date with all of the requested patches: [ 19419.647972] WARNING: lwp 16 (system

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-28 Thread Taylor R Campbell
> Date: Fri, 28 Jul 2023 09:42:30 -0400 > From: Brad Spencer > > Thanks, that allowed the dtrace to execute, but I never have time to > execute the second probe, as this kernel panic occures within a few > seconds of the first probe being run (probably on the order of 4 - 5 > seconds): > > [ 213

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-28 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Fri, 28 Jul 2023 07:42:04 -0400 >> From: Brad Spencer >> >> dtrace: invalid probe specifier sdt:xen:hardclock:jump { @ = quantize(arg1 - >> arg0) } tick-10s { printa(@) }: probe description :::tick-10s does not match >> any probes > > modload dtrace_profile

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-28 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Fri, 28 Jul 2023 07:42:04 -0400 >> From: Brad Spencer >> >> dtrace: invalid probe specifier sdt:xen:hardclock:jump { @ = quantize(arg1 - >> arg0) } tick-10s { printa(@) }: probe description :::tick-10s does not match >> any probes > > modload dtrace_profile

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-28 Thread Taylor R Campbell
> Date: Fri, 28 Jul 2023 07:42:04 -0400 > From: Brad Spencer > > dtrace: invalid probe specifier sdt:xen:hardclock:jump { @ = quantize(arg1 - > arg0) } tick-10s { printa(@) }: probe description :::tick-10s does not match > any probes modload dtrace_profile

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-28 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Thu, 27 Jul 2023 11:17:30 -0400 >> From: Brad Spencer >> >> On the system that I have that exhibits the negative runtime problem, it >> may very well be the case that hardclocks are missed for 4.3sec. The >> system has to have been up for a while and busy as

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-28 Thread Taylor R Campbell
> Date: Thu, 27 Jul 2023 11:17:30 -0400 > From: Brad Spencer > > On the system that I have that exhibits the negative runtime problem, it > may very well be the case that hardclocks are missed for 4.3sec. The > system has to have been up for a while and busy as a prereq., but if I > then run: >

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-27 Thread Brad Spencer
Taylor R Campbell writes: [snip] > I don't know what the sparc timecounter frequency is, but the Xen > system timecounter returns units of nanoseconds, i.e., runs at 1 GHz, > well within these bounds. So this kind of wraparound leading to > apparently negative runtime -- that is, l->l_stime goi

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-27 Thread Taylor R Campbell
> Date: Thu, 27 Jul 2023 15:05:23 +1000 > from: matthew green > > one problem i've seen in kern_tc.c when the timecounter returns > a smaller value is that tc_delta() ends up returning a very large > (underflowed) value, and that makes the consumers of it do a very > wrong thing. eg, -2 becomes

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-27 Thread Brad Spencer
Taylor R Campbell writes: > Can you please try running with the attached patch and share the > warnings it produces? Should give slightly more information. I applied the patch and caught this: [ 76783.161208] WARNING: lwp 16 (system idle/1) flags 0xa020: timecounter went backwards from (7

re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-26 Thread matthew green
one problem i've seen in kern_tc.c when the timecounter returns a smaller value is that tc_delta() ends up returning a very large (underflowed) value, and that makes the consumers of it do a very wrong thing. eg, -2 becomes 2^32-2, and then eg in binuptime: 477 bintime_addx(bt, th

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-25 Thread Taylor R Campbell
Can you please try running with the attached patch and share the warnings it produces? Should give slightly more information. >From b6f360d9b1fdc418105fcc41b41f1206ca04334d Mon Sep 17 00:00:00 2001 From: Taylor R Campbell Date: Wed, 26 Jul 2023 01:36:28 + Subject: [PATCH] WIP: attribution for

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-25 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Thu, 20 Jul 2023 21:50:26 -0400 >> From: Brad Spencer >> >> With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with >> either of those dtrace probes on the DOMU: >> >> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: { >> p

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-21 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Thu, 20 Jul 2023 21:50:26 -0400 >> From: Brad Spencer >> >> With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with >> either of those dtrace probes on the DOMU: >> >> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: { >> p

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-21 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Thu, 20 Jul 2023 21:50:26 -0400 >> From: Brad Spencer >> >> With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with >> either of those dtrace probes on the DOMU: >> >> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: { >> p

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-21 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Thu, 20 Jul 2023 21:50:26 -0400 >> From: Brad Spencer >> >> With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with >> either of those dtrace probes on the DOMU: >> >> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: { >> p

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-20 Thread Taylor R Campbell
> Date: Thu, 20 Jul 2023 21:50:26 -0400 > From: Brad Spencer > > With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with > either of those dtrace probes on the DOMU: > > dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: { > printf("%d %d %d %d %d %d %d", arg0,

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-20 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Sat, 08 Jul 2023 14:34:56 -0400 >> From: Brad Spencer >> >> Taylor R Campbell writes: >> >> > Can you either: >> >> Yes, I can perform as much of this as needed after I get some other >> stuff in life dealt with more towards the end of the month. I really

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-13 Thread Taylor R Campbell
> Date: Thu, 13 Jul 2023 13:39:43 + > From: Taylor R Campbell > > > Date: Sat, 08 Jul 2023 14:34:56 -0400 > > From: Brad Spencer > > > > [ 1792486.921759] kobj_checksyms, 988: [dtrace]: linker error: symbol > > `dtrace_invop_calltrap_addr' not found > > [ 1792486.921759] kobj_checksyms, 98

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-13 Thread Taylor R Campbell
> Date: Sat, 08 Jul 2023 14:34:56 -0400 > From: Brad Spencer > > Taylor R Campbell writes: > > > Can you either: > > Yes, I can perform as much of this as needed after I get some other > stuff in life dealt with more towards the end of the month. I really > won't have any time before then. N

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-08 Thread Brad Spencer
Taylor R Campbell writes: >> Date: Wed, 04 Jan 2023 14:43:25 -0500 >> From: Brad Spencer >> >> So... I have a PV+PVSHIM DOMU running a pretty recent 9.x on a DOM0 >> running a 9.99.xx kernel. The DOM0 is not large, a 4 processor E-2224 >> with 32GB of memory. The DOMU has 2 VCPUs and 8GB of

Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-07-08 Thread Taylor R Campbell
> Date: Wed, 04 Jan 2023 14:43:25 -0500 > From: Brad Spencer > > So... I have a PV+PVSHIM DOMU running a pretty recent 9.x on a DOM0 > running a 9.99.xx kernel. The DOM0 is not large, a 4 processor E-2224 > with 32GB of memory. The DOMU has 2 VCPUs and 8GB of memory. About > every day a very

What to do about "WARNING: negative runtime; monotonic clock has gone backwards"

2023-01-04 Thread Brad Spencer
So... I have a PV+PVSHIM DOMU running a pretty recent 9.x on a DOM0 running a 9.99.xx kernel. The DOM0 is not large, a 4 processor E-2224 with 32GB of memory. The DOMU has 2 VCPUs and 8GB of memory. About every day a very particular DOMU tosses the: WARNING: negative runtime; monotonic clock