Re: TSC as timecounter makes system lag

[John Baldwin]

On Friday, February 24, 2017 10:50:19 PM Jia-Shiun Li wrote:
> On Fri, Feb 24, 2017 at 9:32 PM, Jia-Shiun Li  wrote:
> 
> > On Fri, Feb 24, 2017 at 7:45 PM, Konstantin Belousov 
> > wrote:
> >
> >> On Fri, Feb 24, 2017 at 12:15:26PM +0800, Jia-Shiun Li wrote:
> >> > Tested working on E7400 against r313909. And changing timecounter
> >> from/to
> >> > TSC
> >> > correctly enables/disables C2.
> >> >
> >> > The latter part cpu_disable_c2_sleep++ is not needed. When
> >> > init_TSC_tc() got called timecounter is not yet tsc_timecounter.
> >> > inittimecounter() later will do the work calling tc_windup().
> >> >
> >>
> >> You mean, just this
> >> -   if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
> >> +   if (cpu_vendor_id == CPU_VENDOR_INTEL &&
> >> is enough to fix the issue ?  If yes, we can remove the cpu_deepest_sleep
> >> variable.  This is John' observation, I think he would prefer to prepare
> >> the patch.
> >>
> >
> > Correct. That's enough.
> >
> >
> Since that's simple enough... patch attached.
> Tested against r313909 too.

Thanks for tracking down the cause and the patch!

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

On Fri, Feb 24, 2017 at 9:32 PM, Jia-Shiun Li  wrote:

> On Fri, Feb 24, 2017 at 7:45 PM, Konstantin Belousov 
> wrote:
>
>> On Fri, Feb 24, 2017 at 12:15:26PM +0800, Jia-Shiun Li wrote:
>> > Tested working on E7400 against r313909. And changing timecounter
>> from/to
>> > TSC
>> > correctly enables/disables C2.
>> >
>> > The latter part cpu_disable_c2_sleep++ is not needed. When
>> > init_TSC_tc() got called timecounter is not yet tsc_timecounter.
>> > inittimecounter() later will do the work calling tc_windup().
>> >
>>
>> You mean, just this
>> -   if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
>> +   if (cpu_vendor_id == CPU_VENDOR_INTEL &&
>> is enough to fix the issue ?  If yes, we can remove the cpu_deepest_sleep
>> variable.  This is John' observation, I think he would prefer to prepare
>> the patch.
>>
>
> Correct. That's enough.
>
>
Since that's simple enough... patch attached.
Tested against r313909 too.

-Jia-Shiun.


fix-tsc-timecounter.patch
Description: Binary data
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

On Fri, Feb 24, 2017 at 7:45 PM, Konstantin Belousov 
wrote:

> On Fri, Feb 24, 2017 at 12:15:26PM +0800, Jia-Shiun Li wrote:
> > Tested working on E7400 against r313909. And changing timecounter from/to
> > TSC
> > correctly enables/disables C2.
> >
> > The latter part cpu_disable_c2_sleep++ is not needed. When
> > init_TSC_tc() got called timecounter is not yet tsc_timecounter.
> > inittimecounter() later will do the work calling tc_windup().
> >
>
> You mean, just this
> -   if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
> +   if (cpu_vendor_id == CPU_VENDOR_INTEL &&
> is enough to fix the issue ?  If yes, we can remove the cpu_deepest_sleep
> variable.  This is John' observation, I think he would prefer to prepare
> the patch.
>

Correct. That's enough.


-Jia-Shiun.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Konstantin Belousov]

On Fri, Feb 24, 2017 at 12:15:26PM +0800, Jia-Shiun Li wrote:
> On Fri, Feb 24, 2017 at 12:55 AM, John Baldwin  wrote:
> 
> > On Thursday, February 23, 2017 11:04:58 PM Jia-Shiun Li wrote:
> > >
> > >
> > > This does not work.
> > >
> > > I added a printf before the outer if clause, and it says
> > >
> > > init_TSC_tc:546: deepest  vendor 8086 amd_pminfo 
> > >
> > > full boot dmesg attached. Looks init_TSC_tc() is called too early before
> > > acpi_cpu_attach() initializing cpu_deepest_sleep. Maybe it should be put
> > > after
> > > driver initialization, since it depends on probed ACPI C states?
> >
> > We don't actually need cpu_deepest_sleep.  We could just set C2STOP always.
> > It doesn't hurt to have the flag set if the system only supports C1 except
> > that you get the printf in a verbose boot.
> >
> > Try this slight variation of Konstantin's patch.  If this works we can
> > remove
> > cpu_deepest_sleep entirely as a followup since it will no longer be used:
> >
> > Index: tsc.c
> > ===
> > --- tsc.c   (revision 314113)
> > +++ tsc.c   (working copy)
> > @@ -542,9 +542,11 @@ init_TSC_tc(void)
> >  * result incorrect runtimes for kernel idle threads (but not
> >  * for any non-idle threads).
> >  */
> > -   if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
> > +   if (cpu_vendor_id == CPU_VENDOR_INTEL &&
> > (amd_pminfo & AMDPM_TSC_INVARIANT) == 0) {
> > tsc_timecounter.tc_flags |= TC_FLAGS_C2STOP;
> > +   if (timecounter == &tsc_timecounter)
> > +   cpu_disable_c2_sleep++;
> > if (bootverbose)
> > printf("TSC timecounter disables C2 and C3.\n");
> > }
> >
> 
> Tested working on E7400 against r313909. And changing timecounter from/to
> TSC
> correctly enables/disables C2.
> 
> The latter part cpu_disable_c2_sleep++ is not needed. When
> init_TSC_tc() got called timecounter is not yet tsc_timecounter.
> inittimecounter() later will do the work calling tc_windup().
> 

You mean, just this
-   if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
+   if (cpu_vendor_id == CPU_VENDOR_INTEL &&
is enough to fix the issue ?  If yes, we can remove the cpu_deepest_sleep
variable.  This is John' observation, I think he would prefer to prepare
the patch.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

On Fri, Feb 24, 2017 at 12:55 AM, John Baldwin  wrote:

> On Thursday, February 23, 2017 11:04:58 PM Jia-Shiun Li wrote:
> >
> >
> > This does not work.
> >
> > I added a printf before the outer if clause, and it says
> >
> > init_TSC_tc:546: deepest  vendor 8086 amd_pminfo 
> >
> > full boot dmesg attached. Looks init_TSC_tc() is called too early before
> > acpi_cpu_attach() initializing cpu_deepest_sleep. Maybe it should be put
> > after
> > driver initialization, since it depends on probed ACPI C states?
>
> We don't actually need cpu_deepest_sleep.  We could just set C2STOP always.
> It doesn't hurt to have the flag set if the system only supports C1 except
> that you get the printf in a verbose boot.
>
> Try this slight variation of Konstantin's patch.  If this works we can
> remove
> cpu_deepest_sleep entirely as a followup since it will no longer be used:
>
> Index: tsc.c
> ===
> --- tsc.c   (revision 314113)
> +++ tsc.c   (working copy)
> @@ -542,9 +542,11 @@ init_TSC_tc(void)
>  * result incorrect runtimes for kernel idle threads (but not
>  * for any non-idle threads).
>  */
> -   if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
> +   if (cpu_vendor_id == CPU_VENDOR_INTEL &&
> (amd_pminfo & AMDPM_TSC_INVARIANT) == 0) {
> tsc_timecounter.tc_flags |= TC_FLAGS_C2STOP;
> +   if (timecounter == &tsc_timecounter)
> +   cpu_disable_c2_sleep++;
> if (bootverbose)
> printf("TSC timecounter disables C2 and C3.\n");
> }
>

Tested working on E7400 against r313909. And changing timecounter from/to
TSC
correctly enables/disables C2.

The latter part cpu_disable_c2_sleep++ is not needed. When
init_TSC_tc() got called timecounter is not yet tsc_timecounter.
inittimecounter() later will do the work calling tc_windup().


-Jia-Shiun.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[John Baldwin]

On Thursday, February 23, 2017 11:04:58 PM Jia-Shiun Li wrote:
> On Thu, Feb 23, 2017 at 6:08 PM, Konstantin Belousov 
> wrote:
> 
> >
> > This is a useful analysis.
> >
> > Yes, I think that there is an init ordering issue. Note that
> > cpu_disable_c2_sleep is only changed in tc_windup() when timecounter
> > is changed. If existing and already engadged timecounter suddenly gets
> > TC_FLAG_C2STOP set, tc_windup() ignores the flag.  And with the early
> > AP startup, tsc seems to be set as timecounter too early.
> >
> > Just moving order of init_TSC_tc() would not help, since tsc checks smp
> > consistency, which requires started APs.  Try this for now, but might
> > be John has better idea how to handle the issue.  You might need to add
> > some extern declarations for the patch to compile.
> >
> > diff --git a/sys/x86/x86/tsc.c b/sys/x86/x86/tsc.c
> > index 3f36fbd9f8a..f8e33069c70 100644
> > --- a/sys/x86/x86/tsc.c
> > +++ b/sys/x86/x86/tsc.c
> > @@ -545,6 +545,8 @@ init_TSC_tc(void)
> > if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
> > (amd_pminfo & AMDPM_TSC_INVARIANT) == 0) {
> > tsc_timecounter.tc_flags |= TC_FLAGS_C2STOP;
> > +   if (timecounter == &tsc_timecounter)
> > +   cpu_disable_c2_sleep++;
> > if (bootverbose)
> > printf("TSC timecounter disables C2 and C3.\n");
> > }
> >
> 
> 
> This does not work.
> 
> I added a printf before the outer if clause, and it says
> 
> init_TSC_tc:546: deepest  vendor 8086 amd_pminfo 
> 
> full boot dmesg attached. Looks init_TSC_tc() is called too early before
> acpi_cpu_attach() initializing cpu_deepest_sleep. Maybe it should be put
> after
> driver initialization, since it depends on probed ACPI C states?

We don't actually need cpu_deepest_sleep.  We could just set C2STOP always.
It doesn't hurt to have the flag set if the system only supports C1 except
that you get the printf in a verbose boot.

Try this slight variation of Konstantin's patch.  If this works we can remove
cpu_deepest_sleep entirely as a followup since it will no longer be used:

Index: tsc.c
===
--- tsc.c   (revision 314113)
+++ tsc.c   (working copy)
@@ -542,9 +542,11 @@ init_TSC_tc(void)
 * result incorrect runtimes for kernel idle threads (but not
 * for any non-idle threads).
 */
-   if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
+   if (cpu_vendor_id == CPU_VENDOR_INTEL &&
(amd_pminfo & AMDPM_TSC_INVARIANT) == 0) {
tsc_timecounter.tc_flags |= TC_FLAGS_C2STOP;
+   if (timecounter == &tsc_timecounter)
+   cpu_disable_c2_sleep++;
if (bootverbose)
printf("TSC timecounter disables C2 and C3.\n");
}



-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

On Thu, Feb 23, 2017 at 6:08 PM, Konstantin Belousov 
wrote:

>
> This is a useful analysis.
>
> Yes, I think that there is an init ordering issue. Note that
> cpu_disable_c2_sleep is only changed in tc_windup() when timecounter
> is changed. If existing and already engadged timecounter suddenly gets
> TC_FLAG_C2STOP set, tc_windup() ignores the flag.  And with the early
> AP startup, tsc seems to be set as timecounter too early.
>
> Just moving order of init_TSC_tc() would not help, since tsc checks smp
> consistency, which requires started APs.  Try this for now, but might
> be John has better idea how to handle the issue.  You might need to add
> some extern declarations for the patch to compile.
>
> diff --git a/sys/x86/x86/tsc.c b/sys/x86/x86/tsc.c
> index 3f36fbd9f8a..f8e33069c70 100644
> --- a/sys/x86/x86/tsc.c
> +++ b/sys/x86/x86/tsc.c
> @@ -545,6 +545,8 @@ init_TSC_tc(void)
> if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
> (amd_pminfo & AMDPM_TSC_INVARIANT) == 0) {
> tsc_timecounter.tc_flags |= TC_FLAGS_C2STOP;
> +   if (timecounter == &tsc_timecounter)
> +   cpu_disable_c2_sleep++;
> if (bootverbose)
> printf("TSC timecounter disables C2 and C3.\n");
> }
>


This does not work.

I added a printf before the outer if clause, and it says

init_TSC_tc:546: deepest  vendor 8086 amd_pminfo 

full boot dmesg attached. Looks init_TSC_tc() is called too early before
acpi_cpu_attach() initializing cpu_deepest_sleep. Maybe it should be put
after
driver initialization, since it depends on probed ACPI C states?

-Jia-Shiun.
Table 'FACP' at 0xcdd80290
Table 'APIC' at 0xcdd80390
APIC: Found table at 0xcdd80390
APIC: Using the MADT enumerator.
MADT: Found CPU APIC ID 0 ACPI ID 1: enabled
SMP: Added CPU 0 (AP)
MADT: Found CPU APIC ID 1 ACPI ID 2: enabled
SMP: Added CPU 1 (AP)
MADT: Found CPU APIC ID 130 ACPI ID 3: disabled
MADT: Found CPU APIC ID 131 ACPI ID 4: disabled
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #21 r313909M: Thu Feb 23 22:31:13 CST 2017
jsli@jsli-e5:/usr/obj/usr/src/c2dlag/sys/GENERIC-NODEBUG amd64
FreeBSD clang version 3.9.1 (tags/RELEASE_391/final 289601) (based on LLVM 
3.9.1)
Table 'FACP' at 0xcdd80290
Table 'APIC' at 0xcdd80390
Table 'MCFG' at 0xcdd80400
Table 'OEMB' at 0xcdd8e040
Table 'HPET' at 0xcdd89530
Table 'GSCI' at 0xcdd8e0d0
Table 'OSFR' at 0xcdd89570
Table 'SSDT' at 0xcdd90bd0
ACPI: No SRAT table found
PPIM 0: PA=0xa, VA=0x8261, size=0x1, mode=0
VT(vga): resolution 640x480
Preloaded elf kernel "/boot/c2d.v/kernel" at 0x824ad000.
Preloaded /boot/entropy "/boot/entropy" at 0x824ade40.
Preloaded elf obj module "/boot/c2d.v/zfs.ko" at 0x824ade90.
Preloaded elf obj module "/boot/c2d.v/opensolaris.ko" at 0x824ae678.
Calibrating TSC clock ... TSC clock: 2800156285 Hz
CPU: Intel(R) Core(TM)2 Duo CPU E7400  @ 2.80GHz (2800.16-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10
  
Features=0xbfebfbff
  
Features2=0xc08e39d
  AMD Features=0x20100800
  AMD Features2=0x1
  TSC: P-state invariant, performance statistics
Instruction TLB: 2M pages, 4-way, 8 entries or 4M pages, 4-way, 4 entries
Instruction TLB: 4 KB Pages, 4-way set associative, 128 entries
64-Byte prefetching
Data TLB0: 4 KByte pages, 4-way associative, 16 entries
Data TLB0: 4 MByte pages, 4-way set associative, 16 entries
2nd-level cache: 3MByte, 12-way set associative, 64 byte line size
1st-level instruction cache: 32 KB, 8-way set associative, 64 byte line size
Data TLB1: 4 KByte pages, 4-way associative, 256 entries
1st-level data cache: 32 KB, 8-way set associative, 64 byte line size
L2 cache: 3072 kbytes, 8-way associative, 64 bytes/line
real memory  = 4294967296 (4096 MB)
Physical memory chunk(s):
0x0001 - 0x0009afff, 569344 bytes (139 pages)
0x0010 - 0x001f, 1048576 bytes (256 pages)
0x024f6000 - 0xc614efff, 3284504576 bytes (801881 pages)
0x0001 - 0x00012ffcdfff, 805101568 bytes (196558 pages)
avail memory = 4058112000 (3870 MB)
Event timer "LAPIC" quality 100
LAPIC: ipi_wait() us multiplier 68 (r 4100134 tsc 2800156285)
ACPI APIC Table: 
Package ID shift: 1
L2 cache ID shift: 1
L1 cache ID shift: 0
Core ID shift: 0
INTR: Adding local APIC 1 as a target
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
Package HW ID = 0
Core HW ID = 0
CPU0 (BSP): APIC ID: 0
Core HW ID = 1
CPU1 (AP): APIC ID: 1
APIC: CPU 0 has ACPI ID 1
APIC: CPU 1 has ACPI ID 2
x86bios:  IVT 0x00-0x0004ff at 0x

Re: TSC as timecounter makes system lag

[Konstantin Belousov]

On Thu, Feb 23, 2017 at 01:11:29AM +0800, Jia-Shiun Li wrote:
> I got the impression that TSC was not preferred timecounter
> if it is not C-state invariant. But this apparenly is not the case now.
> Dig a bit and found r277900 chose to prefer TSC over saving power
> by disabling C2 state when TSC is selected as timecounter.
> 
> But with EARLY_AP_STARTUP, and TSC as timecounter,
> CPU still enters C2 state.
> (Observed by sysctl dev.cpu.0.cx_usage_counters)
> With nooptions EARLY_AP_STARTUP, CPU correctly stays 100%
> in C1 and no lower.
> 
> I added a printf in tc_windup() to check. With EARLY_AP_STARTUP,
> cpu_disable_c2_sleep is never increased, so it can not prevent CPU
> from entering C2.
> 
> Guess there's some kind of race or init order issue, but it is
> beyond my understanding for now.

This is a useful analysis.

Yes, I think that there is an init ordering issue. Note that
cpu_disable_c2_sleep is only changed in tc_windup() when timecounter
is changed. If existing and already engadged timecounter suddenly gets
TC_FLAG_C2STOP set, tc_windup() ignores the flag.  And with the early
AP startup, tsc seems to be set as timecounter too early.

Just moving order of init_TSC_tc() would not help, since tsc checks smp
consistency, which requires started APs.  Try this for now, but might
be John has better idea how to handle the issue.  You might need to add
some extern declarations for the patch to compile.

diff --git a/sys/x86/x86/tsc.c b/sys/x86/x86/tsc.c
index 3f36fbd9f8a..f8e33069c70 100644
--- a/sys/x86/x86/tsc.c
+++ b/sys/x86/x86/tsc.c
@@ -545,6 +545,8 @@ init_TSC_tc(void)
if (cpu_deepest_sleep >= 2 && cpu_vendor_id == CPU_VENDOR_INTEL &&
(amd_pminfo & AMDPM_TSC_INVARIANT) == 0) {
tsc_timecounter.tc_flags |= TC_FLAGS_C2STOP;
+   if (timecounter == &tsc_timecounter)
+   cpu_disable_c2_sleep++;
if (bootverbose)
printf("TSC timecounter disables C2 and C3.\n");
}
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

I got the impression that TSC was not preferred timecounter
if it is not C-state invariant. But this apparenly is not the case now.
Dig a bit and found r277900 chose to prefer TSC over saving power
by disabling C2 state when TSC is selected as timecounter.

But with EARLY_AP_STARTUP, and TSC as timecounter,
CPU still enters C2 state.
(Observed by sysctl dev.cpu.0.cx_usage_counters)
With nooptions EARLY_AP_STARTUP, CPU correctly stays 100%
in C1 and no lower.

I added a printf in tc_windup() to check. With EARLY_AP_STARTUP,
cpu_disable_c2_sleep is never increased, so it can not prevent CPU
from entering C2.

Guess there's some kind of race or init order issue, but it is
beyond my understanding for now.


-Jia-Shiun.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

FYI since it was not really solved as of r313090,
I created bug 216833 to keep track of it.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216833

-Jia-Shiun.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Hans Petter Selasky]

FYI:

https://svnweb.freebsd.org/changeset/base/312551

--HPS
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

On Tue, Jan 17, 2017 at 10:05 PM, Hans Petter Selasky 
wrote:

> I've seen something similar. Does the attached patch make any difference?
>
> Can you dump:
>
> vmstat -i
>
> Just after boot w/ and w/o the attached patch, when the keystroke did not
> repeat smoothly.
>
>
Your patch fixes this issue. It is now working as expected.

vmstat output attached. Running w/ kernel r312210.


Thanks,
-Jia-Shiun.
interrupt  total   rate
???0  0
irq1: atkbd0   2  0
stray irq1 0  0
irq0: attimer0 0  0
stray irq0 0  0
irq3:  0  0
stray irq3 0  0
irq4: uart00  0
stray irq4 0  0
irq5:  0  0
stray irq5 0  0
irq6:  0  0
stray irq6 0  0
irq7:  0  0
stray irq7 0  0
irq8: atrtc0   0  0
stray irq8 0  0
irq9: acpi00  0
stray irq9 0  0
irq10: 0  0
stray irq100  0
irq11: 0  0
stray irq110  0
irq12: 0  0
stray irq120  0
irq13: 0  0
stray irq130  0
irq14: 0  0
stray irq140  0
irq15: 0  0
stray irq150  0
irq16: em0:irq0++ 16  0
stray irq160  0
irq17: 0  0
stray irq170  0
irq18: uhci2 ehci0+   18  0
stray irq180  0
irq19: uhci4   0  0
stray irq190  0
irq20: hpet0   28522442
stray irq200  0
irq21: uhci1   0  0
stray irq210  0
irq22: 0  0
stray irq220  0
irq23: uhci3 ehci1 0  0
stray irq230  0
cpu0:timer 0  0
irq256: hdac0110  2
stray irq256   0  0
irq257: pcib1  0  0
stray irq257   0  0
irq258: pcib2  0  0
stray irq258   0  0
irq259: pcib3  0  0
stray irq259   0  0
irq260: re0 6587102
stray irq260   0  0
irq261: ahci0:ch0   4062 63
stray irq261   0  0
irq262: ahci0:ch1  0  0
stray irq262   0  0
irq263: ahci0:ch2  0  0
stray irq263   0  0
irq264: ahci0:ch3  0  0
stray irq264   0  0
irq265: ahci0:ch4  0  0
stray irq265   0  0
irq266: ahci0:ch5  0  0
stray irq266   0  0
irq267: ahci0:60  0
stray irq267   0  0
irq268: ahci0:70  0
stray irq268   0  0
irq269: ahci0:80  0
stray irq269   0  0
irq270: ahci0:90  0
stray irq270   0  0
irq271: ahci0:10   0  0
stray irq271   0  0
irq272: ahci0:11   0  0
stray irq272   0  0
irq273: ahci0:12   0  0
stray irq273   0  0
irq274: ahci0:13   0  0
stray irq274   0  0
irq275: ahci0:14   0  0
stray irq275 

Re: TSC as timecounter makes system lag

[Hans Petter Selasky]

On 01/16/17 15:34, Jia-Shiun Li wrote:

Yes. I noticed this because systat refreshes looked slower,
and keystroke did not repeat smoothly for 30/s.


I've seen something similar. Does the attached patch make any difference?

Can you dump:

vmstat -i

Just after boot w/ and w/o the attached patch, when the keystroke did 
not repeat smoothly.


--HPS
diff --git a/sys/kern/kern_clocksource.c b/sys/kern/kern_clocksource.c
index 7f7769d..454a130 100644
--- a/sys/kern/kern_clocksource.c
+++ b/sys/kern/kern_clocksource.c
@@ -511,7 +511,7 @@ configtimer(int start)
 			state->nexthard = next;
 			state->nextstat = next;
 			state->nextprof = next;
-			state->nextcall = next;
+			state->nextcall = SBT_MAX;
 			state->nextcallopt = next;
 			hardclock_sync(cpu);
 		}
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

On Mon, Jan 16, 2017 at 8:00 PM, Konstantin Belousov 
wrote:

> On Mon, Jan 16, 2017 at 12:28:54PM +0800, Jia-Shiun Li wrote:
> > BTW please see my other mail of this thread. It seems to be related to
> > EARLY_AP_STARTUP option.
> Yes, I noted, I might have an idea, but the report that changing the
> timecounter makes the lags go away still does not fit into my understanding
> of the code.
>
> Most likely this is an interaction between the EARLY_AP_STARTUP and the
> fact that HPET interrupt is global, while most modern systems use LAPIC
> event timer, which is per-cpu, and the testing of the option was done
> on them. There are some differences in handling the configurations, see
> sys/kern/kern_clocksource.c, the option and ET_FLAG_PERCPU.
>
>
And, changing the _timecounter_ fixes the issue ?  Can you double-check
> this ?
>

Yes. I noticed this because systat refreshes looked slower,
and keystroke did not repeat smoothly for 30/s.
I have system clock shown on tmux status line. On c2d it drifted away.
Setting timecounter brings it back to normal.
See also eventtimer & timecounter tests below.


>
> With the settings above, i.e. HPET for both eventtimer and timecounter,
> please show vmstat -ia output for two times with the interval of 2 secs.
>

Attached.


>
> What if you change _eventtimer_ to APIC and then immediately back to HPET,
> does the problem go away ?
>
>
It doesn't look so. But keeping LAPIC as eventtimer helps.

jsli@jsli-bsd:/home/jsli # sysctl kern.eventtimer.timer=LAPIC && sysctl
kern.eventtimer.timer=HPET && ntpdate tw.pool.ntp.org && sleep 30 &&
ntpdate tw.pool.ntp.org
kern.eventtimer.timer: HPET -> LAPIC
kern.eventtimer.timer: LAPIC -> HPET
16 Jan 22:00:21 ntpdate[8472]: step time server 203.71.244.7 offset
18.980716 sec
16 Jan 22:01:56 ntpdate[8601]: step time server 103.226.213.30 offset
58.813079 sec
jsli@jsli-bsd:/home/jsli # sysctl kern.eventtimer.timer=LAPIC && ntpdate
tw.pool.ntp.org && sleep 30 && ntpdate tw.pool.ntp.org
 kern.eventtimer.timer: HPET -> LAPIC
16 Jan 22:02:36 ntpdate[8666]: step time server 103.226.213.30 offset
19.773086 sec
16 Jan 22:03:13 ntpdate[8776]: adjust time server 103.226.213.30 offset
0.000455 sec
jsli@jsli-bsd:/home/jsli # sysctl kern.eventtimer.timer=HPET && ntpdate
tw.pool.ntp.org && sleep 30 && ntpdate tw.pool.ntp.org
kern.eventtimer.timer: LAPIC -> HPET
16 Jan 22:03:47 ntpdate[8853]: step time server 103.226.213.30 offset
6.344004 sec
16 Jan 22:05:18 ntpdate[8975]: step time server 61.216.153.105 offset
54.908872 sec
jsli@jsli-bsd:/home/jsli # sysctl kern.timecounter.hardware=HPET && ntpdate
tw.pool.ntp.org && sleep 30 && ntpdate tw.pool.ntp.org
kern.timecounter.hardware: TSC-low -> HPET
16 Jan 22:06:29 ntpdate[9073]: step time server 59.124.29.241 offset
39.211691 sec
16 Jan 22:07:05 ntpdate[9185]: adjust time server 61.216.153.105 offset
0.001015 sec
jsli@jsli-bsd:/home/jsli # sysctl kern.timecounter.hardware=TSC-low &&
ntpdate tw.pool.ntp.org && sleep 30 && ntpdate tw.pool.ntp.org
kern.timecounter.hardware: HPET -> TSC-low
16 Jan 22:07:28 ntpdate[9244]: step time server 61.216.153.105 offset
3.122954 sec
16 Jan 22:08:58 ntpdate[9357]: step time server 61.216.153.104 offset
53.758451 sec
jsli@jsli-bsd:/home/jsli #



> Also, if you set the loader tunable kern.eventtimer.timer to LAPIC,
> and do not enable C2+, does the system boot into the usable state ?
>
>
Not sure if you mean disabling C2 from BIOS.
Right now I don't have BIOS access to the c2d, and my notebook only
has minimal BIOS settings to play with. Anyway on notebook

  set kern.eventtimer.timer=LAPIC

in loader prompt to boot , and the system still lags. But after setting

  sysctl dev.cpu.0.cx_lowest=C1 &&  sysctl dev.cpu.1.cx_lowest=C1

system clock returns to normal speed.


-Jia-Shiun.
jsli@jsli-bsd:~ % vmstat -ia 2 2
interrupt  total   rate
???0  0
irq1: atkbd0   2  0
stray irq1 0  0
irq0: attimer0 0  0
stray irq0 0  0
irq3:  0  0
stray irq3 0  0
irq4: uart00  0
stray irq4 0  0
irq5:  0  0
stray irq5 0  0
irq6:  0  0
stray irq6 0  0
irq7:  0  0
stray irq7 0  0
irq8: atrtc0   0  0
stray irq8 0  0
irq9: acpi00  0
stray irq9 0  0
irq10: 0  0
stray irq100  0
irq11: 

Re: TSC as timecounter makes system lag

[Konstantin Belousov]

On Mon, Jan 16, 2017 at 12:28:54PM +0800, Jia-Shiun Li wrote:
> BTW please see my other mail of this thread. It seems to be related to
> EARLY_AP_STARTUP option.
Yes, I noted, I might have an idea, but the report that changing the
timecounter makes the lags go away still does not fit into my understanding
of the code.

Most likely this is an interaction between the EARLY_AP_STARTUP and the
fact that HPET interrupt is global, while most modern systems use LAPIC
event timer, which is per-cpu, and the testing of the option was done
on them. There are some differences in handling the configurations, see
sys/kern/kern_clocksource.c, the option and ET_FLAG_PERCPU.

> 
> 
> On Mon, Jan 16, 2017 at 4:20 AM, Konstantin Belousov 
> wrote:
> 
> > I still do not understand.  Is the sysctl output below from the pristine
> > boot where no timecounter/eventtimer reconfiguration were done ?
> >
> > Show me exact command which you used to revive the machine.  Do not
> > describe
> > it by words, copy/paste from the console.
> >
> >
> Don't have access to the exact machine right now.
> Let me reproduce it on another with a c2d E7400.
> 
> login as: jsli
> Authenticating with public key "rsa-key-20160711@jsli-pc"
> Last login: Mon Jan 16 11:11:28 2017 from tmux(1074).%1
> FreeBSD 12.0-CURRENT (GENERIC-NODEBUG) #24 r312210: Sun Jan 15 15:03:40 CST
> 2017
> 
> Welcome to FreeBSD!
> 
> Release Notes, Errata: https://www.FreeBSD.org/releases/
> Security Advisories:   https://www.FreeBSD.org/security/
> FreeBSD Handbook:  https://www.FreeBSD.org/handbook/
> FreeBSD FAQ:   https://www.FreeBSD.org/faq/
> Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-
> questions/
> FreeBSD Forums:https://forums.FreeBSD.org/
> 
> Documents installed with the system are in the /usr/local/share/doc/freebsd/
> directory, or can be installed later with:  pkg install en-freebsd-doc
> For other languages, replace "en" with a language code like de or fr.
> 
> Show the version of FreeBSD installed:  freebsd-version ; uname -a
> Please include that output and any error messages when posting questions.
> Introduction to manual pages:  man man
> FreeBSD directory layout:  man hier
> 
> Edit /etc/motd to change this login announcement.
> jsli@jsli-bsd:~ % sysctl kern.eventtimer kern.timecounter
> kern.eventtimer.periodic: 0
> kern.eventtimer.timer: HPET
> kern.eventtimer.idletick: 0
> kern.eventtimer.singlemul: 2
> kern.eventtimer.choice: HPET(450) HPET1(440) HPET2(440) HPET3(440)
> LAPIC(100) i8254(100) RTC(0)
> kern.eventtimer.et.HPET3.quality: 440
> kern.eventtimer.et.HPET3.frequency: 14318180
> kern.eventtimer.et.HPET3.flags: 3
> kern.eventtimer.et.HPET2.quality: 440
> kern.eventtimer.et.HPET2.frequency: 14318180
> kern.eventtimer.et.HPET2.flags: 3
> kern.eventtimer.et.HPET1.quality: 440
> kern.eventtimer.et.HPET1.frequency: 14318180
> kern.eventtimer.et.HPET1.flags: 3
> kern.eventtimer.et.HPET.quality: 450
> kern.eventtimer.et.HPET.frequency: 14318180
> kern.eventtimer.et.HPET.flags: 3
> kern.eventtimer.et.RTC.quality: 0
> kern.eventtimer.et.RTC.frequency: 32768
> kern.eventtimer.et.RTC.flags: 17
> kern.eventtimer.et.i8254.quality: 100
> kern.eventtimer.et.i8254.frequency: 1193182
> kern.eventtimer.et.i8254.flags: 1
> kern.eventtimer.et.LAPIC.quality: 100
> kern.eventtimer.et.LAPIC.frequency: 0
> kern.eventtimer.et.LAPIC.flags: 15
> kern.timecounter.tsc_shift: 1
> kern.timecounter.smp_tsc_adjust: 0
> kern.timecounter.smp_tsc: 1
> kern.timecounter.invariant_tsc: 1
> kern.timecounter.fast_gettime: 1
> kern.timecounter.tick: 1
> kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(1000)
> dummy(-100)
> kern.timecounter.hardware: TSC-low
> kern.timecounter.alloweddeviation: 5
> kern.timecounter.stepwarnings: 0
> kern.timecounter.tc.ACPI-fast.quality: 900
> kern.timecounter.tc.ACPI-fast.frequency: 3579545
> kern.timecounter.tc.ACPI-fast.counter: 5046106
> kern.timecounter.tc.ACPI-fast.mask: 16777215
> kern.timecounter.tc.HPET.quality: 950
> kern.timecounter.tc.HPET.frequency: 14318180
> kern.timecounter.tc.HPET.counter: 1449012340
> kern.timecounter.tc.HPET.mask: 4294967295
> kern.timecounter.tc.i8254.quality: 0
> kern.timecounter.tc.i8254.frequency: 1193182
> kern.timecounter.tc.i8254.counter: 10698
> kern.timecounter.tc.i8254.mask: 65535
> kern.timecounter.tc.TSC-low.quality: 1000
> kern.timecounter.tc.TSC-low.frequency: 1400076588
> kern.timecounter.tc.TSC-low.counter: 2260820915
> kern.timecounter.tc.TSC-low.mask: 4294967295
> jsli@jsli-bsd:~ % su
> Password:
> jsli@jsli-bsd:/home/jsli # sysctl kern.timecounter.hardware=HPET
> kern.timecounter.hardware: TSC-low -> HPET
> jsli@jsli-bsd:/home/jsli # sysctl kern.eventtimer kern.timecounter
> kern.eventtimer.periodic: 0
> kern.eventtimer.timer: HPET
> kern.eventtimer.idletick: 0
> kern.eventtimer.singlemul: 2
> kern.eventtimer.choice: HPET(450) HPET1(440) HPET2(440) HPET3(440)
> LAPIC(100) i8254(100) RTC(0)
> kern.eventtimer.et.HPET3.quality: 440
> 

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

BTW please see my other mail of this thread. It seems to be related to
EARLY_AP_STARTUP option.


On Mon, Jan 16, 2017 at 4:20 AM, Konstantin Belousov 
wrote:

> I still do not understand.  Is the sysctl output below from the pristine
> boot where no timecounter/eventtimer reconfiguration were done ?
>
> Show me exact command which you used to revive the machine.  Do not
> describe
> it by words, copy/paste from the console.
>
>
Don't have access to the exact machine right now.
Let me reproduce it on another with a c2d E7400.

login as: jsli
Authenticating with public key "rsa-key-20160711@jsli-pc"
Last login: Mon Jan 16 11:11:28 2017 from tmux(1074).%1
FreeBSD 12.0-CURRENT (GENERIC-NODEBUG) #24 r312210: Sun Jan 15 15:03:40 CST
2017

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:  https://www.FreeBSD.org/handbook/
FreeBSD FAQ:   https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-
questions/
FreeBSD Forums:https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:  man hier

Edit /etc/motd to change this login announcement.
jsli@jsli-bsd:~ % sysctl kern.eventtimer kern.timecounter
kern.eventtimer.periodic: 0
kern.eventtimer.timer: HPET
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: HPET(450) HPET1(440) HPET2(440) HPET3(440)
LAPIC(100) i8254(100) RTC(0)
kern.eventtimer.et.HPET3.quality: 440
kern.eventtimer.et.HPET3.frequency: 14318180
kern.eventtimer.et.HPET3.flags: 3
kern.eventtimer.et.HPET2.quality: 440
kern.eventtimer.et.HPET2.frequency: 14318180
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 440
kern.eventtimer.et.HPET1.frequency: 14318180
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 450
kern.eventtimer.et.HPET.frequency: 14318180
kern.eventtimer.et.HPET.flags: 3
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.LAPIC.quality: 100
kern.eventtimer.et.LAPIC.frequency: 0
kern.eventtimer.et.LAPIC.flags: 15
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(1000)
dummy(-100)
kern.timecounter.hardware: TSC-low
kern.timecounter.alloweddeviation: 5
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 5046106
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 14318180
kern.timecounter.tc.HPET.counter: 1449012340
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 10698
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.TSC-low.quality: 1000
kern.timecounter.tc.TSC-low.frequency: 1400076588
kern.timecounter.tc.TSC-low.counter: 2260820915
kern.timecounter.tc.TSC-low.mask: 4294967295
jsli@jsli-bsd:~ % su
Password:
jsli@jsli-bsd:/home/jsli # sysctl kern.timecounter.hardware=HPET
kern.timecounter.hardware: TSC-low -> HPET
jsli@jsli-bsd:/home/jsli # sysctl kern.eventtimer kern.timecounter
kern.eventtimer.periodic: 0
kern.eventtimer.timer: HPET
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: HPET(450) HPET1(440) HPET2(440) HPET3(440)
LAPIC(100) i8254(100) RTC(0)
kern.eventtimer.et.HPET3.quality: 440
kern.eventtimer.et.HPET3.frequency: 14318180
kern.eventtimer.et.HPET3.flags: 3
kern.eventtimer.et.HPET2.quality: 440
kern.eventtimer.et.HPET2.frequency: 14318180
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 440
kern.eventtimer.et.HPET1.frequency: 14318180
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 450
kern.eventtimer.et.HPET.frequency: 14318180
kern.eventtimer.et.HPET.flags: 3
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.LAPIC.quality: 100
kern.eventtimer.et.LAPIC.frequency: 0
kern.eventtimer.et.LAPIC.flags: 15
kern.timecounter.tsc_shift: 1
k

Re: TSC as timecounter makes system lag

[Konstantin Belousov]

On Sun, Jan 15, 2017 at 10:35:26PM +0800, Jia-Shiun Li wrote:
> Sorry just saw this. Bad Gmail.
> 
> 
> On Fri, Jan 13, 2017 at 8:05 PM, Konstantin Belousov 
> wrote:
> 
> > On Fri, Jan 13, 2017 at 08:26:04AM +0800, Jia-Shiun Li wrote:
> > > Hi all,
> > >
> > > since 2 or 3 weeks ago, I noticed that my old Penryn-based Intel Pentium
> > > T4200 notebook lagged a lot. System time was running a lot slower,
> > > sometimes even looked like it freezed. Keystroke repeat rate was slow
> > too.
> > >
> > > Since system time is slow, I tried to change timecounter from default TSC
> > > to HPET. And it resumed normal immediately.
> > Please show the output of sysctl kern.timecounter and kern.eventtimer.
> > I suspect that you changed eventtimer and not timecounter.
> >
> 
> Files attached. I changed it by "sysctl kern.timecounter.hardware=HPET"
> 
> 
> > The same world binary works fine on other Ivybridge and Haswell desktops,
> > > so I assume this may be related to CPU or mainboard generations.
> > >
> > > version is
> > >
> > > FreeBSD jsli-nb 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311687: Mon Jan  9
> > > 04:07:27 CST 2017
> > > jsli@4cbsd:/personal/freebsd/obj/x64/personal/freebsd/
> > fbsdsrc/sys/MINIMAL-NODEBUG
> > > amd64
> > >
> > > and CPU is
> > >
> > > CPU: Pentium(R) Dual-Core CPU   T4200  @ 2.00GHz (1995.04-MHz
> > K8-class
> > > CPU)
> > >   Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10
> > >
> > > Features=0xbfebfbff > APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,
> > MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> > >
> > > Features2=0xc00e39d > SSSE3,CX16,xTPR,PDCM,XSAVE,OSXSAVE>
> > >   AMD Features=0x20100800
> > >   AMD Features2=0x1
> > >   TSC: P-state invariant, performance statistics
> > >
> > > Tested similar OS rev on another Intel Core 2 Duo E7400 Wolfdale (the
> > same
> > > generation as the Pentium T4200). The same lag also happens on it.
> > >
> > > BTW on both system, cpuX:timer interrupts do not fire at all and count
> > > remains 0.
> > It is known that LAPIC is shut down in C2 and deeper CPU sleep states on
> > Core2.  FreeBSD 11 (and HEAD) started using MWAIT and requesting deep
> > wait states from BIOS.  If the configuration uses LAPIC and deep sleeps
> > are enabled, eventtimers do not work reliably.
> >
> 
> > Default configuration should strongly prefer HPET eventtimer over LAPIC for
> > machines which do not have LAPIC armed in Cx states, see r309189.  If you
> > do not have any customizations of eventtimer selection, then please provide
> > verbose dmesg from your boot.
> >
> > If you prefer to not use deep Cx and MWAIT, set loader tunable
> > debug.acpi.disabled to include word "mwait", see acpi(4).  You can check
> > that this worked by looking at sysctl dev.cpu.N output.
> >
> 
> Thanks for the explanation. Looks eventtimer favored HPET over LAPIC
> like you described on this notebook.
I still do not understand.  Is the sysctl output below from the pristine
boot where no timecounter/eventtimer reconfiguration were done ?

Show me exact command which you used to revive the machine.  Do not describe
it by words, copy/paste from the console.

> 
> -Jia-Shiun.

> kern.eventtimer.periodic: 0
> kern.eventtimer.timer: HPET
> kern.eventtimer.idletick: 0
> kern.eventtimer.singlemul: 2
> kern.eventtimer.choice: HPET(450) HPET1(440) HPET2(440) HPET3(440) LAPIC(100) 
> i8254(100) RTC(0)
> kern.eventtimer.et.i8254.quality: 100
> kern.eventtimer.et.i8254.frequency: 1193182
> kern.eventtimer.et.i8254.flags: 1
> kern.eventtimer.et.HPET3.quality: 440
> kern.eventtimer.et.HPET3.frequency: 14318180
> kern.eventtimer.et.HPET3.flags: 3
> kern.eventtimer.et.HPET2.quality: 440
> kern.eventtimer.et.HPET2.frequency: 14318180
> kern.eventtimer.et.HPET2.flags: 3
> kern.eventtimer.et.HPET1.quality: 440
> kern.eventtimer.et.HPET1.frequency: 14318180
> kern.eventtimer.et.HPET1.flags: 3
> kern.eventtimer.et.HPET.quality: 450
> kern.eventtimer.et.HPET.frequency: 14318180
> kern.eventtimer.et.HPET.flags: 3
> kern.eventtimer.et.RTC.quality: 0
> kern.eventtimer.et.RTC.frequency: 32768
> kern.eventtimer.et.RTC.flags: 17
> kern.eventtimer.et.LAPIC.quality: 100
> kern.eventtimer.et.LAPIC.frequency: 0
> kern.eventtimer.et.LAPIC.flags: 15
> kern.timecounter.tsc_shift: 1
> kern.timecounter.smp_tsc_adjust: 0
> kern.timecounter.smp_tsc: 1
> kern.timecounter.invariant_tsc: 1
> kern.timecounter.fast_gettime: 1
> kern.timecounter.tick: 1
> kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC(1000) 
> dummy(-100)
> kern.timecounter.hardware: TSC
> kern.timecounter.alloweddeviation: 5
> kern.timecounter.stepwarnings: 0
> kern.timecounter.tc.ACPI-fast.quality: 900
> kern.timecounter.tc.ACPI-fast.frequency: 3579545
> kern.timecounter.tc.ACPI-fast.counter: 79078
> kern.timecounter.tc.ACPI-fast.mask: 16777215
> kern.timecounter.tc.i8254.quality: 0
> kern.timecounter.tc.i8254.frequency: 1193182
> kern.timecounter.tc.i8254.counter: 55592
> kern.timecounter.tc.i8254.mask: 65535
> kern.

Re: TSC as timecounter makes system lag [-> jhb]

[Larry Rosenman]

On 2017-01-14 23:03, Julian Elischer wrote:

On 15/01/2017 10:11 AM, Jia-Shiun Li wrote:
On Fri, Jan 13, 2017 at 8:26 AM, Jia-Shiun Li  
wrote:



Hi all,

since 2 or 3 weeks ago, I noticed that my old Penryn-based Intel 
Pentium

T4200 notebook lagged a lot. System time was running a lot slower,
sometimes even looked like it freezed. Keystroke repeat rate was slow 
too.


Since system time is slow, I tried to change timecounter from default 
TSC

to HPET. And it resumed normal immediately.



Did a binary search. Turns out it was caused by r310177 "Enable
EARLY_AP_STARTUP on amd64 and i386 kernels by default." r310175 does 
not
have this issue. Removing this option from kernel config also solves 
it.


making sure jhb notices this.
FWIW, I noted similar "slowness", and nooptions EARLY_AP_STARTUP makes 
it "normal"

again.

Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights 
reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #13 r311997: Sat Jan 14 22:35:29 CST 2017
r...@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64
FreeBSD clang version 3.9.1 (tags/RELEASE_391/final 289601) (based on 
LLVM 3.9.1)

MEMGUARD DEBUGGING ALLOCATOR INITIALIZED:
MEMGUARD map base: 0xfe40
MEMGUARD map size: 128600960 KBytes
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU   E5410  @ 2.33GHz (2327.55-MHz 
K8-class CPU)

  Origin="GenuineIntel"  Id=0x10676  Family=0x6  Model=0x17  Stepping=6
  
Features=0xbfebfbff
  
Features2=0xce3bd

  AMD Features=0x20100800
  AMD Features2=0x1
  VT-x: HLT,PAUSE
  TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 65353601024 (62326 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs


--
Larry Rosenman http://people.freebsd.org/~ler
Phone: +1 214-642-9640 E-Mail: l...@freebsd.org
US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

Sorry just saw this. Bad Gmail.


On Fri, Jan 13, 2017 at 8:05 PM, Konstantin Belousov 
wrote:

> On Fri, Jan 13, 2017 at 08:26:04AM +0800, Jia-Shiun Li wrote:
> > Hi all,
> >
> > since 2 or 3 weeks ago, I noticed that my old Penryn-based Intel Pentium
> > T4200 notebook lagged a lot. System time was running a lot slower,
> > sometimes even looked like it freezed. Keystroke repeat rate was slow
> too.
> >
> > Since system time is slow, I tried to change timecounter from default TSC
> > to HPET. And it resumed normal immediately.
> Please show the output of sysctl kern.timecounter and kern.eventtimer.
> I suspect that you changed eventtimer and not timecounter.
>

Files attached. I changed it by "sysctl kern.timecounter.hardware=HPET"


> The same world binary works fine on other Ivybridge and Haswell desktops,
> > so I assume this may be related to CPU or mainboard generations.
> >
> > version is
> >
> > FreeBSD jsli-nb 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311687: Mon Jan  9
> > 04:07:27 CST 2017
> > jsli@4cbsd:/personal/freebsd/obj/x64/personal/freebsd/
> fbsdsrc/sys/MINIMAL-NODEBUG
> > amd64
> >
> > and CPU is
> >
> > CPU: Pentium(R) Dual-Core CPU   T4200  @ 2.00GHz (1995.04-MHz
> K8-class
> > CPU)
> >   Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10
> >
> > Features=0xbfebfbff APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,
> MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> >
> > Features2=0xc00e39d SSSE3,CX16,xTPR,PDCM,XSAVE,OSXSAVE>
> >   AMD Features=0x20100800
> >   AMD Features2=0x1
> >   TSC: P-state invariant, performance statistics
> >
> > Tested similar OS rev on another Intel Core 2 Duo E7400 Wolfdale (the
> same
> > generation as the Pentium T4200). The same lag also happens on it.
> >
> > BTW on both system, cpuX:timer interrupts do not fire at all and count
> > remains 0.
> It is known that LAPIC is shut down in C2 and deeper CPU sleep states on
> Core2.  FreeBSD 11 (and HEAD) started using MWAIT and requesting deep
> wait states from BIOS.  If the configuration uses LAPIC and deep sleeps
> are enabled, eventtimers do not work reliably.
>

> Default configuration should strongly prefer HPET eventtimer over LAPIC for
> machines which do not have LAPIC armed in Cx states, see r309189.  If you
> do not have any customizations of eventtimer selection, then please provide
> verbose dmesg from your boot.
>
> If you prefer to not use deep Cx and MWAIT, set loader tunable
> debug.acpi.disabled to include word "mwait", see acpi(4).  You can check
> that this worked by looking at sysctl dev.cpu.N output.
>

Thanks for the explanation. Looks eventtimer favored HPET over LAPIC
like you described on this notebook.

-Jia-Shiun.
kern.eventtimer.periodic: 0
kern.eventtimer.timer: HPET
kern.eventtimer.idletick: 0
kern.eventtimer.singlemul: 2
kern.eventtimer.choice: HPET(450) HPET1(440) HPET2(440) HPET3(440) LAPIC(100) 
i8254(100) RTC(0)
kern.eventtimer.et.i8254.quality: 100
kern.eventtimer.et.i8254.frequency: 1193182
kern.eventtimer.et.i8254.flags: 1
kern.eventtimer.et.HPET3.quality: 440
kern.eventtimer.et.HPET3.frequency: 14318180
kern.eventtimer.et.HPET3.flags: 3
kern.eventtimer.et.HPET2.quality: 440
kern.eventtimer.et.HPET2.frequency: 14318180
kern.eventtimer.et.HPET2.flags: 3
kern.eventtimer.et.HPET1.quality: 440
kern.eventtimer.et.HPET1.frequency: 14318180
kern.eventtimer.et.HPET1.flags: 3
kern.eventtimer.et.HPET.quality: 450
kern.eventtimer.et.HPET.frequency: 14318180
kern.eventtimer.et.HPET.flags: 3
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.flags: 17
kern.eventtimer.et.LAPIC.quality: 100
kern.eventtimer.et.LAPIC.frequency: 0
kern.eventtimer.et.LAPIC.flags: 15
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 1
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: ACPI-fast(900) i8254(0) HPET(950) TSC(1000) 
dummy(-100)
kern.timecounter.hardware: TSC
kern.timecounter.alloweddeviation: 5
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 79078
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 55592
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 14318180
kern.timecounter.tc.HPET.counter: 1931486323
kern.timecounter.tc.HPET.mask: 4294967295
kern.timecounter.tc.TSC.quality: 1000
kern.timecounter.tc.TSC.frequency: 1995044550
kern.timecounter.tc.TSC.counter: 2329785074
kern.timecounter.tc.TSC.mask: 4294967295
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The Fre

Re: TSC as timecounter makes system lag [-> jhb]

[Julian Elischer]

On 15/01/2017 10:11 AM, Jia-Shiun Li wrote:

On Fri, Jan 13, 2017 at 8:26 AM, Jia-Shiun Li  wrote:


Hi all,

since 2 or 3 weeks ago, I noticed that my old Penryn-based Intel Pentium
T4200 notebook lagged a lot. System time was running a lot slower,
sometimes even looked like it freezed. Keystroke repeat rate was slow too.

Since system time is slow, I tried to change timecounter from default TSC
to HPET. And it resumed normal immediately.



Did a binary search. Turns out it was caused by r310177 "Enable
EARLY_AP_STARTUP on amd64 and i386 kernels by default." r310175 does not
have this issue. Removing this option from kernel config also solves it.


making sure jhb notices this.

-Jia-Shiun.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Jia-Shiun Li]

On Fri, Jan 13, 2017 at 8:26 AM, Jia-Shiun Li  wrote:

> Hi all,
>
> since 2 or 3 weeks ago, I noticed that my old Penryn-based Intel Pentium
> T4200 notebook lagged a lot. System time was running a lot slower,
> sometimes even looked like it freezed. Keystroke repeat rate was slow too.
>
> Since system time is slow, I tried to change timecounter from default TSC
> to HPET. And it resumed normal immediately.
>
>
Did a binary search. Turns out it was caused by r310177 "Enable
EARLY_AP_STARTUP on amd64 and i386 kernels by default." r310175 does not
have this issue. Removing this option from kernel config also solves it.

-Jia-Shiun.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC as timecounter makes system lag

[Konstantin Belousov]

On Fri, Jan 13, 2017 at 08:26:04AM +0800, Jia-Shiun Li wrote:
> Hi all,
> 
> since 2 or 3 weeks ago, I noticed that my old Penryn-based Intel Pentium
> T4200 notebook lagged a lot. System time was running a lot slower,
> sometimes even looked like it freezed. Keystroke repeat rate was slow too.
> 
> Since system time is slow, I tried to change timecounter from default TSC
> to HPET. And it resumed normal immediately.
Please show the output of sysctl kern.timecounter and kern.eventtimer.
I suspect that you changed eventtimer and not timecounter.

> 
> The same world binary works fine on other Ivybridge and Haswell desktops,
> so I assume this may be related to CPU or mainboard generations.
> 
> version is
> 
> FreeBSD jsli-nb 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311687: Mon Jan  9
> 04:07:27 CST 2017
> jsli@4cbsd:/personal/freebsd/obj/x64/personal/freebsd/fbsdsrc/sys/MINIMAL-NODEBUG
> amd64
> 
> and CPU is
> 
> CPU: Pentium(R) Dual-Core CPU   T4200  @ 2.00GHz (1995.04-MHz K8-class
> CPU)
>   Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10
> 
> Features=0xbfebfbff
> 
> Features2=0xc00e39d
>   AMD Features=0x20100800
>   AMD Features2=0x1
>   TSC: P-state invariant, performance statistics
> 
> Tested similar OS rev on another Intel Core 2 Duo E7400 Wolfdale (the same
> generation as the Pentium T4200). The same lag also happens on it.
> 
> BTW on both system, cpuX:timer interrupts do not fire at all and count
> remains 0.
It is known that LAPIC is shut down in C2 and deeper CPU sleep states on
Core2.  FreeBSD 11 (and HEAD) started using MWAIT and requesting deep
wait states from BIOS.  If the configuration uses LAPIC and deep sleeps
are enabled, eventtimers do not work reliably.

Default configuration should strongly prefer HPET eventtimer over LAPIC for 
machines which do not have LAPIC armed in Cx states, see r309189.  If you
do not have any customizations of eventtimer selection, then please provide
verbose dmesg from your boot.

If you prefer to not use deep Cx and MWAIT, set loader tunable
debug.acpi.disabled to include word "mwait", see acpi(4).  You can check
that this worked by looking at sysctl dev.cpu.N output.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

TSC as timecounter makes system lag

[Jia-Shiun Li]

Hi all,

since 2 or 3 weeks ago, I noticed that my old Penryn-based Intel Pentium
T4200 notebook lagged a lot. System time was running a lot slower,
sometimes even looked like it freezed. Keystroke repeat rate was slow too.

Since system time is slow, I tried to change timecounter from default TSC
to HPET. And it resumed normal immediately.

The same world binary works fine on other Ivybridge and Haswell desktops,
so I assume this may be related to CPU or mainboard generations.

version is

FreeBSD jsli-nb 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311687: Mon Jan  9
04:07:27 CST 2017
jsli@4cbsd:/personal/freebsd/obj/x64/personal/freebsd/fbsdsrc/sys/MINIMAL-NODEBUG
amd64

and CPU is

CPU: Pentium(R) Dual-Core CPU   T4200  @ 2.00GHz (1995.04-MHz K8-class
CPU)
  Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10

Features=0xbfebfbff

Features2=0xc00e39d
  AMD Features=0x20100800
  AMD Features2=0x1
  TSC: P-state invariant, performance statistics

Tested similar OS rev on another Intel Core 2 Duo E7400 Wolfdale (the same
generation as the Pentium T4200). The same lag also happens on it.

BTW on both system, cpuX:timer interrupts do not fire at all and count
remains 0.

-Jia-Shiun
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"