Bug#670398: linux-image-2.6.32-5-amd64: SSH logins hang while hpet interrupts multiply on Intel Nehalem CPUs

2012-04-25 Thread John Stultz 

On 04/25/2012 08:49 PM, Ben Hutchings wrote:

On Wed, 2012-04-25 at 10:36 +0200, Sven Hoexter wrote:

Package: linux-image-2.6.32-5-amd64
Version: 2.6.32-41squeeze2
Severity: important

Hi,

since about December 2011 we've seen systems were SSH sessions suddenly hang and
further logins on the physical TTY or via SSH are no longer possible. In some 
cases
ssh logins still work and you see motd and mayeb can even issue one or two 
commands.
(I've brought this issue up on debian-user in march with a private reply from a
fellow DD yesterday http://lists.debian.org/debian-user/2012/03/msg01204.html)


Over time we observed that ssh logins without PTS (ssh -T) still work. Looking 
at
other sessions sshd was in state and D entries in /dev/pts/ were created 
correctly.
Searching through munin graphs we could narrow down the starting point of this 
issue
to the point when the hpet interrupts for one CPU core multiplied. Sometimes 
they
multiplied by six. Looking further we've found the Kernel [events/$x] in state D
where $x is the number of the CPU core which has the high number of hpet 
interrupts.

When we started strace -f on the sshd master process everything works until you 
logout.
Then you'll again see the forked sshd process hanging in state D.

This is strange, because D state means uninterruptible sleep (not
handling signals).  But perhaps the sshd process was repeatedly changing
between uninterruptible and interruptible state.


[snip]

Based on those summary lines, I think the upstream changes are:

commit 08ec0c58fb8a05d3191d5cb6f5d6f81adb419798 (v2.6.38-rc1~480^2~1)
Author: John Stultzjohns...@us.ibm.com
Date:   Tue Jul 27 17:00:00 2010 -0700

 x86: Improve TSC calibration using a delayed workqueue

commit f12a15be63d1de9a35971f35f06b73088fa25c3a (v2.6.36-rc1~514^2~4)
Author: John Stultzjohns...@us.ibm.com
Date:   Tue Jul 13 17:56:27 2010 -0700

 x86: Convert common clocksources to use clocksource_register_hz/khz

commit 7d2f944a2b836c69a9d260a0a5f0d1720d57fdff (v2.6.33-rc1~363^2~12)
Author: Thomas Gleixnert...@linutronix.de
Date:   Wed Nov 11 14:05:29 2009 +

 clocksource: Provide a generic mult/shift factor calculation

The latter two are dependencies for the first, which is presumably the
really important change.  At a guess, better TSC calibration helps us to
avoid switching to the HPET.  John, do you know whether your changes
have that effect?
No, the TSC calibration just makes sure we get the same fine-grained 
cpukhz value every bootup, which avoids time skew. It should not affect 
if we switch to the HPET, which would be a sign of a unsynced or halting 
TSC.


When you can connect to the system that is having problems, do you see 
any problems with the time? ie: does date show the correct time, and 
does it increment normally?


It sounds like if there is some HPET irq issue, it would likely be due 
to some sort of global wakeup to handle local apics that halt in deep 
sleep modes.  Its likely that getting /proc/timer_list output would help 
(both before and after the problem).


Also I'd try to bring in tglx for his thoughts.

thanks
-john




--
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4f98d028.2010...@us.ibm.com

Bug#647095: CPU hyperthreading turned on after soft power-cycle

2011-11-29 Thread John Stultz 
On Tue, 2011-11-29 at 13:26 +0100, Clarinet wrote:
  Using an older known-good kernel, could you build and run the test
  case at the end of Documentation/rtc.txt a few times and see if it
  triggers the same problem?
 
  I'm suspicious that the setting the alarm is whats tripping the BIOS
  into enabling the HT bit. Because with older kernels, we used PIE mode
  irqs which hwclock usually uses at boot, but with newer kernels, we
  emulate PIE via AIE alarm mode. So if the BIOS was broken before, you
  wouldn't have noticed unless you tried to use AIE irqs.
 
  If this doesn't work, I'll get some patches to both 2.6.27 and 2.6.28
  kernels to debug the exact flow of how we're touching the hardware and
  then we can further narrow it down.
 
 I ran the tests the following way:
 
 - boot 2.6.37.6 - check /proc/cpuinfo - 12 processors
 - halt
 - boot 2.6.37.6 - check /proc/cpuinfo - 12 processors
 - run rtctest
 - reboot
 - boot 2.6.37.6 - check /proc/cpuinfo - 12 processors
 - halt
 - boot 2.6.37.6 - check /proc/cpuinfo - 12 processors
 - run rtctest
 - halt
 - boot 2.6.37.6 - check /proc/cpuinfo - 24 processors
 
 So the conclusion is that only if rtctest is run and the machine is 
 halted, it triggers the HT problem. Reboot seems to neutralize 
 whatever rtctest did.

Ok, this also confirms that the board had issues *before* any changes
were made to the RTC core. I'd push the board vendor to update the BIOS
to avoid this issue.

Even so, I'm curious as to what exactly trips it up. Maybe we can
provide a module option for the rtc-cmos driver to disable the alarm
functionality, so you can at least avoid the issue until the board
vendor fixes the problem (if ever).

Assuming its the alarm being set, could you try the following on a
current kernel and let me know if it still shows the problem? hwclock
might throw some odd messages with this test patch, but those can be
ignored.

thanks
-john

diff --git a/drivers/rtc/rtc-cmos.c b/drivers/rtc/rtc-cmos.c
index 05beb6c..d9814aa 100644
--- a/drivers/rtc/rtc-cmos.c
+++ b/drivers/rtc/rtc-cmos.c
@@ -305,8 +305,8 @@ static void cmos_irq_enable(struct cmos_rtc *cmos, unsigned 
char mask)
cmos_checkintr(cmos, rtc_control);
 
rtc_control |= mask;
-   CMOS_WRITE(rtc_control, RTC_CONTROL);
-   hpet_set_rtc_irq_bit(mask);
+// CMOS_WRITE(rtc_control, RTC_CONTROL);
+// hpet_set_rtc_irq_bit(mask);
 
cmos_checkintr(cmos, rtc_control);
 }







-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1322609653.21423.46.camel@work-vm

Bug#647095: CPU hyperthreading turned on after soft power-cycle

2011-11-28 Thread John Stultz 
On Mon, 2011-11-21 at 22:31 +0100, Jiri Polach wrote:
  diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
  index cb9a104..77b5273 100644
  --- a/arch/x86/Kconfig
  +++ b/arch/x86/Kconfig
  @@ -640,7 +640,7 @@ config HPET_TIMER
Choose N to continue using the legacy 8254 timer.
 
config HPET_EMULATE_RTC
  -   def_bool y
  +   def_bool n
  depends on HPET_TIMER  (RTC=y || RTC=m || RTC_DRV_CMOS=m || 
  RTC_DRV_CMOS=y)
 
config APB_TIMER
 
 Applying this patch does not change anything, this kernel is bad.

Using an older known-good kernel, could you build and run the test
case at the end of Documentation/rtc.txt a few times and see if it
triggers the same problem?

I'm suspicious that the setting the alarm is whats tripping the BIOS
into enabling the HT bit. Because with older kernels, we used PIE mode
irqs which hwclock usually uses at boot, but with newer kernels, we
emulate PIE via AIE alarm mode. So if the BIOS was broken before, you
wouldn't have noticed unless you tried to use AIE irqs.

If this doesn't work, I'll get some patches to both 2.6.27 and 2.6.28
kernels to debug the exact flow of how we're touching the hardware and
then we can further narrow it down.

thanks
-john






-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1322533915.24090.69.camel@work-vm

Bug#647095: CPU hyperthreading turned on after soft power-cycle

2011-11-21 Thread John Stultz 
On Mon, 2011-11-21 at 14:27 +0100, Jiri Polach wrote:
  Finally! After another 50+ compilations a have it! It took some time as
  first I had to find a reason why some revisions did not boot (almost 2/3
  were unbootable and the first bad commit was among them). Having this
  solved I have been able to bisect without skipping. The result is
  surprising (at least for me) - believe it or not, the first bad commit
  is 6610e089 RTC: Rework RTC code to use timerqueue for events from
  John Stultz (I am sending him a copy of this message).
 
  I would never expect this would be a problem, but my understanding of
  this commit is very limited, so I am certainly missing the point.
  However, I have tried to compile 2.6.38 (which was bad) with Real
  Time Clock configuration option turned off and it behaves normally
  then (= is good).
 
  Huh. That's *very* odd.  Is your system doing anything in-particular
  with the RTC?  I don't have a clue right off, so probably the next step
 
  Yes, it is very odd. The system does not do anything special with RTC.
  It is a diskless computational workstation.
 
  is doing a bit of instrumentation to try to figure out where exactly we
  trigger the behavior. Could you checkout commit 6610e089 and apply the
  patch below to see if we can't narrow it down?
 
  With the patch applied the system does not show the strange behavior (=
  is good).
 
  Could you also send your .config to me?
 
  Sure. It is attached. I have found that if I turn CONFIG_RTC_DRV_CMOS
  off, the system behaves normally (= is good) too.
 
  Yea. My rough guess is that the BIOS is somehow sensitive to how the
  CMOS RTC is touched.
 
  Does disabling CONFIG_HPET_EMULATE_RTC change the behavior?
 
 But how do I do it? :-)
 
 I have not found a way to disable it in menuconfig. If I comment it 
 out manually in .config, it is automatically set back to y as soon as 
 compilation starts ...

Good point. I forgot on x86_64 you can't disable HPET_TIMER.

Could you then use the following patch (and run make oldconfig before
building).

thanks
-john


diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index cb9a104..77b5273 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -640,7 +640,7 @@ config HPET_TIMER
  Choose N to continue using the legacy 8254 timer.
 
 config HPET_EMULATE_RTC
-   def_bool y
+   def_bool n
depends on HPET_TIMER  (RTC=y || RTC=m || RTC_DRV_CMOS=m || 
RTC_DRV_CMOS=y)
 
 config APB_TIMER






-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1321905728.6445.2.camel@work-vm

Bug#647095: CPU hyperthreading turned on after soft power-cycle

2011-11-17 Thread John Stultz 
On Wed, 2011-11-16 at 23:49 +0100, Clarinet wrote:
 Hi all,
 
  Result of bisecting: v2.6.38-rc1 exhibits the problem. v2.6.37 and
  many of the topic branches merged in the 2.6.38 merge window work ok.
  Some other topic branches do not boot at all.
 
  Jiri: if you have gitk installed, then git bisect visualize can help
  get a sense of what's in the middle of the regression range.
  gitk --bisect --first-parent v2.6.37..v2.6.38-rc1 might be a good way
  to find mainline commits to test before finding a topic branch to delve
  into.
 
  I have been able to narrow the interval manually a little bit from the
  top (the bad side) and I will go on from the bottom now. However,
  there seems to be a large area where kernels are unbootable for me -
  they mostly stop when init is called and I do not know why.
 
 Finally! After another 50+ compilations a have it! It took some time as 
 first I had to find a reason why some revisions did not boot (almost 2/3 
 were unbootable and the first bad commit was among them). Having this 
 solved I have been able to bisect without skipping. The result is 
 surprising (at least for me) - believe it or not, the first bad commit 
 is 6610e089 RTC: Rework RTC code to use timerqueue for events from 
 John Stultz (I am sending him a copy of this message).
 
 I would never expect this would be a problem, but my understanding of 
 this commit is very limited, so I am certainly missing the point. 
 However, I have tried to compile 2.6.38 (which was bad) with Real 
 Time Clock configuration option turned off and it behaves normally 
 then (= is good).

Huh. That's *very* odd.  Is your system doing anything in-particular
with the RTC?  I don't have a clue right off, so probably the next step
is doing a bit of instrumentation to try to figure out where exactly we
trigger the behavior. Could you checkout commit 6610e089 and apply the
patch below to see if we can't narrow it down?

Could you also send your .config to me?

thanks
-john

diff --git a/drivers/rtc/rtc-cmos.c b/drivers/rtc/rtc-cmos.c
index 5856167..d049344 100644
--- a/drivers/rtc/rtc-cmos.c
+++ b/drivers/rtc/rtc-cmos.c
@@ -497,13 +497,13 @@ static int cmos_procfs(struct device *dev, struct 
seq_file *seq)
 static const struct rtc_class_ops cmos_rtc_ops = {
.read_time  = cmos_read_time,
.set_time   = cmos_set_time,
-   .read_alarm = cmos_read_alarm,
-   .set_alarm  = cmos_set_alarm,
-   .proc   = cmos_procfs,
-   .irq_set_freq   = cmos_irq_set_freq,
-   .irq_set_state  = cmos_irq_set_state,
-   .alarm_irq_enable   = cmos_alarm_irq_enable,
-   .update_irq_enable  = cmos_update_irq_enable,
+// .read_alarm = cmos_read_alarm,
+// .set_alarm  = cmos_set_alarm,
+// .proc   = cmos_procfs,
+// .irq_set_freq   = cmos_irq_set_freq,
+// .irq_set_state  = cmos_irq_set_state,
+// .alarm_irq_enable   = cmos_alarm_irq_enable,
+// .update_irq_enable  = cmos_update_irq_enable,
 };
 
 /**/






-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1321561946.25715.16.camel@work-vm

Bug#647095: CPU hyperthreading turned on after soft power-cycle

2011-11-17 Thread John Stultz 
On Fri, 2011-11-18 at 00:42 +0100, Jiri Polach wrote:
 On 11/17/2011 9:32 PM, John Stultz wrote:
  On Wed, 2011-11-16 at 23:49 +0100, Clarinet wrote:
  Hi all,
 
  Result of bisecting: v2.6.38-rc1 exhibits the problem. v2.6.37 and
  many of the topic branches merged in the 2.6.38 merge window work ok.
  Some other topic branches do not boot at all.
 
  Jiri: if you have gitk installed, then git bisect visualize can help
  get a sense of what's in the middle of the regression range.
  gitk --bisect --first-parent v2.6.37..v2.6.38-rc1 might be a good way
  to find mainline commits to test before finding a topic branch to delve
  into.
 
  I have been able to narrow the interval manually a little bit from the
  top (the bad side) and I will go on from the bottom now. However,
  there seems to be a large area where kernels are unbootable for me -
  they mostly stop when init is called and I do not know why.
 
  Finally! After another 50+ compilations a have it! It took some time as
  first I had to find a reason why some revisions did not boot (almost 2/3
  were unbootable and the first bad commit was among them). Having this
  solved I have been able to bisect without skipping. The result is
  surprising (at least for me) - believe it or not, the first bad commit
  is 6610e089 RTC: Rework RTC code to use timerqueue for events from
  John Stultz (I am sending him a copy of this message).
 
  I would never expect this would be a problem, but my understanding of
  this commit is very limited, so I am certainly missing the point.
  However, I have tried to compile 2.6.38 (which was bad) with Real
  Time Clock configuration option turned off and it behaves normally
  then (= is good).
 
  Huh. That's *very* odd.  Is your system doing anything in-particular
  with the RTC?  I don't have a clue right off, so probably the next step
 
 Yes, it is very odd. The system does not do anything special with RTC. 
 It is a diskless computational workstation.
 
  is doing a bit of instrumentation to try to figure out where exactly we
  trigger the behavior. Could you checkout commit 6610e089 and apply the
  patch below to see if we can't narrow it down?
 
 With the patch applied the system does not show the strange behavior (= 
 is good).
 
  Could you also send your .config to me?
 
 Sure. It is attached. I have found that if I turn CONFIG_RTC_DRV_CMOS 
 off, the system behaves normally (= is good) too.

Yea. My rough guess is that the BIOS is somehow sensitive to how the
CMOS RTC is touched.

Does disabling CONFIG_HPET_EMULATE_RTC change the behavior?

thanks
-john







-- 
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1321574019.25715.52.camel@work-vm