Re: High interrupt rate
>>> On 6/3/2009 at 8:18 AM, "David K. Kelly" >>> wrote: > > HI Marcy, > > When you say SuSE's web site did you mean suse.org. I tried Novell's > site but under the link below I couldn't find any of the kernel patches. I'm pretty sure she didn't mean suse.org. :) More likely nu.novell.com, which is where YOU/rug/zmd go for updates. > Do you have a better link? Also when I tried rug I received the following: > ERROR: 'kernel-default' is not available. Do you have an update source configured? "rug sl" If yes, then what level is your kernel currently? "uname -r". Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
HI Marcy,
When you say SuSE's web site did you mean suse.org. I tried Novell's
site but under the link below I couldn't find any of the kernel patches.
Do you have a better link? Also when I tried rug I received the following:
ERROR: 'kernel-default' is not available.
http://download.novell.com/index.jsp?product_id=&search=Search&families=2658&version=19925&date_range=&date_start=03+Jun+2009&date_end=03+Jun+2009&keywords=kernel&sort_by=&results_per_page=&x=36&y=7
(Embedded image moved to file: pic30932.jpg)
Thank you
David Kelly
The Supreme Court of Virginia
Marcy Cortes
To
Sent by: Linux on LINUX-390@VM.MARIST.EDU
390 Port cc
Subject
Re: High interrupt rate
05/29/2009 03:21
PM
Please respond to
Linux on 390 Port
You can either grab the rpm off of SuSE's website or run "rug install
kernel-default" .
Marcy
"This message may contain confidential and/or privileged information. If
you are not the addressee or authorized to receive this for the addressee,
you must not use, copy, disclose, or take any action based on this message
or any information herein. If you have received this message in error,
please advise the sender immediately by reply e-mail and delete this
message. Thank you for your cooperation."
-Original Message-
From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of David
K. Kelly
Sent: Friday, May 29, 2009 12:16 PM
To: LINUX-390@VM.MARIST.EDU
Subject: Re: [LINUX-390] High interrupt rate
Hi Rob
Is it possible to just down load the kernel patch for this and not all the
extra packages
that you get when you request an update?
Thank you
David Kelly
The Supreme Court of Virginia
Rob van der Heij
To
Sent by: Linux on LINUX-390@VM.MARIST.EDU
390 Port cc
Subject
Re: High interrupt rate
05/28/2009 01:48
PM
Please respond to
Linux on 390 Port
On Thu, May 28, 2009 at 5:12 PM, Deric Abel wrote:
> We are running sles10 sp2 not redhat, so I am not sure if this bug was in
any of the red hat releases. The kernel that comes with sp2 was
2.6.16.60-0.21, and the most recent kernel for sp2 is
2.6.16.60-0.37 (remember that the number after the "-" is the rpm
build/release version, so that will be different between distros). I am
not sure want the underlying problem was, but from this mailing list it
seems there was a bug in the sles10sp2 kernel and it's fixed with the
newest. So unless someone else would like to explain the bug, that's all
the information I have.
The cause is that the application asks for a wakeup in 1 ms or so
(this is silly design since it is polling). The kernel only provides
10 ms granularity and rounded incorrectly to 0 ms, thus making the
silly thing pretty bad. It also shows very high CP overhead when this
happens.
The fix is that the kernel rounds it to 10 ms to limit the impact.
Martin just told me the fix is in: 2.6.18-124.el5 für RedHat RHEL5 und
2.6.16.60-0.34 für SuSE SLES10
--
Rob van der Heij
Velocity Software
http://www.velocitysoftware.com/
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-39
Re: High interrupt rate
David, I just grabbed the kernel rpm from novell's site (since my guests don't have outside access) and applied the RPMs. That has solved my problem across the board, where an interrupt rate of 5000/sec was normal they are all now at 105 or less. Deric -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
On Fri, May 29, 2009 at 9:15 PM, David K. Kelly wrote: > Hi Rob > > Is it possible to just down load the kernel patch for this and not all the > extra packages > that you get when you request an update? An easy way to see if this is indeed your problem: set the hz_timer to 1 and see whether it goes away. If you have a buggy application that is polling, you will not drop from queue anyway. So if it fixes the issue you could leave it until the new kernel is there. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
You can either grab the rpm off of SuSE's website or run "rug install
kernel-default" .
Marcy
"This message may contain confidential and/or privileged information. If you
are not the addressee or authorized to receive this for the addressee, you must
not use, copy, disclose, or take any action based on this message or any
information herein. If you have received this message in error, please advise
the sender immediately by reply e-mail and delete this message. Thank you for
your cooperation."
-Original Message-
From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of David K.
Kelly
Sent: Friday, May 29, 2009 12:16 PM
To: LINUX-390@VM.MARIST.EDU
Subject: Re: [LINUX-390] High interrupt rate
Hi Rob
Is it possible to just down load the kernel patch for this and not all the
extra packages
that you get when you request an update?
Thank you
David Kelly
The Supreme Court of Virginia
Rob van der Heij
To
Sent by: Linux on LINUX-390@VM.MARIST.EDU
390 Port cc
Subject
Re: High interrupt rate
05/28/2009 01:48
PM
Please respond to
Linux on 390 Port
On Thu, May 28, 2009 at 5:12 PM, Deric Abel wrote:
> We are running sles10 sp2 not redhat, so I am not sure if this bug was in
any of the red hat releases. The kernel that comes with sp2 was
2.6.16.60-0.21, and the most recent kernel for sp2 is
2.6.16.60-0.37 (remember that the number after the "-" is the rpm
build/release version, so that will be different between distros). I am
not sure want the underlying problem was, but from this mailing list it
seems there was a bug in the sles10sp2 kernel and it's fixed with the
newest. So unless someone else would like to explain the bug, that's all
the information I have.
The cause is that the application asks for a wakeup in 1 ms or so
(this is silly design since it is polling). The kernel only provides
10 ms granularity and rounded incorrectly to 0 ms, thus making the
silly thing pretty bad. It also shows very high CP overhead when this
happens.
The fix is that the kernel rounds it to 10 ms to limit the impact.
Martin just told me the fix is in: 2.6.18-124.el5 für RedHat RHEL5 und
2.6.16.60-0.34 für SuSE SLES10
--
Rob van der Heij
Velocity Software
http://www.velocitysoftware.com/
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
Hi Rob Is it possible to just down load the kernel patch for this and not all the extra packages that you get when you request an update? Thank you David Kelly The Supreme Court of Virginia Rob van der Heij To Sent by: Linux on LINUX-390@VM.MARIST.EDU 390 Port cc Subject Re: High interrupt rate 05/28/2009 01:48 PM Please respond to Linux on 390 Port On Thu, May 28, 2009 at 5:12 PM, Deric Abel wrote: > We are running sles10 sp2 not redhat, so I am not sure if this bug was in any of the red hat releases. The kernel that comes with sp2 was 2.6.16.60-0.21, and the most recent kernel for sp2 is 2.6.16.60-0.37 (remember that the number after the "-" is the rpm build/release version, so that will be different between distros). I am not sure want the underlying problem was, but from this mailing list it seems there was a bug in the sles10sp2 kernel and it's fixed with the newest. So unless someone else would like to explain the bug, that's all the information I have. The cause is that the application asks for a wakeup in 1 ms or so (this is silly design since it is polling). The kernel only provides 10 ms granularity and rounded incorrectly to 0 ms, thus making the silly thing pretty bad. It also shows very high CP overhead when this happens. The fix is that the kernel rounds it to 10 ms to limit the impact. Martin just told me the fix is in: 2.6.18-124.el5 für RedHat RHEL5 und 2.6.16.60-0.34 für SuSE SLES10 -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
On Thu, May 28, 2009 at 5:12 PM, Deric Abel wrote: > We are running sles10 sp2 not redhat, so I am not sure if this bug was in any > of the red hat releases. The kernel that comes with sp2 was 2.6.16.60-0.21, > and the most recent kernel for sp2 is 2.6.16.60-0.37 (remember that the > number after the "-" is the rpm build/release version, so that will be > different between distros). I am not sure want the underlying problem was, > but from this mailing list it seems there was a bug in the sles10sp2 kernel > and it's fixed with the newest. So unless someone else would like to explain > the bug, that's all the information I have. The cause is that the application asks for a wakeup in 1 ms or so (this is silly design since it is polling). The kernel only provides 10 ms granularity and rounded incorrectly to 0 ms, thus making the silly thing pretty bad. It also shows very high CP overhead when this happens. The fix is that the kernel rounds it to 10 ms to limit the impact. Martin just told me the fix is in: 2.6.18-124.el5 für RedHat RHEL5 und 2.6.16.60-0.34 für SuSE SLES10 -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
We are running sles10 sp2 not redhat, so I am not sure if this bug was in any of the red hat releases. The kernel that comes with sp2 was 2.6.16.60-0.21, and the most recent kernel for sp2 is 2.6.16.60-0.37 (remember that the number after the "-" is the rpm build/release version, so that will be different between distros). I am not sure want the underlying problem was, but from this mailing list it seems there was a bug in the sles10sp2 kernel and it's fixed with the newest. So unless someone else would like to explain the bug, that's all the information I have. >>> Alan Ackerman 5/27/2009 2:30 PM >>> I read through this thread. I see information about db2fmcd, which you say did not cause your problem. I don't see anything in the thread that tells me about the problem or fix to the kernel. Can you point me to information about the kernel problem? Just to confuse things, we have Red Hat, not SuSE. >Thanks everyone for your help! Based on everything I've read, I concluded >to upgrade the kernel to 2.6.16.60-0.37. That has solve our issue. As >for the db2fmcd process, I did not find that running nor in the /etc/inittab, >so no changes were made there. Alan Ackerman Alan (dot) Ackerman (at) Bank of America (dot) com -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
I read through this thread. I see information about db2fmcd, which you say did not cause your problem. I don't see anything in the thread that tells me about the problem or fix to the kernel. Can you point me to information about the kernel problem? Just to confuse things, we have Red Hat, not SuSE. >Thanks everyone for your help! Based on everything I've read, I concluded >to upgrade the kernel to 2.6.16.60-0.37. That has solve our issue. As >for the db2fmcd process, I did not find that running nor in the /etc/inittab, >so no changes were made there. Alan Ackerman Alan (dot) Ackerman (at) Bank of America (dot) com -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
On Tue, May 26, 2009 at 7:34 PM, Deric Abel wrote: > Thanks everyone for your help! Based on everything I've read, I concluded to > upgrade the kernel to 2.6.16.60-0.37. That has solve our issue. As for the > db2fmcd process, I did not find that running nor in the /etc/inittab, so no > changes were made there. Thanks again, I've been unable to keep track of which Novel kernel has which fixes from IBM. But if the new kernel fixes the problem, then it sure looks like another db2 component has the same issue. So it's probably a design fault rather than a bug ;-) The kernel issue only shows the problem by actually doing the thing that the application was asking for. Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
Thanks everyone for your help! Based on everything I've read, I concluded to upgrade the kernel to 2.6.16.60-0.37. That has solve our issue. As for the db2fmcd process, I did not find that running nor in the /etc/inittab, so no changes were made there. Thanks again, Deric -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
You can also issue the db2fmcu -d command. This erases the fmc entries in inittab. Also ./sqllib/fm.whatever.reg check the instances and make sure everyone FM_ON and FM_ACTIVE are set to NO. 'Where ever you go - There you are!! ' Richard (Gaz) Gasiorowski System z - Linux Product Manager Portfolio Platform Services CSC 3170 Fairview Park Dr., Falls Church, VA 22042 845-889-8533|Work|845-392-7889 Cell|rgasi...@csc.com|www.csc.com This is a PRIVATE message. If you are not the intended recipient, please delete without copying and kindly advise us by e-mail of the mistake in delivery. NOTE: Regardless of content, this e-mail shall not operate to bind CSC to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose. Rob van der Heij Sent by: Linux on 390 Port 05/26/2009 11:22 AM Please respond to Linux on 390 Port To LINUX-390@VM.MARIST.EDU cc Subject Re: High interrupt rate On Tue, May 26, 2009 at 5:10 PM, Deric Abel wrote: > Hi, I have an issue with one of my linux guests (sles10) where when we enable HADR with db2 the HADR db guest interrupt rate jumps from a few thousand to 300,000 per second. This causes the guest to be using 20-30% of one IFL, despite doing very little. Has anyone seen this issue before, or know of a good way to debug it? Without performance data but you probably have db2fmcd running. This has nothing to do with HADR but frequently interferes with it. We're not sure whether it is a bug or design problem with db2fmcd that it sometimes starts to poll. Combined with a kernel bug in SLES10 SP2, it typically causes a lot of overhead. Since db2fmcd has no function in this environment, the easy way out is to stop it (and remove the entry in /etc/inittab ) Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
> Hi, I have an issue with one of my linux guests (sles10) where when we > enable HADR with db2 the HADR db guest interrupt rate jumps from a few HADR? > (BTW, I am brand new to this mailing list as well as new to the Z > system, but not to linux) Cool. Welcome to the party. -- db -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
> it typically causes a lot of overhead. Since db2fmcd has no function > in this environment, the easy way out is to stop it (and remove the > entry in /etc/inittab ) It would also be really useful if someone sat down with the people in DB/2 development and explained that using /etc/inittab to manage services on Linux is Generally Considered Rude. Grr. What a PITA. --d b -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
On Tue, May 26, 2009 at 5:10 PM, Deric Abel wrote: > Hi, I have an issue with one of my linux guests (sles10) where when we > enable HADR with db2 the HADR db guest interrupt rate jumps from a few > thousand to 300,000 per second. This causes the guest to be using 20-30% of > one IFL, despite doing very little. Has anyone seen this issue before, or > know of a good way to debug it? Without performance data but you probably have db2fmcd running. This has nothing to do with HADR but frequently interferes with it. We're not sure whether it is a bug or design problem with db2fmcd that it sometimes starts to poll. Combined with a kernel bug in SLES10 SP2, it typically causes a lot of overhead. Since db2fmcd has no function in this environment, the easy way out is to stop it (and remove the entry in /etc/inittab ) Rob -- Rob van der Heij Velocity Software http://www.velocitysoftware.com/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: High interrupt rate
>>> David Boyes 5/26/2009 9:14 AM >>> > Hi, I have an issue with one of my linux guests (sles10) where when we > enable HADR with db2 the HADR db guest interrupt rate jumps from a few HADR? High Availability Disaster Recovery It's a DB2 process that replicates db's between servers (guests in our case) Deric > (BTW, I am brand new to this mailing list as well as new to the Z > system, but not to linux) Cool. Welcome to the party. Thanks, very happy to be here! -- db -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
