User proc uses all RAM+swap => kernel panic - shouldn't OS not allow?
Greetings, Several times over past few years I've seen user processes "go mad" (programming error) & use all RAM, then all swap (as ganglia so vividly shows), then the box ends up at a kernel panic. (Server OS is SL5.x 64-bit BTW) What's puzzling is, shouldn't the OS by default not allow users to do "something bad enough" to cause grief to the OS? Possibly some sort of tuning can fix this, but one expects that, out of the box, this should not be needed, users just can't bring OS down. In the past on SL4 I did see oom come into play when box too loaded (killing the mysqld process for instance) & am wondering why this isn't happening on SL5 with badly behaved user processes. Grateful for advice!
Re: User proc uses all RAM+swap => kernel panic - shouldn't OS not allow?
Hello Winnie, On Sep 13, 2012, at 16:01 , Winnie Lacesso wrote: > Several times over past few years I've seen user processes "go mad" > (programming error) & use all RAM, then all swap (as ganglia so vividly > shows), then the box ends up at a kernel panic. > (Server OS is SL5.x 64-bit BTW) we rarely see panics in these cases. The box just becomes unusable. Which effectively makes no difference though. > What's puzzling is, shouldn't the OS by default not allow users to do > "something bad enough" to cause grief to the OS? > > Possibly some sort of tuning can fix this, but one expects that, out of > the box, this should not be needed, users just can't bring OS down. There are plenty of ways to bring a *x system down, or make it deny service to users, by making mistakes in userland. Just think of the classic fork bomb. > In the past on SL4 I did see oom come into play when box too > loaded (killing the mysqld process for instance) & am wondering > why this isn't happening on SL5 with badly behaved user processes. By any chance, were your SL4 systems mostly 32-bit, and your SL5 systems are mostly 64-bit? As much as I do advocate using 64-bit, I have to admit that the x86-64 kernel seems to be handling OOM situations much worse than x86 used to. And I think it started with SL3 already. > Grateful for advice! The one way I know about to reliably prevent these problems is to use syctl to change the value of vm.overcommit_ratio, and possibly adapt vm.overcommit_memory. Both are documented in proc(5). The problem with this approach is that there's more and more software making very generous use of virtual address space without ever using what was allocated. The current Maple and Oracle's Java come to mind. Having sufficient swap space does help. We used to set aside only 2GB for swap even on systems with much more RAM, because they weren't supposed to swap/page much at all. But it turns out that having the recommended amount makes systems much more resilient to memory hogs. Hope this helps, Stephan -- Stephan Wiesand DESY -DV- Platanenenallee 6 15738 Zeuthen, Germany
Re: User proc uses all RAM+swap => kernel panic - shouldn't OS not allow?
Hello, Please take a look at /etc/security/limits.conf if you have not already. And it's manpage; http://linux.die.net/man/5/limits.conf There's always GRSec and the like if you would like to run a modified kernel.** Kind regards, Declan Williams On 13 September 2012 15:32, Stephan Wiesand wrote: > Hello Winnie, > > On Sep 13, 2012, at 16:01 , Winnie Lacesso wrote: > > > Several times over past few years I've seen user processes "go mad" > > (programming error) & use all RAM, then all swap (as ganglia so vividly > > shows), then the box ends up at a kernel panic. > > (Server OS is SL5.x 64-bit BTW) > > we rarely see panics in these cases. The box just becomes unusable. Which > effectively makes no difference though. > > > What's puzzling is, shouldn't the OS by default not allow users to do > > "something bad enough" to cause grief to the OS? > > > > Possibly some sort of tuning can fix this, but one expects that, out of > > the box, this should not be needed, users just can't bring OS down. > > There are plenty of ways to bring a *x system down, or make it deny > service to users, by making mistakes in userland. Just think of the classic > fork bomb. > > > In the past on SL4 I did see oom come into play when box too > > loaded (killing the mysqld process for instance) & am wondering > > why this isn't happening on SL5 with badly behaved user processes. > > By any chance, were your SL4 systems mostly 32-bit, and your SL5 systems > are mostly 64-bit? As much as I do advocate using 64-bit, I have to admit > that the x86-64 kernel seems to be handling OOM situations much worse than > x86 used to. And I think it started with SL3 already. > > > Grateful for advice! > > > The one way I know about to reliably prevent these problems is to use > syctl to change the value of vm.overcommit_ratio, and possibly adapt > vm.overcommit_memory. Both are documented in proc(5). > > The problem with this approach is that there's more and more software > making very generous use of virtual address space without ever using what > was allocated. The current Maple and Oracle's Java come to mind. > > Having sufficient swap space does help. We used to set aside only 2GB for > swap even on systems with much more RAM, because they weren't supposed to > swap/page much at all. But it turns out that having the recommended amount > makes systems much more resilient to memory hogs. > > Hope this helps, > Stephan > > -- > Stephan Wiesand > DESY -DV- > Platanenenallee 6 > 15738 Zeuthen, Germany > -- ID:2048R/76E8D09C pub: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x92D0A99B76E8D09C uid: Declan Williams declan_willi...@lycan-net.eu
ps and IPC
Hello, I am running Scientific Linux 6.2 and am getting a semget error when I run ps as any user, including root. The full message is: semget: No such file or directory semop: Invalid argument [DEVICE -1] The requested IPC message queue is locked. Error in message send = 22 itype, ichan, nwords,2,-1,2 Could someone explain what this error means, and show me how to troubleshoot it? Sorry if this is a basic question, but I am stumped. Thanks! Dave +++ David Fitzgerald Department of Earth Sciences Millersville University Millersville, PA 17551 Phone: 717-871-2394
Re: ps and IPC
On 13 September 2012 10:52, David Fitzgerald wrote: > Hello, > > > > I am running Scientific Linux 6.2 and am getting a semget error when I run > ps as any user, including root. > > The full message is: > > > > semget: No such file or directory > > semop: Invalid argument > > [DEVICE -1] The requested IPC message queue is locked. > > Error in message send = 22 > > itype, ichan, nwords,2,-1,2 > > > > Could someone explain what this error means, and show me how to troubleshoot > it? > > Well that error is not a basic one.. which ps are you running (you can find out by typing which ps). Something is locking your access to the kernels memory so something has changed on the system from a default install. I guess I need a lot more questions answered: Type of system (x86_64, i386) Type of hardware The output of the following commands df -a ls -l /proc/ > > Sorry if this is a basic question, but I am stumped. > > > > Thanks! > > > > Dave > > +++ > > David Fitzgerald > > Department of Earth Sciences > > Millersville University > > Millersville, PA 17551 > > > > Phone: 717-871-2394 > > -- Stephen J Smoogen. "Don't derail a useful feature for the 99% because you're not in it." Linus Torvalds "Years ago my mother used to say to me,... Elwood, you must be oh so smart or oh so pleasant. Well, for years I was smart. I recommend pleasant. You may quote me." —James Stewart as Elwood P. Dowd
Re: User proc uses all RAM+swap => kernel panic - shouldn't OS not allow?
On 09/13/2012 10:32 AM, Stephan Wiesand wrote: Hello Winnie, On Sep 13, 2012, at 16:01 , Winnie Lacesso wrote: Several times over past few years I've seen user processes "go mad" (programming error) & use all RAM, then all swap (as ganglia so vividly shows), then the box ends up at a kernel panic. (Server OS is SL5.x 64-bit BTW) we rarely see panics in these cases. The box just becomes unusable. Which effectively makes no difference though. That's why most of my workstations have swap sizes of just 512 MB. It reduces the time the system is unresponsive when an errant program overuses RAM but before the oom killer kicks in. But 512MB does allow for a little extension when things are getting just a little tight and a clue that something is about to go wrong. That said, most of my experience is with EL5; EL6 is relatively new to the desktops. I'm up to around 60 EL6 systems over the past 6 months with 100 still at EL5, and it's not like I deal with this daily, only once every couple months, so it's possible recent EL6 kernels are less robust in this regard. The problem with this approach is that there's more and more software making very generous use of virtual address space without ever using what was allocated. The current Maple and Oracle's Java come to mind. In our experience, if memory is allocated and never touched, it's like you never allocated it at all (with respect to swap). Allocated but untouched pages will not be swapped. Having sufficient swap space does help. We used to set aside only 2GB for swap even on systems with much more RAM, because they weren't supposed to swap/page much at all. But it turns out that having the recommended amount makes systems much more resilient to memory hogs. As mentioned, I have the opposite experience. Users just turn off the box if it goes unresponsive for more than a few minutes, which is what happens when you have lots of swap allocated and it starts paging itself to death. It is unfortunate that even today, paging seems to cause an abundance of inefficient disk seeking, making writing to swap very slow with traditional "spinning rust" hard drives.
RE: ps and IPC
I solved the problem. Stephen put me on the right track. Turns out my PATH got messed up and was calling the wrong ps. I will go hang my head in shame. -Original Message- From: owner-scientific-linux-us...@listserv.fnal.gov [mailto:owner-scientific-linux-us...@listserv.fnal.gov] On Behalf Of Stephen John Smoogen Sent: Thursday, September 13, 2012 12:58 PM To: owner-scientific-linux-us...@listserv.fnal.gov Cc: scientific-linux-us...@fnal.gov Subject: Re: ps and IPC On 13 September 2012 10:52, David Fitzgerald wrote: > Hello, > > > > I am running Scientific Linux 6.2 and am getting a semget error when I > run ps as any user, including root. > > The full message is: > > > > semget: No such file or directory > > semop: Invalid argument > > [DEVICE -1] The requested IPC message queue is locked. > > Error in message send = 22 > > itype, ichan, nwords,2,-1,2 > > > > Could someone explain what this error means, and show me how to > troubleshoot it? > > Well that error is not a basic one.. which ps are you running (you can find out by typing which ps). Something is locking your access to the kernels memory so something has changed on the system from a default install. I guess I need a lot more questions answered: Type of system (x86_64, i386) Type of hardware The output of the following commands df -a ls -l /proc/ > > Sorry if this is a basic question, but I am stumped. > > > > Thanks! > > > > Dave > > +++ > > David Fitzgerald > > Department of Earth Sciences > > Millersville University > > Millersville, PA 17551 > > > > Phone: 717-871-2394 > > -- Stephen J Smoogen. "Don't derail a useful feature for the 99% because you're not in it." Linus Torvalds "Years ago my mother used to say to me,... Elwood, you must be oh so smart or oh so pleasant. Well, for years I was smart. I recommend pleasant. You may quote me." -James Stewart as Elwood P. Dowd
Re: User proc uses all RAM+swap => kernel panic - shouldn't OS not allow?
On Thu, Sep 13, 2012 at 04:32:00PM +0200, Stephan Wiesand wrote: > Hello Winnie, > > On Sep 13, 2012, at 16:01 , Winnie Lacesso wrote: > > > Several times over past few years I've seen user processes "go mad" > > (programming error) & use all RAM, then all swap (as ganglia so vividly > > shows), then the box ends up at a kernel panic. > > (Server OS is SL5.x 64-bit BTW) > > we rarely see panics in these cases. The box just becomes unusable. Which > effectively makes no difference though. > yes, same here. often, the system starts paging and becomes unresponsive (user pushes the reset button) well before OOM kills the offending process. -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
ntop for EL6
Hi, list I've created some ntop (5.0-2) RPMs for EL6[1] and I have plans to submit it to RPMForge, but, before I do so, I'd like to test it. I'd love if the network administration guys here could give it a try and send me some feedback. Thank you [1] - http://upload.lonelyspooky.com/rpm/EL6/ -- Henrique "LonelySpooky" Junior http://about.me/henriquejunior
Re: ntop for EL6
I used yum to install the RPM Error Processing Dependency: libGeoIP.so.1()(64bit) for package: ntop-5.0-2.el6.x86_64 --> Finished Dependency Resolution Error: Package: ntop-5.0-2.el6.x86_64 (/ntop-5.0-2.el6.x86_64) Requires: libGeoIP.so.1()(64bit) From: Henrique Junior Reply-To: Henrique Junior Date: Thursday, September 13, 2012 2:56 PM To: Scientific Linux Users Subject: ntop for EL6 Hi, list I've created some ntop (5.0-2) RPMs for EL6[1] and I have plans to submit it to RPMForge, but, before I do so, I'd like to test it. I'd love if the network administration guys here could give it a try and send me some feedback. Thank you [1] - http://upload.lonelyspooky.com/rpm/EL6/ -- Henrique "LonelySpooky" Junior http://about.me/henriquejunior
Re: ntop for EL6
Hi, What repos do you have enabled? -- Henrique "LonelySpooky" Junior http://about.me/henriquejunior > > From: Federico Alves >To: Henrique Junior ; Scientific Linux Users > >Sent: Thursday, September 13, 2012 4:07 PM >Subject: Re: ntop for EL6 > > >I used yum to install the RPM >Error > Processing Dependency: libGeoIP.so.1()(64bit) for package: >ntop-5.0-2.el6.x86_64 >--> Finished Dependency Resolution >Error: Package: ntop-5.0-2.el6.x86_64 (/ntop-5.0-2.el6.x86_64) > Requires: libGeoIP.so.1()(64bit) > >From: Henrique Junior >Reply-To: Henrique Junior >Date: Thursday, September 13, 2012 2:56 PM >To: Scientific Linux Users >Subject: ntop for EL6 > > > >Hi, list >I've created some ntop (5.0-2) RPMs for EL6[1] and I have plans to submit it >to RPMForge, but, before I do so, I'd like to test it. >I'd love if the network administration guys here could give it a try and send >me some feedback. > > >Thank you > > >[1] - http://upload.lonelyspooky.com/rpm/EL6/ > >-- >Henrique "LonelySpooky" Junior >http://about.me/henriquejunior > >
Re: ntop for EL6
You'll get libGeoIP.so.1 from EPEL. If you don't have this repo, please, install it: $ su -c 'yum install yum-conf epel' -- Henrique "LonelySpooky" Junior http://about.me/henriquejunior > > From: Federico Alves >To: Henrique Junior ; Scientific Linux Users > >Sent: Thursday, September 13, 2012 4:07 PM >Subject: Re: ntop for EL6 > > >I used yum to install the RPM >Error > Processing Dependency: libGeoIP.so.1()(64bit) for package: >ntop-5.0-2.el6.x86_64 >--> Finished Dependency Resolution >Error: Package: ntop-5.0-2.el6.x86_64 (/ntop-5.0-2.el6.x86_64) > Requires: libGeoIP.so.1()(64bit) > >From: Henrique Junior >Reply-To: Henrique Junior >Date: Thursday, September 13, 2012 2:56 PM >To: Scientific Linux Users >Subject: ntop for EL6 > > > >Hi, list >I've created some ntop (5.0-2) RPMs for EL6[1] and I have plans to submit it >to RPMForge, but, before I do so, I'd like to test it. >I'd love if the network administration guys here could give it a try and send >me some feedback. > > >Thank you > > >[1] - http://upload.lonelyspooky.com/rpm/EL6/ > >-- >Henrique "LonelySpooky" Junior >http://about.me/henriquejunior > >
Re: ntop for EL6
On Thu, 2012-09-13 at 11:56 -0700, Henrique Junior wrote: > Hi, list > I've created some ntop (5.0-2) RPMs for EL6[1] and I have plans to > submit it to RPMForge, but, before I do so, I'd like to test it. Dear Henrique? Have you considered to contribute to EPEL and/or Fedora directly? That'd be the best approach, because most people could profit from your work. You could maintain the EL branches for ntop, if Sven Lankes doesn't like to. https://fedoraproject.org/wiki/Join_the_package_collection_maintainers Volker Fröhlich > I'd love if the network administration guys here could give it a try > and send me some feedback. > > > Thank you > > > [1] - http://upload.lonelyspooky.com/rpm/EL6/ > > -- > Henrique "LonelySpooky" Junior > http://about.me/henriquejunior
Re: ntop for EL6
Hello, Volker I'm already a Fedora packager (not an experient one) and I'd love to put ntop in EPEL (it is in EPEL for EL5). The problem in maintain ntop in EPEL6 is that Fedora now uses systemd to manage our services instead of sysv. It is a Fedora policy to keep only one spec file and that spec is not compatible with EL6 anymore. That is why I believe that RPMForge is the better option. I'm open to ideas to make this package as useful as possible. -- Henrique "LonelySpooky" Junior http://about.me/henriquejunior > > From: Volker Fröhlich >To: Henrique Junior >Cc: Scientific Linux Users >Sent: Thursday, September 13, 2012 5:08 PM >Subject: Re: ntop for EL6 > >On Thu, 2012-09-13 at 11:56 -0700, Henrique Junior wrote: >> Hi, list >> I've created some ntop (5.0-2) RPMs for EL6[1] and I have plans to >> submit it to RPMForge, but, before I do so, I'd like to test it. > >Dear Henrique? > >Have you considered to contribute to EPEL and/or Fedora directly? That'd >be the best approach, because most people could profit from your work. >You could maintain the EL branches for ntop, if Sven Lankes doesn't like >to. > >https://fedoraproject.org/wiki/Join_the_package_collection_maintainers > >Volker Fröhlich > >> I'd love if the network administration guys here could give it a try >> and send me some feedback. >> >> >> Thank you >> >> >> [1] - http://upload.lonelyspooky.com/rpm/EL6/ >> >> -- >> Henrique "LonelySpooky" Junior >> http://about.me/henriquejunior > > > > >
Re: ntop for EL6
Hi, Frederico. It seems ok to me. Look: yum localinstall ntop-5.0-2.el6.x86_64.rpm --nogpgcheck = Package Arch Version Repository Size = Installing: ntop x86_64 5.0-2.el6 /ntop-5.0-2.el6.x86_64 25 M Installing for dependencies: GeoIP x86_64 1.4.8-1.el6 epel 620 k dejavu-lgc-sans-mono-fonts noarch 2.30-2.el6 sl 392 k gd x86_64 2.0.35-10.el6 sl 141 k graphviz x86_64 2.26.0-7.el6 sl 979 k rrdtool x86_64 1.3.8-6.el6 sl 292 k Transaction Summary = Install 6 Package(s) Total size: 27 M Total download size: 2.4 M Installed size: 31 M Is this ok [y/N]: -- Henrique "LonelySpooky" Junior http://about.me/henriquejunior > > From: Federico Alves >To: Henrique Junior ; Scientific Linux Users > >Sent: Thursday, September 13, 2012 4:07 PM >Subject: Re: ntop for EL6 > > >I used yum to install the RPM >Error > Processing Dependency: libGeoIP.so.1()(64bit) for package: >ntop-5.0-2.el6.x86_64 >--> Finished Dependency Resolution >Error: Package: ntop-5.0-2.el6.x86_64 (/ntop-5.0-2.el6.x86_64) > Requires: libGeoIP.so.1()(64bit) > >From: Henrique Junior >Reply-To: Henrique Junior >Date: Thursday, September 13, 2012 2:56 PM >To: Scientific Linux Users >Subject: ntop for EL6 > > > >Hi, list >I've created some ntop (5.0-2) RPMs for EL6[1] and I have plans to submit it >to RPMForge, but, before I do so, I'd like to test it. >I'd love if the network administration guys here could give it a try and send >me some feedback. > > >Thank you > > >[1] - http://upload.lonelyspooky.com/rpm/EL6/ > >-- >Henrique "LonelySpooky" Junior >http://about.me/henriquejunior > >
Re: ntop for EL6
On 09/14/2012 08:20 AM, Henrique Junior wrote: Hello, Volker I'm already a Fedora packager (not an experient one) and I'd love to put ntop in EPEL (it is in EPEL for EL5). The problem in maintain ntop in EPEL6 is that Fedora now uses systemd to manage our services instead of sysv. It is a Fedora policy to keep only one spec file and that spec is not compatible with EL6 anymore. That is why I believe that RPMForge is the better option. I'm open to ideas to make this package as useful as possible. ntop deps are in EPEL already, so EPEL is the most sensible place to put it imo. You'll need to put your EL6-specific bits in conditional clauses within the spec file, the same way you would if you were writing a multi-distro spec (like the magical packages that can install on Vine, EL and SuSE). Playing with spec macros may not be your favorite thing, but this is possible and would open ntop to a broader audience. I think there may still be some multi-distro packages in the core Fedora repository if you need some examples. If not, let me know and I can find something for you.
Re: ntop for EL6
Hi, zxq9 It is nice to have you here! =) Just find a BZ entry [1] proposing a new spec file to build in EL6 too. I'll contact the maintainer and see what I can do to help. [1] - https://bugzilla.redhat.com/show_bug.cgi?id=845195 -- Henrique "LonelySpooky" Junior http://about.me/henriquejunior > > From: zxq9 >To: Scientific Linux Users >Sent: Thursday, September 13, 2012 10:42 PM >Subject: Re: ntop for EL6 > >On 09/14/2012 08:20 AM, Henrique Junior wrote: >> Hello, Volker >> I'm already a Fedora packager (not an experient one) and I'd love to put >> ntop in EPEL (it is in EPEL for EL5). The problem in maintain ntop in >> EPEL6 is that Fedora now uses systemd to manage our services instead of >> sysv. It is a Fedora policy to keep only one spec file and that spec is >> not compatible with EL6 anymore. That is why I believe that RPMForge is >> the better option. >> I'm open to ideas to make this package as useful as possible. > >ntop deps are in EPEL already, so EPEL is the most sensible place to put it >imo. > >You'll need to put your EL6-specific bits in conditional clauses within the >spec file, the same way you would if you were writing a multi-distro spec >(like the magical packages that can install on Vine, EL and SuSE). Playing >with spec macros may not be your favorite thing, but this is possible and >would open ntop to a broader audience. > >I think there may still be some multi-distro packages in the core Fedora >repository if you need some examples. If not, let me know and I can find >something for you. > > >
Re: ntop for EL6
On Thu, 2012-09-13 at 16:20 -0700, Henrique Junior wrote: > Hello, Volker > I'm already a Fedora packager (not an experient one) and I'd love to > put ntop in EPEL (it is in EPEL for EL5). The problem in maintain ntop > in EPEL6 is that Fedora now uses systemd to manage our services > instead of sysv. It is a Fedora policy to keep only one spec file and > that spec is not compatible with EL6 anymore. This is certainly no policy and would be impossible in many cases. Volker > That is why I believe that RPMForge is the better option. > I'm open to ideas to make this package as useful as possible. > > > > -- > Henrique "LonelySpooky" Junior > http://about.me/henriquejunior > > > __ > From: Volker Fröhlich > To: Henrique Junior > Cc: Scientific Linux Users > Sent: Thursday, September 13, 2012 5:08 PM > Subject: Re: ntop for EL6 > > > On Thu, 2012-09-13 at 11:56 -0700, Henrique Junior wrote: > > Hi, list > > I've created some ntop (5.0-2) RPMs for EL6[1] and I have > plans to > > submit it to RPMForge, but, before I do so, I'd like to test > it. > > Dear Henrique? > > Have you considered to contribute to EPEL and/or Fedora > directly? That'd > be the best approach, because most people could profit from > your work. > You could maintain the EL branches for ntop, if Sven Lankes > doesn't like > to. > > https://fedoraproject.org/wiki/Join_the_package_collection_maintainers > > Volker Fröhlich > > > I'd love if the network administration guys here could give > it a try > > and send me some feedback. > > > > > > Thank you > > > > > > [1] - http://upload.lonelyspooky.com/rpm/EL6/ > > > > -- > > Henrique "LonelySpooky" Junior > > http://about.me/henriquejunior > > > > >
