Here's a snippet of regular "top" output, sorted by memory usage:
top - 05:19:13 up 7 days, 19:21, 4 users, load average: 129.56, 129.03, 121.31 Tasks: 249 total, 1 running, 248 sleeping, 0 stopped, 0 zombie Cpu(s): 0.5%us, 11.3%sy, 0.0%ni, 0.0%id, 87.0%wa, 0.0%hi, 1.2%si, 0.0%st Mem: 2055108k total, 1988664k used, 66444k free, 80k buffers Swap: 2064376k total, 2064148k used, 228k free, 7400k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8573 root 20 0 222m 42m 72 D 0.1 2.1 0:00.16 perl 8579 root 20 0 222m 42m 68 D 0.1 2.1 0:00.10 perl 8610 root 20 0 222m 42m 68 D 0.1 2.1 0:00.05 perl 8615 root 20 0 222m 42m 68 D 0.1 2.1 0:00.04 perl 8616 root 20 0 222m 42m 68 D 0.2 2.1 0:00.08 perl 8606 root 20 0 222m 42m 68 D 0.1 2.1 0:00.06 perl 8597 root 20 0 222m 42m 68 D 0.1 2.1 0:00.07 perl ...etc... Andrew --- On Wed, 9/14/11, Andrew Klaassen <[email protected]> wrote: > From: Andrew Klaassen <[email protected]> > Subject: Re: [xcat-user] rinstall memory usage > To: "xCAT Users Mailing list" <[email protected]> > Received: Wednesday, September 14, 2011, 6:59 AM > --- On Tue, 9/13/11, Jarrod Johnson > <[email protected]> > wrote: > > > Might want ps axf output. If they are console > backends, something > > is not right because perl shouldn't be long-lived (it > should nowadays > > be either sleep or ipmitool or ssh depending, with > perl being their > > for only a few milliseconds before not being perl > anymore)). > > > > If it is xcatd, the memory usage in top is misleading, > but you still > > shouldn't have a lot of xcatd all at once normally. > > top -b -n 1 might also be interesting. > > The culprit appears to be getcredentials in the > postscripts. The server ps axf has these: > > 7177 ? Ds > 0:00 xcatd: SSL listener > 8032 ? D > 0:00 \_ xCATd SSL: getcredentials for r063 > 8034 ? D > 0:00 \_ xCATd SSL: getcredentials for r065 > 8037 ? D > 0:00 \_ xCATd SSL: getcredentials for r051 > 8067 ? D > 0:00 | \_ xCATd SSL: getcredentials > for r051 > 8038 ? D > 0:00 \_ xCATd SSL: getcredentials for r081 > 8064 ? D > 0:00 | \_ xCATd SSL: getcredentials > for r081 > 8058 ? D > 0:00 \_ xCATd SSL: getcredentials for r096 > 8065 ? D > 0:00 | \_ xCATd SSL: getcredentials > for r096 > .......etc....... > > and these: > > 7946 ? D > 0:00 xCATd SSL: getpostscript for r070 > 7966 ? D > 0:00 xCATd SSL: getcredentials for r061 > 7968 ? D > 0:00 xCATd SSL: getcredentials for r057 > 7984 ? D > 0:00 xCATd SSL: getpostscript for r063 > 7995 ? D > 0:00 xCATd SSL: getcredentials for r068 > .......etc....... > > The top -b -n 1 output has a bunch of these: > > 8208 root 20 0 > 228m 43m 68 D 0.0 > 2.1 0:00.16 perl > 8209 root 20 0 > 228m 39m 56 D 0.0 > 1.9 0:00.14 perl > 8220 root 20 0 > 228m 23m 0 D 0.0 > 1.1 0:00.07 perl > 8259 root 20 0 > 222m 25m 200 D 0.0 > 1.3 0:00.03 perl > 8260 root 20 0 > 222m 25m 4 D 0.0 > 1.3 0:00.03 perl > 8261 root 20 0 > 222m 25m 4 D 0.0 > 1.3 0:00.03 perl > > Here's a snippet of ps axflwww from a few minutes later > (load average now 128): > > F UID PID PPID > PRI NI VSZ RSS > WCHAN STAT TTY TIME > COMMAND > 1 0 7177 > 1 20 0 228024 18092 > hrtime Ss ? > 0:00 xcatd: SSL listener > 1 0 8265 7177 > 20 0 228024 42564 get_re D > ? 0:00 \_ xcatd: SSL > listener > 1 0 8266 7177 > 20 0 228072 41716 get_re D > ? 0:00 \_ xcatd: SSL > listener > 1 0 8267 7177 > 20 0 228072 39388 get_re D > ? 0:00 \_ xcatd: SSL > listener > 1 0 8268 7177 > 20 0 228024 45004 get_re D > ? 0:00 \_ xcatd: SSL > listener > .......etc....... > 1 0 7946 > 1 20 0 232308 > 9536 conges D ? > 0:00 xCATd SSL: getpostscript for r070 > 1 0 7966 > 1 20 0 232512 > 8212 get_re D ? > 0:00 xCATd SSL: getcredentials for r061 > 1 0 7968 > 1 20 0 231804 10048 > conges D ? > 0:00 xCATd SSL: getcredentials for r057 > 1 0 7984 > 1 20 0 233276 15228 > sync_p D ? > 0:00 xCATd SSL: getpostscript for r063 > 1 0 7995 > 1 20 0 232772 > 9672 sync_p D ? > 0:00 xCATd SSL: getcredentials for r068 > 1 0 7996 > 1 20 0 232772 > 5368 sync_p D ? > 0:00 xCATd SSL: getcredentials for r057 > .......etc....... > > I've attached the full output of each (partial output for > ps axf, since it's still hung after ten minutes or > so). Apologies for the DOS line-endings. > > Andrew > -----Inline Attachment Follows----- > > ------------------------------------------------------------------------------ > BlackBerry® DevCon Americas, Oct. 18-20, San > Francisco, CA > Learn about the latest advances in developing for the > BlackBerry® mobile platform with sessions, labs > & more. > See new tools and technologies. Register for > BlackBerry® DevCon today! > http://p.sf.net/sfu/rim-devcon-copy1 > > -----Inline Attachment Follows----- > > _______________________________________________ > xCAT-user mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/xcat-user > ------------------------------------------------------------------------------ BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA Learn about the latest advances in developing for the BlackBerry® mobile platform with sessions, labs & more. See new tools and technologies. Register for BlackBerry® DevCon today! http://p.sf.net/sfu/rim-devcon-copy1 _______________________________________________ xCAT-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/xcat-user
