Use Rob's cmmflush ! https://zvmperf.wordpress.com/2012/07/06/using-cmm-to-flush-a-linux-guests-memory/
We use it every day in dev to keep the vm paging rate way down. -----Original Message----- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Michael MacIsaac Sent: Thursday, July 09, 2015 7:49 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] How to find a memory leak? Thomas, > Did you use a buffer larger than 32k on those vmcp commands? Yes, I always use 1M (vmcpCmd="/sbin/vmcp --buffer=1M") in the event there is a lot of output from CP. > Vmcp can fail due to memory fragmentation even on a server with lots > of free memory. Hmmm, interesting... could this be considered a bug? As a workaround, is there a command to flush the buffer cache? Thanks. -Mike M. On Thu, Jul 9, 2015 at 10:40 AM, Pavelka, Tomas <tomas.pave...@ca.com> wrote: > > In the past this server has gone to near zero memory, and vmcp > > commands > fail. > > Do you have any specifics? Did you use a buffer larger than 32k on > those vmcp commands? Vmcp can fail due to memory fragmentation even on > a server with lots of free memory. > > Tomas Pavelka > CA Technologies > Sr Software Engineer > > CA CZ, s.r.o > V Parku 12, > 148 00 Praha > Czech Republic > > Office: +25996 | tomas.pave...@ca.com > > > > Id. Císlo 25694073, z obchodního rejstříku, vedeného Městským soudem v > Praze, oddíl C, vložka 61808 / Id. No. 25694073, registered in the > Commercial Register maintained by the Municipal Court in Praque, > Section C, File 61808 > > > > -----Original Message----- > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of > Michael MacIsaac > Sent: Thursday, July 09, 2015 4:15 PM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: How to find a memory leak? > > Thanks Richard for the joke :)) > > Thanks Thomas for the input. I changed the ps command flag to '--sort > -rss', and restarted memusage - will continue to monitor. > > Thanks Dave for the pointer, but I don't have any of my own C/C++ > programs running, just many bash scripts (if they do no 'malloc's, can > they still cause memory leaks?). > > In the past this server has gone to near zero memory, and vmcp > commands fail. I'm guessing the OOM killer was invoked, but by then > it's already too late ... > > -Mike > > On Thu, Jul 9, 2015 at 9:54 AM, Dave Jones <d...@vsoft-software.com> > wrote: > > > Hi, Mike. > > > > if the package AddressSanitizer (ASan) is available, you might want > > to ive it a go. It is a fast memory error detector. that can find > > use-after-free and {heap,stack,global}-buffer overflow bugs in C/C++ > > programs. it's here: > > > > https://code.google.com/p/address-sanitizer/ > > > > Good luck....I still think C/C++ will be the death of us all. :-) > > > > DJ > > > > On 07/09/2015 07:50 AM, Pavelka, Tomas wrote: > > > Look at the " -/+ buffers/cache" line in the free output: > > > > > > Before: > > > -/+ buffers/cache: 41 450 > > > After: > > > -/+ buffers/cache: 48 443 > > > > > > (First number used, second free) > > > > > > Linux has various buffers and caches that are allocated if there > > > is free > > memory. For example for disk reads. These are dropped if the memory > > is needed by processes. The " -/+ buffers/cache" line shows what > > memory is actually used by processes and not the buffers. In your > > case the used memory rose only by 7 MB. > > > > > > BTW I would not look at the virtual memory size of proceses, this > > > may be > > allocated way over the virtual memory size of your machine. The more > > interesting metric is RSS which is how much memory is actually used. > > > > > > HTH, > > > Tomas > > > > > > Tomas Pavelka > > > CA Technologies > > > Sr Software Engineer > > > > > > CA CZ, s.r.o > > > V Parku 12, > > > 148 00 Praha > > > Czech Republic > > > > > > Office: +25996 | tomas.pave...@ca.com > > > > > > > > > > > > Id. Císlo 25694073, z obchodního rejstříku, vedeného Městským > > > soudem v > > Praze, oddíl C, vložka 61808 / Id. No. 25694073, registered in the > > Commercial Register maintained by the Municipal Court in Praque, > > Section C, File 61808 > > > > > > > > > -----Original Message----- > > > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf > > > Of > > Michael MacIsaac > > > Sent: Thursday, July 09, 2015 2:19 PM > > > To: LINUX-390@VM.MARIST.EDU > > > Subject: How to find a memory leak? > > > > > > Hello list, > > > > > > I have a SLES 11 SP3 system that is leaking memory, but I don't > > > know how > > or where. > > > > > > I find a script on the Internet that runs forever, adapt it > > > somewhat, > > and start logging some info to a temp file. Here's the script: > > > > > > # cat memusage > > > #!/bin/bash > > > # > > > # track memory usage > > > # > > > outFile="/tmp/memusage" > > > while true > > > do > > > echo "---------------------------------------------------" >> > $outFile > > > date >> $outFile > > > ps aux --sort -vsz | head -22 >> $outFile > > > echo >> $outFile > > > free -m >> $outFile > > > sleep 300 > > > done > > > > > > After a fresh reboot of a 512 MB virtual machine, I start the > > > script and > > the first entry in the temp file shows about 20 MB (512 - 492) used > > by Linux and 97 MB used by processes: > > > > > > Wed Jul 8 12:37:45 EDT 2015 > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME > COMMAND > > > root 2181 0.0 0.2 115404 1024 ? Ssl 12:36 0:00 > > > /usr/sbin/nscd > > > root 1851 0.0 0.1 11512 692 ? S<sl 12:36 0:00 > > > /sbin/auditd -s disable > > > root 2556 0.3 0.7 11456 4004 ? Ss 12:37 0:00 sshd: > > > root@pts/0 > > > root 2306 0.0 0.7 10720 3700 ? Ss 12:36 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2307 0.0 0.4 10720 2204 ? S 12:36 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2308 0.0 0.4 10720 2204 ? S 12:36 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2309 0.0 0.4 10720 2204 ? S 12:36 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2310 0.0 0.4 10720 2204 ? S 12:36 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2311 0.0 0.4 10720 2204 ? S 12:36 0:00 > > > /usr/sbin/httpd2-prefork > > > root 1853 0.0 0.1 10428 824 ? S<sl 12:36 0:00 > > > /sbin/audispd > > > root 997 0.0 0.6 9036 3224 ? Ssl 12:36 0:00 > > > /usr/sbin/console-kit-da > > > root 2265 0.0 0.5 8136 2532 ? Ss 12:36 0:00 > > > /usr/lib/postfix/master > > > postfix 2277 0.0 0.4 8004 2372 ? S 12:36 0:00 qmgr > -l > > -t > > > fifo -u > > > postfix 2276 0.0 0.4 7948 2352 ? S 12:36 0:00 pickup > > -l > > > -t fifo -u > > > root 2172 0.0 0.3 7916 1532 ? Ss 12:36 0:00 > > > /usr/sbin/sshd -o PidFi > > > 101 994 0.0 0.5 7852 2804 ? Ss 12:36 0:00 > > > /usr/sbin/hald --daemon > > > root 1869 0.0 0.8 6464 4504 ? Ss 12:36 0:00 > > > /sbin/haveged -w 1024 - > > > root 2559 1.0 0.6 6056 3076 pts/0 Ss 12:37 0:00 -bash > > > root 998 0.0 0.2 3980 1332 ? S 12:36 0:00 > > hald-runner > > > root 2591 0.0 0.3 3652 1604 pts/0 S+ 12:37 0:00 > > /bin/bash > > > /usr/local/sb > > > root 2343 0.0 0.1 3508 944 ? Ss 12:36 0:00 > > > /usr/sbin/xinetd -pidfi > > > > > > total used free shared buffers > cached > > > Mem: *492 97* 394 0 5 > > 50 > > > -/+ buffers/cache: 41 450 > > > Swap: 898 0 898 > > > > > > This morning the last entry shows 156 MB used by processes: ~59 MB > > > of > > memory lost in less than a day. But the 'VSZ' of the top 22 > > processes seems to be about the same: > > > > > > Thu Jul 9 07:57:47 EDT 2015 > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME > COMMAND > > > root 2181 0.0 0.2 115404 1088 ? Ssl Jul08 0:00 > > > /usr/sbin/nscd > > > root 1851 0.0 0.1 11512 692 ? S<sl Jul08 0:00 > > > /sbin/auditd -s disable > > > root 44879 0.0 0.7 11456 4008 ? Ss 07:36 0:00 sshd: > > > root@pts/1 > > > root 44913 0.0 0.7 11456 4008 ? Ss 07:37 0:00 sshd: > > > root@pts/2 > > > root 2306 0.0 0.7 10720 3700 ? Ss Jul08 0:01 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2307 0.0 0.4 10720 2204 ? S Jul08 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2308 0.0 0.4 10720 2204 ? S Jul08 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2309 0.0 0.4 10720 2204 ? S Jul08 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2310 0.0 0.4 10720 2204 ? S Jul08 0:00 > > > /usr/sbin/httpd2-prefork > > > wwwrun 2311 0.0 0.4 10720 2204 ? S Jul08 0:00 > > > /usr/sbin/httpd2-prefork > > > root 1853 0.0 0.1 10428 824 ? S<sl Jul08 0:00 > > > /sbin/audispd > > > root 997 0.0 0.6 9036 3224 ? Ssl Jul08 0:00 > > > /usr/sbin/console-kit-da > > > root 2265 0.0 0.5 8136 2532 ? Ss Jul08 0:00 > > > /usr/lib/postfix/master > > > postfix 2277 0.0 0.4 8004 2372 ? S Jul08 0:00 qmgr > -l > > -t > > > fifo -u > > > postfix 44778 0.0 0.4 7948 2356 ? S 06:56 0:00 pickup > > -l > > > -t fifo -u > > > root 2172 0.0 0.3 7916 1664 ? Ss Jul08 0:00 > > > /usr/sbin/sshd -o PidFi > > > 101 994 0.0 0.5 7852 2852 ? Ss Jul08 0:00 > > > /usr/sbin/hald --daemon > > > root 1869 0.0 0.8 6464 4504 ? Ss Jul08 0:00 > > > /sbin/haveged -w 1024 - > > > root 44882 0.0 0.6 6056 3052 pts/1 Ss 07:36 0:00 -bash > > > root 44916 0.0 0.5 6056 2876 pts/2 Ss+ 07:37 0:00 -bash > > > mike 44992 0.0 0.5 6056 3004 pts/1 S+ 07:56 0:00 -bash > > > > > > total used free shared buffers > cached > > > Mem: *492 156 * 335 0 24 > > 83 > > > -/+ buffers/cache: 48 443 > > > Swap: 898 0 898 > > > > > > Is there a better way to detect memory leaks? Are there any > > > known > > 'culprits'? > > > > > > Any help will be appreciated. Thanks. > > > > > > -Mike MacIsaac > > > > > > ------------------------------------------------------------------ > > > -- > > > -- For LINUX-390 subscribe / signoff / archive access > > > instructions, send > > email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > > visit > > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > > ------------------------------------------------------------------ > > > -- > > > -- For more information on Linux on System z, visit > > http://wiki.linuxvm.org/ > > > > > > > -- > > Dave Jones > > Houston, TX > > 281.578.7544 > > > > -------------------------------------------------------------------- > > -- For LINUX-390 subscribe / signoff / archive access instructions, > > send email to lists...@vm.marist.edu with the message: INFO > > LINUX-390 or visit > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > -------------------------------------------------------------------- > > -- For more information on Linux on System z, visit > > http://wiki.linuxvm.org/ > > > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, send > email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
