Thanks Richard for the joke :))
Thanks Thomas for the input. I changed the ps command flag to '--sort
-rss', and restarted memusage - will continue to monitor.
Thanks Dave for the pointer, but I don't have any of my own C/C++ programs
running, just many bash scripts (if they do no 'malloc's, can they still
cause memory leaks?).
In the past this server has gone to near zero memory, and vmcp commands
fail. I'm guessing the OOM killer was invoked, but by then it's already
too late ...
-Mike
On Thu, Jul 9, 2015 at 9:54 AM, Dave Jones <d...@vsoft-software.com> wrote:
> Hi, Mike.
>
> if the package AddressSanitizer (ASan) is available, you might want to
> ive it a go. It is a fast memory error detector. that can find
> use-after-free and {heap,stack,global}-buffer overflow bugs in C/C++
> programs. it's here:
>
> https://code.google.com/p/address-sanitizer/
>
> Good luck....I still think C/C++ will be the death of us all. :-)
>
> DJ
>
> On 07/09/2015 07:50 AM, Pavelka, Tomas wrote:
> > Look at the " -/+ buffers/cache" line in the free output:
> >
> > Before:
> > -/+ buffers/cache: 41 450
> > After:
> > -/+ buffers/cache: 48 443
> >
> > (First number used, second free)
> >
> > Linux has various buffers and caches that are allocated if there is free
> memory. For example for disk reads. These are dropped if the memory is
> needed by processes. The " -/+ buffers/cache" line shows what memory is
> actually used by processes and not the buffers. In your case the used
> memory rose only by 7 MB.
> >
> > BTW I would not look at the virtual memory size of proceses, this may be
> allocated way over the virtual memory size of your machine. The more
> interesting metric is RSS which is how much memory is actually used.
> >
> > HTH,
> > Tomas
> >
> > Tomas Pavelka
> > CA Technologies
> > Sr Software Engineer
> >
> > CA CZ, s.r.o
> > V Parku 12,
> > 148 00 Praha
> > Czech Republic
> >
> > Office: +25996 | tomas.pave...@ca.com
> >
> >
> >
> > Id. Císlo 25694073, z obchodního rejstříku, vedeného Městským soudem v
> Praze, oddíl C, vložka 61808 / Id. No. 25694073, registered in the
> Commercial Register maintained by the Municipal Court in Praque, Section C,
> File 61808
> >
> >
> > -----Original Message-----
> > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of
> Michael MacIsaac
> > Sent: Thursday, July 09, 2015 2:19 PM
> > To: LINUX-390@VM.MARIST.EDU
> > Subject: How to find a memory leak?
> >
> > Hello list,
> >
> > I have a SLES 11 SP3 system that is leaking memory, but I don't know how
> or where.
> >
> > I find a script on the Internet that runs forever, adapt it somewhat,
> and start logging some info to a temp file. Here's the script:
> >
> > # cat memusage
> > #!/bin/bash
> > #
> > # track memory usage
> > #
> > outFile="/tmp/memusage"
> > while true
> > do
> > echo "---------------------------------------------------" >> $outFile
> > date >> $outFile
> > ps aux --sort -vsz | head -22 >> $outFile
> > echo >> $outFile
> > free -m >> $outFile
> > sleep 300
> > done
> >
> > After a fresh reboot of a 512 MB virtual machine, I start the script and
> the first entry in the temp file shows about 20 MB (512 - 492) used by
> Linux and 97 MB used by processes:
> >
> > Wed Jul 8 12:37:45 EDT 2015
> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > root 2181 0.0 0.2 115404 1024 ? Ssl 12:36 0:00
> > /usr/sbin/nscd
> > root 1851 0.0 0.1 11512 692 ? S<sl 12:36 0:00
> > /sbin/auditd -s disable
> > root 2556 0.3 0.7 11456 4004 ? Ss 12:37 0:00 sshd:
> > root@pts/0
> > root 2306 0.0 0.7 10720 3700 ? Ss 12:36 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2307 0.0 0.4 10720 2204 ? S 12:36 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2308 0.0 0.4 10720 2204 ? S 12:36 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2309 0.0 0.4 10720 2204 ? S 12:36 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2310 0.0 0.4 10720 2204 ? S 12:36 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2311 0.0 0.4 10720 2204 ? S 12:36 0:00
> > /usr/sbin/httpd2-prefork
> > root 1853 0.0 0.1 10428 824 ? S<sl 12:36 0:00
> > /sbin/audispd
> > root 997 0.0 0.6 9036 3224 ? Ssl 12:36 0:00
> > /usr/sbin/console-kit-da
> > root 2265 0.0 0.5 8136 2532 ? Ss 12:36 0:00
> > /usr/lib/postfix/master
> > postfix 2277 0.0 0.4 8004 2372 ? S 12:36 0:00 qmgr -l
> -t
> > fifo -u
> > postfix 2276 0.0 0.4 7948 2352 ? S 12:36 0:00 pickup
> -l
> > -t fifo -u
> > root 2172 0.0 0.3 7916 1532 ? Ss 12:36 0:00
> > /usr/sbin/sshd -o PidFi
> > 101 994 0.0 0.5 7852 2804 ? Ss 12:36 0:00
> > /usr/sbin/hald --daemon
> > root 1869 0.0 0.8 6464 4504 ? Ss 12:36 0:00
> > /sbin/haveged -w 1024 -
> > root 2559 1.0 0.6 6056 3076 pts/0 Ss 12:37 0:00 -bash
> > root 998 0.0 0.2 3980 1332 ? S 12:36 0:00
> hald-runner
> > root 2591 0.0 0.3 3652 1604 pts/0 S+ 12:37 0:00
> /bin/bash
> > /usr/local/sb
> > root 2343 0.0 0.1 3508 944 ? Ss 12:36 0:00
> > /usr/sbin/xinetd -pidfi
> >
> > total used free shared buffers cached
> > Mem: *492 97* 394 0 5
> 50
> > -/+ buffers/cache: 41 450
> > Swap: 898 0 898
> >
> > This morning the last entry shows 156 MB used by processes: ~59 MB of
> memory lost in less than a day. But the 'VSZ' of the top 22 processes seems
> to be about the same:
> >
> > Thu Jul 9 07:57:47 EDT 2015
> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > root 2181 0.0 0.2 115404 1088 ? Ssl Jul08 0:00
> > /usr/sbin/nscd
> > root 1851 0.0 0.1 11512 692 ? S<sl Jul08 0:00
> > /sbin/auditd -s disable
> > root 44879 0.0 0.7 11456 4008 ? Ss 07:36 0:00 sshd:
> > root@pts/1
> > root 44913 0.0 0.7 11456 4008 ? Ss 07:37 0:00 sshd:
> > root@pts/2
> > root 2306 0.0 0.7 10720 3700 ? Ss Jul08 0:01
> > /usr/sbin/httpd2-prefork
> > wwwrun 2307 0.0 0.4 10720 2204 ? S Jul08 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2308 0.0 0.4 10720 2204 ? S Jul08 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2309 0.0 0.4 10720 2204 ? S Jul08 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2310 0.0 0.4 10720 2204 ? S Jul08 0:00
> > /usr/sbin/httpd2-prefork
> > wwwrun 2311 0.0 0.4 10720 2204 ? S Jul08 0:00
> > /usr/sbin/httpd2-prefork
> > root 1853 0.0 0.1 10428 824 ? S<sl Jul08 0:00
> > /sbin/audispd
> > root 997 0.0 0.6 9036 3224 ? Ssl Jul08 0:00
> > /usr/sbin/console-kit-da
> > root 2265 0.0 0.5 8136 2532 ? Ss Jul08 0:00
> > /usr/lib/postfix/master
> > postfix 2277 0.0 0.4 8004 2372 ? S Jul08 0:00 qmgr -l
> -t
> > fifo -u
> > postfix 44778 0.0 0.4 7948 2356 ? S 06:56 0:00 pickup
> -l
> > -t fifo -u
> > root 2172 0.0 0.3 7916 1664 ? Ss Jul08 0:00
> > /usr/sbin/sshd -o PidFi
> > 101 994 0.0 0.5 7852 2852 ? Ss Jul08 0:00
> > /usr/sbin/hald --daemon
> > root 1869 0.0 0.8 6464 4504 ? Ss Jul08 0:00
> > /sbin/haveged -w 1024 -
> > root 44882 0.0 0.6 6056 3052 pts/1 Ss 07:36 0:00 -bash
> > root 44916 0.0 0.5 6056 2876 pts/2 Ss+ 07:37 0:00 -bash
> > mike 44992 0.0 0.5 6056 3004 pts/1 S+ 07:56 0:00 -bash
> >
> > total used free shared buffers cached
> > Mem: *492 156 * 335 0 24
> 83
> > -/+ buffers/cache: 48 443
> > Swap: 898 0 898
> >
> > Is there a better way to detect memory leaks? Are there any known
> 'culprits'?
> >
> > Any help will be appreciated. Thanks.
> >
> > -Mike MacIsaac
> >
> > ----------------------------------------------------------------------
> > For LINUX-390 subscribe / signoff / archive access instructions, send
> email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
> > http://www.marist.edu/htbin/wlvindex?LINUX-390
> > ----------------------------------------------------------------------
> > For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
> >
>
> --
> Dave Jones
> Houston, TX
> 281.578.7544
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> ----------------------------------------------------------------------
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
