Re: How to find a memory leak? cmmflush and iib/wmb

2015-11-12 Thread Marcy Cortes
65097&myns=apar&mynp=DOCTYPEcomponent&mync=E&cm_sp=apar-_-DOCTYPEcomponent-_-E -Original Message- From: Cortes, Marcy D. Sent: Tuesday, July 21, 2015 7:08 AM To: LINUX-390@VM.MARIST.EDU Subject: RE: How to find a memory leak? cmmflush and iib/wmb Maybe?!I checked one W

Re: How to find a memory leak? cmmflush and iib/wmb

2015-07-21 Thread Marcy Cortes
@VM.MARIST.EDU Subject: Re: [LINUX-390] How to find a memory leak? cmmflush and iib/wmb Will cmmflush cause WMB or IIB to release memory that may build up over the week in the execution groups? Chris Will -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf

Re: How to find a memory leak? cmmflush and iib/wmb

2015-07-21 Thread Will, Chris
Subject: Re: How to find a memory leak? Easier, but the pages aren't dropped from the zVM side immediately so if you are memory constrained there, cmmflush is your friend. -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Michael MacIsaac

Re: How to find a memory leak?

2015-07-13 Thread Pavelka, Tomas
Let me try a different example than Mike's 'Q DASD DETAILS 0-': Suppose you are writing software for disaster recovery of LVM disks. The Linux that owned them will not come up so you link them from another Linux. Get a list of minidisk addresses and their owners and issue LINK against each.

Re: How to find a memory leak?

2015-07-13 Thread Alan Altmark
On Monday, 07/13/2015 at 11:12 EDT, Michael MacIsaac wrote: > I thought you also pointed out that CMS found a way to work around it (or > maybe that was Chuckie). I thought, perhaps naively, that Linux may be > able to work around it also. Sorry, Mike. I thought you were still talking about the

Re: How to find a memory leak?

2015-07-13 Thread Marcy Cortes
: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] How to find a memory leak? How about dividing it up in a little loop? Issue Q D DETAILS 000-FFF, Q DA DETAILS 1000-1FFF -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Michael MacIsaac Sent

Re: How to find a memory leak?

2015-07-13 Thread Michael MacIsaac
Alan, > Mike, I already posted that the vmcp problem is caused by the DIAGNOSE > 0x08 requirement for contiguous memory. Thanks. I thought you also pointed out that CMS found a way to work around it (or maybe that was Chuckie). I thought, perhaps naively, that Linux may be able to work around i

Re: How to find a memory leak?

2015-07-13 Thread Christian Borntraeger
Am 10.07.2015 um 14:18 schrieb Bruce Hayden: > The message sent to stderr is not documented in the device drivers book. > It tells you about the response code of 2, but the description of that > response code doesn't say anything about the error message to stderr or > that the message tells you how

Re: How to find a memory leak?

2015-07-13 Thread Christian Borntraeger
Am 13.07.2015 um 16:42 schrieb Michael MacIsaac: > Any chance the requirement for contiguous memory in vmcp could be relaxed? That would require a change in the diagnose definition of z/VM. diag 8 does require the buffer contiguous in guest real storage. > In my test case, 'Q DA DETAILS 0-' r

Re: How to find a memory leak?

2015-07-13 Thread Pavelka, Tomas
> the qeth driver has been improved in 2014 to reduce its demand for contiguous > storage: Thanks Ursula, the memories keep coming back ;-) I remembered that the vmcp is still problematic but managed to forget that the qeth driver got fixed.

Re: How to find a memory leak?

2015-07-13 Thread Alan Altmark
On Monday, 07/13/2015 at 10:43 EDT, Michael MacIsaac wrote: > Any chance the requirement for contiguous memory in vmcp could be relaxed? > In my test case, 'Q DA DETAILS 0-' requires ~2.5 MB and, as I > understand it, that output can simply not be obtained through vmcp. Mike, I already posted

Re: How to find a memory leak?

2015-07-13 Thread Marcy Cortes
-390] How to find a memory leak? Ursula, Any chance the requirement for contiguous memory in vmcp could be relaxed? In my test case, 'Q DA DETAILS 0-' requires ~2.5 MB and, as I understand it, that output can simply not be obtained through vmcp. Thanks. -Mike MacIsaac On M

Re: How to find a memory leak?

2015-07-13 Thread Michael MacIsaac
Ursula, Any chance the requirement for contiguous memory in vmcp could be relaxed? In my test case, 'Q DA DETAILS 0-' requires ~2.5 MB and, as I understand it, that output can simply not be obtained through vmcp. Thanks. -Mike MacIsaac On Mon, Jul 13, 2015 at 10:28 AM, Ursula Braun wro

Re: How to find a memory leak?

2015-07-13 Thread Ursula Braun
Tomas, the qeth driver has been improved in 2014 to reduce its demand for contiguous storage: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/s390/net/qeth_core.h?id=d445a4e28c0ff740e946ae22860be85428814c39 Regards, Ursula Braun, IBM Germany, Linux on System z Dev.

Re: How to find a memory leak?

2015-07-13 Thread Alan Altmark
On Monday, 07/13/2015 at 09:40 EDT, "Pavelka, Tomas" wrote: > > I wouldn't really put that at the feet of s390 (z/Architecture). > > Bad wording on my part. When I said s390 I meant the s390 part of the Linux > kernel implementation, not the entire architecture. I meant to point out that > the oth

Re: How to find a memory leak?

2015-07-13 Thread Pavelka, Tomas
> I wouldn't really put that at the feet of s390 (z/Architecture). Bad wording on my part. When I said s390 I meant the s390 part of the Linux kernel implementation, not the entire architecture. I meant to point out that the other parts of the kernel are working on getting out of the requirement

Re: How to find a memory leak?

2015-07-13 Thread Alan Altmark
On Friday, 07/10/2015 at 02:09 EDT, "Pavelka, Tomas" wrote: > Where the s390 is different is that it uses large continuous buffers all over. I wouldn't really put that at the feet of s390 (z/Architecture). As you noted, the DIAGNOSE 0x08 problem is because CP requires the buffer to be contiguous

Re: How to find a memory leak?

2015-07-10 Thread Alan Altmark
On Friday, 07/10/2015 at 03:03 EDT, Rick Troth wrote: > Same thing *has* been said of VM ... by an IBMer ... to me ... as he > deflected a legit inquiry/requirement. I certainly hope that's ancient history. If not, feel free to contact me offline and maybe we can figure out what went wrong and h

Re: How to find a memory leak?

2015-07-10 Thread Rick Troth
-389-4601 > ● lstew...@visa.com > > -Original Message- > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Alan > Altmark > Sent: Friday, July 10, 2015 7:23 AM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: How to find a memory leak? > > This is L

Re: How to find a memory leak?

2015-07-10 Thread Stewart, Lee
:23 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: How to find a memory leak? This is Linux. You have source code. Who needs documentation? -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists

Re: How to find a memory leak?

2015-07-10 Thread Alan Altmark
On Friday, 07/10/2015 at 08:20 EDT, Bruce Hayden wrote: > The message sent to stderr is not documented in the device drivers book. > It tells you about the response code of 2, but the description of that > response code doesn't say anything about the error message to stderr or > that the message t

Re: How to find a memory leak?

2015-07-10 Thread Michael MacIsaac
OK, y'all convinced me that always using a 1M buffer for vmcp is not the best idea. I appreciate all the feedback. Here's my first take at a bash function to retry a vmcp command with a few boundary test cases: # cat testvmcp #!/bin/bash function zVerbose { echo "$@" } #+---

Re: How to find a memory leak?

2015-07-10 Thread Bruce Hayden
The message sent to stderr is not documented in the device drivers book. It tells you about the response code of 2, but the description of that response code doesn't say anything about the error message to stderr or that the message tells you how long the output was. For use in scripts, it would b

Re: How to find a memory leak?

2015-07-10 Thread Christian Borntraeger
Am 10.07.2015 um 13:19 schrieb Christian Borntraeger: > Anothing thing: 1M is quite large (the larges contiguous memory that Linux > wants > to allocate as slab). So try to not use that unless you need it. If you want > to > know how much memory is needed, then vmcp gives you back the result of t

Re: How to find a memory leak?

2015-07-10 Thread Pavelka, Tomas
> Only admins would have access to those sudo commands. But the sudoers line shows an intent to restrict access to tee only: %zoom ALL=NOPASSWD:/usr/bin/tee The hole that Karsten has shown is that the line in sudoers is really the security equivalent of: %zoom ALL=NOPASSWD:ALL Whether it is a

Re: How to find a memory leak?

2015-07-10 Thread Christian Borntraeger
Am 09.07.2015 um 19:16 schrieb Michael MacIsaac: > I'm going to stop here for now. I've learned a lot about Linux memory from > this thread (but that's easy when you don't know much to begin with :)). > > I guess a question to the Linux developers in Germany would be: > > If vmcp is called with a

Re: How to find a memory leak?

2015-07-10 Thread Michael MacIsaac
> That's a huge security hole, btw. Don't do that! this is an attempt to be more security minded, not less. Only admins would have access to those sudo commands. For example, if I am root and issue the command: echo "newroot:x:0:0:root:/root:/bin/bash" >> /etc/passwd then who added the newroot u

Re: How to find a memory leak?

2015-07-10 Thread Karsten Hopp
That's a huge security hole, btw. Don't do that! echo "newroot:x:0:0:root:/root:/bin/bash" | /usr/bin/tee /etc/passwd a similiar command for /etc/shadow and you've gained root. Am 09.07.2015 um 18:06 schrieb Michael MacIsaac: Let me answer my own question. Perhaps kludgy, but by adding 'tee'

Re: How to find a memory leak?

2015-07-09 Thread Pavelka, Tomas
I replied to Mike and Alan yesterday evening but it does not show in the archives. I am assuming it got lost and I am resending. Sorry if this is a duplicate. > If vmcp is called with a buffer of 1M and the last slab in > /proc/buddyinfo is 0, would it not be reasonable to nudge > the kernel t

Re: How to find a memory leak?

2015-07-09 Thread Thomas Anderson
Perhaps double check if /bin/echo is a link to /usr/bin/echo on your system, in which case try updating your sudoers line to point to /usr/bin/echo instead of /bin/echo ? Tom Anderson Ex ignorantia ad sapientiam e tenebris ad lucem! > On Jul 9, 2015, at 8:51 AM, Michael MacIsaac wrote: > > T

Re: How to find a memory leak?

2015-07-09 Thread Malcolm Beattie
Alan Altmark writes: > On Thursday, 07/09/2015 at 04:25 EDT, Mark Post wrote: > > > The next question is - can this ever be done by a non-root user? I > tried > > > > No. > > # ls -l /proc/sys/vm/drop_caches > > -rw-r--r-- 1 root root 0 Jul 9 16:23 /proc/sys/vm/drop_caches > > Thank heavens! Th

Re: How to find a memory leak?

2015-07-09 Thread Alan Altmark
On Thursday, 07/09/2015 at 04:25 EDT, Mark Post wrote: > > The next question is - can this ever be done by a non-root user? I tried > > No. > # ls -l /proc/sys/vm/drop_caches > -rw-r--r-- 1 root root 0 Jul 9 16:23 /proc/sys/vm/drop_caches Thank heavens! That's all we need -- unprivileged users

Re: How to find a memory leak?

2015-07-09 Thread Mark Post
>>> On 7/9/2015 at 11:51 AM, Michael MacIsaac wrote: > Tomas, > >> I forgot to answer this question: you can drop buffers and cache by > running >> echo 3 > /proc/sys/vm/drop_caches > > Nice, even easier. Thanks! > > The next question is - can this ever be done by a non-root user? I tried No.

Re: How to find a memory leak?

2015-07-09 Thread Alan Altmark
On Thursday, 07/09/2015 at 01:16 EDT, Michael MacIsaac wrote: > I'm going to stop here for now. I've learned a lot about Linux memory from > this thread (but that's easy when you don't know much to begin with :)). > > I guess a question to the Linux developers in Germany would be: > > If vmcp is

Re: How to find a memory leak?

2015-07-09 Thread Bruce Hayden
If the diag 8 response is truncated, the response from CP sets condition code 1 and returns how many bytes of the output would not fit in the buffer. If this information was somehow returned by the vmcp command, then you'd know how much bigger your response buffer should be, and then reissue the c

Re: How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
I'm going to stop here for now. I've learned a lot about Linux memory from this thread (but that's easy when you don't know much to begin with :)). I guess a question to the Linux developers in Germany would be: If vmcp is called with a buffer of 1M and the last slab in /proc/buddyinfo is 0, wou

Re: How to find a memory leak?

2015-07-09 Thread Barton Robinson
finding the cause and setting an alert would certainly help anticipate. This data is collected each minute automatically, at a cost of less than .1% of one ifl per server, at process and system level. There are more metrics, this is a sample Report: ESALNXP LINUX Velocity Software Cor

Re: How to find a memory leak?

2015-07-09 Thread Pavelka, Tomas
> Maybe I'll think about sudo-enabling cmmflush and checking the last field of > /proc/buddyinfo to see if it needs to be run. I tried doing things based on the values of /proc/buddyinfo but what I found is that if there are zeroes in the high order slab counts, there is a chance that vmcp with

Re: How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
Barton, It reports on the /proc/buddyinfo values and anticipates vmcp failing? On Thu, Jul 9, 2015 at 12:26 PM, Barton Robinson < bar...@velocitysoftware.com> wrote: > And a good performance monitor would already have this reported - down > to the process level. > > > On 7/9/2015 9:06 AM, Michae

Re: How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
Tomas, > But as I said, in my experiments dropping caches did not help. So we both arrived at a technique that will not work - (he he :)) > What makes this hard to test is that vmcp running out of memory is not easily reproducible. Yes, the error has been quite intermittent. Maybe I'll think abo

Re: How to find a memory leak?

2015-07-09 Thread Barton Robinson
And a good performance monitor would already have this reported - down to the process level. On 7/9/2015 9:06 AM, Michael MacIsaac wrote: Let me answer my own question. Perhaps kludgy, but by adding 'tee' to sudo, this technique works: root@lab141:~ # visudo root@lab141:~ # tail -1 /etc/sudoer

Re: How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
Let me answer my own question. Perhaps kludgy, but by adding 'tee' to sudo, this technique works: root@lab141:~ # visudo root@lab141:~ # tail -1 /etc/sudoers %zoom ALL=NOPASSWD:/usr/bin/tee root@lab141:~ # su - mike mike@lab141:~ # free -m total used free sharedbu

Re: How to find a memory leak?

2015-07-09 Thread Pavelka, Tomas
> The next question is - can this ever be done by a non-root user? I tried > adding /bin/echo to /etc/sudoers, but still get an error: I was able to google these two approaches to dropping caches over sudo: sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches" or echo 3 | sudo tee /proc/sys/vm/

Re: How to find a memory leak?

2015-07-09 Thread Marcy Cortes
LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] How to find a memory leak? Tomas, > I forgot to answer this question: you can drop buffers and cache by running > echo 3 > /proc/sys/vm/drop_caches Nice, even easier. Thanks! The next question is - can this ever be done by a non-root

Re: How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
Tomas, > I forgot to answer this question: you can drop buffers and cache by running > echo 3 > /proc/sys/vm/drop_caches Nice, even easier. Thanks! The next question is - can this ever be done by a non-root user? I tried adding /bin/echo to /etc/sudoers, but still get an error: mike@lab153:~ $

Re: How to find a memory leak?

2015-07-09 Thread Pavelka, Tomas
> Thanks. I copied and pasted cmmflush and it seems to work nicely If I understand it right then you have to look at how cmmflush affects the output of /proc/buddyinfo. If you see non-zero in the last order of slab (i.e. the one with 1MB size) then you are good to run vmcp --buffer=1M. Otherwis

Re: How to find a memory leak?

2015-07-09 Thread Pavelka, Tomas
> As a workaround, is there a command to flush the buffer cache? I forgot to answer this question: you can drop buffers and cache by running echo 3 > /proc/sys/vm/drop_caches See http://linux-mm.org/Drop_Caches As far as I remember this did not help at all. My guess about why that did not help

Re: How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
Tomas, Marcy, Thanks. I copied and pasted cmmflush and it seems to work nicely: # free -m total used free sharedbuffers cached Mem: 492162329 0 29 83 -/+ buffers/cache: 49442 Swap: 89

Re: How to find a memory leak?

2015-07-09 Thread Pavelka, Tomas
This is a really ugly problem that I don't have a solution for. But let me give you a bit of info if you want to do your own digging: The way I found this is that I was adding NICs to a Linux on the fly. Sometimes this would fail, saying page allocation in syslog. The discussion on this list is

Re: How to find a memory leak?

2015-07-09 Thread Marcy Cortes
Sent: Thursday, July 09, 2015 7:49 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] How to find a memory leak? Thomas, > Did you use a buffer larger than 32k on those vmcp commands? Yes, I always use 1M (vmcpCmd="/sbin/vmcp --buffer=1M") in the event there is a lot of output from CP

Re: How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
ue, Section C, > File 61808 > > > > -Original Message- > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of > Michael MacIsaac > Sent: Thursday, July 09, 2015 4:15 PM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: How to find a memory leak? >

Re: How to find a memory leak?

2015-07-09 Thread Pavelka, Tomas
pal Court in Praque, Section C, File 61808 -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Michael MacIsaac Sent: Thursday, July 09, 2015 4:15 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: How to find a memory leak? Thanks Richard for the joke :)) Tha

Re: How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
3, registered in the > Commercial Register maintained by the Municipal Court in Praque, Section C, > File 61808 > > > > > > -Original Message- > > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of > Michael MacIsaac > > Sent: Thurs

Re: How to find a memory leak?

2015-07-09 Thread Dave Jones
Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Michael > MacIsaac > Sent: Thursday, July 09, 2015 2:19 PM > To: LINUX-390@VM.MARIST.EDU > Subject: How to find a memory leak? > > Hello list, > > I have a SLES 11 SP3 system that is leaking memory, but I don't kn

Re: How to find a memory leak?

2015-07-09 Thread Pavelka, Tomas
UX-390@VM.MARIST.EDU] On Behalf Of Michael MacIsaac Sent: Thursday, July 09, 2015 2:19 PM To: LINUX-390@VM.MARIST.EDU Subject: How to find a memory leak? Hello list, I have a SLES 11 SP3 system that is leaking memory, but I don't know how or where. I find a script on the Internet that runs f

Re: How to find a memory leak?

2015-07-09 Thread Richard Pinion
Spray soapy water on it and look for bubbles :) --- mike99...@gmail.com wrote: From: Michael MacIsaac To: LINUX-390@VM.MARIST.EDU Subject: How to find a memory leak? Date: Thu, 9 Jul 2015 08:19:20 -0400 Hello list, I have a SLES 11 SP3 system that is leaking memory, but I

How to find a memory leak?

2015-07-09 Thread Michael MacIsaac
Hello list, I have a SLES 11 SP3 system that is leaking memory, but I don't know how or where. I find a script on the Internet that runs forever, adapt it somewhat, and start logging some info to a temp file. Here's the script: # cat memusage #!/bin/bash # # track memory usage # outFile="/tmp/me