RE: Huge memory leak, probably related to making new processes
Dave, I take that back - it still might be one from your list. The machine has Symantec antivirus - it was hidden behind the "Show hidden icons". (Doh, I am making too many obvious mistakes.) I'm not an admin, so I cannot check which one - maybe both? The machine doesn't seem to have PC Tool's Spyware doctor - at least judging by Start menu, System tray and Add/Remove programs. -- View this message in context: http://www.nabble.com/Huge-memory-leak%2C-probably-related-to-making-new-processes-tf4557470.html#a13019625 Sent from the Cygwin Users mailing list archive at Nabble.com. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Huge memory leak, probably related to making new processes
On 03 October 2007 14:39, wimxa wrote: > Dave, > > Nope, this one is called Webroot Spy Sweeper (if it is the problem with it). It almost certainly is! Morgan Gangwere mentioned that software before but I forgot to add it to TBLODA; it's definitely going on now. http://cygwin.com/ml/cygwin/2007-09/msg00044.html On the other hand, Dave Silvia once had a similar problem, and decided it wasn't Spy Sweeper but PCTools' Spyware doctor that was the underlying cause; is it possible you have that one installed as well? http://cygwin.com/ml/cygwin/2006-12/msg00347.html cheers, DaveK -- Can't think of a witty .sigline today -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Huge memory leak, probably related to making new processes
Dave, Nope, this one is called Webroot Spy Sweeper (if it is the problem with it). -- View this message in context: http://www.nabble.com/Huge-memory-leak%2C-probably-related-to-making-new-processes-tf4557470.html#a13019192 Sent from the Cygwin Users mailing list archive at Nabble.com. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Huge memory leak, probably related to making new processes
I have just tried this on another machine with Cygwin - no leaks. OK, so it is not Cygwin - my bad, but I'm glad. Just to test, I have made a script for WIndows' cmd.exe: @echo off set c=1 :loop cmd /c echo > nul echo %c% set /a c=%c%+1 if %c%==123456 goto end goto loop :end Leaks... Not so fast as bash, but that is because executing "cmd /c" takes much longer. It leaks approximately the same considering the numbers represented by %c%. So, forking in Windows itself (not Cygwin) it apparently is the whole problem - in fact, as you mentioned - it's probably some other process (AV, whatever) messing up with Windows' normal CreateProcessA, as bash politely explained - and I assume the above script will also - didn't finish yet :(. Sorry for bugging you for something that apparently is not Cygwin-related - should have tested more. -- View this message in context: http://www.nabble.com/Huge-memory-leak%2C-probably-related-to-making-new-processes-tf4557470.html#a13019145 Sent from the Cygwin Users mailing list archive at Nabble.com. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Huge memory leak, probably related to making new processes
On 03 October 2007 13:38, wimxa wrote: > Guys, > > Thank you for the trouble of testing this. Unfortunately, I cannot confirm > whether this is a problem with any AV - I am not an admin on the machine in > question, cannot shut AV down. Is it one of the ones listed on TBLODA at http://cygwin.com/ml/cygwin-talk/2007-q3/msg00174.html ? > About the kernel memory figures - they are increasing - much slower and in a > very unpredictable see-saw way, but they are. In fact, I ran it the second > time after a while (without restarting the computer) and it slowly released > a great deal of the kernel memory it had "allocated" - quite strange > behavior - one time it increases, the next time it decreases - couldn't find > any pattern. This is very strong evidence that a faulty device driver is responsible. > At number 43500 (this must be hard-coded in Cygwin somewhere :)) the > "COUNTER test" (the last one Lewis also tried) died with: > > 11831 [main] bash 3820 fork: child -1 - CreateProcessA failed, errno 11 > bash: fork: Resource temporarily unavailable > > Restarts would allow only at most a few more runs with the same error > following, e.g.: > > 136705676 [main] bash 3820 fork: child -1 - CreateProcessA failed, errno 11 > bash: fork: Resource temporarily unavailable This is one of the very typical symptoms of BLODA interference. cheers, DaveK -- Can't think of a witty .sigline today -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Huge memory leak, probably related to making new processes
Guys, Thank you for the trouble of testing this. Unfortunately, I cannot confirm whether this is a problem with any AV - I am not an admin on the machine in question, cannot shut AV down. About the kernel memory figures - they are increasing - much slower and in a very unpredictable see-saw way, but they are. In fact, I ran it the second time after a while (without restarting the computer) and it slowly released a great deal of the kernel memory it had "allocated" - quite strange behavior - one time it increases, the next time it decreases - couldn't find any pattern. At number 43500 (this must be hard-coded in Cygwin somewhere :)) the "COUNTER test" (the last one Lewis also tried) died with: 11831 [main] bash 3820 fork: child -1 - CreateProcessA failed, errno 11 bash: fork: Resource temporarily unavailable Restarts would allow only at most a few more runs with the same error following, e.g.: 136705676 [main] bash 3820 fork: child -1 - CreateProcessA failed, errno 11 bash: fork: Resource temporarily unavailable I tried to restart Cygwin, but could not run it again (the same kind of error on startup). What I can do is test this on my home machine (yes, I should have done that already - doh) in the evening and let you know if this occurring on it too - I will shut down anything necessary to get the offender if it occurs there. However, I am glad that this is not a Cygwin-only problem, based on two independent tests. Thanks again! -- View this message in context: http://www.nabble.com/Huge-memory-leak%2C-probably-related-to-making-new-processes-tf4557470.html#a13018087 Sent from the Cygwin Users mailing list archive at Nabble.com. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
RE: Huge memory leak, probably related to making new processes
On 03 October 2007 05:31, Christopher Faylor wrote: > On Wed, Oct 03, 2007 at 04:20:18AM +, Lewis Hyatt wrote: >>> Anyway, can I ask you to do this yourself - just do the last test: >>> >>> COUNTER=1 >>> while [ $COUNTER -lt 123456 ]; do (echo $COUNTER); let >>> COUNTER=$COUNTER+1; done >>> >>> and wait a little (couple of minutes). If necessary, repeat it until your >>> memory drops to 10-20 MB range and your HDD should start whining. Then >>> close cygwin and wait 10 minutes. The memory is still "occupied". I don't >>> know when Windows would free it, but I did not get that behavior with any >>> other program (e.g. try to open & close Firefox or such - it will show a >>> peak in both directions regarding memory and will do that almost >>> immediately). That's exactly what I see when I try your testcase. No continuous increase in any of the figures, I just see the commit charge go up a bit each time an echo is launched and down a bit each time it terminates, about two meg variation. The same slight up-and-down variation in the physical memory available/cache figures. No change at all in the kernel memory figures. > I've been waiting for someone to make the observation that Cygwin has no > magic powers which allow it to allocate memory and never release it - > even on process exit. No modern OS allows you to get away with that. > > If the OP is really seeing that then either there is something else on his > system which is responsible for the behavior. I suspect BLODA interference. I'd particularly like the OP to report if there is any leak in the kernel memory figures, because that would certainly point the finger at a dodgy antivirus disk filter device driver or similar. > The other alternative is > misunderstanding of what is going on. I'd say that it was probably > 50/59 which is the case here. That's inflation for ya - 100% aint what it used to be. cheers, DaveK -- Can't think of a witty .sigline today -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Huge memory leak, probably related to making new processes
On Wed, Oct 03, 2007 at 04:20:18AM +, Lewis Hyatt wrote: >> Anyway, can I ask you to do this yourself - just do the last test: >> >> COUNTER=1 >> while [ $COUNTER -lt 123456 ]; do (echo $COUNTER); let COUNTER=$COUNTER+1; >> done >> >> and wait a little (couple of minutes). If necessary, repeat it until your >> memory drops to 10-20 MB range and your HDD should start whining. Then close >> cygwin and wait 10 minutes. The memory is still "occupied". I don't know >> when Windows would free it, but I did not get that behavior with any other >> program (e.g. try to open & close Firefox or such - it will show a peak in >> both directions regarding memory and will do that almost immediately). >> >> Thanks for the note. > >I ran it for about 15 minutes, no problem. > >In general, you should be wary of what Task Manager is telling you. In this >example, it doesn't make sense to me that memory could leak, because each of >the >(echo) processes is executing in a subshell, which terminates after completion. >No matter what a process does in Windows, its memory is always returned when it >terminates, unless it is doing something very unusual. (But, of course, >cygwin's >fork() does do something unusual...) You could try running some simple c++ >program, after it looks like your memory is exhausted, to allocate 1 GB of >memory, and see if it succeeds. When it returns, the OS will probably report >more free memory. This was the basis of some scam-ish "RAM cleaner" programs >that really didn't do anything other than make Task Manager's output look >nicer. > >That said, you are also describing some real symptoms of a problem, such as >exhausting the physical memory and swapping. That should not occur, so there >may >be some problem that's beyond my experience. (In your example, I would have >said >that maybe the disk thrashing you experienced was caused by find itself, but if >you also see it just from running this subshell loop, that is strange.) I guess >it could be something going wrong with fork(), etc, that someone else on the >list might know about. They'll probably ask you to check all the usual suspects >first... are there any firewalls, AV scanners, etc, running in the background? >If so, try getting rid of all of them and see if the problem still occurs. >There >are some poorly written ones that have been known to interfere with cygwin. >Hope >that helps... I've been waiting for someone to make the observation that Cygwin has no magic powers which allow it to allocate memory and never release it - even on process exit. No modern OS allows you to get away with that. If the OP is really seeing that then either there is something else on his system which is responsible for the behavior. The other alternative is misunderstanding of what is going on. I'd say that it was probably 50/59 which is the case here. cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Huge memory leak, probably related to making new processes
> Anyway, can I ask you to do this yourself - just do the last test: > > COUNTER=1 > while [ $COUNTER -lt 123456 ]; do (echo $COUNTER); let COUNTER=$COUNTER+1; > done > > and wait a little (couple of minutes). If necessary, repeat it until your > memory drops to 10-20 MB range and your HDD should start whining. Then close > cygwin and wait 10 minutes. The memory is still "occupied". I don't know > when Windows would free it, but I did not get that behavior with any other > program (e.g. try to open & close Firefox or such - it will show a peak in > both directions regarding memory and will do that almost immediately). > > Thanks for the note. I ran it for about 15 minutes, no problem. In general, you should be wary of what Task Manager is telling you. In this example, it doesn't make sense to me that memory could leak, because each of the (echo) processes is executing in a subshell, which terminates after completion. No matter what a process does in Windows, its memory is always returned when it terminates, unless it is doing something very unusual. (But, of course, cygwin's fork() does do something unusual...) You could try running some simple c++ program, after it looks like your memory is exhausted, to allocate 1 GB of memory, and see if it succeeds. When it returns, the OS will probably report more free memory. This was the basis of some scam-ish "RAM cleaner" programs that really didn't do anything other than make Task Manager's output look nicer. That said, you are also describing some real symptoms of a problem, such as exhausting the physical memory and swapping. That should not occur, so there may be some problem that's beyond my experience. (In your example, I would have said that maybe the disk thrashing you experienced was caused by find itself, but if you also see it just from running this subshell loop, that is strange.) I guess it could be something going wrong with fork(), etc, that someone else on the list might know about. They'll probably ask you to check all the usual suspects first... are there any firewalls, AV scanners, etc, running in the background? If so, try getting rid of all of them and see if the problem still occurs. There are some poorly written ones that have been known to interfere with cygwin. Hope that helps... -Lewis -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Huge memory leak, probably related to making new processes
> How do you know it is leaking memory? If you are looking at Windows Task Manager or some similar program, then you're probably just being misled. The OS will automatically free the memory from each "echo" process after it terminates, but it may not always immediately report it as available. -Lewis < Yes, I am using Windows Task Manager. Actually, it is leaking. I discovered the leak when I had like 10 programs running when I started a simple find just like the ones I posted. I searched a big collection of files (several tens of MB). After a while, I noticed it still didn't finish (I expected it to be done at the time) and that my computer is behaving somewhat strangely. I started closing one program at the time, but that was quite slow and getting worse. Then I noticed that HDD led was on most of the time. I launched Task Manager and it showed that my memory is <15MB (on a 2GB machine). I was assuming (with the programs launched) that around 1GB or so would be free - well, it was not. I restarted the machine, ~1.5GB free. All after that is history (i.e. I performed the tests from my previous post and I got the offender). Does this sound as a reasonable test to you? Anyway, can I ask you to do this yourself - just do the last test: COUNTER=1 while [ $COUNTER -lt 123456 ]; do (echo $COUNTER); let COUNTER=$COUNTER+1; done and wait a little (couple of minutes). If necessary, repeat it until your memory drops to 10-20 MB range and your HDD should start whining. Then close cygwin and wait 10 minutes. The memory is still "occupied". I don't know when Windows would free it, but I did not get that behavior with any other program (e.g. try to open & close Firefox or such - it will show a peak in both directions regarding memory and will do that almost immediately). Thanks for the note. -- View this message in context: http://www.nabble.com/Huge-memory-leak%2C-probably-related-to-making-new-processes-tf4557470.html#a13010714 Sent from the Cygwin Users mailing list archive at Nabble.com. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Huge memory leak, probably related to making new processes
wimxa wrote:
Try executing:
find -exec echo {} \;
Simple command. This one, however, leaks at about 5kB/s. I tried the
following:
How do you know it is leaking memory? If you are looking at Windows Task
Manager or some similar program, then you're probably just being misled.
The OS will automatically free the memory from each "echo" process after
it terminates, but it may not always immediately report it as available.
-Lewis
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
