Re: reproducible cygwin memory problems
ok, let me try again. here is a case in which I try to run latest cvs gnu emacs under gdb and this just immediately freezes x windows. surely someone in the cygwin/ emacs teams would be interested in figuring this out...? ~ $ startx [1] 304 ~ $ Welcome to the XWin X Server Vendor: The Cygwin/X Project Release: 6.8.99.901-4 Contact: cygwin-xfree@cygwin.com XWin was started with the following command line: X :0 -multiwindow -clipboard -emulate3buttons 500 _XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root winValidateArgs - g_iNumScreens: 1 iMaxConsecutiveScreen: 1 (II) XF86Config is not supported (II) See http://x.cygwin.com/docs/faq/cygwin-x-faq.html for more information winDetectSupportedEngines - Windows NT/2000/XP winDetectSupportedEngines - DirectDraw installed winDetectSupportedEngines - DirectDraw4 installed winDetectSupportedEngines - Returning, supported engines 0007 winSetEngine - Multi Window or Rootless = ShadowGDI winAdjustVideoModeShadowGDI - Using Windows display depth of 32 bits per pixel winAllocateFBShadowGDI - Creating DIB with width: 1280 height: 768 depth: 32 winFinishScreenInitFB - Masks: 00ff ff00 00ff winInitVisualsShadowGDI - Masks 00ff ff00 00ff BPRGB 8 d 24 bpp 32 null screen fn ReparentWindow null screen fn RestackWindow InitQueue - Calling pthread_mutex_init InitQueue - pthread_mutex_init returned InitQueue - Calling pthread_cond_init InitQueue - pthread_cond_init returned winInitMultiWindowWM - Hello winInitMultiWindowWM - Calling pthread_mutex_lock () winMultiWindowXMsgProc - Hello winMultiWindowXMsgProc - Calling pthread_mutex_lock () MIT-SHM extension disabled due to lack of kernel support XFree86-Bigfont extension local-client optimization disabled due to lack of shar ed memory support in the kernel (--) Setting autorepeat to delay=500, rate=31 (--) winConfigKeyboard - Layout: 0409 (0409) (--) Using preset keyboard for English (USA) (409), type 4 Rules = xorg Model = pc105 Layout = us Variant = (null) Options = (null ) /usr/local/emacs/src $ /usr/local/emacs/src $ xterm xterm Xt error: Can't open display: xterm: DISPLAY is not set /usr/local/emacs/src $ setenv DISPLAY :0.0 bash: setenv: command not found /usr/local/emacs/src $ export DISPLAY=:0.0 /usr/local/emacs/src $ xterm /usr/local/emacs/src $ gdb emacs.exe GNU gdb 6.5.50.20060706-cvs (cygwin-special) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as i686-pc-cygwin... DISPLAY = :0.0 TERM = cygwin Breakpoint 1 at 0x2009d8d6: file emacs.c, line 464. Breakpoint 2 at 0x200b6e29: file sysdep.c, line 1395. (gdb) run -Q Starting program: /usr/local/emacs/src/emacs.exe -Q Loaded symbols for /cygdrive/c/WINDOWS/system32/ntdll.dll Loaded symbols for /cygdrive/c/WINDOWS/system32/kernel32.dll Loaded symbols for /usr/bin/cygncurses-8.dll Loaded symbols for /usr/bin/cygwin1.dll Loaded symbols for /cygdrive/c/WINDOWS/system32/advapi32.dllid Window parameter) Loaded symbols for /cygdrive/c/WINDOWS/system32/rpcrt4.dll Loaded symbols for /usr/bin/cygjpeg-62.dllas NULL Loaded symbols for /usr/bin/cygpng12.dll was NULL Loaded symbols for /usr/bin/cygz.dlld: The handle is invalid. Loaded symbols for /usr/bin/cygtiff-5.dll Loaded symbols for /usr/bin/cygungif-4.dll handle is invalid. Loaded symbols for /usr/X11R6/bin/cygX11-6.dll Loaded symbols for /usr/X11R6/bin/cygICE-6.dll2, last bpp: 32, new bpp: 32 Loaded symbols for /usr/X11R6/bin/cygSM-6.dll 1280 new height: 768 Loaded symbols for /usr/X11R6/bin/cygXaw3d-7.dll last bpp: 32, new bpp: 32 Loaded symbols for /usr/X11R6/bin/cygXext-6.dll280 new height: 768 Loaded symbols for /usr/X11R6/bin/cygXmu-6.dlldle is invalid. Loaded symbols for /usr/X11R6/bin/cygXt-6.dll Loaded symbols for /usr/X11R6/bin/cygXpm-4.dll2, last bpp: 32, new bpp: 32 winWindowProc - WM_DISPLAYCHANGE - new width: 1280 new height: 768 Program received signal SIGSEGV, Segmentation fault.t bpp: 32, new bpp: 32 [Switching to thread 5892.0x1520]- new width: 1280 new height: 768 0x in ?? ()DISPLAYCHANGE - orig bpp: 32, last bpp: 32, new bpp: 32 (gdb) xinit: connection to X server lost. xterm: fatal IO error 104 (Connection reset by peer) or KillClient on X server :0.0dowProc - WM_DISPLAYCHANGE - new width: 1280 new height: 768 xterm: fatal IO error 104 (Connection reset by peer) or KillClient on X server :0.0dowProc - WM_DISPLAYCHANGE - new width: 1280 new height: 768 xterm: fatal IO error 104 (Connection reset by peer) or KillClient on X server :0.0dowProc - WM_DISPLAYCHANGE - new width: 1280 new height: 768 winMultiWindowXMsgProcErrorHandler - ERROR: BadWindow (invalid Window parameter)
Re: reproducible cygwin memory problems
Date: Mon, 14 Aug 2006 08:58:12 +0200 From: Corinna Vinschen [EMAIL PROTECTED] As far as I could tell from the OP's report about Emacs configuration results, available here: http://www.deas.harvard.edu/climate/eli/emacs-debug/ Emacs he built does not use mmap. Here's the relevant excerpt: Should Emacs use the GNU version of malloc? yes Should Emacs use a relocating allocator for buffers?yes Should Emacs use mmap(2) for buffer allocation? no So it looks like the above hypothesis about mmap is not relevant, and You missed what I told about Doug Lea's malloc implementation. It uses mmap for mappings 256K. It does not matter whether emacs is configured for using mmap or not. Right. Sorry. Cygwin uses the Doug Lea version of malloc/free. The latest Cygwin versions uses the DEFAULT_MMAP_THRESHOLD of 256K, so mmap/munmap are used for mappings 256K. I assume that's the case here, since we're talking about images. Otherwise, for mallocs 256K, Cygwin is using sbrk and sbrk'ed memory is never returned to the OS. seems to say that the fact that memory is not returned to the OS is a known issue with Cygwin's malloc when sbrk is used. Am I missing something? There is no known issue. The heap is an area of reserved memory, which is on demand commited when sbrk is called. If the heap is too small to fit the new allocation, more memory is reserved/commited. When sbrk is called with negative value, memory is decommited, but it's not released. Maybe that's why the OP doesn't see a decrease in the memory footprint. However, this works as designed. AFAIK, decommitting memory does decrease its memory footprint as shown by the task manager. The emulation of sbrk used by the native Windows port of Emacs does the same as what you described: it calls VirtualFree with MEM_DECOMMIT flag (see w32heap.c:sbrk), and I do see decrease in memory footprint when a large buffer is killed. So, at least in theory, the Cygwin build of Emacs should exhibit the same behavior. Sbrk resp. munmap are fairly basic operations. I explained how they work or are supposed to work. If they don't work for emacs/lisp, they don't work for hundreds of applications and it should be quite easy to come up with a simple testcase in C. So far I didn't see one, and my local tests work as expected. FWIW, I looked into the Emacs image-support code, and I don't see any memory allocation when it loads and displays an image, except the allocation that happens inside the external image libraries used for image support (libtiff.dll, jpeg62.dll, etc.). So what's been said earlier in this thread -- that Emacs allocates memory for images, but doesn't release all of it -- sounds less and less likely, especially since Cygwin uses the same C code as any other Posix platform, including GNU/Linux. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
On Aug 18 14:40, Eli Zaretskii wrote:
From: Corinna Vinschen [EMAIL PROTECTED]
There is no known issue. The heap is an area of reserved memory, which
is on demand commited when sbrk is called. If the heap is too small to
fit the new allocation, more memory is reserved/commited. When sbrk is
called with negative value, memory is decommited, but it's not released.
Maybe that's why the OP doesn't see a decrease in the memory footprint.
However, this works as designed.
AFAIK, decommitting memory does decrease its memory footprint as shown
by the task manager. The emulation of sbrk used by the native Windows
port of Emacs does the same as what you described: it calls
VirtualFree with MEM_DECOMMIT flag (see w32heap.c:sbrk), and I do see
decrease in memory footprint when a large buffer is killed. So, at
least in theory, the Cygwin build of Emacs should exhibit the same
behavior.
Yes, right, sorry for jumping to a conclusion.
Sbrk resp. munmap are fairly basic operations. I explained how they
work or are supposed to work. If they don't work for emacs/lisp, they
don't work for hundreds of applications and it should be quite easy to
come up with a simple testcase in C. So far I didn't see one, and my
local tests work as expected.
FWIW, I looked into the Emacs image-support code, and I don't see any
memory allocation when it loads and displays an image, except the
allocation that happens inside the external image libraries used for
image support (libtiff.dll, jpeg62.dll, etc.). So what's been said
earlier in this thread -- that Emacs allocates memory for images, but
doesn't release all of it -- sounds less and less likely, especially
since Cygwin uses the same C code as any other Posix platform,
including GNU/Linux.
Well, since a testcase is still missing, I wrote my own.
SNIP
#include stdio.h
#include stdlib.h
#include malloc.h
#define COUNT 320
int
main (int argc, char **argv)
{
char *p[COUNT];
/* Use sbrk. Set argv[1] to a value 256K to use mmap/munmap. */
size_t size = 20;
int i;
if (argc 1)
size = strtoul (argv[1], NULL, 0);
#if 1
/* This early getchar is necessary because otherwise the next
getchar, called after the malloc calls, would malloc another 1K
chunk. This chunk is then at the top of the sbrk'ed heap and
would in turn disable the memory trimming in free. In other
words, sbrk with negative value could never be called.
Consequentially, disable this getchar call and you'll see how the
free calls never call sbrk. However, the memory would still be
free'd if argv[1] 256K, since munmap works even if getchar
allocates sbrk'ed memory at the top of the heap. */
fprintf (stderr, Press key...\n);
getchar ();
#endif
while (1)
{
for (i = 0; i COUNT; ++i)
{
p[i] = malloc (size);
if (!p[i])
fprintf(stderr, Couldn't allocate %lu bytes\n, size);
else
fprintf(stderr, allocate %lu bytes at %p\n, size, p[i]);
}
getchar ();
for (i = 0; i COUNT; ++i)
{
free (p[COUNT - i - 1]);
fprintf(stderr, freed %lu bytes at %p\n, size, p[COUNT - i - 1]);
}
getchar ();
}
return 0;
}
SNAP
Try this on Cygwin and open a task manager window in parallel. After
the mallocs have been called, you'll see how the memory is free'd
again, which proves that sbrk with negative value is called and works
as expected. Use argv[1] to set the chunk size to a value 256K to
test the mmap/munmap implementation. You'll see in task manager that
the result is again as expected, free'ing OS memory.
Now, disable the first call to getchar, and observe how an unlucky
placed call to malloc, resp. an unlucky placed library call which also
malloc's memory, can spoil all the efforts.
Now the testcase is available and still, I don't see anything unusual.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
From: emacs user [EMAIL PROTECTED] Cc: cygwin@cygwin.com, emacs-pretest-bug@gnu.org, [EMAIL PROTECTED] Bcc: Date: Mon, 14 Aug 2006 08:11:25 -0400 well, as you probably remember one cannot run cvs emacs in gdb as this immediately crashes x windows in cygwin. No, I didn't remember. Why would that crash X? -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
On Aug 13 22:11, Eli Zaretskii wrote: Date: Sun, 13 Aug 2006 11:23:16 +0200 From: Reini Urban [EMAIL PROTECTED] Corinna Vinschen schrieb: ... Ok, back to mmap/munmap. Private anonymous mappings are implemented in Cygwin by calling VirtualAlloc/VirtualFree. However, due to restrictions in the Win32 API, VirtualFree is only called by munmap, as soon as *all* pages within a single mmap'ed area are free'ed by this or preceding calls to munmap. In other words, the mmap'ed memory will not be returned to the OS, if not the complete mmap'ed area is munmap'ed. Therefore I assume that the implementation in Lisp keeps bits of memory around which are not munmap'ed for some internal reason. As far as I could tell from the OP's report about Emacs configuration results, available here: http://www.deas.harvard.edu/climate/eli/emacs-debug/ Emacs he built does not use mmap. Here's the relevant excerpt: Should Emacs use the GNU version of malloc? yes Should Emacs use a relocating allocator for buffers?yes Should Emacs use mmap(2) for buffer allocation? no So it looks like the above hypothesis about mmap is not relevant, and You missed what I told about Doug Lea's malloc implementation. It uses mmap for mappings 256K. It does not matter whether emacs is configured for using mmap or not. this portion of Corinna's message: Cygwin uses the Doug Lea version of malloc/free. The latest Cygwin versions uses the DEFAULT_MMAP_THRESHOLD of 256K, so mmap/munmap are used for mappings 256K. I assume that's the case here, since we're talking about images. Otherwise, for mallocs 256K, Cygwin is using sbrk and sbrk'ed memory is never returned to the OS. seems to say that the fact that memory is not returned to the OS is a known issue with Cygwin's malloc when sbrk is used. Am I missing something? There is no known issue. The heap is an area of reserved memory, which is on demand commited when sbrk is called. If the heap is too small to fit the new allocation, more memory is reserved/commited. When sbrk is called with negative value, memory is decommited, but it's not released. Maybe that's why the OP doesn't see a decrease in the memory footprint. However, this works as designed. Sbrk resp. munmap are fairly basic operations. I explained how they work or are supposed to work. If they don't work for emacs/lisp, they don't work for hundreds of applications and it should be quite easy to come up with a simple testcase in C. So far I didn't see one, and my local tests work as expected. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
On Aug 13 11:23, Reini Urban wrote: Corinna Vinschen schrieb: ... Ok, back to mmap/munmap. Private anonymous mappings are implemented in Cygwin by calling VirtualAlloc/VirtualFree. However, due to restrictions in the Win32 API, VirtualFree is only called by munmap, as soon as *all* pages within a single mmap'ed area are free'ed by this or preceding calls to munmap. In other words, the mmap'ed memory will not be returned to the OS, if not the complete mmap'ed area is munmap'ed. Therefore I assume that the implementation in Lisp keeps bits of memory around which are not munmap'ed for some internal reason. Of course it's also possible that there's a bug in Cygwin related to the way mmap/munmap is called. But I have so far no evidence for that and simple mmap/munmap tests show that VirtualFree is called as soon as all pages in a mmap'ed area have been munmap'ed. If somebody thinks there's a bug in this implementation, please send a simple, self-contained testcase in pure C source code, which allows to reproduce the problem. Of course, patches are welcome, too :) And there's the known mmap() under cygwin limitation, esp. harmful to clisp, that mmap() can only be properly aligned to a base address modulo 64k (windows allocation granularity), and not the usual pagesize, if it's 4k or 8k. windows pagesize is still 4k though. http://www.cygwin.com/ml/cygwin/2000-09/msg00380.html http://www.cygwin.com/ml/cygwin/2004-09/msg00741.html This is a non-issue since 1.5.19: http://cygwin.com/ml/cygwin-announce/2006-01/msg00016.html Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
[Sorry, I forgot all the CCs] On Aug 13 22:11, Eli Zaretskii wrote: Date: Sun, 13 Aug 2006 11:23:16 +0200 From: Reini Urban [EMAIL PROTECTED] Corinna Vinschen schrieb: ... Ok, back to mmap/munmap. Private anonymous mappings are implemented in Cygwin by calling VirtualAlloc/VirtualFree. However, due to restrictions in the Win32 API, VirtualFree is only called by munmap, as soon as *all* pages within a single mmap'ed area are free'ed by this or preceding calls to munmap. In other words, the mmap'ed memory will not be returned to the OS, if not the complete mmap'ed area is munmap'ed. Therefore I assume that the implementation in Lisp keeps bits of memory around which are not munmap'ed for some internal reason. As far as I could tell from the OP's report about Emacs configuration results, available here: http://www.deas.harvard.edu/climate/eli/emacs-debug/ Emacs he built does not use mmap. Here's the relevant excerpt: Should Emacs use the GNU version of malloc? yes Should Emacs use a relocating allocator for buffers?yes Should Emacs use mmap(2) for buffer allocation? no So it looks like the above hypothesis about mmap is not relevant, and You missed what I told about Doug Lea's malloc implementation. It uses mmap for mappings 256K. It does not matter whether emacs is configured for using mmap or not. this portion of Corinna's message: Cygwin uses the Doug Lea version of malloc/free. The latest Cygwin versions uses the DEFAULT_MMAP_THRESHOLD of 256K, so mmap/munmap are used for mappings 256K. I assume that's the case here, since we're talking about images. Otherwise, for mallocs 256K, Cygwin is using sbrk and sbrk'ed memory is never returned to the OS. seems to say that the fact that memory is not returned to the OS is a known issue with Cygwin's malloc when sbrk is used. Am I missing something? There is no known issue. The heap is an area of reserved memory, which is on demand commited when sbrk is called. If the heap is too small to fit the new allocation, more memory is reserved/commited. When sbrk is called with negative value, memory is decommited, but it's not released. Maybe that's why the OP doesn't see a decrease in the memory footprint. However, this works as designed. Sbrk resp. munmap are fairly basic operations. I explained how they work or are supposed to work. If they don't work for emacs/lisp, they don't work for hundreds of applications and it should be quite easy to come up with a simple testcase in C. So far I didn't see one, and my local tests work as expected. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
could you please write which .c files and which routines should I be looking at slightly more specifically? thanks... These are library functions, so no .c files are involved. Just (gdb) break sbrk (gdb) break mmap then visit text and image files as suggested and see which of these breakpoint actually breaks. well, as you probably remember one cannot run cvs emacs in gdb as this immediately crashes x windows in cygwin. I do wish there would be someone more knowledgeble then me doing the testing with cygwin, now that we've made some progress. I am worried that I may be missing something elementary and waisting your time. installing cygwin and then cvs gnu emacs would take a few minutes only... anyway, am happy to keep trying. _ Dont just search. Find. Check out the new MSN Search! http://search.msn.click-url.com/go/onm00200636ave/direct/01/ -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
Corinna Vinschen schrieb: ... Ok, back to mmap/munmap. Private anonymous mappings are implemented in Cygwin by calling VirtualAlloc/VirtualFree. However, due to restrictions in the Win32 API, VirtualFree is only called by munmap, as soon as *all* pages within a single mmap'ed area are free'ed by this or preceding calls to munmap. In other words, the mmap'ed memory will not be returned to the OS, if not the complete mmap'ed area is munmap'ed. Therefore I assume that the implementation in Lisp keeps bits of memory around which are not munmap'ed for some internal reason. Of course it's also possible that there's a bug in Cygwin related to the way mmap/munmap is called. But I have so far no evidence for that and simple mmap/munmap tests show that VirtualFree is called as soon as all pages in a mmap'ed area have been munmap'ed. If somebody thinks there's a bug in this implementation, please send a simple, self-contained testcase in pure C source code, which allows to reproduce the problem. Of course, patches are welcome, too :) And there's the known mmap() under cygwin limitation, esp. harmful to clisp, that mmap() can only be properly aligned to a base address modulo 64k (windows allocation granularity), and not the usual pagesize, if it's 4k or 8k. windows pagesize is still 4k though. http://www.cygwin.com/ml/cygwin/2000-09/msg00380.html http://www.cygwin.com/ml/cygwin/2004-09/msg00741.html This has nothing directly to do with the possible xemacs munmap() problem, but maybe there's a logic flaw somewhere. -- Reini Urban http://phpwiki.org/ http://murbreak.at/ http://helsinki.at/ http://spacemovie.mur.at/ -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
Date: Sun, 13 Aug 2006 11:23:16 +0200 From: Reini Urban [EMAIL PROTECTED] Corinna Vinschen schrieb: ... Ok, back to mmap/munmap. Private anonymous mappings are implemented in Cygwin by calling VirtualAlloc/VirtualFree. However, due to restrictions in the Win32 API, VirtualFree is only called by munmap, as soon as *all* pages within a single mmap'ed area are free'ed by this or preceding calls to munmap. In other words, the mmap'ed memory will not be returned to the OS, if not the complete mmap'ed area is munmap'ed. Therefore I assume that the implementation in Lisp keeps bits of memory around which are not munmap'ed for some internal reason. As far as I could tell from the OP's report about Emacs configuration results, available here: http://www.deas.harvard.edu/climate/eli/emacs-debug/ Emacs he built does not use mmap. Here's the relevant excerpt: Should Emacs use the GNU version of malloc? yes Should Emacs use a relocating allocator for buffers?yes Should Emacs use mmap(2) for buffer allocation? no So it looks like the above hypothesis about mmap is not relevant, and this portion of Corinna's message: Cygwin uses the Doug Lea version of malloc/free. The latest Cygwin versions uses the DEFAULT_MMAP_THRESHOLD of 256K, so mmap/munmap are used for mappings 256K. I assume that's the case here, since we're talking about images. Otherwise, for mallocs 256K, Cygwin is using sbrk and sbrk'ed memory is never returned to the OS. seems to say that the fact that memory is not returned to the OS is a known issue with Cygwin's malloc when sbrk is used. Am I missing something? If I don't miss anything, it remains to be explained why the OP didn't see problems with normal buffers. Could the OP please try visiting several large text files, then kill their buffers, invoke the `garbage-collect' function (M-x garbage-collect RET), and see if the memory footprint returns to its original value? A few more tests come to mind: . Put a breakpoint in sbrk and in mmap, and find out which one is being called when normal text files and image files (both small and large) are visited. . Visit many small image files (smaller than 256K) and see whether the memory is being returned to the OS. I'd like to ask the OP to try these and report the results. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
From: Eli Zaretskii [EMAIL PROTECTED] If I don't miss anything, it remains to be explained why the OP didn't see problems with normal buffers. Could the OP please try visiting several large text files, then kill their buffers, invoke the `garbage-collect' function (M-x garbage-collect RET), and see if the memory footprint returns to its original value? starting emacs -Q: 12,832k editing a large test file (43,085,936), emacs memory goes up to 55,028k killing : emacs uses 13,070 and M-x garbage-collect 13,100 or so, so far it seems that tmemory is returned. however, using emacs a bit more, opening and killing buffers, eventually does lead to memory increase even after I kill all buffers and do garbage-collect. A few more tests come to mind: . Put a breakpoint in sbrk and in mmap, and find out which one is being called when normal text files and image files (both small and large) are visited. could you please write which .c files and which routines should I be looking at slightly more specifically? thanks... . Visit many small image files (smaller than 256K) and see whether the memory is being returned to the OS. I'd like to ask the OP to try these and report the results. _ On the road to retirement? Check out MSN Life Events for advice on how to get there! http://lifeevents.msn.com/category.aspx?cid=Retirement -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
From: emacs user [EMAIL PROTECTED] Bcc: Date: Sun, 13 Aug 2006 17:32:38 -0400 From: Eli Zaretskii [EMAIL PROTECTED] If I don't miss anything, it remains to be explained why the OP didn't see problems with normal buffers. Could the OP please try visiting several large text files, then kill their buffers, invoke the `garbage-collect' function (M-x garbage-collect RET), and see if the memory footprint returns to its original value? starting emacs -Q: 12,832k editing a large test file (43,085,936), emacs memory goes up to 55,028k killing : emacs uses 13,070 and M-x garbage-collect 13,100 or so, so far it seems that tmemory is returned. Yes. however, using emacs a bit more, opening and killing buffers, eventually does lead to memory increase even after I kill all buffers and do garbage-collect. Assuming that ``using a bit more'' loads some Lisp packages, this is normal. Also, font-lock, which works in the background, adds text properties to buffers, which also uses up some memory. A few more tests come to mind: . Put a breakpoint in sbrk and in mmap, and find out which one is being called when normal text files and image files (both small and large) are visited. could you please write which .c files and which routines should I be looking at slightly more specifically? thanks... These are library functions, so no .c files are involved. Just (gdb) break sbrk (gdb) break mmap then visit text and image files as suggested and see which of these breakpoint actually breaks. . Visit many small image files (smaller than 256K) and see whether the memory is being returned to the OS. Same here. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
emacs user schrieb: Dear cygwin friends, while debugging some emacs related problem, we seem to come to the conclusion that there is a cygwin issue here. can someone please comment on this? From: Eli Zaretskii [EMAIL PROTECTED] From: emacs user [EMAIL PROTECTED] Cc: emacs-pretest-bug@gnu.org Bcc: Date: Thu, 10 Aug 2006 01:06:22 -0400 #kill it #(clear-image-cache t) I am here1 I am here2 I am here4 I am here5 I am here6 I am here5 I am here6 Okay, thanks. This shows that free_image does get called. What image types did you try till now? If you tried with only one image type, please see if other image types exhibit the same problem (i.e. that memory is not returned to system when an image is killed and the image cache cleared). problem occurs with both jpeg and gif. I didnt try others If you already tried all the supported image types, I cannot think of any reason but one: that the Cygwin implementation of malloc/free does not return freed memory to the OS. If this is the conclusion, then it should probably be taken up with Cygwin maintainers on their mailing list. Look Eli, that's really a bold statement, esp. when elisp is involved. cygwin uses the newlib libc. How does it come to your mind that newlib's free() and subsequently Microsoft's VirtualFree (,MEM_RELEASE) doesn't work as advertised? Please see the src for newlib's free() at src/newlib/libc/stdlib/mallocr.c http://cygwin.com/cgi-bin/cvsweb.cgi/src/newlib/libc/stdlib/mallocr.c?rev=1.15content-type=text/x-cvsweb-markupcvsroot=src POSIX free() is not like perl or lisp, where there's no such concept of free(), if by lazyness or better strategies. POSIX free() and cygwin's free() does give back it's malloc'ed buffers back to the OS, and the OS does list them correctly as free. What you have here is an ELISP issue, which doesn't free it's interim's memory, and you falsely accuse the C part of the image library. Please have a look again in some lisp allocated buffers and then check if some free() is missing. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
Eli, here is a response from the cygwin list. thanks to Reini Urban for this. Date: Sat, 12 Aug 2006 11:32:01 +0200 From: Reini Urban rurban at x-ray dot at To: cygwin at cygwin dot com Subject: Re: reproducible cygwin memory problems Sender: cygwin-owner at cygwin dot com emacs user schrieb: Dear cygwin friends, while debugging some emacs related problem, we seem to come to the conclusion that there is a cygwin issue here. can someone please comment on this? From: Eli Zaretskii [EMAIL PROTECTED] From: emacs user [EMAIL PROTECTED] Cc: emacs-pretest-bug@gnu.org Bcc: Date: Thu, 10 Aug 2006 01:06:22 -0400 #kill it #(clear-image-cache t) I am here1 I am here2 I am here4 I am here5 I am here6 I am here5 I am here6 Okay, thanks. This shows that free_image does get called. What image types did you try till now? If you tried with only one image type, please see if other image types exhibit the same problem (i.e. that memory is not returned to system when an image is killed and the image cache cleared). problem occurs with both jpeg and gif. I didnt try others If you already tried all the supported image types, I cannot think of any reason but one: that the Cygwin implementation of malloc/free does not return freed memory to the OS. If this is the conclusion, then it should probably be taken up with Cygwin maintainers on their mailing list. Look Eli, that's really a bold statement, esp. when elisp is involved. cygwin uses the newlib libc. How does it come to your mind that newlib's free() and subsequently Microsoft's VirtualFree (,MEM_RELEASE) doesn't work as advertised? Please see the src for newlib's free() at src/newlib/libc/stdlib/mallocr.c http://cygwin.com/cgi-bin/cvsweb.cgi/src/newlib/libc/stdlib/mallocr.c?rev=1.15content-type=text/x-cvsweb-markupcvsroot=src POSIX free() is not like perl or lisp, where there's no such concept of free(), if by lazyness or better strategies. POSIX free() and cygwin's free() does give back it's malloc'ed buffers back to the OS, and the OS does list them correctly as free. What you have here is an ELISP issue, which doesn't free it's interim's memory, and you falsely accuse the C part of the image library. Please have a look again in some lisp allocated buffers and then check if some free() is missing. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ _ Dont just search. Find. Check out the new MSN Search! http://search.msn.click-url.com/go/onm00200636ave/direct/01/ -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
From: Eli Zaretskii [EMAIL PROTECTED] [I'm not subscribed to the Cygwin list, so please CC me directly.] From: emacs user [EMAIL PROTECTED] Bcc: Date: Sat, 12 Aug 2006 07:46:33 -0400 Cc: cygwin@cygwin.com Eli, here is a response from the cygwin list. thanks to Reini Urban for this. Look Eli, that's really a bold statement, esp. when elisp is involved. cygwin uses the newlib libc. How does it come to your mind that newlib's free() and subsequently Microsoft's VirtualFree (,MEM_RELEASE) doesn't work as advertised? Please see the src for newlib's free() at src/newlib/libc/stdlib/mallocr.c http://cygwin.com/cgi-bin/cvsweb.cgi/src/newlib/libc/stdlib/mallocr.c?rev=1.15content-type=text/x-cvsweb-markupcvsroot=src POSIX free() is not like perl or lisp, where there's no such concept of free(), if by lazyness or better strategies. POSIX free() and cygwin's free() does give back it's malloc'ed buffers back to the OS, and the OS does list them correctly as free. What you have here is an ELISP issue, which doesn't free it's interim's memory, and you falsely accuse the C part of the image library. Please have a look again in some lisp allocated buffers and then check if some free() is missing. It's possible that I'm wrong, of course. Let me state the facts and observations that led me to my conclusion: . The OP reported that the problem happens only with image files. When he edits text files, the Emacs memory footprint stays at a reasonable value. . After killing the image buffer, the OP invoked the Lisp function clear-image-cache, which is supposed to release any memory used by Emacs for storing images. There's no other Lisp memory that I can see in the sources that is used for images, and that could be left unreleased. To be sure, I'd like to ask [EMAIL PROTECTED], who is the OP, to invoke M-x garbage-collect RET after clear-image-cache, and see if that changes anything. tried that, no effect on memory usage. I used a fresh copy of latest (as of right now) gnu emacs from cvs, latest cygwin, after make distclean, configure, make. . The OP traced the Emacs code in clear-image-cache, and saw that the function in the image libraries that frees the image is indeed called. . Other Emacs builds, including the native Windows build and a GNU/Linux build, do not exhibit this behavior. Given these observations, and the fact that the Cygwin build of Emacs runs almost the same C and Lisp code as the GNU/Linux build, what other factors can be involved here? _ Is your PC infected? Get a FREE online computer virus scan from McAfee® Security. http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963 -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
On Aug 12 17:52, Eli Zaretskii wrote: It's possible that I'm wrong, of course. Let me state the facts and observations that led me to my conclusion: . The OP reported that the problem happens only with image files. When he edits text files, the Emacs memory footprint stays at a reasonable value. . After killing the image buffer, the OP invoked the Lisp function clear-image-cache, which is supposed to release any memory used by Emacs for storing images. There's no other Lisp memory that I can see in the sources that is used for images, and that could be left unreleased. To be sure, I'd like to ask [EMAIL PROTECTED], who is the OP, to invoke M-x garbage-collect RET after clear-image-cache, and see if that changes anything. . The OP traced the Emacs code in clear-image-cache, and saw that the function in the image libraries that frees the image is indeed called. . Other Emacs builds, including the native Windows build and a GNU/Linux build, do not exhibit this behavior. Given these observations, and the fact that the Cygwin build of Emacs runs almost the same C and Lisp code as the GNU/Linux build, what other factors can be involved here? Cygwin uses the Doug Lea version of malloc/free. The latest Cygwin versions uses the DEFAULT_MMAP_THRESHOLD of 256K, so mmap/munmap are used for mappings 256K. I assume that's the case here, since we're talking about images. Otherwise, for mallocs 256K, Cygwin is using sbrk and sbrk'ed memory is never returned to the OS. Ok, back to mmap/munmap. Private anonymous mappings are implemented in Cygwin by calling VirtualAlloc/VirtualFree. However, due to restrictions in the Win32 API, VirtualFree is only called by munmap, as soon as *all* pages within a single mmap'ed area are free'ed by this or preceding calls to munmap. In other words, the mmap'ed memory will not be returned to the OS, if not the complete mmap'ed area is munmap'ed. Therefore I assume that the implementation in Lisp keeps bits of memory around which are not munmap'ed for some internal reason. Of course it's also possible that there's a bug in Cygwin related to the way mmap/munmap is called. But I have so far no evidence for that and simple mmap/munmap tests show that VirtualFree is called as soon as all pages in a mmap'ed area have been munmap'ed. If somebody thinks there's a bug in this implementation, please send a simple, self-contained testcase in pure C source code, which allows to reproduce the problem. Of course, patches are welcome, too :) Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
[I'm not subscribed to the Cygwin list, so please CC me directly.] From: emacs user [EMAIL PROTECTED] Bcc: Date: Sat, 12 Aug 2006 07:46:33 -0400 Cc: cygwin@cygwin.com Eli, here is a response from the cygwin list. thanks to Reini Urban for this. Look Eli, that's really a bold statement, esp. when elisp is involved. cygwin uses the newlib libc. How does it come to your mind that newlib's free() and subsequently Microsoft's VirtualFree (,MEM_RELEASE) doesn't work as advertised? Please see the src for newlib's free() at src/newlib/libc/stdlib/mallocr.c http://cygwin.com/cgi-bin/cvsweb.cgi/src/newlib/libc/stdlib/mallocr.c?rev=1.15content-type=text/x-cvsweb-markupcvsroot=src POSIX free() is not like perl or lisp, where there's no such concept of free(), if by lazyness or better strategies. POSIX free() and cygwin's free() does give back it's malloc'ed buffers back to the OS, and the OS does list them correctly as free. What you have here is an ELISP issue, which doesn't free it's interim's memory, and you falsely accuse the C part of the image library. Please have a look again in some lisp allocated buffers and then check if some free() is missing. It's possible that I'm wrong, of course. Let me state the facts and observations that led me to my conclusion: . The OP reported that the problem happens only with image files. When he edits text files, the Emacs memory footprint stays at a reasonable value. . After killing the image buffer, the OP invoked the Lisp function clear-image-cache, which is supposed to release any memory used by Emacs for storing images. There's no other Lisp memory that I can see in the sources that is used for images, and that could be left unreleased. To be sure, I'd like to ask [EMAIL PROTECTED], who is the OP, to invoke M-x garbage-collect RET after clear-image-cache, and see if that changes anything. . The OP traced the Emacs code in clear-image-cache, and saw that the function in the image libraries that frees the image is indeed called. . Other Emacs builds, including the native Windows build and a GNU/Linux build, do not exhibit this behavior. Given these observations, and the fact that the Cygwin build of Emacs runs almost the same C and Lisp code as the GNU/Linux build, what other factors can be involved here? -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: reproducible cygwin memory problems
Dear cygwin friends, while debugging some emacs related problem, we seem to come to the conclusion that there is a cygwin issue here. can someone please comment on this? From: Eli Zaretskii [EMAIL PROTECTED] From: emacs user [EMAIL PROTECTED] Cc: emacs-pretest-bug@gnu.org Bcc: Date: Thu, 10 Aug 2006 01:06:22 -0400 #kill it #(clear-image-cache t) I am here1 I am here2 I am here4 I am here5 I am here6 I am here5 I am here6 Okay, thanks. This shows that free_image does get called. What image types did you try till now? If you tried with only one image type, please see if other image types exhibit the same problem (i.e. that memory is not returned to system when an image is killed and the image cache cleared). problem occurs with both jpeg and gif. I didnt try others If you already tried all the supported image types, I cannot think of any reason but one: that the Cygwin implementation of malloc/free does not return freed memory to the OS. If this is the conclusion, then it should probably be taken up with Cygwin maintainers on their mailing list. cygwin list copied above _ Express yourself instantly with MSN Messenger! Download today - it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
Re: Cygwin memory problems
At 04:32 PM 2/6/2002, David Guaspari wrote: A web search turned up a brief (2-message) thread about memory problems using cygwin. Was there any resolution? Hard to say since it's not clear what your memory problem is or which reference you found on the list. I'm going to guess you're referencing this thread: http://sources.redhat.com/ml/cygwin/2001-02/msg01627.html If that's the case, the reply I gave to Chris Althouse is the only one I can give to you. Certainly more details are necessary to make a more specific diagnosis. Perhaps Chris can enlighten you and the rest of us with the results of his debugging session. I'm trying to run a 350Kbyte executable on a machine with 512Meg of RAM and get the message Program too big to fit in memory. I just installed the latest cygwin today. Perhaps the output of cygcheck -s -r -v might help, along with some details of what the problems are that you're seeing. Sorry, that's all I can offer at this point. Larry Hall [EMAIL PROTECTED] RFK Partners, Inc. http://www.rfk.com 838 Washington Street (508) 893-9779 - RFK Office Holliston, MA 01746 (508) 893-9889 - FAX -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/
