Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6?
Am Mittwoch, den 23.02.2005, 12:28 +0900 schrieb Horms: On Fri, Feb 18, 2005 at 03:33:30PM +0100, Ingo Strüwing wrote: Am Donnerstag, den 10.02.2005, 18:28 +0900 schrieb Horms: On Thu, Feb 10, 2005 at 09:07:00AM +0100, Ingo Strüwing wrote: Am Donnerstag, den 10.02.2005, 10:33 +0900 schrieb Horms: On Wed, Feb 09, 2005 at 04:50:49PM +0100, Ingo Strüwing wrote: I have additional information: - The problem exists in 2.6.10 too. (BTW. should I report it there too?) - The problem exists on ext3 filesystems too. - The problem does not exist in 2.4.27. Thanks, all good information. You should probably just reasign the bug to kernel-source-2.6.10 as 2.6.9 is slowly on its way out of the debian tree. What does 'reasign' mean? Is there a way to move a report from one packet to another? Or is it just reporting it again against 2.6.10? Yes, you can move the bug. http://www.debian.org/Bugs/server-control Or someone else (like say, me) can do it for you. Yes. Please. I would appreciate if you would move it to either 2.6.10 or 2.6.8. Wichever has the higher chances to solve it. Also any feedback on 2.6.8 would be good, that is the target for sarge, 2.6.10 is just in sid for people to experiment and so we can see what has been fixed upstream when problems crop up in 2.6.8. I tested this now successfully on 2.6.8-13 and 2.6.10-5. Does that mean the bug has been resolved and can be closed? No. Of course not. The quotes around successfully mean that it is not really a success, but that I was successful in reproducing the problem. I hoped that this was clear in conjuction with my sentence 9 lines above that one. The bug is still unresolved and I desire a solution heavily. Regards, Ingo -- Ingo Strüwing, Senior Software Developer MySQL AB, www.mysql.com Office: +49 30 43672407 Are you MySQL certified? www.mysql.com/certification
Processed: Re: Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6?
Processing commands for [EMAIL PROTECTED]: reassign 286803 kernel-source-2.6.8 Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6? Bug reassigned from package `kernel-source-2.6.9' to `kernel-source-2.6.8'. thanks Stopping processing here. Please contact me if you need assistance. Debian bug tracking system administrator (administrator, Debian Bugs database) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6?
reassign 286803 kernel-source-2.6.8 thanks On Wed, Feb 23, 2005 at 10:09:53AM +0100, Ingo Strüwing wrote: Am Mittwoch, den 23.02.2005, 12:28 +0900 schrieb Horms: On Fri, Feb 18, 2005 at 03:33:30PM +0100, Ingo Strüwing wrote: Am Donnerstag, den 10.02.2005, 18:28 +0900 schrieb Horms: On Thu, Feb 10, 2005 at 09:07:00AM +0100, Ingo Strüwing wrote: Am Donnerstag, den 10.02.2005, 10:33 +0900 schrieb Horms: On Wed, Feb 09, 2005 at 04:50:49PM +0100, Ingo Strüwing wrote: I have additional information: - The problem exists in 2.6.10 too. (BTW. should I report it there too?) - The problem exists on ext3 filesystems too. - The problem does not exist in 2.4.27. Thanks, all good information. You should probably just reasign the bug to kernel-source-2.6.10 as 2.6.9 is slowly on its way out of the debian tree. What does 'reasign' mean? Is there a way to move a report from one packet to another? Or is it just reporting it again against 2.6.10? Yes, you can move the bug. http://www.debian.org/Bugs/server-control Or someone else (like say, me) can do it for you. Yes. Please. I would appreciate if you would move it to either 2.6.10 or 2.6.8. Wichever has the higher chances to solve it. Also any feedback on 2.6.8 would be good, that is the target for sarge, 2.6.10 is just in sid for people to experiment and so we can see what has been fixed upstream when problems crop up in 2.6.8. I tested this now successfully on 2.6.8-13 and 2.6.10-5. Does that mean the bug has been resolved and can be closed? No. Of course not. The quotes around successfully mean that it is not really a success, but that I was successful in reproducing the problem. I hoped that this was clear in conjuction with my sentence 9 lines above that one. The bug is still unresolved and I desire a solution heavily. Sorry for my misunderstanding. I have reassigned it to 2.6.8, as it is the target for Sarge and thus should get the most attention. -- Horms
Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6?
On Fri, Feb 18, 2005 at 03:33:30PM +0100, Ingo Strüwing wrote: Am Donnerstag, den 10.02.2005, 18:28 +0900 schrieb Horms: On Thu, Feb 10, 2005 at 09:07:00AM +0100, Ingo Strüwing wrote: Am Donnerstag, den 10.02.2005, 10:33 +0900 schrieb Horms: On Wed, Feb 09, 2005 at 04:50:49PM +0100, Ingo Strüwing wrote: I have additional information: - The problem exists in 2.6.10 too. (BTW. should I report it there too?) - The problem exists on ext3 filesystems too. - The problem does not exist in 2.4.27. Thanks, all good information. You should probably just reasign the bug to kernel-source-2.6.10 as 2.6.9 is slowly on its way out of the debian tree. What does 'reasign' mean? Is there a way to move a report from one packet to another? Or is it just reporting it again against 2.6.10? Yes, you can move the bug. http://www.debian.org/Bugs/server-control Or someone else (like say, me) can do it for you. Yes. Please. I would appreciate if you would move it to either 2.6.10 or 2.6.8. Wichever has the higher chances to solve it. Also any feedback on 2.6.8 would be good, that is the target for sarge, 2.6.10 is just in sid for people to experiment and so we can see what has been fixed upstream when problems crop up in 2.6.8. I tested this now successfully on 2.6.8-13 and 2.6.10-5. Does that mean the bug has been resolved and can be closed? -- Horms
Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6?
On Thu, Feb 10, 2005 at 09:07:00AM +0100, Ingo Strüwing wrote: Am Donnerstag, den 10.02.2005, 10:33 +0900 schrieb Horms: On Wed, Feb 09, 2005 at 04:50:49PM +0100, Ingo Strüwing wrote: I have additional information: - The problem exists in 2.6.10 too. (BTW. should I report it there too?) - The problem exists on ext3 filesystems too. - The problem does not exist in 2.4.27. Thanks, all good information. You should probably just reasign the bug to kernel-source-2.6.10 as 2.6.9 is slowly on its way out of the debian tree. What does 'reasign' mean? Is there a way to move a report from one packet to another? Or is it just reporting it again against 2.6.10? Yes, you can move the bug. http://www.debian.org/Bugs/server-control Or someone else (like say, me) can do it for you. Also any feedback on 2.6.8 would be good, that is the target for sarge, 2.6.10 is just in sid for people to experiment and so we can see what has been fixed upstream when problems crop up in 2.6.8. In my original report I mentioned that I watched the behaviour in 2.6.8 Sorry, I missed that. and 2.6.7 too. There may be a new version of 2.6.8 available though. When I have your answer to the above question, I'll make a new 2.6.8 kernel, test and report accordingly. Yes, there have been 3 updates to 2.6.8 since your report on December 22nd. The current version is 2.6.8-13. -- Horms
Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6?
Hi Thiemo, Am Mi, den 22.12.2004 schrieb Thiemo Seufer um 12:22: MySQL Development wrote: [snip] The problem is that this did not work out. Every time when I run a make on a MySQL source tree, I loose some of my memory. More specifically spoken, the amount of memory, called Active increases, while Cached decreases (names are taken from /proc/meminfo). Just to avoid misunderstandings: Active does not fall back to what it was before the the make process started. After running MySQL makes a couple of times, Active occupies about 80 to 90% of the RAM. After that, it does not grow any more. Nevertheless, only a small fraction of my expensive RAM is left for the file system cache. Have you tried to adjust /proc/sys/vm/vfs_cache_pressure ? linux/Documentation/filesystems/proc.txt might be worth a read. thank you for your suggestion. I made several experiments with it, but it didn't help. I configured vfs_cache_pressure in some steps down to zero and tried 1000 finally. Going down didn't change anything, going to 1000 resulted in even less cache (what one would expect from the documentation). So I ask you to handle my bug report like usual. I hope that the problem will be fixed one day. With best wishes for the new year and kind regards, Ingo -- Ingo Strüwing, Senior Software Developer MySQL AB, www.mysql.com Office: +49 30 43672407 Are you MySQL certified? www.mysql.com/certification
Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6?
Package: kernel-source-2.6.9
Version: 2.6.9-3
Severity: normal
Hi. I do not even know if this is a bug. I can only say that the kernel does
not behave according to my expectations. The problem is not easy to explain,
nor easy to reproduce. I'll try to explain with increasing detail. Stop to
read whenever you can tell if my expectations are wrong, or you know what the
bug is.
My situation is the following: I work as a developer for MySQL AB. I have to
work with a couple of source trees. One or more for each MySQL version,
depending on the number of projects I am working on. I need to ./configure,
make, test, edit, grep through sources, make, test and so forth.
My expectation was that a lot of RAM would provide much memory for the file
system cache. That way, I hoped, lots of disk reads would become unnecessary
and everything would go faster.
The problem is that this did not work out. Every time when I run a make on a
MySQL source tree, I loose some of my memory. More specifically spoken, the
amount of memory, called Active increases, while Cached decreases (names
are taken from /proc/meminfo). Just to avoid misunderstandings: Active does
not fall back to what it was before the the make process started. After
running MySQL makes a couple of times, Active occupies about 80 to 90% of
the RAM. After that, it does not grow any more. Nevertheless, only a small
fraction of my expensive RAM is left for the file system cache.
Fortunately, it is possible to get back the lost memory by massive file
system operations. But this implies disk reads, which I wanted to diminish.
This problem, which might be a bug or a misunderstanding of how Linux uses
RAM, is _not_ new to kernel 2.6.9. I watched it at least with 2.6.8 and 2.6.7.
I am not sure about earlier versions, since I do not work with MySQL that long
yet. I did not yet report the problem, since I thought that everyone has
already noticed it and it would be solved in the next version. Today I buried
that hope, sat down and wrote this report.
My hope is that a kernel developer has special tools to analyze, what kind of
pages are left over as Active after the make. So he would find a bug and
make me happy with the next kernel version. Or that someone tells me to switch
on or off some kernel parameter or something alike.
I prepared two ways to reproduce the problem. Step 1.) and 2.) use a MySQL
source tree and build a binary. Step 3.) is an alternative way without MySQL
sources, but it is much less efficient to reproduce the memory leak.
Before starting, my file system cache is beautifully filled:
sleep 40 cat /proc/meminfo
MemTotal: 2075608 kB
MemFree: 94448 kB
Buffers:244608 kB
Cached:1555968 kB
SwapCached: 0 kB
Active: 256340 kB
Inactive: 1598516 kB
HighTotal: 1179072 kB
HighFree:74432 kB
LowTotal: 896536 kB
LowFree: 20016 kB
SwapTotal: 1052248 kB
SwapFree: 1052248 kB
Dirty: 0 kB
Writeback: 0 kB
Mapped: 71880 kB
Slab: 101580 kB
Committed_AS:91712 kB
PageTables: 1212 kB
VmallocTotal: 114680 kB
VmallocUsed: 14352 kB
VmallocChunk:99316 kB
1.) Get a MySQL source tree and build a MySQL version:
You need quite current versions of the autotools. I have:
ii autoconf 2.59-8
ii automake1.7 1.7.9-6
ii libtool 1.5.6-2
wget http://ftp.gwdg.de/pub/misc/mysql/Downloads/MySQL-4.0/mysql-4.0.23.tar.gz
tar xzf mysql-4.0.23.tar.gz
cd mysql-4.0.23
sleep 40 cat /proc/meminfo meminfo-1.txt
BUILD/compile-pentium-debug-max # runs about 10 to 20 minutes
sleep 40 cat /proc/meminfo meminfo-2.txt
The result is (meminfo-1, meminfo-2, difference, unchanged rows left out):
diff -y meminfo-1.txt meminfo-2.txt | perl -n -e 's/^(.{13}\s+)(\d+)
kB\s+\|\s.{13}(\s+)(\d+).*// printf(%s%s kB%s%s kB = %10d\n, $1, $2, $3,
$4, $4-$2)'
MemFree: 94448 kB 175308 kB = 80860
Buffers:244724 kB 283420 kB = 38696
Cached:1555968 kB 1178168 kB =-377800
Active: 256416 kB 869988 kB = 613572
Inactive: 1598628 kB 902712 kB =-695916
HighFree:74432 kB 107328 kB = 32896
LowFree: 20016 kB67980 kB = 47964
Dirty: 8 kB49672 kB = 49664
Mapped: 71956 kB72804 kB =848
Slab: 101584 kB 102960 kB = 1376
Which, if I understand correctly, means that the active memory increased by
600 MB. The file system cache lost 370 MB, wich is OK of course, since the
build processes needed memory during their work.
I expected that a second build on the same tree would result in no major
changes in the memory printout. The active memory, used during the first run,
should be reused for the second run. And the cache should stay at 1GB (unless
I build two or more trees at once). But:
2.) After building MySQL once like above you can concentrate on the small
fraction of the build
Bug#286803: kernel-source-2.6.9: Kind of a memory leak in kernel 2.6?
MySQL Development wrote: [snip] The problem is that this did not work out. Every time when I run a make on a MySQL source tree, I loose some of my memory. More specifically spoken, the amount of memory, called Active increases, while Cached decreases (names are taken from /proc/meminfo). Just to avoid misunderstandings: Active does not fall back to what it was before the the make process started. After running MySQL makes a couple of times, Active occupies about 80 to 90% of the RAM. After that, it does not grow any more. Nevertheless, only a small fraction of my expensive RAM is left for the file system cache. Have you tried to adjust /proc/sys/vm/vfs_cache_pressure ? linux/Documentation/filesystems/proc.txt might be worth a read. Thiemo
