On Fri, 29 Jul 2005, Jigal van Hemert wrote: > I do not know exactly which speedup optimizations might be taken in > Fedora Core 4 (as mentioned in your first posting) in general, or in a > 64 bit version specifically, so I am speculating: > > A running MySQL server as configured by you, with 7 GB buffer pool, will > occupy substantial amounts of RAM, probably backed in the "swap area" > (even though this is really a paging area). When the process terminates, > all its resources need to be freed, including flushing files, closing > file descriptors, and releasing these 7 GB. This may take some time. > > Consider that there are file systems that delay writes in order to > optimize disk I/O and to favor reads on which other processes might be > waiting. I suspect that similar strategies might be used on the page device. > > IOW: I doubt that the removal of a process from "ps" output implies that > all its resources have already been freed, and are available. > I admit that the Linux kernel should detect such a situation and delay > the new request (rather than reject it) as the scarce resources are just > getting available, but maybe this is not (yet) done?
Hi Joerg, I am a colleague of Jigal van Hemert with which you had this discussion earlier. I subscribed to the MySQL list to clarify the situation, as I'm the one actually experiencing the problems. I'd like to start with the following: http://lists.debian.org/debian-kernel/2004/12/msg00410.html This implies I'm not the only one strugling with 'Active' memory on a 2.6 kernel. Interesting detail: the problem report is issued by a MySQL developer named Ingo Strüwing, maybe you know him? Either way, I already contacted him to share my experiences. Furthermore I have started a thread on the Fedora mailing list about this, as it seems to be related to somewhat whacky memory management. Now back to the problem, what I've found out is basically the following: - When doing a malloc call it appears the requested amount of memory is tested against the total amount of memory minus the amount of 'Active' (according to /proc/meminfo) memory. So when 6GB of Active memory has piled up on my system after a couple of compiles, the largest block of memory allocatable through malloc seems to be roughly 8GB-6GB=2GB. This is why the single malloc call for 7GB from InnoDB fails. - Interestingly enough it is perfectly possible to allocate multiple 2GB blocks in above situation. This can be done almost without limit, because the memory is not actually in use yet, it is only allocated. I have been able to allocate up to 12GB (did not try any higher) this way. As longs as the single malloc calls request blocks which fall within the Total - Active equation, this will succeed. I tested this by modifying the 'fillmem' utility from the 'memtest' package: http://carpanta.dc.fi.udc.es/~quintela/memtest/ - Even more interesting is the fact that 'fillmem' is in fact able to reclaim the Active memory. If I instruct fillmem to allocate (and actually use it by filling it with random values) near to 8GB of RAM, it does so with success and in the end the total amount of Active memory is near zero. After this I can restart MySQL again. This is a temporary workaround. Now the only question that remains is why the Active memory goes close to zero when exiting fillmem and is not when ending a compile run. I asked this question on the Fedora list to find out if this is a normal situation or if there could be a memory leak somewhere in the compiler, linker etc chain. In the meanwhile I can use the mentioned workaround, but it's still a bit weird situation. > Have you ever tried to wait a bit after a failing restart and then > attempt it again, rather than rebooting? We have waited up to 48 hours, but alas the Active memory never returned... Thanks for your time, -- Matthijs van der Klip System Administrator Spill E-Projects The Netherlands -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/[EMAIL PROTECTED]
