> > Well I have a script that loops through all running processes, etc. > > checked the used swap space and calculates the sum from all together. > > When running the script, it always tells me: "Swap usage overall: 0" - > > checking all processes on single base I only see 0 > > So I really don't understand how swap can be in use when no process has > > a reference to it. > > Maybe a little background the XML database requires having swap > > available and it also seems to use it. > > My low-level kernel internal knowledge (beyond basic tuning) is even > more dated than my JRE/VM internals (I really need to change that), so > I'd have to dive deeper into how various vm/proc information is > instrumented in the kernel before I answer further. This could > include how pages are allocated and marked by the program itself. Don't worry too much - I'll take a look at it with the application guys and probably with the vendor of the XML DB. They probably already have an answer for that.
> > The problem is, when we stop the XML database (which is java) and the > > Oracle Database, swap isn't freed up. > > When I do a swapoff -a (when all applications are shutdown) it takes > > like 2 hours to free it up and have it available for maintenance > > Then they really are pages that are virtually never utilized. My > apologies, as I might have missed it, but are you seeing any > performance issues? Is there a "wake-up" event that causes paging en > masse? > > Things like sysstat's sa reporting, where you capture the disk > statistics would be extremely helpful, or just run its vmstat for 24 > hours and looking for spikes in paging. If you see reads, and > reduction of swap usage, it could be the event where that data and/or > object is finally utilized. We have systat running on that server, but I don't see any peak in there, except during backup window of the filesystem and XML DB! But both of them are only visible on the I/O information. Swap is not touched at all > > I already raised a change today morning to bring swappiness down to 1 - > > but my concern is that it won't fix the problem (change not yet approved > > but hopefully soon). > > Again, it's probably best to "define" the problem. Is this affecting > performance at all? That is the question. Sysstat helps > tremendously. > > Just looking back at your "snapshot," you are using only 8GiB for > resident objects/data, and another 56GiB is used for read caching. > Your buffers are less than 1GiB in the snapshot. So your system is > easily not swamped, at least at that point-in-time, and using nearly > all of your memory for caching reads. > > > Also, the server isn't very busy during normal operation (from I/O and > > Load perspective) - only if they load the XML database with new data, > > the load and I/O goes up. > > Is that XML database largely in memory? Or is it sizable, but > possibly accounting for much of your read cache? I need to check that with the XML DB guys - looking at the stuff I'd say Yes, the biggest part is in the memory ... or somewhere in the swap area ;-) But I also saw that lots of data are being swapped after initial startup of the application. The interesting part is, that the process I mentioned earlier is the gui Application which is supposed to be used for the XML DB Management > > Anyhow, I'd like to share some more kernel settings with you - maybe you > > can see something we are doing wrong or at least where we can improve > > kernel.shmmax = 56968879104 (this value is set by puppet to 75% of the > > real available memory) > > kernel.shmall = 4294967296 > > vm.max_reclaims_in_progress = 0 > > vm.pagecache = 100 > > vm.swap_token_timeout = 300 0 > > vm.vfs_cache_pressure = 100 > > vm.max_map_count = 65536 > > vm.percpu_pagelist_fraction = 0 > > vm.min_free_kbytes = 34511 > > Yep, only 0.03GiB must remain free, so you literally have plenty of room. > > > vm.lowmem_reserve_ratio = 256 256 32 > > vm.swappiness = 60 > > Default swappiness is the root of all evil. If you don't want to use > swap aggressively, you never leave this in the double digits. If you > leave it at 60, expect swap to always be utilized. > > Just because swap is utilized does _not_ mean you're out of physical > memory. It's just the kernel, aggressively in this case, leaving free > memory for buffers that may need to be immediately used, etc.. > > In fact, considering your dirty background ratio is 10%, that's > 7.2GiB, and roughly the amount of free memory you have. I.e., the > kernel is likely reserving that amount, and not using it for read > cache (stopping at 56GiB, with 8GiB of other resident usage), in case > your system receives off a multi-GiB write in a fraction of a second. I'll change the value of vm.swappiness to 1 and see what's happening But as mentioned above the applications starts swapping right after initialization and even then, the server has enough free memory available. So I think, that it's best to talk to the vendor about that ... > > vm.dirty_expire_centisecs = 2999 > > vm.dirty_writeback_centisecs = 499 > > vm.dirty_ratio = 40 > > vm.dirty_background_ratio = 10 > > vm.page-cluster = 3 > > These are all defaults and never ideal for such large memory systems. > _However_, as I mentioned earlier, this likely does not impact your > symptoms at all. At most, as I mentioned above, the dirty ratio of > 10% could be a factor in why the kernel is leaving around 8GiB free, > and not using more for read cache. It's on my todo list to configure the server according the the give hardware configuration - right now I did not have time for it and the customer did not complain never wake-up a sleeping dog ;-) But I'll look at this too and will hopefully find some time soon to-do proper tuning. Thanks for your help and your suggestions and all the best, Si _______________________________________________ rhelv5-list mailing list rhelv5-list@redhat.com https://www.redhat.com/mailman/listinfo/rhelv5-list