Re: [qubes-users] swappiness, caches
On 10/19/2016 11:54 AM, johnyju...@sigaint.org wrote: It always seemed a bit "off" to me that there should be any swap usage or significant buffers/caches inside VM's. dom0 already caches the virtual .img files, so having the kernel inside each VM also buffering/caching files and metadata is really just a waste of CPU and disk space. I don't quite agree. Disk cache is a dilemma, actually. That is because there are different advantages to having the cache in either dom0 or domUs. For example, booting a whole system where you have between 2 and 4 VMs starting in rapid succession goes much quicker if dom0 is caching the VM images. OTOH, having VMs cache more of their own data means less buffer-copying between dom0 and domUs while running applications. More importantly, swap activity is a major performance killer on bare hardware; even more so on a virtual machine. There's a case to let some stuff swap out (unused x-servers in sys-net/sys-firewall, etc.) but the default is way too aggressive for good performance, IMHO. The kernel has a "vm.swappiness" (0 to 100) parameter that controls the balance between grabbing a new free page from the cache or swapping a page out to re-use. The default is 60, which leans towards swapping instead of cache use. Not ideal. In dom0 and all my VM's, I tried changing the swappiness factor to see what the effect would be: # echo 0 >/proc/sys/vm/swappiness or $ sudo sysctl -w vm.swappiness=0 To empty existing swap space, I did a "swapoff -av && swapon -av" Performance is noticeably better, with no swapping happening in any of the VM's, nor in dom0. And memory usage reported by the Vm's seems to be smaller; a heavy browsing session used to crank a VM up to 1.5G; now it seems to be more like 800M. So I can run more VM's, get more done. I have experimented with this myself. How well it works depends on how much physical RAM you have. I suspect the effects of dom0 swapping are being felt more in R3.2 because dom0 now has a size limit. In 3.1 and prior, swapping seemed less common. There is also vm.vfs_cache_pressure which affects cache size. Chris -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/bf953f40-05a6-63a0-bf5c-c861892c38a2%40openmailbox.org. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] swappiness, caches
> Interesting that the Wiki page for swappiness (this kernel parameter is > officially more famous than I am) recommends setting it to at least 1. > > https://en.wikipedia.org/wiki/Swappiness I'm going to stick with vm.swappiness=0 for a few days just to see if any reliability problems or app failures do pop up, out of curiosity. I think a setting of 1 (or at least <10) is probably best for the long run, letting some very infrequently used stuff creep off to swap. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/87ef6832599ed40108b3708ea2ee3580.webmail%40localhost. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] swappiness, caches
> Interesting, sounds reasonable. > > Running with absolutely 0 swap however can lead to unexpected problems > from my experience: Interesting that the Wiki page for swappiness (this kernel parameter is officially more famous and I am) recommends setting it to at least 1. https://en.wikipedia.org/wiki/Swappiness More on the vm.swapiness parameter. It's a bit more subtle than I thought. Some references: 1) http://askubuntu.com/questions/103915/how-do-i-configure-swappiness This page describes it as being how prone Linux is to start swapping out processes. At 0, nothing will be swapped until memory is fully exhausted; at 100, swapping out of processes will begin almost immediately. It indicates the default setting of 60 means swapping will start when memory is around "half" full. So a setting of zero doesn't prevent swapping, it just puts it off until there is no memory available. This is the old-school Unix behaviour I'm used to, and probably best for VM. 2) http://unix.stackexchange.com/questions/88693/why-is-swappiness-set-to-60-by-default This page talks about it in relation to choosing pages with backing store (programs, libraries, cached/mem-mapped data files, already-swapped data) vs. "anonymous" pages of allocated memory. Cached files have a weight of 200-vm.swappiness, and anonymous pages a weight of vm.swapiness. That may be saying the same thing as #1 but in a different; possibly more prescise way, since a setting of 100 gives a 50/50 weight for new page acquisition betreen swap and cache. 3) http://www.cloudibee.com/linux-performance-tuning-vm-swappiness/ This one talks about it as a balance between "swapping and freeing cache," which is the same, I think. - Any anonymous page is going to need to be written to swap before being given to the VM needing memory. (As well as a read when the page is used in the future.) And writes are usually more expensive than reads to start with. A cached file/program/library doesn't need to be written, the page can be discarded and re-used immediately since it can be retrieved from the backing file/program/library when needed in the future. Having a swap/anon page swapped and retrieved has a cost of 1w+1r Having a file/prog page discarded and later retrieved has a cost of 1r. So swapping has a r/w cost of at least 2x that of stealing from the file-backed cache. (Writes are usually a bit more costly than reads, as well.) Obviously the nature of your machine (server/desktop) affects things. That 60 default setting means file-backed cached pages have a weight of 200-60, or 140, while the anonymous/to-be-swapped pages have a weight of 60, a 70%/30% balance in favour of resuing file-backed cached pages versus swapping something out to get free pages. Not a bad compromise for running on bare hardware, or a server; but not appropriate/necessary for a VM. With vm.swappiness set to 0 and the same swap space as before, swap can get used when needed; and as much as before, but not until memory is exhausted. And when the free memory is exhausted, that also implies that all of the cache has also been re-allocated as assigned memory as well. Since the VM's really shouldn't be caching in the first place (double-caching in both dom0 and the VM has to be slower than just one level of cache). I'm still looking around for options to disable file caching, but having vm.swappiness low at least gives any running program priority over the memory being used as cache. The qmemman load balancer won't consider the memory used for a VM's cache as part of it's "required" memory (but it does include swapped out stuff, giving it reasonable chance of getting back into memory without thrashing), so with low vm.swapiness a VM will not be given extra memory to maintain any significant level of cache, unless there's free memory around to be doled out between VM's overall. I can't help but think the original intent was for vm.swappiness=0 behaviour. Once vm.swappiness is >0, then some level swapping will occur resulting in free pages for the VM, and Linux will then go and use these pages for additional (and unnecessary) cache space; an all-around waste of disk-access, CPU time, memory. Running with vm.swappiness=0 seems to work in practice so far. I'm still amazed at the difference in memory/performance I'm seeing. Zero swap on a system that used to have 40M-ish swapped on all VM's and in dom0. And smaller VM's, allowing more to be started. I'm surprised that this hasn't been a default, or at least some similar tuning done by default. In the source code, the 350M "dom0 memory boost" is mentioned as being specifically to give dom0 free memory (that will inherently be used as cache) beyond its actual needs (used memory+used swap). So there is intent to let dom0 do the file caching. But not a similar effort to prevent unnecessary caching in the VM's. Also, it's worth verifying the benefitting from a low vm.swappiness in dom0 itself. Swapping can just
Re: [qubes-users] swappiness, caches
On 10/19/2016 05:54 PM, johnyju...@sigaint.org wrote: It always seemed a bit "off" to me that there should be any swap usage or significant buffers/caches inside VM's. dom0 already caches the virtual .img files, so having the kernel inside each VM also buffering/caching files and metadata is really just a waste of CPU and disk space. More importantly, swap activity is a major performance killer on bare hardware; even more so on a virtual machine. There's a case to let some stuff swap out (unused x-servers in sys-net/sys-firewall, etc.) but the default is way too aggressive for good performance, IMHO. The kernel has a "vm.swappiness" (0 to 100) parameter that controls the balance between grabbing a new free page from the cache or swapping a page out to re-use. The default is 60, which leans towards swapping instead of cache use. Not ideal. In dom0 and all my VM's, I tried changing the swappiness factor to see what the effect would be: # echo 0 >/proc/sys/vm/swappiness or $ sudo sysctl -w vm.swappiness=0 To empty existing swap space, I did a "swapoff -av && swapon -av" Performance is noticeably better, with no swapping happening in any of the VM's, nor in dom0. And memory usage reported by the Vm's seems to be smaller; a heavy browsing session used to crank a VM up to 1.5G; now it seems to be more like 800M. So I can run more VM's, get more done. (I'm not sure why this is, but firefox seems to be less of a memory pig with this change. Maybe with the former aggressive swapping out, Firefox thought free memory was a bit cheaper than it is, and was more aggressive in its own use, or more lax in freeing things up. With a more realistic picture of the memory situation, by changing vm.swappiness to 0, it doesn't seem to be quite the pig it was.) You can set the parameter permanently by adding "vm.swappiness=0" to /etc/sysctl.conf in dom0 and your templates. Poking around 'net, I see a few comments where 0 swappiness is best for guest VM's. I wonder if a little higher might not be bad, for the case of an unused X server or whatever, being able to swap out. I might play a bit with different settings in different VM's. It would be nice to disable caching in the VM's, but that seems to be a difficult thing to do in Linux. So far I've found that the system is pretty good about keeping the VM size to the minimum/startup size, and giving up buffers/cache when needed. (buffers/cache aren't included in the 'memory-used' calculation when mem-balancing between VM's, which keeps the buffers/cache under control a bit, I think. Excessive cache use is not given weight and rewarded by additional memory in the balancing. Any real memory needs from an existing or new VM would effectively take priority over buffer space, in the memory balancing allocations.) Real quick and dirty way of checking your swap usage across VM's (I might add this info to the VM manager for fun, since it's pretty critical performance information; will submit any changes): $ sudo xenstore-ls -f /local/domain | awk '/meminfo/ { print $12-$14, $1 }' The # reported in the path is the domid, which you can see with "qvm-ls -i" I'd be interested in hearing others' thoughts on this. Related reading: https://www.xenproject.org/directory/directory/research/118-geiger-monitoring-the-buffer-cache-in-a-virtual-machine-envrionement.html http://www.sersc.org/journals/IJCA/vol6_no1/12.pdf Interesting, sounds reasonable. Running with absolutely 0 swap however can lead to unexpected problems from my experience: Some applications (e.g. firefox downloads the last time I tested it) use memory-mapped files of arbitrary size assuming swapping is enabled, i.e. you'll store the entire files in memory without swapping. If however the files are getting too large (say you download a multi Gig file via firefox), your memory will run out and the application will report an error (firefox will cancel the download due to a write error). Most applications work though and reducing the swap size to an absolute minimum certainly helps to improve speed. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/2efe4523-86f1-02a9-4a7a-313bc1a0733a%40hackingthe.net. For more options, visit https://groups.google.com/d/optout. smime.p7s Description: S/MIME Cryptographic Signature