Re: [gentoo-user] Strive for zero swap usage?
On Friday, October 07, 2016 04:43:56 PM Grant wrote: > >> >>> Swap usage on Linux always seems a little tricky to me. Should my > >> >>> goal on a web server be zero swap usage, meaning the attached graph > >> >>> should show no green lines at all if I'm doing it right? > >> >> > >> >> No. You want things that aren't in use to be swapped, like memory > >> >> leaks and such. You don't want things that will be used to be > >> >> swapped. > >> > > >> > Does this look OK? It looks to me like heavy swapping in and out with > >> > plenty of free memory (minus buffers/cache). > >> > >> Or put another way, how do I know when swapping is a problem? I'm > >> running munin so I can look over graphs of my system's characteristics > >> but I'm not sure what to look for to determine if I'm swapping > >> excessively. > > > > "Swapping excessively" is inherently a use-case-specific problem, but it > > comes down to two questions: > > > > * Do you notice your system spending time in iowait swapping data in while > > you're waiting on it? > > * Do you notice your system spending time in iowait swapping data out > > while > > you're waiting on it? (I.e. as it tries to make room for new memory > > allocations) > > I can't find a good graph for iowait in munin. Is watching wa in top > my best bet? I've never used Munin. I use Zabbix, which monitors iowait just fine. > > If I do find a correlation between iowait and web server response > times, should I just decrease memory usage until the problem goes > away? There's more than one cause for iowait, so you can't just assume iowait is related to memory consumption. iowait can happen while waiting for files to load in from disk. In this case, freeing up memory to be used by the page cache can help. iowait can happen while waiting for data to be written *to* disk. In this case, finding what's writing to disk and reducing that can help. iowait can happen during swap. In this case, reducing things going into swap can help. You can reduce things going into swap by reducing vm.swappiness (I prefer to set it to 0, myself; swap will be used if and only if there isn't enough memory at the moment). You can reduce things going into swap by tuning applications to use less memory. (With a web server, there are going to be a lot of things to tune here. What is the webserver doing? What web server is it running? Is it running a dynamic application? What language?) > > What I do notice is that my web server's response time increases along > with the swapping peaks in the graph I posted before. Then you're using too much memory, which is leading you to use too much swap, which is causing an I/O bottleneck for you. Can you tell me more about your stack? What is the web server doing? What HTTPd are you using? Are you running dynamic applications? What language? Are you using any caching? -- :wq signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Strive for zero swap usage?
On Thu, Oct 06, 2016 at 03:31:59PM -0700, Grant wrote: > > Does this look OK? It looks to me like heavy swapping in and out with > plenty of free memory (minus buffers/cache). > Have you looked at the _units_ displayed by vmstat and munin? You're looking at **kilobytes** being swapped and calling that _heavy swapping_... In general your system shows that you have a lot of memory used as cache, and I would guess that the kernel thinks its better to keep using that memory for caching and swap out the occasional page of application data. The alternative would be dropping some cache - might be worse for performance. Swapping does not seem to be something that could be causing any issues for you. Do you even have an issue with response times of your webservers? If so (which you haven't explicitly stated here) it would be better for you to look at things from that end - what is taking long in the webserver and what is causing those things to take that long. As it stands, you're using anti-methods of performance analysis: Streetlight Anti-Method: 1. Pick observability tools there are familiar 2. Run tools 3. Look for obvious issues and Traffic Light Anti-Method 1. Open dashboard (munin) 2. All green? Assume everything is good 3. Something red? Assume that's a problem If you're really interested in this kind of stuff I recommend checking out the materials that Brendan Gregg has gathered over the years. http://www.brendangregg.com/methodology.html and other links on his page.
Re: [gentoo-user] Strive for zero swap usage?
>> "Swapping excessively" is inherently a use-case-specific problem, but it >> comes >> down to two questions: >> >> * Do you notice your system spending time in iowait swapping data in while >> you're waiting on it? >> * Do you notice your system spending time in iowait swapping data out while >> you're waiting on it? (I.e. as it tries to make room for new memory >> allocations) I just ran sar from the sysstat package and this looks like a lot of iowait to me: 00:00:02CPU %user %nice %system %iowait%steal %idle 00:10:01all 48.11 0.86 0.83 1.38 0.00 48.82 00:20:01all 43.98 0.85 0.64 0.54 0.00 53.99 00:30:01all 48.17 0.90 1.04 0.82 0.00 49.07 00:40:01all 48.69 0.85 1.06 0.48 0.00 48.92 00:50:01all 49.74 0.87 0.58 0.49 0.00 48.33 01:00:01all 46.21 0.85 0.48 0.41 0.00 52.05 01:10:01all 48.10 0.86 0.79 0.61 0.00 49.64 01:20:01all 54.00 0.86 0.60 0.65 0.00 43.89 01:30:01all 45.81 0.85 0.49 0.49 0.00 52.36 01:40:01all 52.04 0.86 0.56 0.56 0.00 45.99 01:50:01all 48.49 0.85 0.52 0.47 0.00 49.66 02:00:01all 43.18 0.85 0.48 0.50 0.00 54.99 02:10:01all 45.48 1.12 1.74 20.65 0.00 31.01 02:20:02all 46.20 10.22 1.97 9.70 0.00 31.90 02:30:01all 64.93 0.88 1.98 12.54 0.00 19.67 02:40:01all 46.24 0.86 0.93 5.08 0.00 46.90 02:50:01all 43.49 0.85 0.45 0.60 0.00 54.60 03:00:01all 43.28 0.85 0.45 0.45 0.00 54.97 03:10:01all 39.58 0.85 0.81 5.22 0.00 53.54 03:20:01all 42.04 0.91 0.72 3.97 0.00 52.35 03:30:01all 46.60 0.85 0.74 0.49 0.00 51.31 03:40:01all 47.30 0.85 0.82 0.82 0.00 50.22 03:50:01all 49.43 0.85 0.84 0.59 0.00 48.29 04:00:01all 45.50 0.85 1.02 0.71 0.00 51.91 04:10:01all 44.35 0.86 1.13 2.32 0.00 51.35 04:20:01all 44.29 0.85 1.17 4.91 0.00 48.77 04:30:01all 42.69 0.85 0.47 1.41 0.00 54.59 04:40:01all 48.22 0.85 1.00 7.23 0.00 42.70 04:50:01all 44.70 0.86 0.49 1.49 0.00 52.45 Average:all 46.92 1.19 0.86 2.95 0.00 48.08 > If I do find a correlation between iowait and web server response > times, should I just decrease memory usage until the problem goes > away? > > What I do notice is that my web server's response time increases along > with the swapping peaks in the graph I posted before. > > >> There are ways other than swap to find yourself in iowait, though. I wonder >> what might a good metric of combining iowait numbers with swap event counts. >> Swap events without iowait are likely imperceptible. I do see a clear correlation between iowait above and swap in on the munin graph. Is that enough to conclude that swap activity is slowing down the system and I need to reduce memory usage or perhaps tune swappiness? - Grant
Re: [gentoo-user] Strive for zero swap usage?
>>> Swap usage on Linux always seems a little tricky to me. Should my >>> goal on a web server be zero swap usage, meaning the attached graph >>> should show no green lines at all if I'm doing it right? >> ... > > Have you tuned swappiness? Thanks Bill I'll give swappiness a try too. - Grant
Re: [gentoo-user] Strive for zero swap usage?
On 08/10/16 08:26, Bill Kenworthy wrote: > On 08/10/16 07:43, Grant wrote: >>> Swap usage on Linux always seems a little tricky to me. Should my >>> goal on a web server be zero swap usage, meaning the attached graph >>> should show no green lines at all if I'm doing it right? >> ... > > Have you tuned swappiness? > > e.g.: > vm.swappiness=1 > vm.vfs_cache_pressure=50 > > You can force minimal use of swap as well as turn it off. I have tried > to go swap-less a few times and have come to the conclusion you are > trading occasional slow downs for the certainty of a crash and burn when > you hit an unexpected memory requirement. I have swap (2xram - mostly > for hibernation), but tune it as above. > > rattus ~ # free > totalusedfree shared buff/cache > available > Mem:4041520 1605108 147588 19500 2288824 > 2083800 > Swap: 4194300 356 4193944 > rattus ~ # > > BillK > > Also, this is a read that may help: http://www.linuxjournal.com/article/10678 BillK
Re: [gentoo-user] Strive for zero swap usage?
On 08/10/16 07:43, Grant wrote: >> Swap usage on Linux always seems a little tricky to me. Should my >> goal on a web server be zero swap usage, meaning the attached graph >> should show no green lines at all if I'm doing it right? > ... Have you tuned swappiness? e.g.: vm.swappiness=1 vm.vfs_cache_pressure=50 You can force minimal use of swap as well as turn it off. I have tried to go swap-less a few times and have come to the conclusion you are trading occasional slow downs for the certainty of a crash and burn when you hit an unexpected memory requirement. I have swap (2xram - mostly for hibernation), but tune it as above. rattus ~ # free totalusedfree shared buff/cache available Mem:4041520 1605108 147588 19500 2288824 2083800 Swap: 4194300 356 4193944 rattus ~ # BillK
Re: [gentoo-user] Strive for zero swap usage?
>> >>> Swap usage on Linux always seems a little tricky to me. Should my >> >>> goal on a web server be zero swap usage, meaning the attached graph >> >>> should show no green lines at all if I'm doing it right? >> >> >> >> No. You want things that aren't in use to be swapped, like memory >> >> leaks and such. You don't want things that will be used to be >> >> swapped. >> > >> > Does this look OK? It looks to me like heavy swapping in and out with >> > plenty of free memory (minus buffers/cache). >> >> Or put another way, how do I know when swapping is a problem? I'm >> running munin so I can look over graphs of my system's characteristics >> but I'm not sure what to look for to determine if I'm swapping >> excessively. > > "Swapping excessively" is inherently a use-case-specific problem, but it comes > down to two questions: > > * Do you notice your system spending time in iowait swapping data in while > you're waiting on it? > * Do you notice your system spending time in iowait swapping data out while > you're waiting on it? (I.e. as it tries to make room for new memory > allocations) I can't find a good graph for iowait in munin. Is watching wa in top my best bet? If I do find a correlation between iowait and web server response times, should I just decrease memory usage until the problem goes away? What I do notice is that my web server's response time increases along with the swapping peaks in the graph I posted before. - Grant > If the answer to those questions is yes, then you're swapping excessively. If > not, you're not. > > There are ways other than swap to find yourself in iowait, though. I wonder > what might a good metric of combining iowait numbers with swap event counts. > Swap events without iowait are likely imperceptible. > > But it does all come down to perception and how you want to manage it. I have > some nodes that swap a *lot*, but I don't care as long as they don't fall > behind in their workload. And I have some nodes that I don't permit to swap at > all, as that causes latency spikes that are difficult to nail down, or can > cause > snowballing cascade events across several nodes and processes that interact > with each other.
Re: [gentoo-user] Strive for zero swap usage?
On Friday, October 07, 2016 04:33:27 AM Grant wrote: > >>> Swap usage on Linux always seems a little tricky to me. Should my > >>> goal on a web server be zero swap usage, meaning the attached graph > >>> should show no green lines at all if I'm doing it right? > >> > >> No. You want things that aren't in use to be swapped, like memory > >> leaks and such. You don't want things that will be used to be > >> swapped. > > > > Does this look OK? It looks to me like heavy swapping in and out with > > plenty of free memory (minus buffers/cache). > > Or put another way, how do I know when swapping is a problem? I'm > running munin so I can look over graphs of my system's characteristics > but I'm not sure what to look for to determine if I'm swapping > excessively. "Swapping excessively" is inherently a use-case-specific problem, but it comes down to two questions: * Do you notice your system spending time in iowait swapping data in while you're waiting on it? * Do you notice your system spending time in iowait swapping data out while you're waiting on it? (I.e. as it tries to make room for new memory allocations) If the answer to those questions is yes, then you're swapping excessively. If not, you're not. There are ways other than swap to find yourself in iowait, though. I wonder what might a good metric of combining iowait numbers with swap event counts. Swap events without iowait are likely imperceptible. But it does all come down to perception and how you want to manage it. I have some nodes that swap a *lot*, but I don't care as long as they don't fall behind in their workload. And I have some nodes that I don't permit to swap at all, as that causes latency spikes that are difficult to nail down, or can cause snowballing cascade events across several nodes and processes that interact with each other. -- :wq signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Strive for zero swap usage?
>>> Swap usage on Linux always seems a little tricky to me. Should my >>> goal on a web server be zero swap usage, meaning the attached graph >>> should show no green lines at all if I'm doing it right? >>> >> >> No. You want things that aren't in use to be swapped, like memory >> leaks and such. You don't want things that will be used to be >> swapped. > > > Does this look OK? It looks to me like heavy swapping in and out with > plenty of free memory (minus buffers/cache). Or put another way, how do I know when swapping is a problem? I'm running munin so I can look over graphs of my system's characteristics but I'm not sure what to look for to determine if I'm swapping excessively. - Grant > # vmstat 1 20 > procs ---memory-- ---swap-- -io -system-- > --cpu- > r b swpd free buff cache si sobibo in cs us sy id wa > st > 0 1 155308 71328 29200 243726010 1276300 48 > 1 50 1 0 > 0 1 155308 4 29204 246516400 27772 0 1483 1236 1 > 1 74 24 0 > 2 2 155480 32904 29220 24763920 172 33684 172 4166 2790 44 > 2 30 24 0 > 4 0 155828 30332 29220 24797560 348 41388 348 5319 1884 76 > 2 1 21 0 > 1 1 156184 37668 29244 24707520 356 28960 424 5033 3299 53 > 2 15 29 0 > 0 2 156412 32912 29264 24773640 228 30976 252 2909 2516 14 > 2 56 29 0 > 2 5 156640 29372 29400 24803280 228 23180 344 3949 3156 32 > 2 35 31 0 > 1 4 157072 36372 29624 24699240 432 28032 488 4120 3289 50 > 2 10 37 0 > 2 4 157260 37372 29812 24660360 188 12188 316 3789 2540 54 > 1 7 38 0 > 2 3 157260 30236 29912 246767200 1637248 3730 2077 57 > 2 8 34 0 > 2 3 157304 29684 28940 2459688 24 44 1089244 4271 2808 44 > 2 11 43 0 > 5 2 157824 36448 28372 24401764 520 13036 620 4796 3185 56 > 2 10 33 0 > 2 3 157928 34552 28516 24370324 104 19620 148 4446 4108 48 > 3 15 35 0 > 2 6 157928 38620 28444 2385828 4360 4808 136 4615 2851 39 > 5 6 51 0 > 3 5 157928 36936 28604 2377112 7320 807284 4773 2970 64 > 2 0 34 0 > 5 7 157928 39068 28832 2375944 400 2040 492 4877 3341 71 > 2 1 26 0 > 3 4 157928 81820 28984 2376868 1880 19320 0 5122 6990 66 > 5 0 30 0 > > # free -h > total used free sharedbuffers cached > Mem: 3.9G 3.8G41M88M20M 2.4G > -/+ buffers/cache: 1.4G 2.4G > Swap: 1.0G 154M 869M
Re: [gentoo-user] Strive for zero swap usage?
>> Swap usage on Linux always seems a little tricky to me. Should my >> goal on a web server be zero swap usage, meaning the attached graph >> should show no green lines at all if I'm doing it right? >> > > No. You want things that aren't in use to be swapped, like memory > leaks and such. You don't want things that will be used to be > swapped. Does this look OK? It looks to me like heavy swapping in and out with plenty of free memory (minus buffers/cache). # vmstat 1 20 procs ---memory-- ---swap-- -io -system-- --cpu- r b swpd free buff cache si sobibo in cs us sy id wa st 0 1 155308 71328 29200 243726010 1276300 48 1 50 1 0 0 1 155308 4 29204 246516400 27772 0 1483 1236 1 1 74 24 0 2 2 155480 32904 29220 24763920 172 33684 172 4166 2790 44 2 30 24 0 4 0 155828 30332 29220 24797560 348 41388 348 5319 1884 76 2 1 21 0 1 1 156184 37668 29244 24707520 356 28960 424 5033 3299 53 2 15 29 0 0 2 156412 32912 29264 24773640 228 30976 252 2909 2516 14 2 56 29 0 2 5 156640 29372 29400 24803280 228 23180 344 3949 3156 32 2 35 31 0 1 4 157072 36372 29624 24699240 432 28032 488 4120 3289 50 2 10 37 0 2 4 157260 37372 29812 24660360 188 12188 316 3789 2540 54 1 7 38 0 2 3 157260 30236 29912 246767200 1637248 3730 2077 57 2 8 34 0 2 3 157304 29684 28940 2459688 24 44 1089244 4271 2808 44 2 11 43 0 5 2 157824 36448 28372 24401764 520 13036 620 4796 3185 56 2 10 33 0 2 3 157928 34552 28516 24370324 104 19620 148 4446 4108 48 3 15 35 0 2 6 157928 38620 28444 2385828 4360 4808 136 4615 2851 39 5 6 51 0 3 5 157928 36936 28604 2377112 7320 807284 4773 2970 64 2 0 34 0 5 7 157928 39068 28832 2375944 400 2040 492 4877 3341 71 2 1 26 0 3 4 157928 81820 28984 2376868 1880 19320 0 5122 6990 66 5 0 30 0 # free -h total used free sharedbuffers cached Mem: 3.9G 3.8G41M88M20M 2.4G -/+ buffers/cache: 1.4G 2.4G Swap: 1.0G 154M 869M - Grant
Re: [gentoo-user] Strive for zero swap usage?
On Thu, Oct 6, 2016 at 5:51 PM, Grant wrote: > Swap usage on Linux always seems a little tricky to me. Should my > goal on a web server be zero swap usage, meaning the attached graph > should show no green lines at all if I'm doing it right? > No. You want things that aren't in use to be swapped, like memory leaks and such. You don't want things that will be used to be swapped. -- Rich
[gentoo-user] Strive for zero swap usage?
Swap usage on Linux always seems a little tricky to me. Should my goal on a web server be zero swap usage, meaning the attached graph should show no green lines at all if I'm doing it right? - Grant
