Thank you for the feedback! Appreciate it. Yes, you are right. The intention was not to show that THP is an awesome feature but to share techniques to measure and control risks. I made the changes <https://github.com/alexandrnikitin/blog/compare/2139c405f0c50a3ab907fb2530421bf352caa412...3e58094386b14d19e06752d9faa0435be2cbe651>to highlight the purpose and risks.
The experiment is indeed interesting. I believe the "defer" option should help in that environment. I'm really keen to try the latest kernel (related not only to THP). *Frankly, I still don't have strong opinion about huge latency spikes in allocation path in general. I'm not sure whether it's a THP issue or application/environment itself. Likely it's high memory pressure in general that causes spikes. Or the root of the issues is in something else, e.g. the jemalloc case.* On Friday, August 18, 2017 at 6:32:40 PM UTC+3, Gil Tene wrote: > > This is very well written and quite detailed. It has all the makings of a > great post I'd point people to. However, as currently stated, I'd worry > that it would (mis)lead readers into using THP with "always" > /sys/kernel/mm/transparent_hugepage/defrag settings (instead of "defer"), > and/or on older (pre-4.6) kernels with a false sense that the many-msec > slow path allocation latency problems many people warn about don't actually > exist. You do link to the discussions on the subject, but the measurements > and summary conclusion of the posting alone would not end up warning people > who don't actually follow those links. > > I assume your intention is not to have the reader conclude that "there is > lots of advise out there telling you to turn off THP, and it is wrong. > Turning it on is perfectly safe, and may significantly speed up your > application", but are instead are aiming for something like "THP used to be > problematic enough to cause wide ranging recommendations to simply turn it > off, but this has changed with recent Linux kernels. It is now safe to use > in widely applicable ways (will th the right settings) and can really help > application performance without risking huge stalls". Unfortunately, I > think that many readers would understand the current text as the former, > not the latter. > > Here is what I'd change to improve on the current text: > > 1. Highlight the risk of high slow path allocation latencies with the > "always" (and even "madvise") setting in > /sys/kernel/mm/transparent_hugepage/defrag, the fact that the "defer" > option is intended to address those risks, and this defer option is > available with Linux kernel versions 4.6 or later. > > 2. Create an environment that would actually demonstrate these very high > (many msec or worse) latencies in the allocation slow path with defrag set > to "always". This is the part that will probably take some extra work, but > it will also be a very valuable contribution. The issues are so widely > reported (into the 100s of msec or more, and with a wide verity of > workloads as your links show) that intentional reproduction *should* be > possible. And being able to demonstrate it actually happening will also > allow you to demonstrate how newer kernels address it with the defer > setting. > > 3. Show how changing the defrag setting to "defer" removes the high > latencies seen by the allocation slow path under the same conditions. > > For (2) above, I'd look to induce a situation where the allocation slow > path can't find a free 2MB page without having to defragment one directly. > E.g. > - I'd start by significantly slowing down the background defragmentation > in khugepaged (e.g set > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs > to 3600000). I'd avoid turning it off completely in order to make sure you > are still measuring the system in a configuration that believes it does > background defragmentation. > - I'd add some static physical memory pressure (e.g. allocate and touch a > bunch of anonymous memory in a process that would just sit on it) such that > the system would only have 2-3GB free for buffers and your netty workload's > heap. A sleeping jvm launched with an empirically sized and big enough -Xmx > and -Xms and with AlwaysPretouch on is an easy way to do that. > - I'd then create an intentional and spiky fragmentation load (e.g. > perform spikes of a scanning through a 20GB file every minute or so). > - with all that in place, I'd then repeatedly launch and run your Netty > workload without the PreTouch flag, in order to try to induce situations > where an on-demand allocated 2MB heap page hits the slow path, and the > effect shows up in your netty latency measurements. > > All the above are obviously experimentation starting points, and may take > some iteration to actually induce the demonstrated high latencies we are > looking for. But once you are able to demonstrate the impact of on-demand > allocation doing direct (synchronous) compaction both in your application > latency measurement and in your kernel tracing data, you would then be able > to try the same experiment with the defrag setting set to "defer" to show > how newer kernels and this new setting now make it safe (or at least much > more safe) to use THP. And with that actually demonstrated, everything > about THP recommendations for freeze-averse applications can change, making > for a really great posting. > > Sent from my iPad > > On Aug 18, 2017, at 3:00 AM, Alexandr Nikitin <nikitin.a...@gmail.com > <javascript:>> wrote: > > I decided to write a post about measuring the performance impact > (otherwise it stays in my messy notes forever) > Any feedback is appreciated. > > https://alexandrnikitin.github.io/blog/transparent-hugepages-measuring-the-performance-impact/ > > On Saturday, August 12, 2017 at 1:01:31 PM UTC+3, Alexandr Nikitin wrote: >> >> I played with Transparent Hugepages some time ago and I want to share >> some numbers based on real world high-load applications. >> We have a JVM application: high-load tcp server based on netty. No clear >> bottleneck, CPU, memory and network are equally highly loaded. The amount >> of work depends on request content. >> The following numbers are based on normal server load ~40% of maximum >> number of requests one server can handle. >> >> *When THP is off:* >> End-to-end application latency in microseconds: >> "p50" : 718.891, >> "p95" : 4110.26, >> "p99" : 7503.938, >> "p999" : 15564.827, >> >> perf stat -e dTLB-load-misses,iTLB-load-misses -p PID -I 1000 >> ... >> ... 25,164,369 iTLB-load-misses >> ... 81,154,170 dTLB-load-misses >> ... >> >> *When THP is always on:* >> End-to-end application latency in microseconds: >> "p50" : 601.196, >> "p95" : 3260.494, >> "p99" : 7104.526, >> "p999" : 11872.642, >> >> perf stat -e dTLB-load-misses,iTLB-load-misses -p PID -I 1000 >> ... >> ... 21,400,513 dTLB-load-misses >> ... 4,633,644 iTLB-load-misses >> ... >> >> As you can see THP performance impact is measurable and too significant >> to ignore. 4.1 ms vs 3.2 ms 99%% and 100M vs 25M TLB misses. >> I also used SytemTap to measure few kernel functions like >> collapse_huge_page, clear_huge_page, split_huge_page. There were no >> significant spikes using THP. >> AFAIR that was 3.10 kernel which is 4 years old now. I can repeat >> experiments with the newer kernels if there's interest. (I don't know what >> was changed there though) >> >> >> On Monday, August 7, 2017 at 6:42:21 PM UTC+3, Peter Veentjer wrote: >>> >>> Hi Everyone, >>> >>> I'm failing to understand the problem with transparent huge pages. >>> >>> I 'understand' how normal pages work. A page is typically 4kb in a >>> virtual address space; each process has its own. >>> >>> I understand how the TLB fits in; a cache providing a mapping of virtual >>> to real addresses to speed up address conversion. >>> >>> I understand that using a large page e.g. 2mb instead of a 4kb page can >>> reduce pressure on the TLB. >>> >>> So till so far it looks like huge large pages makes a lot of sense; of >>> course at the expensive of wasting memory if only a small section of a page >>> is being used. >>> >>> The first part I don't understand is: why is it called transparent huge >>> pages? So what is transparent about it? >>> >>> The second part I'm failing to understand is: why can it cause problems? >>> There are quite a few applications that recommend disabling THP and I >>> recently helped a customer that was helped by disabling it. It seems there >>> is more going on behind the scene's than having an increased page size. Is >>> it caused due to fragmentation? So if a new page is needed and memory is >>> fragmented (due to smaller pages); that small-pages need to be compacted >>> before a new huge page can be allocated? But if this would be the only >>> thing; this shouldn't be a problem once all pages for the application have >>> been touched and all pages are retained. >>> >>> So I'm probably missing something simple. >>> >>> -- > You received this message because you are subscribed to a topic in the > Google Groups "mechanical-sympathy" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/mechanical-sympathy/sljzehnCNZU/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > mechanical-sympathy+unsubscr...@googlegroups.com <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
