Hi Gopal , >I still do not understand the "only" mode. In the "only" mode, where the query >fragment run, LLAP daemon or tez container? >I change the execution mode from "all" to "only" and disable the >HybridGraceHashJoin. The execution time of q1 from 76s to 456s. Now, in "all" mode , we set the llap container size to 210g(llap java heap size:180g, cache size:20g). If we change to "only" mode, Whether we change the llap container size? And do you have some recommended value?
Regards, Ke -----Original Message----- From: Gopal Vijayaraghavan [mailto:gop...@apache.org] Sent: Saturday, November 25, 2017 3:39 AM To: user@hive.apache.org Subject: Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez Hi, > In our test, we found the shuffle stage of LLAP is very slow. Whether need to > configure some related shuffle value or not? Shuffle is the one hit by the 2nd, 3rd and 4th resource starvation issues listed earlier (FDs, somaxconn & DNS UDP packet loss). > And we get the following log from the LLAP daemon in shuffle stage: > 2017-11-23T12:48:40,718 INFO [New I/O worker #128 ()] > org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler: Setting connection > close header... This is from a Tez setting (& is unexpected with LLAP). Enable keep-alive on the Tez client-side https://github.com/t3rmin4t0r/tez-autobuild/blob/llap/tez-site.xml.frag#L171 FYI, the whole autobuild repo was made to allow easy rebuilding of Hive-LLAP with some sort of basic settings for a machine with 256Gb RAM & 32 HT cores. > Now " hive.llap.execution.mode" have "auto,none,all,map,only" mode. About the > four mode, do you have some suggestions? Whether the "all" mode can gain the > best performance or not? And how the "auto" and "only" mode work? The fastest mode is "only" - "auto" was designed for hybrid execution of ETL queries (i.e small tasks run in LLAP, large tasks run in Tez) & slows down BI queries (i.e slower queries, higher throughput, assuming bulk ETL). The "all" model was designed to throw as much work into LLAP as possible, without failing queries. The trouble with this mode is that the performance of queries will be unpredictable, if some part of them falls out of LLAP. The "only" mode fails those queries, so that you can report a bug or at least, know that the performance loss is because LLAP is not active. Once you switch to the "only" mode, it is a good idea to disable the HybridGraceHashJoin, to get much more performance out of the engine. Because the HybridGrace is a stateful hashtable, it produces 1 hashtable per-thread to avoid race conditions, while disabling it produces 1 read-only stateless hashtable which ends up being interleaved across all NUMA zones, but built exactly once per LLAP daemon. This will cause a reduction in CPU usage (which is a good thing, unlike the lock starvation problems). Cheers, Gopal
