Hi Gopal ,
>I still do not understand the "only" mode. In the "only" mode, where the query
>fragment run, LLAP daemon or tez container?
>I change the execution mode from "all" to "only" and disable the
>HybridGraceHashJoin. The execution time of q1 from 76s to 456s.
Now, in "all" mode , we set the llap container size to 210g(llap java heap
size:180g, cache size:20g). If we change to "only" mode, Whether we change the
llap container size? And do you have some recommended value?
Regards,
Ke
-Original Message-
From: Gopal Vijayaraghavan [mailto:gop...@apache.org]
Sent: Saturday, November 25, 2017 3:39 AM
To: user@hive.apache.org
Subject: Re: Hive +Tez+LLAP does not have obvious performance improvement than
HIVE + Tez
Hi,
> In our test, we found the shuffle stage of LLAP is very slow. Whether need to
> configure some related shuffle value or not?
Shuffle is the one hit by the 2nd, 3rd and 4th resource starvation issues
listed earlier (FDs, somaxconn & DNS UDP packet loss).
> And we get the following log from the LLAP daemon in shuffle stage:
> 2017-11-23T12:48:40,718 INFO [New I/O worker #128 ()]
> org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler: Setting connection
> close header...
This is from a Tez setting (& is unexpected with LLAP). Enable keep-alive on
the Tez client-side
https://github.com/t3rmin4t0r/tez-autobuild/blob/llap/tez-site.xml.frag#L171
FYI, the whole autobuild repo was made to allow easy rebuilding of Hive-LLAP
with some sort of basic settings for a machine with 256Gb RAM & 32 HT cores.
> Now " hive.llap.execution.mode" have "auto,none,all,map,only" mode. About the
> four mode, do you have some suggestions? Whether the "all" mode can gain the
> best performance or not? And how the "auto" and "only" mode work?
The fastest mode is "only" - "auto" was designed for hybrid execution of ETL
queries (i.e small tasks run in LLAP, large tasks run in Tez) & slows down BI
queries (i.e slower queries, higher throughput, assuming bulk ETL).
The "all" model was designed to throw as much work into LLAP as possible,
without failing queries. The trouble with this mode is that the performance of
queries will be unpredictable, if some part of them falls out of LLAP. The
"only" mode fails those queries, so that you can report a bug or at least, know
that the performance loss is because LLAP is not active.
Once you switch to the "only" mode, it is a good idea to disable the
HybridGraceHashJoin, to get much more performance out of the engine. Because
the HybridGrace is a stateful hashtable, it produces 1 hashtable per-thread to
avoid race conditions, while disabling it produces 1 read-only stateless
hashtable which ends up being interleaved across all NUMA zones, but built
exactly once per LLAP daemon.
This will cause a reduction in CPU usage (which is a good thing, unlike the
lock starvation problems).
Cheers,
Gopal
