RE: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-28 Thread Jia, Ke A
che.org Subject: Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez Hi, If you've got the 1st starvation fixed (with Hadoop 2.8 patch), all these configs + enable log4j2 async logging, you should definitely see a performance improvement. Here's the

Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-27 Thread Gopal Vijayaraghavan
Hi, If you've got the 1st starvation fixed (with Hadoop 2.8 patch), all these configs + enable log4j2 async logging, you should definitely see a performance improvement. Here's the log patches, which need a corresponding LLAP config (& have to be disabled in HS2, for the progress bar to work)

RE: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-26 Thread Jia, Ke A
hive.apache.org Subject: Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez Hi, > With these configurations, the cpu utilization of llap is very low. Low CPU usage has been observed with LLAP due to RPC starvation. I'm going to assume that the build you

RE: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-25 Thread Jia, Ke A
-- From: Gopal Vijayaraghavan [mailto:gop...@apache.org] Sent: Saturday, November 25, 2017 3:39 AM To: user@hive.apache.org Subject: Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez Hi, > In our test, we found the shuffle stage of LLAP is very slow. Whether n

Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-24 Thread Gopal Vijayaraghavan
Hi, > In our test, we found the shuffle stage of LLAP is very slow. Whether need to > configure some related shuffle value or not? Shuffle is the one hit by the 2nd, 3rd and 4th resource starvation issues listed earlier (FDs, somaxconn & DNS UDP packet loss). > And we get the following log f

RE: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-22 Thread Jia, Ke A
uggestions? Whether the "all" mode can gain the >best performance or not? And how the "auto" and "only" mode work? Regards, Jia Ke -Original Message- From: Gopal Vijayaraghavan [mailto:gop...@apache.org] Sent: Thursday, November 23, 2017 4:03 AM To:

Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-22 Thread Gopal Vijayaraghavan
Hi, > With these configurations, the cpu utilization of llap is very low. Low CPU usage has been observed with LLAP due to RPC starvation. I'm going to assume that the build you're testing is a raw Hadoop 2.7.3 with no additional patches? Hadoop-RPC is single-threaded & has a single mutex loc

RE: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-21 Thread Jia, Ke A
p is very low. Regards, Jia Ke -Original Message- From: Gopal Vijayaraghavan [mailto:gop...@apache.org] Sent: Wednesday, November 22, 2017 11:20 AM To: user@hive.apache.org Subject: Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez Hi, > Please help us

Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-21 Thread Gopal Vijayaraghavan
Hi, > Please help us find whether we use the wrong configuration. Thanks for your > help. Since there are no details, I'm not sure what configuration you are discussing here. A first step would be to check if LLAP cache is actually being used (the LLAP IO in the explain), vectorization is be

Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez

2017-11-21 Thread Jia, Ke A
Hi all, Now, we are running the benchmark of Hive +Tez+LLAP and Apache Tez in TPC-DS with 3TB orc data. But the result of Hive +Tez+LLAP is almost similar with hive+Tez and some queries may be poorer than hive+tez . The following is our cluster and llap configuration, Cluster: 1 master + 7 slave