Hi Gopal, > I have upgrade hive version to 3.0 and the somaxconn value of shuffle > port(15551) has been 16384 not 50. Thank you very much. > But I encounter the following problem when run llap, and this is same with > https://issues.apache.org/jira/browse/HIVE-10693 . Whether it is a bug of > llap in hive 3.0? 2017-11-28T22:25:41,875 WARN [Fetcher_B {Map_9} #0 ()] org.apache.tez.runtime.library.common.shuffle.Fetcher: Fetch Failure from host while connecting: bdpe91, attempt: InputAttemptIdentifier [inputIdentifier=0, attemptNumber=0, pathComponent=attempt_1511868736816_0074_2_00_000000_0_10003, spillType=0, spillId=-1] Informing ShuffleManager: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_112] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_112] at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_112] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_112] at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_112] at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_112] at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_112] at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) ~[?:1.8.0_112] at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[?:1.8.0_112] at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) ~[?:1.8.0_112] at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[?:1.8.0_112] at org.apache.tez.http.HttpConnection.getInputStream(HttpConnection.java:260) ~[tez-runtime-library-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at org.apache.tez.runtime.library.common.shuffle.Fetcher.setupConnection(Fetcher.java:529) ~[tez-runtime-library-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:561) ~[tez-runtime-library-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at org.apache.tez.runtime.library.common.shuffle.Fetcher.doHttpFetch(Fetcher.java:486) ~[tez-runtime-library-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:284) ~[tez-runtime-library-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:76) ~[tez-runtime-library-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) ~[tez-common-0.9.1-SNAPSHOT.jar:0.9.1-SNAPSHOT] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
-----Original Message----- From: Gopal Vijayaraghavan [mailto:[email protected]] Sent: Tuesday, November 28, 2017 12:32 AM To: [email protected] Subject: Re: Hive +Tez+LLAP does not have obvious performance improvement than HIVE + Tez Hi, If you've got the 1st starvation fixed (with Hadoop 2.8 patch), all these configs + enable log4j2 async logging, you should definitely see a performance improvement. Here's the log patches, which need a corresponding LLAP config (& have to be disabled in HS2, for the progress bar to work) https://issues.apache.org/jira/browse/HIVE-13027 + https://issues.apache.org/jira/browse/HIVE-16061 You might want to share the explain plans + a jstack, if you want me to narrow down on the issue. Pick a simple query like Query55 or 52, to get a sane baseline. > And For somaxconn, why the somaxconn value of the shuffle port (15551) is 50 > not 16384? Thanks for your help. Good catch, I think you might have more backports to match my GA build (which spikes ~80% cpu on 16 cores easily) https://issues.apache.org/jira/browse/HIVE-16737 Cheers, Gopal
