[ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516382#comment-15516382 ]
Peter Vary commented on HIVE-9423: ---------------------------------- Hi [~ctang.ma], I had finished my other tasks, and was about to collect the logs for you. HiveServer2: {code} 2016-09-23T14:59:23,142 WARN [Thread-7] server.TThreadPoolServer: Task has been rejected by ExecutorService 10 times till timedout, reason: java.util.concurrent.RejectedExecutionException: Task org.apache.thrift.server.TThreadPoolServer$WorkerProcess@798d4da3 rejected from org.apache.hive.service.cli.thrift.ThreadPoolExecutorWithOomHook@f7ed8ef[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1] {code} Beeline without patch, with --verbose=true: {code} Connecting to jdbc:hive2://localhost:10000 16/09/23 14:59:23 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000 HS2 may be unavailable, check server status Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: null (state=08S01,code=0) java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: null at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:209) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:208) at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) at org.apache.hive.beeline.Commands.connect(Commands.java:1524) at org.apache.hive.beeline.Commands.connect(Commands.java:1419) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1128) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1167) at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:798) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:512) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:495) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:227) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:182) ... 24 more Beeline version 2.2.0-SNAPSHOT by Apache Hive {code} In my test cases the server side of Thrift closes the connection without writing any data to the output in case of reaching the max # of thrift threads. As for your question, I have not tried kerberized Beeline/HS2. If there is another timeout somewhere, which shorter/equal than the one defined by hive.server2.thrift.login.timeout which is used by ThriftBinaryCLIService, then it might throw the TTransportException with an inner "Connection reset" IOException and with the type of TTransportException.UNKNOWN. It would be good to find the source of the Connection reset. Could you please post your stack trace? Meanwhile I am trying to reproduce your output. Thanks, Peter > HiveServer2: Provide the user with different error messages depending on the > Thrift client exception code > --------------------------------------------------------------------------------------------------------- > > Key: HIVE-9423 > URL: https://issues.apache.org/jira/browse/HIVE-9423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 > Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0 > Reporter: Vaibhav Gumashta > Assignee: Peter Vary > Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch, > HIVE-9423.patch > > > An example of where it is needed: it has been reported that when # of client > connections is greater than {{hive.server2.thrift.max.worker.threads}}, > HiveServer2 stops accepting new connections and ends up having to be > restarted. This should be handled more gracefully by the server and the JDBC > driver, so that the end user gets aware of the problem and can take > appropriate steps (either close existing connections or bump of the config > value or use multiple server instances with dynamic service discovery > enabled). Similarly, we should also review the behaviour of background thread > pool to have a well defined behavior on the the pool getting exhausted. > Ideally implementing some form of general admission control will be a better > solution, so that we do not accept new work unless sufficient resources are > available and display graceful degradation under overload. -- This message was sent by Atlassian JIRA (v6.3.4#6332)