[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15516382#comment-15516382
]
Peter Vary commented on HIVE-9423:
----------------------------------
Hi [~ctang.ma],
I had finished my other tasks, and was about to collect the logs for you.
HiveServer2:
{code}
2016-09-23T14:59:23,142 WARN [Thread-7] server.TThreadPoolServer: Task has
been rejected by ExecutorService 10 times till timedout, reason:
java.util.concurrent.RejectedExecutionException: Task
org.apache.thrift.server.TThreadPoolServer$WorkerProcess@798d4da3 rejected from
org.apache.hive.service.cli.thrift.ThreadPoolExecutorWithOomHook@f7ed8ef[Running,
pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1]
{code}
Beeline without patch, with --verbose=true:
{code}
Connecting to jdbc:hive2://localhost:10000
16/09/23 14:59:23 [main]: WARN jdbc.HiveConnection: Failed to connect to
localhost:10000
HS2 may be unavailable, check server status
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:10000: null (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:10000: null
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:209)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
at
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
at org.apache.hive.beeline.Commands.connect(Commands.java:1524)
at org.apache.hive.beeline.Commands.connect(Commands.java:1419)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)
at
org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1128)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1167)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:798)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:512)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:227)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:182)
... 24 more
Beeline version 2.2.0-SNAPSHOT by Apache Hive
{code}
In my test cases the server side of Thrift closes the connection without
writing any data to the output in case of reaching the max # of thrift threads.
As for your question, I have not tried kerberized Beeline/HS2. If there is
another timeout somewhere, which shorter/equal than the one defined by
hive.server2.thrift.login.timeout which is used by ThriftBinaryCLIService, then
it might throw the TTransportException with an inner "Connection reset"
IOException and with the type of TTransportException.UNKNOWN. It would be good
to find the source of the Connection reset. Could you please post your stack
trace? Meanwhile I am trying to reproduce your output.
Thanks,
Peter
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> ---------------------------------------------------------------------------------------------------------
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
> Reporter: Vaibhav Gumashta
> Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
