Matt McCline created THRIFT-5435:
------------------------------------
Summary: Hive Server 2 crashes when Thrift library encounters
particular security error
Key: THRIFT-5435
URL: https://issues.apache.org/jira/browse/THRIFT-5435
Project: Thrift
Issue Type: Improvement
Reporter: Matt McCline
These portion of a blog called "NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN
HIVESERVER2 LOG" by Eric Liu. NOTE: The web link causes the Apache bot to
remove it. You will need to Google it yourself.
It addresses HIve Server 2 crashes our customers have seen and filed customer
service complaints on. The Apache Thrift library raises a fatal
RuntimeException when In my opinion it should raise a non-fatal Exception
instead.
Portion of the blog:
-----------------------------------------------------------------------------------------------------
"I have seen lots of users complain about seeing lots of “No data or no sasl
data in the stream” errors in the HiveServer2 server log, yet they have not
noticed any performance impact nor query failure for Hive. So I think it would
be good to write a blog about the possible reason behind this to clarify and
remove the concerns that users have. The following shows the full error message
and stacktrace taken from HiveServer2 log:
ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool:
Thread-533556]: Error occurred during processing of message.
java.lang.RuntimeException:
org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in
the stream
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:765)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:762)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1687)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:762)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no
sasl data in the stream
The likely cause is below:
# You have kerberos enabled
# You have multiple HiveServer2 hosts
# You have Load Balancer enabled in front of all HS2 servers that have such
errors
If you have above setup, the error message you saw in HiveServer2 is harmless
and can be safely ignored. This just indicated that SASL negotiation failed for
one particular Hive client, which in this case would be the Load Balancer who
pings regularly to those HiveServer2’s to check for connectivity. Those pings
from LB were trying with PLAIN TCP connection, hence those messages.
-----------------------------------------------------------------------------------------------------
This is similar exception information from Hive Server 2 that crashed on our
customer's machine:
ExceptionClassName
java.lang.RuntimeException
ExceptionStackTrace
java.lang.RuntimeException:
org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in
the stream
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no
sasl data in the stream
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
