Hi, (Refer my same question in stackoverflow
<https://stackoverflow.com/questions/49270419/hiveserver2-is-shutting-down-frequently-in-hadoop-cluster>
)
favorite
<https://stackoverflow.com/questions/49270419/hiveserver2-is-shutting-down-frequently-in-hadoop-cluster#>Facing
this issue from quite sometime now and not able to track the reason why is
it happening. Whenever we start hiveserver2 using command ->
./hiveserver2 &
It starts and stays up for sometime but then shuts down. In hive logs it
does show the following error while hiveserver is up and running.
2018-03-12 04:44:57,029 ERROR [HiveServer2-Handler-Pool: Thread-33]:
server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Erro
r occurred during processing of message.
java.lang.RuntimeException:
org.apache.thrift.transport.TSaslTransportException: No data or no
sasl data in the stream
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TSaslTransportException: No
data or no sasl data in the stream
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:328)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 4 more
2018-03-12 04:45:55,361 INFO [main]: SessionState
(SessionState.java:printInfo(951)) -
Logging initialized using configuration in
file:/usr/local/hive/conf/hive-log4j.properties
But I'm not really sure that the shutting down of hiveserver is due to
above error as it keeps on running for hours before shutting down.
Following are the hive logs that comes when hiveserver shuts down
2018-03-12 04:46:25,285 INFO [main]: ql.Driver
(SessionState.java:printInfo(951)) - Stage-Stage-1: Map: 4 Reduce: 1
Cumulative CPU
: 18.09 sec HDFS Read: 763046 HDFS Write: 2217 SUCCESS
2018-03-12 04:46:25,286 INFO [main]: ql.Driver
(SessionState.java:printInfo(951)) - Total MapReduce CPU Time Spent:
18 seconds 90 mse
c
2018-03-12 04:46:25,286 INFO [main]: ql.Driver
(SessionState.java:printInfo(951)) - OK
2018-03-12 04:46:25,286 INFO [main]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=releaseLocks
from=org.apach
e.hadoop.hive.ql.Driver>
2018-03-12 04:46:25,295 INFO [main]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=releaseLocks
start=152082998
5286 end=1520829985295 duration=9 from=org.apache.hadoop.hive.ql.Driver>
2018-03-12 04:46:25,295 INFO [main]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=Driver.run
start=15208299614
77 end=1520829985295 duration=23818 from=org.apache.hadoop.hive.ql.Driver>
2018-03-12 04:46:25,304 INFO [main]: CliDriver
(SessionState.java:printInfo(951)) - Time taken: 23.818 seconds
2018-03-12 04:46:25,304 INFO [main]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=releaseLocks
from=org.apach
e.hadoop.hive.ql.Driver>
2018-03-12 04:46:25,305 INFO [main]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=releaseLocks
start=152082998
5304 end=1520829985305 duration=1 from=org.apache.hadoop.hive.ql.Driver>
2018-03-12 04:46:36,351 INFO [Thread-9]: server.HiveServer2
(HiveServer2.java:stop(305)) - Shutting down HiveServer2
2018-03-12 04:46:36,351 INFO [Thread-9]: thrift.ThriftCLIService
(ThriftCLIService.java:stop(201)) - Thrift server has stopped
2018-03-12 04:46:36,351 INFO [Thread-9]: service.AbstractService
(AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is
stopped.
2018-03-12 04:46:36,351 INFO [Thread-9]: service.AbstractService
(AbstractService.java:stop(125)) - Service:OperationManager is stopp
ed.
2018-03-12 04:46:36,351 INFO [Thread-9]: service.AbstractService
(AbstractService.java:stop(125)) - Service:SessionManager is stopped
.
2018-03-12 04:46:36,351 INFO [Thread-3]: server.HiveServer2
(HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at ADG-HADOOP-MASTER/192.168.128.31
************************************************************/
2018-03-12 04:46:46,352 WARN [Thread-9]: service.CompositeService
(SessionManager.java:cleanupLoggingRootDir(213)) - Failed to cleanu
p root dir of HS2 logging: /usr/local/hive/log
java.io.FileNotFoundException: File does not exist: /usr/local/hive/log
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2275)
at
org.apache.hive.service.cli.session.SessionManager.cleanupLoggingRootDir(SessionManager.java:211)
at
org.apache.hive.service.cli.session.SessionManager.stop(SessionManager.java:205)
at
org.apache.hive.service.CompositeService.stop(CompositeService.java:102)
at
org.apache.hive.service.CompositeService.stop(CompositeService.java:92)
at org.apache.hive.service.cli.CLIService.stop(CLIService.java:165)
at
org.apache.hive.service.CompositeService.stop(CompositeService.java:102)
at
org.apache.hive.service.CompositeService.stop(CompositeService.java:92)
at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:307)
at
org.apache.hive.service.server.HiveServer2$1.run(HiveServer2.java:107)
2018-03-12 04:46:46,353 INFO [Thread-9]: service.AbstractService
(AbstractService.java:stop(125)) - Service:CLIService is stopped.
2018-03-12 04:46:46,353 INFO [Thread-9]: service.AbstractService
(AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
2018-03-12 04:51:07,336 INFO [main]: SessionState
(SessionState.java:printInfo(951)) -
Logging initialized using configuration in
file:/usr/local/hive/conf/hive-log4j.properties
If the issue is actually because of...
ERROR [HiveServer2-Handler-Pool: Thread-33]: server.TThreadPoolServer
(TThreadPoolServer.java:run(296)) - Erro
r occurred during processing of message.
java.lang.RuntimeException:
org.apache.thrift.transport.TSaslTransportException: No data or no
sasl data in the stream
...then here are my hive-site.xml settings which are related to it as
mentioned in many other related posts.
<name>hive.server2.authentication</name>
<value>PAM</value>
<name>hive.server2.authentication.pam.services</name>
<value>sshd,sudo</value>
<name>hive.server2.thrift.sasl.qop</name>
<value>auth</value>
<name>hive.metastore.sasl.enabled</name>
<value>false</value>
Appreciate some help here,
Thanks.
