It's clear from the Ambati alert the thrift server connection it's trying to use binary connection (and missing the trustore). Are you using hive in binary with SSL or https?
On Fri., Dec. 13, 2019, 14:13 Matt Andruff, <[email protected]> wrote: > What happens when you try to connect via beeline with: > > > jdbc:hive2://<host>:<port>/<db>;ssl=true;sslTrustStore=<trust_store_path>;trustStorePassword=<trust_store_password> > > ? > > Or are you just using the zookeeper namespace to connect? > > On Thu., Dec. 12, 2019, 14:03 Jan Hentschel, < > [email protected]> wrote: > >> Hello everybody, >> >> >> >> we're having problems with the Ambari alert for the Spark Thriftserver, >> complaining about the Beeline connection to the Thriftserver (see the alert >> below): >> >> >> >> Connection failed on host master01:10016 (Traceback (most recent call >> last): >> >> File >> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts/alert_spark2_thrift_port.py", >> line 147, in execute >> >> Execute(cmd, user=sparkuser, path=[beeline_cmd], >> timeout=CHECK_COMMAND_TIMEOUT_DEFAULT) >> >> File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line >> 166, in __init__ >> >> self.env.run() >> >> File >> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line >> 160, in run >> >> self.run_action(resource, action) >> >> File >> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line >> 124, in run_action >> >> provider_action() >> >> File >> "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", >> line 263, in action_run >> >> returns=self.resource.returns) >> >> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", >> line 72, in inner >> >> result = function(command, **kwargs) >> >> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", >> line 102, in checked_call >> >> tries=tries, try_sleep=try_sleep, >> timeout_kill_strategy=timeout_kill_strategy, returns=returns) >> >> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", >> line 150, in _call_wrapper >> >> result = _call(command, **kwargs_copy) >> >> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", >> line 314, in _call >> >> raise ExecutionFailed(err_msg, code, out, err) >> >> ExecutionFailed: Execution of '! >> /usr/hdp/current/spark2-client/bin/beeline -u >> 'jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary' >> -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL' >> ' returned 1. Error: Could not open client transport with JDBC Uri: >> jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: >> java.net.ConnectException: Connection refused (Connection refused) >> (state=08S01,code=0) >> >> Error: Could not open client transport with JDBC Uri: >> jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: >> java.net.ConnectException: Connection refused (Connection refused) >> (state=08S01,code=0) >> >> ) >> >> >> >> Running it with a user the same problem comes up: >> >> >> >> dr_who@master01:/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/alerts$ >> ! /usr/hdp/current/spark2-client/bin/beeline -u >> 'jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary' >> -e '' >> >> Connecting to jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/09/30 10:40:26 INFO Utils: Supplied authorities: master01:10016 >> >> 19/09/30 10:40:26 INFO Utils: Resolved authority: master01:10016 >> >> 19/09/30 10:40:26 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> >> 19/09/30 10:40:26 INFO HiveConnection: Will try to open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC >> connection: binary >> >> Error: Could not open client transport with JDBC Uri: >> jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: >> java.net.SocketException: Connection reset (state=08S01,code=0) >> >> Beeline version 1.21.2.3.0.1.0-187 by Apache Hive >> >> 19/09/30 10:40:27 INFO Utils: Supplied authorities: master01:10016 >> >> 19/09/30 10:40:27 INFO Utils: Resolved authority: master01:10016 >> >> 19/09/30 10:40:27 INFO HiveConnection: Will try to open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/09/30 10:40:27 INFO HiveConnection: Could not open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/09/30 10:40:27 INFO HiveConnection: Transport Used for JDBC >> connection: binary >> >> Error: Could not open client transport with JDBC Uri: >> jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: >> java.net.SocketException: Broken pipe (Write failed) (state=08S01,code=0) >> >> >> >> When trying to run the same with the Spark user from the command line, a >> similar issues arises: >> >> >> >> spark@master01:~$ /usr/hdp/current/spark2-client/bin/beeline -u >> 'jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary' >> -e 'show databases' >> >> Connecting to jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016 >> >> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016 >> >> 19/11/13 14:42:27 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> >> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC >> connection: binary >> >> Error: Could not open client transport with JDBC Uri: >> jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: >> Invalid status 21 (state=08S01,code=0) >> >> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016 >> >> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016 >> >> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC >> connection: binary >> >> No current connection >> >> 19/11/13 14:42:27 INFO Utils: Supplied authorities: master01:10016 >> >> 19/11/13 14:42:27 INFO Utils: Resolved authority: master01:10016 >> >> 19/11/13 14:42:27 INFO HiveConnection: Will try to open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/11/13 14:42:27 INFO HiveConnection: Could not open client transport >> with JDBC Uri: jdbc:hive2://master01:10016/default;principal=spark/ >> [email protected];transportMode=binary >> >> 19/11/13 14:42:27 INFO HiveConnection: Transport Used for JDBC >> connection: binary >> >> Error: Could not open client transport with JDBC Uri: >> jdbc:hive2://master01:10016/default;principal=spark/[email protected];transportMode=binary: >> Invalid status 21 (state=08S01,code=0) >> >> >> >> Looking at the Thriftserver logs reveals the following: >> >> >> >> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during >> processing of message. >> >> java.lang.RuntimeException: >> org.apache.thrift.transport.TTransportException: >> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? >> >> at >> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) >> >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739) >> >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at javax.security.auth.Subject.doAs(Subject.java:360) >> >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) >> >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736) >> >> at >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: org.apache.thrift.transport.TTransportException: >> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? >> >> at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) >> >> at >> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) >> >> at >> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) >> >> at >> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) >> >> at >> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) >> >> at >> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) >> >> at >> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) >> >> ... 10 more >> >> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, >> plaintext connection? >> >> at >> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) >> >> at sun.security.ssl.InputRecord.read(InputRecord.java:527) >> >> at >> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) >> >> at >> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) >> >> at >> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) >> >> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) >> >> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) >> >> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) >> >> at java.io.BufferedInputStream.read(BufferedInputStream.java:345) >> >> at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) >> >> ... 16 more >> >> 19/11/13 14:43:48 ERROR TThreadPoolServer: Error occurred during >> processing of message. >> >> java.lang.RuntimeException: >> org.apache.thrift.transport.TTransportException: >> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? >> >> at >> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) >> >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739) >> >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at javax.security.auth.Subject.doAs(Subject.java:360) >> >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) >> >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736) >> >> at >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: org.apache.thrift.transport.TTransportException: >> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? >> >> at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) >> >> at >> org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) >> >> at >> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) >> >> at >> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) >> >> at >> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) >> >> at >> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) >> >> at >> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) >> >> ... 10 more >> >> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, >> plaintext connection? >> >> at >> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) >> >> at sun.security.ssl.InputRecord.read(InputRecord.java:527) >> >> at >> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) >> >> at >> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) >> >> at >> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) >> >> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) >> >> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) >> >> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) >> >> at java.io.BufferedInputStream.read(BufferedInputStream.java:345) >> >> at >> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) >> >> ... 16 more >> >> >> >> TLS is configured on the Hive side, but still it seems that instead of a >> TLS connection Beeline or Spark tries to establish a plaintext connection. >> Does anyone know how to resolve this problem? Besides the points >> mentioned above the Thriftserver, f.e. when connecting to it via ODBC. >> >> >> >
