[
https://issues.apache.org/jira/browse/HIVE-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Chung updated HIVE-23153:
--------------------------------
Description:
Deregistering from Zookeeper, initiated by the command 'hive --service
hiveserver2 -deregister <hostname>', is not properly worked when HiveServer2
and Zookeeper are kerberized. Even though hive-site.xml has configuration for
Zookeeper Kerberos login (hive.server2.authentication.kerberos.principal and
keytab), it isn't used. I know that kinit with hiveserver2 keytab would make it
work. But as I said, hive-site.xml already has so that the user doesn't need to
do kinit.
* When Kerberos login to Zookeeper is failed and serverUri is not actually
removed from hiveserver2 namespace; {{Will not attempt to authenticate using
SASL (unknown error)}}
{code:java}
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper:
-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-azure-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-azure-datalake-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-mapreduce-client-common-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-mapreduce-client-core-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/jersey-client-1.19.jar:/usr/hdp/3.1.0.0-78/tez/lib/jersey-json-1.19.jar:/usr/hdp/3.1.0.0-78/tez/lib/jettison-1.3.4.jar:/usr/hdp/3.1.0.0-78/tez/lib/jetty-server-9.3.22.v20171030.jar:/usr/hdp/3.1.0.0-78/tez/lib/jetty-util-9.3.22.v20171030.jar:/usr/hdp/3.1.0.0-78/tez/lib/jsr305-3.0.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/servlet-api-2.5.jar:/usr/hdp/3.1.0.0-78/tez/lib/slf4j-api-1.7.10.jar:/usr/hdp/3.1.0.0-78/tez/conf
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:java.library.path=:....
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:os.version=...
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:user.name=...
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:user.home=...
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:user.dir=...
2020-04-08 04:45:44,699 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=... sessionTimeout=60000
watcher=org.apache.curator.ConnectionState@706eab5d
2020-04-08 04:45:44,725 INFO [main-SendThread(...)] zookeeper.ClientCnxn:
Opening socket connection to server ...:2181. Will not attempt to authenticate
using SASL (unknown error)
2020-04-08 04:45:44,731 INFO [main-SendThread(...:2181)] zookeeper.ClientCnxn:
Socket connection established to ...:2181, initiating session
2020-04-08 04:45:44,743 INFO [main-SendThread(...:2181)] zookeeper.ClientCnxn:
Session establishment complete on server ...:2181, sessionid =
0x27148fd2ab1002e, negotiated timeout = 60000
2020-04-08 04:45:44,751 INFO [main-EventThread] state.ConnectionStateManager:
State change: CONNECTED
2020-04-08 04:45:44,760 WARN [main] server.HiveServer2: Will attempt to remove
the znode: /hiveserver2/serverUri=...;version=3.1.2;sequence=0000000049 from
ZooKeeper
Will attempt to remove the znode:
/hiveserver2/serverUri=...;version=3.1.2;sequence=0000000049 from ZooKeeper
2020-04-08 04:45:44,768 INFO [Curator-Framework-0] imps.CuratorFrameworkImpl:
backgroundOperationsLoop exiting
2020-04-08 04:45:44,771 INFO [main] zookeeper.ZooKeeper: Session:
0x27148fd2ab1002e closed
2020-04-08 04:45:44,771 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2020-04-08 04:45:44,794 INFO [shutdown-hook-0] server.HiveServer2: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at ...
************************************************************/{code}
Plus, the result code of CuratorEvent from DeleteCallback is not checked. In
the case that kerberos login is not done, the code is -102
([KeeperException.Code.NoAuth|#NOAUTH]]).
!Screen Shot 2020-04-08 at 5.00.40.png|width=742,height=201!
was:
Deregistering from Zookeeper, initiated by the command 'hive --service
hiveserver2 -deregister <hostname>', is not properly worked when HiveServer2
and Zookeeper are kerberized. Even though hive-site.xml has configuration for
Zookeeper Kerberos login (hive.server2.authentication.kerberos.principal and
keytab), it isn't used. I know that kinit with hiveserver2 keytab would make it
work. But as I said, hive-site.xml already has so that the user doesn't need to
do kinit.
* When Kerberos login to Zookeeper is failed and serverUri is not actually
removed from hiveserver2 namespace
{code:java}
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper:
-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-azure-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-azure-datalake-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-mapreduce-client-common-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-mapreduce-client-core-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/jersey-client-1.19.jar:/usr/hdp/3.1.0.0-78/tez/lib/jersey-json-1.19.jar:/usr/hdp/3.1.0.0-78/tez/lib/jettison-1.3.4.jar:/usr/hdp/3.1.0.0-78/tez/lib/jetty-server-9.3.22.v20171030.jar:/usr/hdp/3.1.0.0-78/tez/lib/jetty-util-9.3.22.v20171030.jar:/usr/hdp/3.1.0.0-78/tez/lib/jsr305-3.0.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/servlet-api-2.5.jar:/usr/hdp/3.1.0.0-78/tez/lib/slf4j-api-1.7.10.jar:/usr/hdp/3.1.0.0-78/tez/conf
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:java.library.path=:....
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:os.version=...
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:user.name=...
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:user.home=...
2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
environment:user.dir=...
2020-04-08 04:45:44,699 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=... sessionTimeout=60000
watcher=org.apache.curator.ConnectionState@706eab5d
2020-04-08 04:45:44,725 INFO [main-SendThread(...)] zookeeper.ClientCnxn:
Opening socket connection to server ...:2181. Will not attempt to authenticate
using SASL (unknown error)
2020-04-08 04:45:44,731 INFO [main-SendThread(...:2181)] zookeeper.ClientCnxn:
Socket connection established to ...:2181, initiating session
2020-04-08 04:45:44,743 INFO [main-SendThread(...:2181)] zookeeper.ClientCnxn:
Session establishment complete on server ...:2181, sessionid =
0x27148fd2ab1002e, negotiated timeout = 60000
2020-04-08 04:45:44,751 INFO [main-EventThread] state.ConnectionStateManager:
State change: CONNECTED
2020-04-08 04:45:44,760 WARN [main] server.HiveServer2: Will attempt to remove
the znode: /hiveserver2/serverUri=...;version=3.1.2;sequence=0000000049 from
ZooKeeper
Will attempt to remove the znode:
/hiveserver2/serverUri=...;version=3.1.2;sequence=0000000049 from ZooKeeper
2020-04-08 04:45:44,768 INFO [Curator-Framework-0] imps.CuratorFrameworkImpl:
backgroundOperationsLoop exiting
2020-04-08 04:45:44,771 INFO [main] zookeeper.ZooKeeper: Session:
0x27148fd2ab1002e closed
2020-04-08 04:45:44,771 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2020-04-08 04:45:44,794 INFO [shutdown-hook-0] server.HiveServer2: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at ...
************************************************************/{code}
Plus, the result code of CuratorEvent from DeleteCallback is not checked. In
the case that kerberos login is not done, the code is -102
([KeeperException.Code.NoAuth|#NOAUTH]]).
!Screen Shot 2020-04-08 at 5.00.40.png|width=742,height=201!
> deregister from zookeeper is not properly worked on kerberized environment
> --------------------------------------------------------------------------
>
> Key: HIVE-23153
> URL: https://issues.apache.org/jira/browse/HIVE-23153
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Eugene Chung
> Assignee: Eugene Chung
> Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-23153.01.patch, HIVE-23153.02.patch, Screen Shot
> 2020-04-08 at 5.00.40.png
>
>
> Deregistering from Zookeeper, initiated by the command 'hive --service
> hiveserver2 -deregister <hostname>', is not properly worked when HiveServer2
> and Zookeeper are kerberized. Even though hive-site.xml has configuration for
> Zookeeper Kerberos login (hive.server2.authentication.kerberos.principal and
> keytab), it isn't used. I know that kinit with hiveserver2 keytab would make
> it work. But as I said, hive-site.xml already has so that the user doesn't
> need to do kinit.
> * When Kerberos login to Zookeeper is failed and serverUri is not actually
> removed from hiveserver2 namespace; {{Will not attempt to authenticate using
> SASL (unknown error)}}
> {code:java}
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper:
> -78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-azure-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-azure-datalake-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-mapreduce-client-common-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-mapreduce-client-core-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.0.0-78.jar:/usr/hdp/3.1.0.0-78/tez/lib/jersey-client-1.19.jar:/usr/hdp/3.1.0.0-78/tez/lib/jersey-json-1.19.jar:/usr/hdp/3.1.0.0-78/tez/lib/jettison-1.3.4.jar:/usr/hdp/3.1.0.0-78/tez/lib/jetty-server-9.3.22.v20171030.jar:/usr/hdp/3.1.0.0-78/tez/lib/jetty-util-9.3.22.v20171030.jar:/usr/hdp/3.1.0.0-78/tez/lib/jsr305-3.0.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/3.1.0.0-78/tez/lib/servlet-api-2.5.jar:/usr/hdp/3.1.0.0-78/tez/lib/slf4j-api-1.7.10.jar:/usr/hdp/3.1.0.0-78/tez/conf
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.library.path=:....
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.version=...
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.name=...
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.home=...
> 2020-04-08 04:45:44,698 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.dir=...
> 2020-04-08 04:45:44,699 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=... sessionTimeout=60000
> watcher=org.apache.curator.ConnectionState@706eab5d
> 2020-04-08 04:45:44,725 INFO [main-SendThread(...)] zookeeper.ClientCnxn:
> Opening socket connection to server ...:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2020-04-08 04:45:44,731 INFO [main-SendThread(...:2181)]
> zookeeper.ClientCnxn: Socket connection established to ...:2181, initiating
> session
> 2020-04-08 04:45:44,743 INFO [main-SendThread(...:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server ...:2181,
> sessionid = 0x27148fd2ab1002e, negotiated timeout = 60000
> 2020-04-08 04:45:44,751 INFO [main-EventThread] state.ConnectionStateManager:
> State change: CONNECTED
> 2020-04-08 04:45:44,760 WARN [main] server.HiveServer2: Will attempt to
> remove the znode:
> /hiveserver2/serverUri=...;version=3.1.2;sequence=0000000049 from ZooKeeper
> Will attempt to remove the znode:
> /hiveserver2/serverUri=...;version=3.1.2;sequence=0000000049 from ZooKeeper
> 2020-04-08 04:45:44,768 INFO [Curator-Framework-0] imps.CuratorFrameworkImpl:
> backgroundOperationsLoop exiting
> 2020-04-08 04:45:44,771 INFO [main] zookeeper.ZooKeeper: Session:
> 0x27148fd2ab1002e closed
> 2020-04-08 04:45:44,771 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> 2020-04-08 04:45:44,794 INFO [shutdown-hook-0] server.HiveServer2:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down HiveServer2 at ...
> ************************************************************/{code}
>
> Plus, the result code of CuratorEvent from DeleteCallback is not checked. In
> the case that kerberos login is not done, the code is -102
> ([KeeperException.Code.NoAuth|#NOAUTH]]).
> !Screen Shot 2020-04-08 at 5.00.40.png|width=742,height=201!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
