Rushabh Shah created HBASE-28757:
------------------------------------
Summary: Understand how supportplaintext property works in TLS
setup.
Key: HBASE-28757
URL: https://issues.apache.org/jira/browse/HBASE-28757
Project: HBase
Issue Type: Improvement
Components: security
Affects Versions: 2.6.0
Reporter: Rushabh Shah
We are testing TLS feature and I am confused on how
hbase.server.netty.tls.supportplaintext property works.
Here is our current setup. This is a fresh cluster deployment.
hbase.server.netty.tls.enabled --> true
hbase.client.netty.tls.enabled --> true
hbase.server.netty.tls.supportplaintext --> false (We don't want to fallback on
kerberos)
We still have our kerberos related configuration enabled.
hbase.security.authentication --> kerberos
*Our expectation:*
During regionserver startup, regionserver will use TLS for authentication and
the communication will succeed.
*Actual observation*
During regionserver startup, hmaster authenticates regionserver* via kerberos
authentication*and *regionserver's reportForDuty RPC fails*.
RS logs:
{noformat}
2024-07-25 16:59:55,098 INFO [regionserver/regionserver-0:60020]
regionserver.HRegionServer - reportForDuty to
master=hmaster-0,60000,1721926791062 with
isa=regionserver-0/<rs-ip-address>:60020, startcode=1721926793434
2024-07-25 16:59:55,548 DEBUG [RS-EventLoopGroup-1-2] ssl.SslHandler - [id:
0xa48e3487, L:/<rs-ip-address>:39837 - R:hmaster-0/<hmaster-ip-address>:60000]
HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
2024-07-25 16:59:55,578 DEBUG [RS-EventLoopGroup-1-2]
security.UserGroupInformation - PrivilegedAction [as: hbase/regionserver-0.
(auth:KERBEROS)][action:
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$2@3769e55]
java.lang.Exception
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1896)
at
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:161)
at
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:43)
...
...
2024-07-25 16:59:55,581 DEBUG [RS-EventLoopGroup-1-2]
security.UserGroupInformation - PrivilegedAction [as: hbase/regionserver-0
(auth:KERBEROS)][action:
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$2@c6f0806]
java.lang.Exception
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1896)
at
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:161)
at
org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:43)
at
org.apache.hbase.thirdparty.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
2024-07-25 16:59:55,602 WARN [regionserver/regionserver-0:60020]
regionserver.HRegionServer - error telling master we are up
org.apache.hbase.thirdparty.com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to
address=hmaster-0:60000 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:340)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:595)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:16398)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2997)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.lambda$run$2(HRegionServer.java:1084)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1079)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call
to address=hmaster-0:60000 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:233)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:129)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:231)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:239)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
{noformat}
Hmaster logs
{noformat}
2024-07-25 16:59:55,378 DEBUG [RS-EventLoopGroup-1-2] ipc.NettyRpcServer - SSL
handler added for channel: [id: 0xd4071764, L:/<hmaster-ip>:60000 -
R:regionserver-0/<rs-ip>:39837]
2024-07-25 16:59:55,526 DEBUG [RS-EventLoopGroup-1-2] ssl.SslHandler - [id:
0xd4071764, L:/<hmaster-ip>:60000 - R:regionserver-0/<rs-ip>:39837] HANDSHAKEN:
protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
2024-07-25 16:59:55,583 INFO [RS-EventLoopGroup-1-2] hbase.Server - Auth
successful for hbase/regionserver-0 (auth:KERBEROS)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
