[
https://issues.apache.org/jira/browse/HBASE-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329035#comment-14329035
]
zhangduo commented on HBASE-12953:
----------------------------------
This is found in a failed run,
https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
{noformat}
2015-02-20 01:44:17,961 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=47118]
visibility.DefaultVisibilityLabelServiceImpl(224): Adding the label
TEST_VISIBILITY
2015-02-20 01:44:17,966 INFO [main]
client.ConnectionManager$HConnectionImplementation(1661): Closing zookeeper
sessionid=0x14ba4a6e11b015f
2015-02-20 01:44:17,967 DEBUG [main] ipc.AsyncRpcClient(244): Stopping async
HBase RPC client
{noformat}
And this is in a success run,
https://builds.apache.org/job/PreCommit-HBASE-Build/12926/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
{noformat}
2015-02-20 14:12:48,681 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=43394]
visibility.DefaultVisibilityLabelServiceImpl(224): Adding the label
TEST_VISIBILITY
2015-02-20 14:12:48,684 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(381): master:57132-0x14ba75420fb0000,
quorum=localhost:63493, baseZNode=/hbase Received ZooKeeper Event,
type=NodeDataChanged, state=SyncConnected, path=/hbase/visibility/labels
2015-02-20 14:12:48,684 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(381): regionserver:43394-0x14ba75420fb0001,
quorum=localhost:63493, baseZNode=/hbase Received ZooKeeper Event,
type=NodeDataChanged, state=SyncConnected, path=/hbase/visibility/labels
2015-02-20 14:12:48,686 INFO [main]
client.ConnectionManager$HConnectionImplementation(1661): Closing zookeeper
sessionid=0x14ba75420fb015f
2015-02-20 14:12:48,688 DEBUG [main] ipc.AsyncRpcClient(244): Stopping async
HBase RPC client
{noformat}
The failed one missing the zk watcher logs.
I'm not familiar with Visibility Labels. I skimmed the code in
DefaultVisibilityLabelServiceImpl, seems we do not update labelsCache directly
when addLabels and just write it to zk and then let a zk watcher notify us to
update labelsCache? So there could be situation that after addLabels returns
successfully, the labelsCache still does not contain the added labels? And
setAuth will check labelsCache so there could be a cache miss?
[~stack], is there anybody can give some suggestions? Thanks.
> RegionServer is not functionally working with AysncRpcClient in secure mode
> ---------------------------------------------------------------------------
>
> Key: HBASE-12953
> URL: https://issues.apache.org/jira/browse/HBASE-12953
> Project: HBase
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0, 1.1.0
> Reporter: Ashish Singhi
> Assignee: zhangduo
> Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-12953.patch, HBASE-12953_1.patch,
> HBASE-12953_2.patch, HBASE-12953_2.patch, HBASE-12953_2.patch,
> HBASE-12953_2.patch, HBASE-12953_2.patch, HBASE-12953_3 (2).patch,
> HBASE-12953_3 (2).patch, HBASE-12953_3.patch, HBASE-12953_3.patch,
> HBASE-12953_3.patch, testcase.patch
>
>
> HBase version 2.0.0
> Default value for {{hbase.rpc.client.impl}} is set to AsyncRpcClient.
> When trying to install HBase with Kerberos, RegionServer is not working
> functionally.
> The following log is logged in its log file
> {noformat}
> 2015-02-02 14:59:05,407 WARN [AsyncRpcChannel-pool1-t1]
> channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it
> reached at the tail of the pipeline. It usually means the last handler in the
> pipeline did not handle the exception.
> io.netty.channel.ChannelPipelineException:
> org.apache.hadoop.hbase.security.SaslClientHandler.handlerAdded() has thrown
> an exception; removed.
> at
> io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:499)
> at
> io.netty.channel.DefaultChannelPipeline.callHandlerAdded(DefaultChannelPipeline.java:481)
> at
> io.netty.channel.DefaultChannelPipeline.addFirst0(DefaultChannelPipeline.java:114)
> at
> io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:97)
> at
> io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:235)
> at
> io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:214)
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete(AsyncRpcChannel.java:194)
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete(AsyncRpcChannel.java:157)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:253)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:288)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
> at
> org.apache.hadoop.hbase.security.SaslClientHandler.handlerAdded(SaslClientHandler.java:154)
> at
> io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:486)
> ... 20 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
> {noformat}
> When set hbase.rpc.client.impl to RpcClientImpl, there seems to be no issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
