[
https://issues.apache.org/jira/browse/HBASE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336422#comment-14336422
]
ramkrishna.s.vasudevan commented on HBASE-13084:
------------------------------------------------
What I found is that even a sleep of 10 secs was not enough some times. The
reason should be because of the ZK error that we got in the Replication
related test cases. And that is why I went on with commenting this testcase
out and look out what is happening with the replication test cases in
TestShell.
May be a better ideal would be to create TestReplicatonShell? And move those
test cases to a seperated one so that other testcases are running fine?
> Add labels to VisibilityLabelsCache asynchronously causes TestShell flakey
> --------------------------------------------------------------------------
>
> Key: HBASE-13084
> URL: https://issues.apache.org/jira/browse/HBASE-13084
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: zhangduo
> Assignee: zhangduo
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-13084.patch, HBASE-13084_1.patch,
> HBASE-13084_2.patch, HBASE-13084_2.patch, HBASE-13084_2.patch,
> HBASE-13084_2.patch, HBASE-13084_2_disable_test.patch
>
>
> As discussed in HBASE-12953, we found this error in PreCommit log
> https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
> {noformat}
> 1) Error:
> test_The_get/put_methods_should_work_for_data_written_with_Visibility(Hbase::VisibilityLabelsAdminMethodsTest):
> ArgumentError: org.apache.hadoop.hbase.DoNotRetryIOException:
> org.apache.hadoop.hbase.security.visibility.InvalidLabelException: Label
> 'TEST_VISIBILITY' doesn't exists
> at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.setAuths(VisibilityController.java:808)
> at
> org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.setAuths(VisibilityLabelsProtos.java:6036)
> at
> org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:6219)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6867)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1707)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1689)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31309)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2038)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:744)
>
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:84:in
> `set_auths'
> ./src/test/ruby/hbase/visibility_labels_admin_test.rb:77:in
> `test_The_get/put_methods_should_work_for_data_written_with_Visibility'
> org/jruby/RubyProc.java:270:in `call'
> org/jruby/RubyKernel.java:2105:in `send'
> org/jruby/RubyArray.java:1620:in `each'
> org/jruby/RubyArray.java:1620:in `each'
> 2) Error:
> test_The_set/clear_methods_should_work_with_authorizations(Hbase::VisibilityLabelsAdminMethodsTest):
> ArgumentError: No authentication set for the given user jenkins
>
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-shell/src/main/ruby/hbase/visibility_labels.rb:97:in
> `get_auths'
> ./src/test/ruby/hbase/visibility_labels_admin_test.rb:57:in
> `test_The_set/clear_methods_should_work_with_authorizations'
> org/jruby/RubyProc.java:270:in `call'
> org/jruby/RubyKernel.java:2105:in `send'
> org/jruby/RubyArray.java:1620:in `each'
> org/jruby/RubyArray.java:1620:in `each'
> {noformat}
> This is the test code
> {code:title=visibility_labels_admin_test.rb}
> label = 'TEST_VISIBILITY'
> user = org.apache.hadoop.hbase.security.User.getCurrent().getName();
> visibility_admin.add_labels(label)
> visibility_admin.set_auths(user, label)
> {code}
> It says 'label does not exists' when calling set_auths.
> Then I add some ugly logs in DefaultVisibilityLabelServiceImpl and
> VisibilityLabelsCache.
> {code:title=DefaultVisibilityLabelServiceImpl.java}
> public OperationStatus[] addLabels(List<byte[]> labels) throws IOException {
> ...
> if (mutateLabelsRegion(puts, finalOpStatus)) {
> updateZk(true);
> }
> for (byte[] label : labels) {
> String labelStr = Bytes.toString(label);
> LOG.info(labelStr + "=====" +
> this.labelsCache.getLabelOrdinal(labelStr));
> }
> ...
> }
> {code}
> {code:title=VisibilityLabelsCache.java}
> public void refreshLabelsCache(byte[] data) throws IOException {
> LOG.info("========refresh", new Exception());
> ...
> }
> {code}
> And I modified TestVisibilityLabelsWithCustomVisLabService to use
> DefaultVisibilityLabelServiceImpl, then collected the logs of setupBeforeClass
> {noformat}
> 2015-02-21 20:39:16,362 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): secret=====0
> 2015-02-21 20:39:16,362 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): topsecret=====0
> 2015-02-21 20:39:16,362 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): confidential=====0
> 2015-02-21 20:39:16,362 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): public=====0
> 2015-02-21 20:39:16,362 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): private=====0
> 2015-02-21 20:39:16,362 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): ©ABC=====0
> 2015-02-21 20:39:16,362 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): ु=====0
> 2015-02-21 20:39:16,362 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): ©ABC"ु\secret"'&\=====0
> 2015-02-21 20:39:16,363 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): '"+=====0
> 2015-02-21 20:39:16,363 INFO
> [B.defaultRpcServer.handler=0,queue=0,port=42678]
> visibility.DefaultVisibilityLabelServiceImpl(236): -?=====0
> 2015-02-21 20:39:16,363 INFO [main-EventThread]
> visibility.VisibilityLabelsCache(115): ========refresh
> java.lang.Exception
> at
> org.apache.hadoop.hbase.security.visibility.VisibilityLabelsCache.refreshLabelsCache(VisibilityLabelsCache.java:115)
> at
> org.apache.hadoop.hbase.security.visibility.ZKVisibilityLabelWatcher.refreshVisibilityLabelsCache(ZKVisibilityLabelWatcher.java:76)
> at
> org.apache.hadoop.hbase.security.visibility.ZKVisibilityLabelWatcher.nodeDataChanged(ZKVisibilityLabelWatcher.java:115)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:412)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> {noformat}
> We can see that, refresh is triggered from a background zk watcher, and it is
> possible that the labels are not in labelsCache after addLabels being called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
