[jira] [Commented] (HBASE-14809) Namespace permission granted to group

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005263#comment-15005263
 ] 

Hadoop QA commented on HBASE-14809:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12772352/14809-v2.txt
  against master branch at commit 7c3c9ac9c67cd03f9a915f528d22cb4ed81cb6e8.
  ATTACHMENT ID: 12772352

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.security.access.TestNamespaceCommands

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16519//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16519//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16519//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16519//console

This message is automatically generated.

> Namespace permission granted to group 
> --
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
> Attachments: 14809-v1.txt, 14809-v2.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> The way I read the documentation a NS admin should be able to grant as it has 
> 

[jira] [Commented] (HBASE-14355) Scan different TimeRange for each column family

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005296#comment-15005296
 ] 

Hudson commented on HBASE-14355:


SUCCESS: Integrated in HBase-1.3 #365 (See 
[https://builds.apache.org/job/HBase-1.3/365/])
HBASE-14355 Scan different TimeRange for each column family (stack: rev 
336eb92016430752b8a5a6771efc476dba92d706)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* hbase-protocol/src/main/protobuf/Client.proto
* hbase-protocol/src/main/protobuf/HBase.proto
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Query.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Get.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java


> Scan different TimeRange for each column family
> ---
>
> Key: HBASE-14355
> URL: https://issues.apache.org/jira/browse/HBASE-14355
> Project: HBase
>  Issue Type: New Feature
>  Components: Client, regionserver, Scanners
>Reporter: Dave Latham
>Assignee: churro morales
> Fix For: 2.0.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-14355-v1.patch, HBASE-14355-v10.patch, 
> HBASE-14355-v11.patch, HBASE-14355-v2.patch, HBASE-14355-v3.patch, 
> HBASE-14355-v4.patch, HBASE-14355-v5.patch, HBASE-14355-v6.patch, 
> HBASE-14355-v7.patch, HBASE-14355-v8.patch, HBASE-14355-v9.patch, 
> HBASE-14355.branch-1.patch, HBASE-14355.patch
>
>
> At present the Scan API supports only table level time range. We have 
> specific use cases that will benefit from per column family time range. (See 
> background discussion at 
> https://mail-archives.apache.org/mod_mbox/hbase-user/201508.mbox/%3ccaa4mzom00ef5eoxstk0hetxeby8mqss61gbvgttgpaspmhq...@mail.gmail.com%3E)
> There are a couple of choices that would be good to validate.  First - how to 
> update the Scan API to support family and table level updates.  One proposal 
> would be to add Scan.setTimeRange(byte family, long minTime, long maxTime), 
> then store it in a Map.  When executing the scan, if a 
> family has a specified TimeRange, then use it, otherwise fall back to using 
> the table level TimeRange.  Clients using the new API against old region 
> servers would not get the families correctly filterd.  Old clients sending 
> scans to new region servers would work correctly.
> The other question is how to get StoreFileScanner.shouldUseScanner to match 
> up the proper family and time range.  It has the Scan available but doesn't 
> currently have available which family it is a part of.  One option would be 
> to try to pass down the column family in each constructor path.  Another 
> would be to instead alter shouldUseScanner to pass down the specific 
> TimeRange to use (similar to how it currently passes down the columns to use 
> which also appears to be a workaround for not having the family available). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14807) TestWALLockup is flakey

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005295#comment-15005295
 ] 

Hudson commented on HBASE-14807:


SUCCESS: Integrated in HBase-1.3 #365 (See 
[https://builds.apache.org/job/HBase-1.3/365/])
HBASE-14807 TestWALLockup is flakey (stack: rev 
72fbfb589ede3786de1d635476bb24f51b1548da)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java


> TestWALLockup is flakey
> ---
>
> Key: HBASE-14807
> URL: https://issues.apache.org/jira/browse/HBASE-14807
> Project: HBase
>  Issue Type: Bug
>  Components: flakey, test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14807.patch
>
>
> Fails frequently. 
> Looks like this:
> {code}
> 2015-11-12 10:38:51,812 DEBUG [Time-limited test] regionserver.HRegion(3882): 
> Found 0 recovered edits file(s) under 
> /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/c8694b53368f3301a8d370089120388d
> 2015-11-12 10:38:51,821 DEBUG [Time-limited test] 
> regionserver.FlushLargeStoresPolicy(56): 
> hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use 
> global config(16777216) instead
> 2015-11-12 10:38:51,880 DEBUG [Time-limited test] wal.WALSplitter(729): Wrote 
> region 
> seqId=/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/c8694b53368f3301a8d370089120388d/recovered.edits/2.seqid
>  to file, newSeqId=2, maxSeqId=0
> 2015-11-12 10:38:51,881 INFO  [Time-limited test] regionserver.HRegion(868): 
> Onlined c8694b53368f3301a8d370089120388d; next sequenceid=2
> 2015-11-12 10:38:51,994 ERROR [sync.1] wal.FSHLog$SyncRunner(1226): Error 
> syncing, request close of WAL
> java.io.IOException: FAKE! Failed to replace a bad datanode...SYNC
>   at 
> org.apache.hadoop.hbase.regionserver.TestWALLockup$1DodgyFSLog$1.sync(TestWALLockup.java:162)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1222)
>   at java.lang.Thread.run(Thread.java:745)
> 2015-11-12 10:38:51,997 DEBUG [Thread-4] regionserver.LogRoller(139): WAL 
> roll requested
> 2015-11-12 10:38:52,019 DEBUG [flusher] 
> regionserver.FlushLargeStoresPolicy(100): Since none of the CFs were above 
> the size, flushing all.
> 2015-11-12 10:38:52,192 INFO  [Thread-4] 
> regionserver.TestWALLockup$1DodgyFSLog(129): LATCHED
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:146)
>   at 
> org.apache.hadoop.hbase.regionserver.TestWALLockup.testLockupWhenSyncInMiddleOfZigZagSetup(TestWALLockup.java:245)
> 2015-11-12 10:39:18,609 INFO  [main] regionserver.TestWALLockup(91): Cleaning 
> test directory: 
> /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> ... then times out after being locked up for 30 seconds.  Writes 50+MB of 
> logs while spinning.
> Reported as this:
> {code}
> ---
> Test set: org.apache.hadoop.hbase.regionserver.TestWALLockup
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 198.23 sec 
> <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestWALLockup
> 

[jira] [Commented] (HBASE-14802) Replaying server crash recovery procedure after a failover causes incorrect handling of deadservers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005297#comment-15005297
 ] 

Hudson commented on HBASE-14802:


SUCCESS: Integrated in HBase-1.3 #365 (See 
[https://builds.apache.org/job/HBase-1.3/365/])
HBASE-14802 Replaying server crash recovery procedure after a failover (stack: 
rev bb9fbdb2d2967a001e3c3f2613b82c85c5125199)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDeadServer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java


> Replaying server crash recovery procedure after a failover causes incorrect 
> handling of deadservers
> ---
>
> Key: HBASE-14802
> URL: https://issues.apache.org/jira/browse/HBASE-14802
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14802-1.patch, HBASE-14802-2.patch, 
> HBASE-14802-3.patch, HBASE-14802.patch
>
>
> The way dead servers are processed is that a ServerCrashProcedure is launched 
> for a server after it is added to the dead servers list. 
> Every time a server is added to the dead list, a counter "numProcessing" is 
> incremented and it is decremented when a crash recovery procedure finishes. 
> Since, adding a dead server and recovering it are two separate events, it can 
> cause inconsistencies.
> If a master failover occurs in the middle of the crash recovery, the 
> numProcessing counter resets but the ServerCrashProcedure is replayed by the 
> new master. This causes the counter to go negative and makes the master think 
> that dead servers are still in process of recovery. 
> This has ramifications on the balancer that the balancer ceases to run after 
> such a failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14798) NPE reporting server load causes regionserver abort; causes TestAcidGuarantee to fail

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005268#comment-15005268
 ] 

Hadoop QA commented on HBASE-14798:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12772355/14798.patch
  against master branch at commit 1fa7b71cf82cc30757ecf5d2a8e0cfba654ed469.
  ATTACHMENT ID: 12772355

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16520//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16520//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16520//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16520//console

This message is automatically generated.

> NPE reporting server load causes regionserver abort; causes TestAcidGuarantee 
> to fail
> -
>
> Key: HBASE-14798
> URL: https://issues.apache.org/jira/browse/HBASE-14798
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Attachments: 14798.patch, 14798.patch
>
>
> Below crashed out a RS. Caused TestAcidGuarantees to fail because then there 
> were not RS to assign too... 
> {code}
> 2015-11-11 11:36:23,092 ERROR 
> [B.defaultRpcServer.handler=4,queue=0,port=58655] 
> master.MasterRpcServices(388): Region server 
> asf907.gq1.ygridcore.net,55184,1447241756717 reported a fatal error:
> ABORTING region server asf907.gq1.ygridcore.net,55184,1447241756717: 
> Unhandled: null
> Cause:
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getOldestHfileTs(HRegion.java:1643)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:1503)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1210)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Here is the failure: 
> 

[jira] [Commented] (HBASE-14807) TestWALLockup is flakey

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005321#comment-15005321
 ] 

Hudson commented on HBASE-14807:


SUCCESS: Integrated in HBase-1.2 #368 (See 
[https://builds.apache.org/job/HBase-1.2/368/])
HBASE-14807 TestWALLockup is flakey (stack: rev 
4e6e93f26b417073fb7fb2d471ef2d8baf5422d9)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java


> TestWALLockup is flakey
> ---
>
> Key: HBASE-14807
> URL: https://issues.apache.org/jira/browse/HBASE-14807
> Project: HBase
>  Issue Type: Bug
>  Components: flakey, test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14807.patch
>
>
> Fails frequently. 
> Looks like this:
> {code}
> 2015-11-12 10:38:51,812 DEBUG [Time-limited test] regionserver.HRegion(3882): 
> Found 0 recovered edits file(s) under 
> /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/c8694b53368f3301a8d370089120388d
> 2015-11-12 10:38:51,821 DEBUG [Time-limited test] 
> regionserver.FlushLargeStoresPolicy(56): 
> hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use 
> global config(16777216) instead
> 2015-11-12 10:38:51,880 DEBUG [Time-limited test] wal.WALSplitter(729): Wrote 
> region 
> seqId=/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/c8694b53368f3301a8d370089120388d/recovered.edits/2.seqid
>  to file, newSeqId=2, maxSeqId=0
> 2015-11-12 10:38:51,881 INFO  [Time-limited test] regionserver.HRegion(868): 
> Onlined c8694b53368f3301a8d370089120388d; next sequenceid=2
> 2015-11-12 10:38:51,994 ERROR [sync.1] wal.FSHLog$SyncRunner(1226): Error 
> syncing, request close of WAL
> java.io.IOException: FAKE! Failed to replace a bad datanode...SYNC
>   at 
> org.apache.hadoop.hbase.regionserver.TestWALLockup$1DodgyFSLog$1.sync(TestWALLockup.java:162)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1222)
>   at java.lang.Thread.run(Thread.java:745)
> 2015-11-12 10:38:51,997 DEBUG [Thread-4] regionserver.LogRoller(139): WAL 
> roll requested
> 2015-11-12 10:38:52,019 DEBUG [flusher] 
> regionserver.FlushLargeStoresPolicy(100): Since none of the CFs were above 
> the size, flushing all.
> 2015-11-12 10:38:52,192 INFO  [Thread-4] 
> regionserver.TestWALLockup$1DodgyFSLog(129): LATCHED
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:146)
>   at 
> org.apache.hadoop.hbase.regionserver.TestWALLockup.testLockupWhenSyncInMiddleOfZigZagSetup(TestWALLockup.java:245)
> 2015-11-12 10:39:18,609 INFO  [main] regionserver.TestWALLockup(91): Cleaning 
> test directory: 
> /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> ... then times out after being locked up for 30 seconds.  Writes 50+MB of 
> logs while spinning.
> Reported as this:
> {code}
> ---
> Test set: org.apache.hadoop.hbase.regionserver.TestWALLockup
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 198.23 sec 
> <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestWALLockup
> 

[jira] [Commented] (HBASE-14355) Scan different TimeRange for each column family

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005322#comment-15005322
 ] 

Hudson commented on HBASE-14355:


SUCCESS: Integrated in HBase-1.2 #368 (See 
[https://builds.apache.org/job/HBase-1.2/368/])
HBASE-14355 Scan different TimeRange for each column family (stack: rev 
76187d116110c0d67d3a9b73c9e5c25cdfea5ea6)
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Query.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* hbase-protocol/src/main/protobuf/Client.proto
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Get.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
* hbase-protocol/src/main/protobuf/HBase.proto
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java


> Scan different TimeRange for each column family
> ---
>
> Key: HBASE-14355
> URL: https://issues.apache.org/jira/browse/HBASE-14355
> Project: HBase
>  Issue Type: New Feature
>  Components: Client, regionserver, Scanners
>Reporter: Dave Latham
>Assignee: churro morales
> Fix For: 2.0.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-14355-v1.patch, HBASE-14355-v10.patch, 
> HBASE-14355-v11.patch, HBASE-14355-v2.patch, HBASE-14355-v3.patch, 
> HBASE-14355-v4.patch, HBASE-14355-v5.patch, HBASE-14355-v6.patch, 
> HBASE-14355-v7.patch, HBASE-14355-v8.patch, HBASE-14355-v9.patch, 
> HBASE-14355.branch-1.patch, HBASE-14355.patch
>
>
> At present the Scan API supports only table level time range. We have 
> specific use cases that will benefit from per column family time range. (See 
> background discussion at 
> https://mail-archives.apache.org/mod_mbox/hbase-user/201508.mbox/%3ccaa4mzom00ef5eoxstk0hetxeby8mqss61gbvgttgpaspmhq...@mail.gmail.com%3E)
> There are a couple of choices that would be good to validate.  First - how to 
> update the Scan API to support family and table level updates.  One proposal 
> would be to add Scan.setTimeRange(byte family, long minTime, long maxTime), 
> then store it in a Map.  When executing the scan, if a 
> family has a specified TimeRange, then use it, otherwise fall back to using 
> the table level TimeRange.  Clients using the new API against old region 
> servers would not get the families correctly filterd.  Old clients sending 
> scans to new region servers would work correctly.
> The other question is how to get StoreFileScanner.shouldUseScanner to match 
> up the proper family and time range.  It has the Scan available but doesn't 
> currently have available which family it is a part of.  One option would be 
> to try to pass down the column family in each constructor path.  Another 
> would be to instead alter shouldUseScanner to pass down the specific 
> TimeRange to use (similar to how it currently passes down the columns to use 
> which also appears to be a workaround for not having the family available). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14802) Replaying server crash recovery procedure after a failover causes incorrect handling of deadservers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005323#comment-15005323
 ] 

Hudson commented on HBASE-14802:


SUCCESS: Integrated in HBase-1.2 #368 (See 
[https://builds.apache.org/job/HBase-1.2/368/])
HBASE-14802 Replaying server crash recovery procedure after a failover (stack: 
rev b22feba7fe2b176ef2578264f8b95947df592bba)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDeadServer.java


> Replaying server crash recovery procedure after a failover causes incorrect 
> handling of deadservers
> ---
>
> Key: HBASE-14802
> URL: https://issues.apache.org/jira/browse/HBASE-14802
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14802-1.patch, HBASE-14802-2.patch, 
> HBASE-14802-3.patch, HBASE-14802.patch
>
>
> The way dead servers are processed is that a ServerCrashProcedure is launched 
> for a server after it is added to the dead servers list. 
> Every time a server is added to the dead list, a counter "numProcessing" is 
> incremented and it is decremented when a crash recovery procedure finishes. 
> Since, adding a dead server and recovering it are two separate events, it can 
> cause inconsistencies.
> If a master failover occurs in the middle of the crash recovery, the 
> numProcessing counter resets but the ServerCrashProcedure is replayed by the 
> new master. This causes the counter to go negative and makes the master think 
> that dead servers are still in process of recovery. 
> This has ramifications on the balancer that the balancer ceases to run after 
> such a failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14533) Thrift client gets "AsyncProcess: Failed to get region location .... closed"

[Pankaj Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005344#comment-15005344
 ] 

Pankaj Kumar commented on HBASE-14533:
--

I also met same issue today.
Thrift server log,
{code}
2015-11-14 19:04:04,631 ERROR [pool-1-thread-136] client.AsyncProcess: Failed 
to get region location
java.io.IOException: hconnection-0x17f35451 closed
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1149)
at 
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
at 
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
at 
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
at 
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1432)
{code}

Thrift Java Client error,
{code}
2015-11-14 19:30:48,930 INFO  [pool-1-thread-4] Thrift2Client$RunSomeThing(60): 
name_7:123
TIOError(message:Failed 100 actions: IOException: 100 times, )
at 
org.apache.hadoop.hbase.thrift2.generated.THBaseService$putMultiple_result$putMultiple_resultStandardScheme.read(THBaseService.java:7918)
at 
org.apache.hadoop.hbase.thrift2.generated.THBaseService$putMultiple_result$putMultiple_resultStandardScheme.read(THBaseService.java:1)
at 
org.apache.hadoop.hbase.thrift2.generated.THBaseService$putMultiple_result.read(THBaseService.java:7854)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at 
org.apache.hadoop.hbase.thrift2.generated.THBaseService$Client.recv_putMultiple(THBaseService.java:436)
at 
org.apache.hadoop.hbase.thrift2.generated.THBaseService$Client.putMultiple(THBaseService.java:422)
{code}

Reproduce steps, doing batchput continuously for more than 10 minutes.

> Thrift client gets "AsyncProcess: Failed to get region location  closed"
> 
>
> Key: HBASE-14533
> URL: https://issues.apache.org/jira/browse/HBASE-14533
> Project: HBase
>  Issue Type: Bug
>  Components: REST, Thrift
>Affects Versions: 1.0.0
>Reporter: stack
>Assignee: stack
> Attachments: 14533.test.patch, 14533v2.branch-1.patch, test.patch
>
>
> An internal python client has been getting below stack trace since 
> HBASE-134347
> {code}
> 2015-09-30 11:27:31,670 runnerERROR   : scheduler 
> executor error
> 2015-09-30 11:27:31,674 runnerERROR   : Traceback (most 
> recent call last):
>   File 
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/runner.py",
>  line 82, in run
> fetch_list = self.__scheduler_executor.run()
>   File 
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/scheduler.py",
>  line 35, in run
> with self.__fetch_db_dao.get_scanner() as scanner:
>   File 
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_dao.py",
>  line 57, in get_scanner
> caching=caching, field_filter_list=field_filter_list)
>   File 
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_client_template.py",
>  line 104, in get_entity_scanner
> self.__fix_cfs(self.__filter_columns(field_filter_list)), caching)
>   File 
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_entity_scanner.py",
>  line 81, in open
> self.__scanner_id = client.scannerOpenWithScan(table_name, scan)
>   File 
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
>  line 1494, in scannerOpenWithScan
> return self.recv_scannerOpenWithScan()
>   File 
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
>  line 1518, in recv_scannerOpenWithScan
> raise result.io
> IOError: 
> IOError(message="org.apache.hadoop.hbase.client.RetriesExhaustedException: 
> Can't get the location\n\tat 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308)\n\tat
>  
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149)\n\tat
>  
> 

[jira] [Commented] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable

[Pankaj Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005327#comment-15005327
 ] 

Pankaj Kumar commented on HBASE-14498:
--

How does the test replicate what the original description describes? It is a 
tricky scenario. Thanks for reporting it. I am afraid that we may not have 
actually fixed the scenario described.
>> As per the issue,
- ZKs were not reachable to HM
- HM received DISCONNECT event continuously. 
- On DISCONNECT we were just ignoring and keep retrying regardless of 
zookeeper.session.timeout. That is why master was not aborted even after 
zookeeper.session.timeout.
   I tried to simulate the same scenario in the test case. 

isConnected is the name of a method you would invoke to check a boolean named 
connected. It is not what you should name a variable.
>> I will modify the variable name.

Is this right?
connWaitTimeOut = this.conf.getLong("zookeeper.session.timeout", 9) * 2 / 3;
IIRC, you ask zk for a session timeout and it may give you something other than 
what you asked for (it is a while since I dug in here)
>> The idea is,
The time interval (t) should be less than the ZK Session time out. (May 
be 2/3rd of session time out value ) , This is to make sure that standby HM 
will not become active within this time period.

You drop the prefix here:
LOG.debug("Received Disconnected from ZooKeeper.");
prefix helps debugging... otherwise these zk logs are hard to trace to their 
origin.
>> my bad, will revert this.

Every call into a disconnect is going to spawn a new one of these unnamed 
threads?
>> Yeah daemon thread will be spawned and will be active util connWaitTimeOut 
>> or SyncConnected.


Did you see the below message in your log output?
LOG.debug(prefix("Received Disconnected from ZooKeeper, ignoring"));
>>  Yes, it was written.

The idea is that we could disconnect but we'll keep trying to reconnect for zk 
session timeout and may succeed? Has the zk session timeout expired when we get 
this disconnect message? Should we abort as soon as we get one of these (I 
wonder why we have the comment that says abort when we get such a message but 
we don't actually? Because the abort is done elsewhere?)
>> In this scenario, ZK session will not expire for HM (zk-client) because as 
>> far as I know session expire task is initiated by ZK server (please correct 
>> me if i'm wrong), zk-client don't handle this. 
   So HM will receive DISCONNECT even and keep trying forever util it connect 
to ZK.
   
Thanks.

> Master stuck in infinite loop when all Zookeeper servers are unreachable
> 
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: HBASE-14498-V2.patch, HBASE-14498-V3.patch, 
> HBASE-14498-V4.patch, HBASE-14498.patch
>
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to 
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO 
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)] 
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in 
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and 
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO 
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)] 
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)] 
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN 
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO 
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Opening socket connection to server 
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown 
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Opening socket connection to server 
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown 
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)] 
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in 
> 30023ms for sessionid 0x2045762cc710006, 

[jira] [Commented] (HBASE-14807) TestWALLockup is flakey

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005339#comment-15005339
 ] 

Hudson commented on HBASE-14807:


FAILURE: Integrated in HBase-Trunk_matrix #463 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/463/])
HBASE-14807 TestWALLockup is flakey (stack: rev 
1fa7b71cf82cc30757ecf5d2a8e0cfba654ed469)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java


> TestWALLockup is flakey
> ---
>
> Key: HBASE-14807
> URL: https://issues.apache.org/jira/browse/HBASE-14807
> Project: HBase
>  Issue Type: Bug
>  Components: flakey, test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14807.patch
>
>
> Fails frequently. 
> Looks like this:
> {code}
> 2015-11-12 10:38:51,812 DEBUG [Time-limited test] regionserver.HRegion(3882): 
> Found 0 recovered edits file(s) under 
> /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/c8694b53368f3301a8d370089120388d
> 2015-11-12 10:38:51,821 DEBUG [Time-limited test] 
> regionserver.FlushLargeStoresPolicy(56): 
> hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use 
> global config(16777216) instead
> 2015-11-12 10:38:51,880 DEBUG [Time-limited test] wal.WALSplitter(729): Wrote 
> region 
> seqId=/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/c8694b53368f3301a8d370089120388d/recovered.edits/2.seqid
>  to file, newSeqId=2, maxSeqId=0
> 2015-11-12 10:38:51,881 INFO  [Time-limited test] regionserver.HRegion(868): 
> Onlined c8694b53368f3301a8d370089120388d; next sequenceid=2
> 2015-11-12 10:38:51,994 ERROR [sync.1] wal.FSHLog$SyncRunner(1226): Error 
> syncing, request close of WAL
> java.io.IOException: FAKE! Failed to replace a bad datanode...SYNC
>   at 
> org.apache.hadoop.hbase.regionserver.TestWALLockup$1DodgyFSLog$1.sync(TestWALLockup.java:162)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1222)
>   at java.lang.Thread.run(Thread.java:745)
> 2015-11-12 10:38:51,997 DEBUG [Thread-4] regionserver.LogRoller(139): WAL 
> roll requested
> 2015-11-12 10:38:52,019 DEBUG [flusher] 
> regionserver.FlushLargeStoresPolicy(100): Since none of the CFs were above 
> the size, flushing all.
> 2015-11-12 10:38:52,192 INFO  [Thread-4] 
> regionserver.TestWALLockup$1DodgyFSLog(129): LATCHED
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:146)
>   at 
> org.apache.hadoop.hbase.regionserver.TestWALLockup.testLockupWhenSyncInMiddleOfZigZagSetup(TestWALLockup.java:245)
> 2015-11-12 10:39:18,609 INFO  [main] regionserver.TestWALLockup(91): Cleaning 
> test directory: 
> /home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.7/label/Hadoop/hbase-server/target/test-data/8b8f8f12-1819-47e3-b1f1-8ffa789438ad
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> ... then times out after being locked up for 30 seconds.  Writes 50+MB of 
> logs while spinning.
> Reported as this:
> {code}
> ---
> Test set: org.apache.hadoop.hbase.regionserver.TestWALLockup
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 198.23 sec 
> <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestWALLockup
> 

[jira] [Commented] (HBASE-14802) Replaying server crash recovery procedure after a failover causes incorrect handling of deadservers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005340#comment-15005340
 ] 

Hudson commented on HBASE-14802:


FAILURE: Integrated in HBase-Trunk_matrix #463 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/463/])
HBASE-14802 Replaying server crash recovery procedure after a failover (stack: 
rev 7c3c9ac9c67cd03f9a915f528d22cb4ed81cb6e8)
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDeadServer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java


> Replaying server crash recovery procedure after a failover causes incorrect 
> handling of deadservers
> ---
>
> Key: HBASE-14802
> URL: https://issues.apache.org/jira/browse/HBASE-14802
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14802-1.patch, HBASE-14802-2.patch, 
> HBASE-14802-3.patch, HBASE-14802.patch
>
>
> The way dead servers are processed is that a ServerCrashProcedure is launched 
> for a server after it is added to the dead servers list. 
> Every time a server is added to the dead list, a counter "numProcessing" is 
> incremented and it is decremented when a crash recovery procedure finishes. 
> Since, adding a dead server and recovering it are two separate events, it can 
> cause inconsistencies.
> If a master failover occurs in the middle of the crash recovery, the 
> numProcessing counter resets but the ServerCrashProcedure is replayed by the 
> new master. This causes the counter to go negative and makes the master think 
> that dead servers are still in process of recovery. 
> This has ramifications on the balancer that the balancer ceases to run after 
> such a failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14597) Fix Groups cache in multi-threaded env

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005357#comment-15005357
 ] 

Hudson commented on HBASE-14597:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1132 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1132/])
HBASE-14597 Fix Groups cache in multi-threaded env (apurtell: rev 
d29f68dc1cfc12adbb9348629428291736ae24c0)
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUser.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/security/UserProvider.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java


> Fix Groups cache in multi-threaded env
> --
>
> Key: HBASE-14597
> URL: https://issues.apache.org/jira/browse/HBASE-14597
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-14597-v1.patch, HBASE-14597-v2.patch, 
> HBASE-14597-v4.patch, HBASE-14597-v5.patch, HBASE-14597-v6.patch, 
> HBASE-14597.patch
>
>
> UGI doesn't hash based on the user as expected so since we have lots of ugi 
> potentially created the cache doesn't do it's job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14512) Cache UGI groups

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005356#comment-15005356
 ] 

Hudson commented on HBASE-14512:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1132 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1132/])
HBASE-14512 Cache UGI groups (apurtell: rev 
8d94773044a68ba601eecd4a08b02c48df5cbaa7)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/security/UserProvider.java


> Cache UGI groups
> 
>
> Key: HBASE-14512
> URL: https://issues.apache.org/jira/browse/HBASE-14512
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, security
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-14512-v1.patch, HBASE-14512-v2.patch, 
> HBASE-14512-v3.patch, HBASE-14512-v4.patch, HBASE-14512.patch
>
>
> Right now every call gets a new User object.
> We should keep the same user for the life of a connection. We should also 
> cache the group names. However we can't cache the groups for forever as that 
> would mean groups don't get refreshed every 5 mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14809) Grant / revoke Namespace admin permission to group

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005738#comment-15005738
 ] 

Ashish Singhi commented on HBASE-14809:
---

+1 (non-binding)

> Grant / revoke Namespace admin permission to group 
> ---
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt, 
> 14809-v4.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005742#comment-15005742
 ] 

Ashish Singhi commented on HBASE-13153:
---

Attached patch(v14) which corrects the logic for bulk load entry filter in 
WALEntryFilter's.
Please review.

> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, 
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, 
> HBASE-13153-v14.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, 
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, 
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, 
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load 
> Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load 
> Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster 
> tolerance scenario.But we encounter an issue that we will use bulkload very 
> frequently,because bulkload bypass write path, and will not generate WAL, so 
> the data will not be replicated to backup cluster. It's inappropriate to 
> bukload twice both on active cluster and backup cluster. So i advise do some 
> modification to bulkload feature to enable bukload to both active cluster and 
> backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14809) Grant / revoke Namespace admin permission to group

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005755#comment-15005755
 ] 

Jerry He commented on HBASE-14809:
--

Patch looks good.
+1

> Grant / revoke Namespace admin permission to group 
> ---
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt, 
> 14809-v4.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14815) TestMobExportSnapshot.testExportFailure timeout occasionally

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005773#comment-15005773
 ] 

Heng Chen commented on HBASE-14815:
---

The reason is that  we set  mapreduce.map.maxattempts to 10 in 
TestMobExportSnapshot.

It is reasonable for other tests but not suitable for testExportFailure. 
Because each mapper will failed in testExportFailure,  MRCluster will retry too 
many times.
IMO we should decrease this number for testExportFailure.

> TestMobExportSnapshot.testExportFailure timeout occasionally
> 
>
> Key: HBASE-14815
> URL: https://issues.apache.org/jira/browse/HBASE-14815
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
>
> On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.
> See
> https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14815) TestMobExportSnapshot.testExportFailure timeout occasionally

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14815:
--
Attachment: HBASE-14815.patch

> TestMobExportSnapshot.testExportFailure timeout occasionally
> 
>
> Key: HBASE-14815
> URL: https://issues.apache.org/jira/browse/HBASE-14815
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14815.patch
>
>
> On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.
> See
> https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-14807) TestWALLockup is flakey

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-14807:
---

Looks like its possible for the test to run in a manner not expected. See 
recent fail on master build: 
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=Hadoop/lastCompletedBuild/testReport/
 where the latch in FSHLog goes into place only after all the drama has passed. 
Deal with it.


2015-11-15 01:59:20,121 DEBUG [StoreOpener-b654ec7a15b8c66e584fed20463b2fcc-1] 
regionserver.HRegionFileSystem(202): No StoreFiles for: 
/home/jenkins/jenkins-slave/workspace/HBase-Trunk_matrix/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/cf948850-365d-462e-bf8f-e0561fc9e6d8/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/b654ec7a15b8c66e584fed20463b2fcc/MyCF
2015-11-15 01:59:20,156 DEBUG [Time-limited test] regionserver.HRegion(3869): 
Found 0 recovered edits file(s) under 
/home/jenkins/jenkins-slave/workspace/HBase-Trunk_matrix/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/cf948850-365d-462e-bf8f-e0561fc9e6d8/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/b654ec7a15b8c66e584fed20463b2fcc
2015-11-15 01:59:20,161 DEBUG [Time-limited test] 
regionserver.FlushLargeStoresPolicy(56): 
hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use 
global config(16777216) instead
2015-11-15 01:59:20,219 DEBUG [Time-limited test] wal.WALSplitter(720): Wrote 
region 
seqId=/home/jenkins/jenkins-slave/workspace/HBase-Trunk_matrix/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/cf948850-365d-462e-bf8f-e0561fc9e6d8/data/default/testLockupWhenSyncInMiddleOfZigZagSetup/b654ec7a15b8c66e584fed20463b2fcc/recovered.edits/2.seqid
 to file, newSeqId=2, maxSeqId=0
2015-11-15 01:59:20,220 INFO  [Time-limited test] regionserver.HRegion(866): 
Onlined b654ec7a15b8c66e584fed20463b2fcc; next sequenceid=2
2015-11-15 01:59:20,696 INFO  [Time-limited test] 
regionserver.TestWALLockup(221): SET throwing of exception on append
2015-11-15 01:59:20,699 WARN  [Time-limited test.append-pool1-t1] 
wal.FSHLog$RingBufferEventHandler(1880): Append sequenceId=-1, requesting roll 
of WAL
java.io.IOException: FAKE! Failed to replace a bad datanode...APPEND
at 
org.apache.hadoop.hbase.regionserver.TestWALLockup$1DodgyFSLog$1.append(TestWALLockup.java:170)
at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1869)
at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1737)
at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1659)
at 
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-11-15 01:59:20,712 DEBUG [Thread-5] regionserver.LogRoller(135): WAL roll 
requested
2015-11-15 01:59:20,713 DEBUG [Flusher] 
regionserver.FlushLargeStoresPolicy(100): Since none of the CFs were above the 
size, flushing all.
2015-11-15 01:59:20,714 INFO  [Flusher] regionserver.HRegion(2155): Flushing 
1/1 column families, memstore=226.56 KB
2015-11-15 01:59:20,730 INFO  [Flusher] regionserver.TestWALLockup$1(248): In 
flush
org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: On sync
at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1782)
at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1659)
at 
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append 
sequenceId=-1, requesting roll of WAL
at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1882)
at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1737)
... 5 more
Caused by: java.io.IOException: FAKE! Failed to replace a bad datanode...APPEND
at 
org.apache.hadoop.hbase.regionserver.TestWALLockup$1DodgyFSLog$1.append(TestWALLockup.java:170)
at 
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1869)
... 6 more
2015-11-15 01:59:20,731 INFO  [Flusher] regionserver.TestWALLockup$1(250): 
Exiting
2015-11-15 01:59:20,734 INFO  [Thread-5] 
regionserver.TestWALLockup$1DodgyFSLog(129): LATCHED
java.lang.InterruptedException: sleep 

[jira] [Commented] (HBASE-14805) status should show the master in shell

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005753#comment-15005753
 ] 

Andrew Purtell commented on HBASE-14805:


+1 on the patch provided it passes a precommit build

> status should show the master in shell
> --
>
> Key: HBASE-14805
> URL: https://issues.apache.org/jira/browse/HBASE-14805
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: hbase-14805_v1.patch
>
>
> {{status 'simple'}} or {{'detailed'}} only shows the regionservers and 
> regions, but not the active master. Actually, there is no way to know about 
> the active masters from the shell it seems. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14805) status should show the master in shell

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14805:
---
Fix Version/s: 0.98.17

> status should show the master in shell
> --
>
> Key: HBASE-14805
> URL: https://issues.apache.org/jira/browse/HBASE-14805
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: hbase-14805_v1.patch
>
>
> {{status 'simple'}} or {{'detailed'}} only shows the regionservers and 
> regions, but not the active master. Actually, there is no way to know about 
> the active masters from the shell it seems. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14815) TestMobExportSnapshot.testExportFailure timeout occasionally

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005776#comment-15005776
 ] 

Heng Chen commented on HBASE-14815:
---

Locally,  if mapreduce.map.maxattempts is 10,  testExportFailure costs about 2 
minutes.
After set it to be 3, it costs about 40s.

> TestMobExportSnapshot.testExportFailure timeout occasionally
> 
>
> Key: HBASE-14815
> URL: https://issues.apache.org/jira/browse/HBASE-14815
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
> Attachments: HBASE-14815.patch
>
>
> On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.
> See
> https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14815) TestMobExportSnapshot.testExportFailure timeout occasionally

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14815:
--
 Assignee: Heng Chen
Fix Version/s: 2.0.0
Affects Version/s: 2.0.0
   Status: Patch Available  (was: Open)

> TestMobExportSnapshot.testExportFailure timeout occasionally
> 
>
> Key: HBASE-14815
> URL: https://issues.apache.org/jira/browse/HBASE-14815
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14815.patch
>
>
> On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.
> See
> https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14802) Replaying server crash recovery procedure after a failover causes incorrect handling of deadservers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005720#comment-15005720
 ] 

Hudson commented on HBASE-14802:


FAILURE: Integrated in HBase-Trunk_matrix #466 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/466/])
Revert "HBASE-14802 Replaying server crash recovery procedure after a (stack: 
rev bb6581345fd9ecac964e19cea2293477162801ca)
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/DeadServer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDeadServer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java


> Replaying server crash recovery procedure after a failover causes incorrect 
> handling of deadservers
> ---
>
> Key: HBASE-14802
> URL: https://issues.apache.org/jira/browse/HBASE-14802
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14802-1.patch, HBASE-14802-2.patch, 
> HBASE-14802-3.patch, HBASE-14802.patch
>
>
> The way dead servers are processed is that a ServerCrashProcedure is launched 
> for a server after it is added to the dead servers list. 
> Every time a server is added to the dead list, a counter "numProcessing" is 
> incremented and it is decremented when a crash recovery procedure finishes. 
> Since, adding a dead server and recovering it are two separate events, it can 
> cause inconsistencies.
> If a master failover occurs in the middle of the crash recovery, the 
> numProcessing counter resets but the ServerCrashProcedure is replayed by the 
> new master. This causes the counter to go negative and makes the master think 
> that dead servers are still in process of recovery. 
> This has ramifications on the balancer that the balancer ceases to run after 
> such a failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14815) TestMobExportSnapshot.testExportFailure timeout occasionally

[Heng Chen (JIRA)]

Heng Chen created HBASE-14815:
-

 Summary: TestMobExportSnapshot.testExportFailure timeout 
occasionally
 Key: HBASE-14815
 URL: https://issues.apache.org/jira/browse/HBASE-14815
 Project: HBase
  Issue Type: Bug
Reporter: Heng Chen


On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.

See
https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13153) Bulk Loaded HFile Replication

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-13153:
--
Attachment: HBASE-13153-v14.patch

> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, 
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, 
> HBASE-13153-v14.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, 
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, 
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, 
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load 
> Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load 
> Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster 
> tolerance scenario.But we encounter an issue that we will use bulkload very 
> frequently,because bulkload bypass write path, and will not generate WAL, so 
> the data will not be replicated to backup cluster. It's inappropriate to 
> bukload twice both on active cluster and backup cluster. So i advise do some 
> modification to bulkload feature to enable bukload to both active cluster and 
> backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14777:
---
Affects Version/s: (was: 0.98.16)
   (was: 1.3.0)
   (was: 1.1.2)
   (was: 1.2.0)
   (was: 1.0.2)
   (was: 2.0.0)
Fix Version/s: 1.3.0
   1.2.0
   2.0.0

> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14355) Scan different TimeRange for each column family

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005431#comment-15005431
 ] 

Hudson commented on HBASE-14355:


SUCCESS: Integrated in HBase-1.3-IT #311 (See 
[https://builds.apache.org/job/HBase-1.3-IT/311/])
HBASE-14355 Scan different TimeRange for each column family (stack: rev 
336eb92016430752b8a5a6771efc476dba92d706)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
* hbase-protocol/src/main/protobuf/Client.proto
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* hbase-protocol/src/main/protobuf/HBase.proto
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Get.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Query.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java


> Scan different TimeRange for each column family
> ---
>
> Key: HBASE-14355
> URL: https://issues.apache.org/jira/browse/HBASE-14355
> Project: HBase
>  Issue Type: New Feature
>  Components: Client, regionserver, Scanners
>Reporter: Dave Latham
>Assignee: churro morales
> Fix For: 2.0.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-14355-v1.patch, HBASE-14355-v10.patch, 
> HBASE-14355-v11.patch, HBASE-14355-v2.patch, HBASE-14355-v3.patch, 
> HBASE-14355-v4.patch, HBASE-14355-v5.patch, HBASE-14355-v6.patch, 
> HBASE-14355-v7.patch, HBASE-14355-v8.patch, HBASE-14355-v9.patch, 
> HBASE-14355.branch-1.patch, HBASE-14355.patch
>
>
> At present the Scan API supports only table level time range. We have 
> specific use cases that will benefit from per column family time range. (See 
> background discussion at 
> https://mail-archives.apache.org/mod_mbox/hbase-user/201508.mbox/%3ccaa4mzom00ef5eoxstk0hetxeby8mqss61gbvgttgpaspmhq...@mail.gmail.com%3E)
> There are a couple of choices that would be good to validate.  First - how to 
> update the Scan API to support family and table level updates.  One proposal 
> would be to add Scan.setTimeRange(byte family, long minTime, long maxTime), 
> then store it in a Map.  When executing the scan, if a 
> family has a specified TimeRange, then use it, otherwise fall back to using 
> the table level TimeRange.  Clients using the new API against old region 
> servers would not get the families correctly filterd.  Old clients sending 
> scans to new region servers would work correctly.
> The other question is how to get StoreFileScanner.shouldUseScanner to match 
> up the proper family and time range.  It has the Scan available but doesn't 
> currently have available which family it is a part of.  One option would be 
> to try to pass down the column family in each constructor path.  Another 
> would be to instead alter shouldUseScanner to pass down the specific 
> TimeRange to use (similar to how it currently passes down the columns to use 
> which also appears to be a workaround for not having the family available). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14809) Namespace permission granted to group

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14809:
---
Attachment: 14809-v3.txt

> Namespace permission granted to group 
> --
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14809) Namespace permission granted to group

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-14809:
--

Assignee: Ted Yu

> Namespace permission granted to group 
> --
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005449#comment-15005449
 ] 

Heng Chen commented on HBASE-14790:
---

IMO  we could fix HBASE-14004 as the first step.
Before we rollback memstore, we should close WAL with some certain length.
If close WAL failed,  we can set a timeout,  if exceeds this limitation, RS 
should close.
If close WAL successfully,  we go  on with a new WAL.

All we should do about DFSOutputStream is  a new close method just calling 
completeFile.

Thoughts?


> Implement a new DFSOutputStream for logging WAL only
> 
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all 
> purposes. But in fact, we do not need most of the features if we only want to 
> log WAL. For example, we do not need pipeline recovery since we could just 
> close the old logger and open a new one. And also, we do not need to write 
> multiple blocks since we could also open a new logger if the old file is too 
> large.
> And the most important thing is that, it is hard to handle all the corner 
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when 
> using original DFSOutputStream due to its complicated logic. And the 
> complicated logic also force us to use some magical tricks to increase 
> performance. For example, we need to use multiple threads to call {{hflush}} 
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when 
> logging WAL. For correctness, and also for performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14809) Namespace permission granted to group

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14809:
---
Description: 
Hi, 

We are looking to roll out HBase and are in the process to design the security 
model. 
We are looking to implement global DBAs and Namespace specific administrators. 
So for example the global dba would create a namespace and grant a user/group 
admin privileges within that ns. 
So that a given ns admin can in turn create objects and grant permission within 
the given ns only. 

We have run into some issues at the ns admin level. It appears that a ns admin 
can NOT grant to a grop unless it also has global admin privilege. But once it 
has global admin privilege it can grant in any NS not just the one where it has 
admin privileges. 

Based on the HBase documentation at 
http://hbase.apache.org/book.html#appendix_acl_matrix 

Table 13. ACL Matrix 
Interface   Operation   Permissions 
AccessController grant(global level) global(A) 
grant(namespace level) global(A)|NS(A) 

grant at a namespace level should be possible for someone with global A OR (|) 
NS A permission. 
As you will see in our test it does not work if NS A permission is granted but 
global A permission is not. 

Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
{code}
hbase(main):011:0> scan 'hbase:acl' 
ROW COLUMN+CELL 
@ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
{code}
However: 
Here you can see that a user who is member of the group hbaseappltest_ns1admin 
can not grant a WRX privilege to a group as it is missing global A privilege. 
{code}
$hbase shell 
15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
deprecated. Instead, use io.native.lib.available 
HBase Shell; enter 'help' for list of supported commands. 
Type "exit" to leave the HBase Shell 
Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 

hbase(main):001:0> whoami 
ns1ad...@wlab.net (auth:KERBEROS) 
groups: hbaseappltest_ns1admin 

hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 

ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
permissions for user 'ns1admin' (global, action=ADMIN) 
{code}
The way I read the documentation a NS admin should be able to grant as it has 
ns level A privilege not only object level permission.

CDH is a version 5.4.7 and Hbase is version 1.0. 

Regards, 
Steven

  was:
Hi, 

We are looking to roll out HBase and are in the process to design the security 
model. 
We are looking to implement global DBAs and Namespace specific administrators. 
So for example the global dba would create a namespace and grant a user/group 
admin privileges within that ns. 
So that a given ns admin can in turn create objects and grant permission within 
the given ns only. 

We have run into some issues at the ns admin level. It appears that a ns admin 
can NOT grant to a grop unless it also has global admin privilege. But once it 
has global admin privilege it can grant in any NS not just the one where it has 
admin privileges. 

Based on the HBase documentation at 
http://hbase.apache.org/book.html#appendix_acl_matrix 

Table 13. ACL Matrix 
Interface   Operation   Permissions 
AccessController grant(global level) global(A) 
grant(namespace level) global(A)|NS(A) 

grant at a namespace level should be possible for someone with global A OR (|) 
NS A permission. 
As you will see in our test it does not work if NS A permission is granted but 
global A permission is not. 

Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 

hbase(main):011:0> scan 'hbase:acl' 
ROW COLUMN+CELL 
@ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 

However: 
Here you can see that a user who is member of the group hbaseappltest_ns1admin 
can not grant a WRX privilege to a group as it is missing global A privilege. 

$hbase shell 
15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
deprecated. Instead, use io.native.lib.available 
HBase Shell; enter 'help' for list of supported commands. 
Type "exit" to leave the HBase Shell 
Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 

hbase(main):001:0> whoami 
ns1ad...@wlab.net (auth:KERBEROS) 
groups: hbaseappltest_ns1admin 

hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 

ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
permissions for user 'ns1admin' (global, action=ADMIN) 

The way I read the documentation a NS admin should be able to grant as it has 
ns level A privilege not only object level permission.

CDH is a version 5.4.7 and Hbase is version 1.0. 

Regards, 
Steven


> Namespace permission granted to group 
> --
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> 

[jira] [Updated] (HBASE-14809) Namespace admin permission granted to group

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14809:
---
Summary: Namespace admin permission granted to group   (was: Namespace 
permission granted to group )

> Namespace admin permission granted to group 
> 
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14809) Namespace permission granted to group

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005434#comment-15005434
 ] 

Ted Yu commented on HBASE-14809:


Patch v3 adds new test to TestNamespaceCommands

> Namespace permission granted to group 
> --
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14756) Break out ClientBackoffPolicy factors into configurable and stackable components

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005452#comment-15005452
 ] 

Heng Chen commented on HBASE-14756:
---

Suggestions? [~apurtell]

> Break out ClientBackoffPolicy factors into configurable and stackable 
> components
> 
>
> Key: HBASE-14756
> URL: https://issues.apache.org/jira/browse/HBASE-14756
> Project: HBase
>  Issue Type: Improvement
>Reporter: Andrew Purtell
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14756.patch
>
>
> Currently ExponentialClientBackoffPolicy evaluates three load parameters sent 
> back in results from the server. The policy is fixed in implementation. 
> Instead it should be possible to define the collection of considered load 
> factors via configuration, and for each selected term parameterize how the 
> load factor should contribute to the backoff calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14597) Fix Groups cache in multi-threaded env

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005477#comment-15005477
 ] 

Hudson commented on HBASE-14597:


FAILURE: Integrated in HBase-0.98-matrix #259 (See 
[https://builds.apache.org/job/HBase-0.98-matrix/259/])
HBASE-14597 Fix Groups cache in multi-threaded env (apurtell: rev 
d29f68dc1cfc12adbb9348629428291736ae24c0)
* hbase-common/src/main/java/org/apache/hadoop/hbase/security/UserProvider.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUser.java


> Fix Groups cache in multi-threaded env
> --
>
> Key: HBASE-14597
> URL: https://issues.apache.org/jira/browse/HBASE-14597
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-14597-v1.patch, HBASE-14597-v2.patch, 
> HBASE-14597-v4.patch, HBASE-14597-v5.patch, HBASE-14597-v6.patch, 
> HBASE-14597.patch
>
>
> UGI doesn't hash based on the user as expected so since we have lots of ugi 
> potentially created the cache doesn't do it's job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14512) Cache UGI groups

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005476#comment-15005476
 ] 

Hudson commented on HBASE-14512:


FAILURE: Integrated in HBase-0.98-matrix #259 (See 
[https://builds.apache.org/job/HBase-0.98-matrix/259/])
HBASE-14512 Cache UGI groups (apurtell: rev 
8d94773044a68ba601eecd4a08b02c48df5cbaa7)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/security/User.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/security/UserProvider.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java


> Cache UGI groups
> 
>
> Key: HBASE-14512
> URL: https://issues.apache.org/jira/browse/HBASE-14512
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, security
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-14512-v1.patch, HBASE-14512-v2.patch, 
> HBASE-14512-v3.patch, HBASE-14512-v4.patch, HBASE-14512.patch
>
>
> Right now every call gets a new User object.
> We should keep the same user for the life of a connection. We should also 
> cache the group names. However we can't cache the groups for forever as that 
> would mean groups don't get refreshed every 5 mins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14809) Namespace admin permission granted to group

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14809:
---
Attachment: 14809-v3.txt

> Namespace admin permission granted to group 
> 
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14798) NPE reporting server load causes regionserver abort; causes TestAcidGuarantee to fail

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14798:
--
   Resolution: Fixed
Fix Version/s: 1.3.0
   1.2.0
   2.0.0
   Status: Resolved  (was: Patch Available)

Pushed to 1.2+

> NPE reporting server load causes regionserver abort; causes TestAcidGuarantee 
> to fail
> -
>
> Key: HBASE-14798
> URL: https://issues.apache.org/jira/browse/HBASE-14798
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14798.patch, 14798.patch
>
>
> Below crashed out a RS. Caused TestAcidGuarantees to fail because then there 
> were not RS to assign too... 
> {code}
> 2015-11-11 11:36:23,092 ERROR 
> [B.defaultRpcServer.handler=4,queue=0,port=58655] 
> master.MasterRpcServices(388): Region server 
> asf907.gq1.ygridcore.net,55184,1447241756717 reported a fatal error:
> ABORTING region server asf907.gq1.ygridcore.net,55184,1447241756717: 
> Unhandled: null
> Cause:
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getOldestHfileTs(HRegion.java:1643)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:1503)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1210)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Here is the failure: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/457/jdk=latest1.8,label=Hadoop/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14809) Namespace admin permission granted to group

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005471#comment-15005471
 ] 

Hadoop QA commented on HBASE-14809:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12772369/14809-v3.txt
  against master branch at commit 1fa7b71cf82cc30757ecf5d2a8e0cfba654ed469.
  ATTACHMENT ID: 12772369

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16521//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16521//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16521//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16521//console

This message is automatically generated.

> Namespace admin permission granted to group 
> 
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as 

[jira] [Commented] (HBASE-14809) Namespace admin permission granted to group

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005494#comment-15005494
 ] 

Ted Yu commented on HBASE-14809:


Printing hanging tests
Hanging test : 
org.apache.hadoop.hbase.procedure2.store.wal.TestWALProcedureStore

> Namespace admin permission granted to group 
> 
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14809) Namespace admin permission granted to group

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005496#comment-15005496
 ] 

Ashish Singhi commented on HBASE-14809:
---

v3 lgtm.
[~tedyu], I see that we have same issue with revoke api also. Do you want to 
fix that one also as part of this ?

> Namespace admin permission granted to group 
> 
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14809) Namespace admin permission granted to group

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005541#comment-15005541
 ] 

Ted Yu commented on HBASE-14809:


Patch v4 handles revoke API.

> Namespace admin permission granted to group 
> 
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt, 
> 14809-v4.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14809) Namespace admin permission granted to group

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14809:
---
Attachment: 14809-v4.txt

> Namespace admin permission granted to group 
> 
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt, 
> 14809-v4.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14806) Missing sources.jar for several modules when building HBase

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005645#comment-15005645
 ] 

Sean Busbey commented on HBASE-14806:
-

+1 both modules check out for correct contents of LICENSE and NOTICE in source 
jars.

> Missing sources.jar for several modules when building HBase
> ---
>
> Key: HBASE-14806
> URL: https://issues.apache.org/jira/browse/HBASE-14806
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-14806.patch
>
>
> Introduced by HBASE-14085. The problem is, for example, in 
> hbase-common/pom.xml, we have
> {code:title=pom.xml}
> 
>   org.apache.maven.plugins
>   maven-source-plugin
>   
> true
> 
>   src/main/java
>   ${project.build.outputDirectory}/META-INF
> 
>   
> 
> {code}
> But in fact, the path inside {{}} tag is relative to source 
> directories, not the project directory. So the maven-source-plugin always end 
> with
> {noformat}
> No sources in project. Archive not created.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14803) Add some debug logs to StoreFileScanner

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005647#comment-15005647
 ] 

Sean Busbey commented on HBASE-14803:
-

{code}
+  static final Log LOG = LogFactory.getLog(StoreFileScanner.class);
+
{code}

please make private

{code}
+if (!reader.passesTimerangeFilter(timeRange, oldestUnexpiredTS)) {
+   if (LOG.isDebugEnabled()) {
+   LOG.debug (reader.getHFileReader().getName() + " skipped 
because of Timerange Filter.");
+   }
+   return false;
{code}

please use 2 spaces for indent and no tabs. (here and a few other instances)

> Add some debug logs to StoreFileScanner
> ---
>
> Key: HBASE-14803
> URL: https://issues.apache.org/jira/browse/HBASE-14803
> Project: HBase
>  Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
>  Labels: beginner
> Fix For: 1.2.0
>
> Attachments: HBASE-14803.v0-trunk.patch, HBASE-14803.v1-trunk.patch, 
> HBASE-14803.v2-trunk.patch
>
>
> To validate some behaviors I had to add some logs into StoreFileScanner.
> I think it can be interesting for other people looking for debuging. So 
> sharing the modifications here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14798) NPE reporting server load causes regionserver abort; causes TestAcidGuarantee to fail

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005653#comment-15005653
 ] 

Hudson commented on HBASE-14798:


FAILURE: Integrated in HBase-1.3 #366 (See 
[https://builds.apache.org/job/HBase-1.3/366/])
HBASE-14798 NPE reporting server load causes regionserver abort; causes (stack: 
rev aaa2294126cb2ebab35ab680951dc9bb79bc0633)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java


> NPE reporting server load causes regionserver abort; causes TestAcidGuarantee 
> to fail
> -
>
> Key: HBASE-14798
> URL: https://issues.apache.org/jira/browse/HBASE-14798
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14798.patch, 14798.patch
>
>
> Below crashed out a RS. Caused TestAcidGuarantees to fail because then there 
> were not RS to assign too... 
> {code}
> 2015-11-11 11:36:23,092 ERROR 
> [B.defaultRpcServer.handler=4,queue=0,port=58655] 
> master.MasterRpcServices(388): Region server 
> asf907.gq1.ygridcore.net,55184,1447241756717 reported a fatal error:
> ABORTING region server asf907.gq1.ygridcore.net,55184,1447241756717: 
> Unhandled: null
> Cause:
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getOldestHfileTs(HRegion.java:1643)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:1503)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1210)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Here is the failure: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/457/jdk=latest1.8,label=Hadoop/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14798) NPE reporting server load causes regionserver abort; causes TestAcidGuarantee to fail

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005655#comment-15005655
 ] 

Hudson commented on HBASE-14798:


SUCCESS: Integrated in HBase-1.3-IT #313 (See 
[https://builds.apache.org/job/HBase-1.3-IT/313/])
HBASE-14798 NPE reporting server load causes regionserver abort; causes (stack: 
rev aaa2294126cb2ebab35ab680951dc9bb79bc0633)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> NPE reporting server load causes regionserver abort; causes TestAcidGuarantee 
> to fail
> -
>
> Key: HBASE-14798
> URL: https://issues.apache.org/jira/browse/HBASE-14798
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14798.patch, 14798.patch
>
>
> Below crashed out a RS. Caused TestAcidGuarantees to fail because then there 
> were not RS to assign too... 
> {code}
> 2015-11-11 11:36:23,092 ERROR 
> [B.defaultRpcServer.handler=4,queue=0,port=58655] 
> master.MasterRpcServices(388): Region server 
> asf907.gq1.ygridcore.net,55184,1447241756717 reported a fatal error:
> ABORTING region server asf907.gq1.ygridcore.net,55184,1447241756717: 
> Unhandled: null
> Cause:
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getOldestHfileTs(HRegion.java:1643)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:1503)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1210)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Here is the failure: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/457/jdk=latest1.8,label=Hadoop/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14798) NPE reporting server load causes regionserver abort; causes TestAcidGuarantee to fail

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005624#comment-15005624
 ] 

Hudson commented on HBASE-14798:


FAILURE: Integrated in HBase-Trunk_matrix #465 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/465/])
HBASE-14798 NPE reporting server load causes regionserver abort; causes (stack: 
rev 43506320a1bb6ca2193162edfb5dee21fffc08a9)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> NPE reporting server load causes regionserver abort; causes TestAcidGuarantee 
> to fail
> -
>
> Key: HBASE-14798
> URL: https://issues.apache.org/jira/browse/HBASE-14798
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14798.patch, 14798.patch
>
>
> Below crashed out a RS. Caused TestAcidGuarantees to fail because then there 
> were not RS to assign too... 
> {code}
> 2015-11-11 11:36:23,092 ERROR 
> [B.defaultRpcServer.handler=4,queue=0,port=58655] 
> master.MasterRpcServices(388): Region server 
> asf907.gq1.ygridcore.net,55184,1447241756717 reported a fatal error:
> ABORTING region server asf907.gq1.ygridcore.net,55184,1447241756717: 
> Unhandled: null
> Cause:
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getOldestHfileTs(HRegion.java:1643)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:1503)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1210)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Here is the failure: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/457/jdk=latest1.8,label=Hadoop/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14798) NPE reporting server load causes regionserver abort; causes TestAcidGuarantee to fail

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005570#comment-15005570
 ] 

Hudson commented on HBASE-14798:


SUCCESS: Integrated in HBase-1.2-IT #282 (See 
[https://builds.apache.org/job/HBase-1.2-IT/282/])
HBASE-14798 NPE reporting server load causes regionserver abort; causes (stack: 
rev c0acfe3a84e1ee14d84df8ae79a6c50400f11be4)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> NPE reporting server load causes regionserver abort; causes TestAcidGuarantee 
> to fail
> -
>
> Key: HBASE-14798
> URL: https://issues.apache.org/jira/browse/HBASE-14798
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14798.patch, 14798.patch
>
>
> Below crashed out a RS. Caused TestAcidGuarantees to fail because then there 
> were not RS to assign too... 
> {code}
> 2015-11-11 11:36:23,092 ERROR 
> [B.defaultRpcServer.handler=4,queue=0,port=58655] 
> master.MasterRpcServices(388): Region server 
> asf907.gq1.ygridcore.net,55184,1447241756717 reported a fatal error:
> ABORTING region server asf907.gq1.ygridcore.net,55184,1447241756717: 
> Unhandled: null
> Cause:
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getOldestHfileTs(HRegion.java:1643)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:1503)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1210)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Here is the failure: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/457/jdk=latest1.8,label=Hadoop/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14798) NPE reporting server load causes regionserver abort; causes TestAcidGuarantee to fail

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005634#comment-15005634
 ] 

Hudson commented on HBASE-14798:


FAILURE: Integrated in HBase-1.2 #370 (See 
[https://builds.apache.org/job/HBase-1.2/370/])
HBASE-14798 NPE reporting server load causes regionserver abort; causes (stack: 
rev c0acfe3a84e1ee14d84df8ae79a6c50400f11be4)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java


> NPE reporting server load causes regionserver abort; causes TestAcidGuarantee 
> to fail
> -
>
> Key: HBASE-14798
> URL: https://issues.apache.org/jira/browse/HBASE-14798
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14798.patch, 14798.patch
>
>
> Below crashed out a RS. Caused TestAcidGuarantees to fail because then there 
> were not RS to assign too... 
> {code}
> 2015-11-11 11:36:23,092 ERROR 
> [B.defaultRpcServer.handler=4,queue=0,port=58655] 
> master.MasterRpcServices(388): Region server 
> asf907.gq1.ygridcore.net,55184,1447241756717 reported a fatal error:
> ABORTING region server asf907.gq1.ygridcore.net,55184,1447241756717: 
> Unhandled: null
> Cause:
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getOldestHfileTs(HRegion.java:1643)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.createRegionLoad(HRegionServer.java:1503)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1210)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:360)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
>   at 
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> Here is the failure: 
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/457/jdk=latest1.8,label=Hadoop/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.TestAcidGuarantees-output.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14806) Missing sources.jar for several modules when building HBase

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005636#comment-15005636
 ] 

Sean Busbey commented on HBASE-14806:
-

What the LICENSE and NOTICE files contain is important. I'll take a look this 
weekend; might take me a bit.

> Missing sources.jar for several modules when building HBase
> ---
>
> Key: HBASE-14806
> URL: https://issues.apache.org/jira/browse/HBASE-14806
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-14806.patch
>
>
> Introduced by HBASE-14085. The problem is, for example, in 
> hbase-common/pom.xml, we have
> {code:title=pom.xml}
> 
>   org.apache.maven.plugins
>   maven-source-plugin
>   
> true
> 
>   src/main/java
>   ${project.build.outputDirectory}/META-INF
> 
>   
> 
> {code}
> But in fact, the path inside {{}} tag is relative to source 
> directories, not the project directory. So the maven-source-plugin always end 
> with
> {noformat}
> No sources in project. Archive not created.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14809) Namespace admin permission granted to group

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005637#comment-15005637
 ] 

Hadoop QA commented on HBASE-14809:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12772376/14809-v4.txt
  against master branch at commit 43506320a1bb6ca2193162edfb5dee21fffc08a9.
  ATTACHMENT ID: 12772376

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16523//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16523//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16523//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16523//console

This message is automatically generated.

> Namespace admin permission granted to group 
> 
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt, 
> 14809-v4.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin 

[jira] [Updated] (HBASE-14809) Grant / revoke Namespace admin permission to group

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14809:
---
Summary: Grant / revoke Namespace admin permission to group   (was: 
Namespace admin permission granted to group )

> Grant / revoke Namespace admin permission to group 
> ---
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt, 
> 14809-v4.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14809) Grant / revoke Namespace admin permission to group

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14809:
---
 Hadoop Flags: Reviewed
Fix Version/s: 1.1.4
   1.3.0
   1.2.0
   2.0.0

> Grant / revoke Namespace admin permission to group 
> ---
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt, 
> 14809-v4.txt
>
>
> Hi, 
> We are looking to roll out HBase and are in the process to design the 
> security model. 
> We are looking to implement global DBAs and Namespace specific 
> administrators. 
> So for example the global dba would create a namespace and grant a user/group 
> admin privileges within that ns. 
> So that a given ns admin can in turn create objects and grant permission 
> within the given ns only. 
> We have run into some issues at the ns admin level. It appears that a ns 
> admin can NOT grant to a grop unless it also has global admin privilege. But 
> once it has global admin privilege it can grant in any NS not just the one 
> where it has admin privileges. 
> Based on the HBase documentation at 
> http://hbase.apache.org/book.html#appendix_acl_matrix 
> Table 13. ACL Matrix 
> Interface Operation   Permissions 
> AccessController grant(global level) global(A) 
> grant(namespace level) global(A)|NS(A) 
> grant at a namespace level should be possible for someone with global A OR 
> (|) NS A permission. 
> As you will see in our test it does not work if NS A permission is granted 
> but global A permission is not. 
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1. 
> {code}
> hbase(main):011:0> scan 'hbase:acl' 
> ROW COLUMN+CELL 
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA 
> {code}
> However: 
> Here you can see that a user who is member of the group 
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is 
> missing global A privilege. 
> {code}
> $hbase shell 
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is 
> deprecated. Instead, use io.native.lib.available 
> HBase Shell; enter 'help' for list of supported commands. 
> Type "exit" to leave the HBase Shell 
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015 
> hbase(main):001:0> whoami 
> ns1ad...@wlab.net (auth:KERBEROS) 
> groups: hbaseappltest_ns1admin 
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1' 
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'ns1admin' (global, action=ADMIN) 
> {code}
> The way I read the documentation a NS admin should be able to grant as it has 
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0. 
> Regards, 
> Steven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14803) Add some debug logs to StoreFileScanner

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005680#comment-15005680
 ] 

Heng Chen commented on HBASE-14803:
---

IMO we should check whether reader.getHFileReader() is null.  Maybe failed test 
case has relates with it.

> Add some debug logs to StoreFileScanner
> ---
>
> Key: HBASE-14803
> URL: https://issues.apache.org/jira/browse/HBASE-14803
> Project: HBase
>  Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
>  Labels: beginner
> Fix For: 1.2.0
>
> Attachments: HBASE-14803.v0-trunk.patch, HBASE-14803.v1-trunk.patch, 
> HBASE-14803.v2-trunk.patch
>
>
> To validate some behaviors I had to add some logs into StoreFileScanner.
> I think it can be interesting for other people looking for debuging. So 
> sharing the modifications here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14814) HDFS UI port clashes cause test fails

[stack (JIRA)]

stack created HBASE-14814:
-

 Summary: HDFS UI port clashes cause test fails
 Key: HBASE-14814
 URL: https://issues.apache.org/jira/browse/HBASE-14814
 Project: HBase
  Issue Type: Sub-task
  Components: flakey, test
Reporter: stack
Assignee: stack


In test runs, HDFS UI can clash when concurrent test runs.  





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14814) HDFS UI port clashes cause test fails

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005690#comment-15005690
 ] 

stack commented on HBASE-14814:
---

Here is how it looks on fail:

{code}
Error Message

Port in use: localhost:0
Stacktrace

java.net.BindException: Port in use: localhost:0
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at 
org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at 
org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:885)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:693)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:583)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:751)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:735)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:998)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:869)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:704)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:642)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:577)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:533)
at 
org.apache.hadoop.hbase.util.TestFSUtils.testIsHDFS(TestFSUtils.java:111)
{code}


Here is rest of log:

{code}
015-11-14 21:09:09,080 ERROR [main] hdfs.MiniDFSCluster(709): IOE creating 
namenodes. Permissions dump:
path 
'/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/c54ebf05-cbf8-40e6-a25e-213289f6d51b/dfscluster_85511ee0-65cc-4850-8b87-6ecf97f1cffc/dfs/data':
 

absolute:/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/c54ebf05-cbf8-40e6-a25e-213289f6d51b/dfscluster_85511ee0-65cc-4850-8b87-6ecf97f1cffc/dfs/data
permissions: 
path 
'/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/c54ebf05-cbf8-40e6-a25e-213289f6d51b/dfscluster_85511ee0-65cc-4850-8b87-6ecf97f1cffc/dfs':
 

absolute:/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/c54ebf05-cbf8-40e6-a25e-213289f6d51b/dfscluster_85511ee0-65cc-4850-8b87-6ecf97f1cffc/dfs
permissions: drwx
path 
'/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/c54ebf05-cbf8-40e6-a25e-213289f6d51b/dfscluster_85511ee0-65cc-4850-8b87-6ecf97f1cffc':
 

absolute:/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/c54ebf05-cbf8-40e6-a25e-213289f6d51b/dfscluster_85511ee0-65cc-4850-8b87-6ecf97f1cffc
permissions: drwx
path 
'/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/c54ebf05-cbf8-40e6-a25e-213289f6d51b':
 

absolute:/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data/c54ebf05-cbf8-40e6-a25e-213289f6d51b
permissions: drwx
path 
'/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data':
 

absolute:/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target/test-data
permissions: drwx
path 
'/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target':
 

absolute:/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server/target
permissions: drwx
path 
'/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server':
 

absolute:/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop/hbase-server
permissions: drwx
path 
'/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop': 

absolute:/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label/Hadoop
permissions: drwx
path '/home/jenkins/jenkins-slave/workspace/HBase-1.2/jdk/latest1.8/label': 

[jira] [Reopened] (HBASE-14802) Replaying server crash recovery procedure after a failover causes incorrect handling of deadservers

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-14802:
---

Reverted from master branch. Since this went in, the related TestDeadServers is 
failing.

See 
https://builds.apache.org/job/HBase-Trunk_matrix/lastCompletedBuild/jdk=latest1.8,label=Hadoop/testReport/org.apache.hadoop.hbase.master/TestDeadServer/testCrashProcedureReplay/history/

Failed twice in master jdk8 builds (passed once).

Failed once on jdk7 
https://builds.apache.org/job/HBase-Trunk_matrix/464/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.master/TestDeadServer/testCrashProcedureReplay/history/

Seems to be just timing out the start up which is odd.

I have not reverted from branch-1 or branch-1.2 because have had a successful 
build in the latter. In the former it seemed to be something else.





> Replaying server crash recovery procedure after a failover causes incorrect 
> handling of deadservers
> ---
>
> Key: HBASE-14802
> URL: https://issues.apache.org/jira/browse/HBASE-14802
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 2.0.0, 1.2.0, 1.2.1
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14802-1.patch, HBASE-14802-2.patch, 
> HBASE-14802-3.patch, HBASE-14802.patch
>
>
> The way dead servers are processed is that a ServerCrashProcedure is launched 
> for a server after it is added to the dead servers list. 
> Every time a server is added to the dead list, a counter "numProcessing" is 
> incremented and it is decremented when a crash recovery procedure finishes. 
> Since, adding a dead server and recovering it are two separate events, it can 
> cause inconsistencies.
> If a master failover occurs in the middle of the crash recovery, the 
> numProcessing counter resets but the ServerCrashProcedure is replayed by the 
> new master. This causes the counter to go negative and makes the master think 
> that dead servers are still in process of recovery. 
> This has ramifications on the balancer that the balancer ceases to run after 
> such a failover.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)