[jira] [Commented] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151077#comment-13151077
 ] 

Hudson commented on HBASE-4795:
---

Integrated in HBase-TRUNK #2446 (See 
[https://builds.apache.org/job/HBase-TRUNK/2446/])
HBASE-4795 Fix TestHFileBlock when running on a 32-bit JVM

stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/ClassSize.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java


> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: D459.1.patch, D459.2.patch, 
> Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151051#comment-13151051
 ] 

Lars Hofhansl commented on HBASE-4213:
--

Thanks Ted.

{code}
protected void handleInstantSchemaChanges(List regions) {
  ...
  try {
...
master.synchronousBalanceSwitch(false);
waitForInflightSplit(regions, status);
...
  } finally {
master.synchronousBalanceSwitch(true); 
  }
{code}

Since balancing might be disabled for other reasons, should probably remember 
the old balancer state and restore that in the finally block.


> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4796) Race between SplitRegionHandlers for the same region kills the master

[ramkrishna.s.vasudevan (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan reassigned HBASE-4796:
-

Assignee: ramkrishna.s.vasudevan

> Race between SplitRegionHandlers for the same region kills the master
> -
>
> Key: HBASE-4796
> URL: https://issues.apache.org/jira/browse/HBASE-4796
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.0, 0.94.0
>
>
> I just saw that multiple SplitRegionHandlers can be created for the same 
> region because of the RS tickling, but it becomes deadly when more than 1 are 
> trying to delete the znode at the same time:
> {quote}
> 2011-11-16 02:25:28,778 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r7s38,62023,1321410237387, 
> region=f80b6a904048a99ce88d61420b8906d1
> 2011-11-16 02:25:28,780 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r7s38,62023,1321410237387, 
> region=f80b6a904048a99ce88d61420b8906d1
> 2011-11-16 02:25:28,796 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for f80b6a904048a99ce88d61420b8906d1; deleting node
> 2011-11-16 02:25:28,798 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde094b Deleting existing unassigned node for 
> f80b6a904048a99ce88d61420b8906d1 that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-16 02:25:28,804 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for f80b6a904048a99ce88d61420b8906d1; deleting node
> 2011-11-16 02:25:28,806 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde094b Deleting existing unassigned node for 
> f80b6a904048a99ce88d61420b8906d1 that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-16 02:25:28,821 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde094b Successfully deleted unassigned node for 
> region f80b6a904048a99ce88d61420b8906d1 in expected state RS_ZK_REGION_SPLIT
> 2011-11-16 02:25:28,821 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,006304,1321409743253.f80b6a904048a99ce88d61420b8906d1. 
> daughter 
> a=TestTable,006304,1321410325564.e0f5d201683bcabe14426817224334b8.daughter
>  b=TestTable,007054,1321410325564.1b82eeb5d230c47ccc51c08256134839.
> 2011-11-16 02:25:28,829 WARN 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node 
> /hbase/unassigned/f80b6a904048a99ce88d61420b8906d1 already deleted, and this 
> is not a retry
> 2011-11-16 02:25:28,830 FATAL org.apache.hadoop.hbase.master.HMaster: Error 
> deleting SPLIT node in ZK for transition ZK node 
> (f80b6a904048a99ce88d61420b8906d1)
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /hbase/unassigned/f80b6a904048a99ce88d61420b8906d1
>   at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>   at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>   at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
>   at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:107)
>   at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:884)
>   at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:506)
>   at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:453)
>   at 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler.process(SplitRegionHandler.java:95)
>   at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:168)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>   at java.lang.Thread.run(Thread.java:662)
> {quote}
> Stack and I came up with the solution that we need just manage that exception 
> because handleSplitReport is an in-memory thing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151049#comment-13151049
 ] 

jirapos...@reviews.apache.org commented on HBASE-4213:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1786/
---

(Updated 2011-11-16 06:12:33.670038)


Review request for Todd Lipcon, Andrew Purtell and Subbu Iyer.


Changes
---

New patch remembers prevBalanceSwitch and uses it to restore load balancer 
switch.


Summary
---

bq. From Subbu:
here is the latest patch that support alter_instant, an instant schema change 
command that supports (Add, Modify, Delete column and Modify table) actions 
through ZK.

1. This pattern capitalizes on the fact that HRI's are now in HDFS and need not 
be sent over again from Master to RS cloud on every schema change event.

2. Offers real time instant schema change as we bypass the explicit bulk 
reassign (unassign + assign) of regions from master to RS.

3. Offers fault tolerant schema change support as schema changes now go through 
ZK. Secondary master taking over a failed schema change will be addressed 
through a separate JIRA.


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/MasterServices.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 
1202381 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java
 1202381 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java
 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java 
1202523 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java
 1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java 
1202381 
  
/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
1202381 
  /src/main/resources/hbase-default.xml 1202381 
  /src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java 
PRE-CREATION 
  
/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java
 PRE-CREATION 
  /src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1202381 
  /src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
1202381 

Diff: https://reviews.apache.org/r/1786/diff


Testing
---

Unit tests pass.


Thanks,

Ted



> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.

[jira] [Commented] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151047#comment-13151047
 ] 

Hudson commented on HBASE-4792:
---

Integrated in HBase-TRUNK #2445 (See 
[https://builds.apache.org/job/HBase-TRUNK/2445/])
Adding the 0.92 entry for HBASE-4792
HBASE-4792  SplitRegionHandler doesn't care if it deletes the znode or not,
   leaves the parent region stuck offline

jdcryans : 
Files : 
* /hbase/trunk/CHANGES.txt

jdcryans : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/SplitRegionHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java


> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
> parent region stuck offline
> --
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
> daughter 
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
>  b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Region 
> e5be6551c8584a6a1065466e520faf4e not found on server 
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619 
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it 
> doesn't check the boolean that's returned. In this case it was false. So for 
> the master the split was completed, but for the region server it's another 
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)

[jira] [Commented] (HBASE-4793) HBase shell still using deprecated methods removed in HBASE-4436

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151046#comment-13151046
 ] 

Hudson commented on HBASE-4793:
---

Integrated in HBase-TRUNK #2445 (See 
[https://builds.apache.org/job/HBase-TRUNK/2445/])
HBASE-4793  HBase shell still using deprecated methods removed in HBASE-4436

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/ruby/hbase/admin.rb


> HBase shell still using deprecated methods removed in HBASE-4436
> 
>
> Key: HBASE-4793
> URL: https://issues.apache.org/jira/browse/HBASE-4793
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4793.txt
>
>
> The patch applied in HBASE-4622 (subtask of HBASE-4436) to remove deprecated 
> methods seems to have missed some usage of those methods by the HBase shell.  
> At least src/main/ruby/hbase/admin.rb is still using some of the removed 
> methods, breaking some shell commands:
> {noformat}
> hbase(main):007:0> alter 'privatetable', { NAME => 'f1', VERSIONS => 2}
> ERROR: wrong number of arguments (3 for 2)
> Backtrace: /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:344:in `alter'
>org/jruby/RubyArray.java:1572:in `each'
>/usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:317:in `alter'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:79:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:68:in 
> `format_simple_command'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:78:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:74:in 
> `translate_hbase_exceptions'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell.rb:110:in `command'
>(eval):2:in `alter'
> {noformat}
> This trace translates to the line:
> {code}
>   @admin.modifyColumn(table_name, column_name, descriptor)
> {code}
> which is calling one of the removed methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4436) Remove methods deprecated in 0.90 from TRUNK and 0.92

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151045#comment-13151045
 ] 

Hudson commented on HBASE-4436:
---

Integrated in HBase-TRUNK #2445 (See 
[https://builds.apache.org/job/HBase-TRUNK/2445/])
HBASE-4793  HBase shell still using deprecated methods removed in HBASE-4436

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/ruby/hbase/admin.rb


> Remove methods deprecated in 0.90 from TRUNK and 0.92
> -
>
> Key: HBASE-4436
> URL: https://issues.apache.org/jira/browse/HBASE-4436
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: Jonathan Hsieh
>  Labels: noob
> Fix For: 0.92.0
>
>
> Remove methods deprecated in 0.90 from codebase.  i took a quick look.  The 
> messy bit is thrift referring to old stuff; will take a little work to do the 
> convertion over to the new methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4654) [replication] Add a check to make sure we don't replicate to ourselves

[Lars Hofhansl (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4654:
-

Attachment: 4654-trunk.txt

For 0.92 and trunk, this should be sufficient (but haven't tested, yet)

> [replication] Add a check to make sure we don't replicate to ourselves
> --
>
> Key: HBASE-4654
> URL: https://issues.apache.org/jira/browse/HBASE-4654
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
> Fix For: 0.90.5
>
> Attachments: 4654-trunk.txt
>
>
> It's currently possible to add a peer for replication and point it to the 
> local cluster, which I believe could very well happen for those like us that 
> use only one ZK ensemble per DC so that only the root znode changes when you 
> want to set up replication intra-DC.
> I don't think comparing just the cluster ID would be enough because you would 
> normally use a different one for another cluster and nothing will block you 
> from pointing elsewhere.
> Comparing the ZK ensemble address doesn't work either when you have multiple 
> DNS entries that point at the same place.
> I think this could be resolved by looking up the master address in the 
> relevant znode as it should be exactly the same thing in the case where you 
> have the same cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151042#comment-13151042
 ] 

jirapos...@reviews.apache.org commented on HBASE-4213:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1786/
---

(Updated 2011-11-16 06:05:02.324776)


Review request for hbase, Todd Lipcon, Andrew Purtell, and Subbu Iyer.


Changes
---

New patch uses master.synchronousBalanceSwitch() to ensure that balancer is 
switched off during instant schema change

TestInstantSchemaChange#testLoadBalancerBlocksDuringSchemaChangeRequests passed.


Summary
---

bq. From Subbu:
here is the latest patch that support alter_instant, an instant schema change 
command that supports (Add, Modify, Delete column and Modify table) actions 
through ZK.

1. This pattern capitalizes on the fact that HRI's are now in HDFS and need not 
be sent over again from Master to RS cloud on every schema change event.

2. Offers real time instant schema change as we bypass the explicit bulk 
reassign (unassign + assign) of regions from master to RS.

3. Offers fault tolerant schema change support as schema changes now go through 
ZK. Secondary master taking over a failed schema change will be addressed 
through a separate JIRA.


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/MasterServices.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 
1202381 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java
 1202381 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java
 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java 
1202523 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java
 1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java 
1202381 
  
/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
1202381 
  /src/main/resources/hbase-default.xml 1202381 
  /src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java 
PRE-CREATION 
  
/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java
 PRE-CREATION 
  /src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1202381 
  /src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
1202381 

Diff: https://reviews.apache.org/r/1786/diff


Testing
---

Unit tests pass.


Thanks,

Ted



> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-I

[jira] [Updated] (HBASE-4655) Document architecture of backups

[Karthik Ranganathan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Ranganathan updated HBASE-4655:
---

Attachment: HBase Backups Architecture.docx

Basic HBase backup architecture and the various levels of protection it would 
offer.

> Document architecture of backups
> 
>
> Key: HBASE-4655
> URL: https://issues.apache.org/jira/browse/HBASE-4655
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, regionserver
>Reporter: Karthik Ranganathan
>Assignee: Karthik Ranganathan
> Attachments: HBase Backups Architecture.docx
>
>
> Basic idea behind the backup architecture for HBase

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151026#comment-13151026
 ] 

Ted Yu commented on HBASE-4213:
---

I wasn't able to find switchBalancer() or synchronousBalanceSwitch(created for 
instant schema change) in the patch.
I think synchronousBalanceSwitch(false) should be called at line 258.
synchronousBalanceSwitch(true) should be called in a finally block at line 285.

Thanks for the finding, Lars.

> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151020#comment-13151020
 ] 

Lars Hofhansl commented on HBASE-4213:
--

@Ted: Hmm... But we haven't started the schema change at that point, right? So 
the master would not know about the intention.


> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151018#comment-13151018
 ] 

Ted Yu commented on HBASE-4213:
---

@Lars:
Master would check whether schema change is in progress before running balancer:
{code}
+  if (schemaChangeTracker.isSchemaChangeInProgress()) {
+LOG.debug("Schema change operation is in progress. Waiting for " +
+"it to complete before running the load balancer.");
+return false;
+  }
+  loadBalancerRunning = true;
{code}
In handleInstantSchemaChanges() where you asked the question:
{code}
waitForLoadBalancerToComplete(status);
waitForInflightSplit(regions, status);
{code}
See also testLoadBalancerBlocksDuringSchemaChangeRequests()

> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4793) HBase shell still using deprecated methods removed in HBASE-4436

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151015#comment-13151015
 ] 

Hudson commented on HBASE-4793:
---

Integrated in HBase-0.92 #134 (See 
[https://builds.apache.org/job/HBase-0.92/134/])
HBASE-4793  HBase shell still using deprecated methods removed in HBASE-4436

tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/ruby/hbase/admin.rb


> HBase shell still using deprecated methods removed in HBASE-4436
> 
>
> Key: HBASE-4793
> URL: https://issues.apache.org/jira/browse/HBASE-4793
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4793.txt
>
>
> The patch applied in HBASE-4622 (subtask of HBASE-4436) to remove deprecated 
> methods seems to have missed some usage of those methods by the HBase shell.  
> At least src/main/ruby/hbase/admin.rb is still using some of the removed 
> methods, breaking some shell commands:
> {noformat}
> hbase(main):007:0> alter 'privatetable', { NAME => 'f1', VERSIONS => 2}
> ERROR: wrong number of arguments (3 for 2)
> Backtrace: /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:344:in `alter'
>org/jruby/RubyArray.java:1572:in `each'
>/usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:317:in `alter'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:79:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:68:in 
> `format_simple_command'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:78:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:74:in 
> `translate_hbase_exceptions'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell.rb:110:in `command'
>(eval):2:in `alter'
> {noformat}
> This trace translates to the line:
> {code}
>   @admin.modifyColumn(table_name, column_name, descriptor)
> {code}
> which is calling one of the removed methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4436) Remove methods deprecated in 0.90 from TRUNK and 0.92

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151014#comment-13151014
 ] 

Hudson commented on HBASE-4436:
---

Integrated in HBase-0.92 #134 (See 
[https://builds.apache.org/job/HBase-0.92/134/])
HBASE-4793  HBase shell still using deprecated methods removed in HBASE-4436

tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/ruby/hbase/admin.rb


> Remove methods deprecated in 0.90 from TRUNK and 0.92
> -
>
> Key: HBASE-4436
> URL: https://issues.apache.org/jira/browse/HBASE-4436
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: Jonathan Hsieh
>  Labels: noob
> Fix For: 0.92.0
>
>
> Remove methods deprecated in 0.90 from codebase.  i took a quick look.  The 
> messy bit is thrift referring to old stuff; will take a little work to do the 
> convertion over to the new methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151016#comment-13151016
 ] 

Hudson commented on HBASE-4792:
---

Integrated in HBase-0.92 #134 (See 
[https://builds.apache.org/job/HBase-0.92/134/])
HBASE-4792  SplitRegionHandler doesn't care if it deletes the znode or not,
   leaves the parent region stuck offline

jdcryans : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/SplitRegionHandler.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java


> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
> parent region stuck offline
> --
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
> daughter 
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
>  b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Region 
> e5be6551c8584a6a1065466e520faf4e not found on server 
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619 
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it 
> doesn't check the boolean that's returned. In this case it was false. So for 
> the master the split was completed, but for the region server it's another 
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't real

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151012#comment-13151012
 ] 

jirapos...@reviews.apache.org commented on HBASE-4213:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1786/#review3293
---


Late to this review...
Awesome feature. Thanks for doing this Subbu!

Looks good to me in general.

Minor comments inline.

As a general comment, I think it would be better if we could drive from the ZK 
notifications, rather than doing all this sleeping in a loop waiting for some 
other action to complete by rechecking the ZK node; would probably require a 
lot of refactoring to make the code event based.



/src/main/java/org/apache/hadoop/hbase/master/HMaster.java


whitespace



/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java


whitespace... here and below



/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java


How do you know for sure that while we waited for inflightSplits no new 
round of load balancing has started?
Or maybe that is not a problem?



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


Maybe put this method on schema tracker?
Similar code is found CompactSplitThread.java.

Also, can we use the ZK notifications here instead of waiting in a loop?




/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java


We're not added the "year" line anymore.



/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java


year line...



/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java


Year line ...



/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java


I think we're not putting this line anymore.


- Lars


On 2011-11-15 23:28:55, Ted Yu wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1786/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 23:28:55)
bq.  
bq.  
bq.  Review request for Todd Lipcon, Andrew Purtell and Subbu Iyer.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  From Subbu:
bq.  here is the latest patch that support alter_instant, an instant schema 
change command that supports (Add, Modify, Delete column and Modify table) 
actions through ZK.
bq.  
bq.  1. This pattern capitalizes on the fact that HRI's are now in HDFS and 
need not be sent over again from Master to RS cloud on every schema change 
event.
bq.  
bq.  2. Offers real time instant schema change as we bypass the explicit bulk 
reassign (unassign + assign) of regions from master to RS.
bq.  
bq.  3. Offers fault tolerant schema change support as schema changes now go 
through ZK. Secondary master taking over a failed schema change will be 
addressed through a separate JIRA.
bq.  
bq.  
bq.  This addresses bug HBASE-4213.
bq.  https://issues.apache.org/jira/browse/HBASE-4213
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 1202381 
bq./src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java 
1202381 
bq./src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 1202381 
bq./src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
1202381 
bq./src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1202381 
bq./src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 
1202381 
bq./src/main/java/org/apache/hadoop/hbase/master/MasterServices.java 
1202381 
bq./src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 1202381 
bq.
/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 
1202381 
bq.
/src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 
1202381 
bq.
/src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java
 1202381 
bq.
/src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java
 1202381 
bq.
/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java 
1202381 
bq.
/src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java
 1202381 
bq.
/

[jira] [Updated] (HBASE-3715) Book.xml - adding architecture section on client, adding section on spec-ex under mapreduce

[Jonathan Hsieh (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-3715:
--

  Description: 
Small changes to book.xml

* added small section under MapReduce saying that it's generally advisable to 
turn off speculative execution when using HBase as a source

* Adding 'client' section under architecture that is a simplified port of the 
client section in the HBaseArchitecture wiki page. 

  was:

Small changes to book.xml

* added small section under MapReduce saying that it's generally advisable to 
turn off speculative execution when using HBase as a source

* Adding 'client' section under architecture that is a simplified port of the 
client section in the HBaseArchitecture wiki page. 

Fix Version/s: 0.92.0

> Book.xml - adding architecture section on client, adding section on spec-ex 
> under mapreduce
> ---
>
> Key: HBASE-3715
> URL: https://issues.apache.org/jira/browse/HBASE-3715
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: book.xml.patch
>
>
> Small changes to book.xml
> * added small section under MapReduce saying that it's generally advisable to 
> turn off speculative execution when using HBase as a source
> * Adding 'client' section under architecture that is a simplified port of the 
> client section in the HBaseArchitecture wiki page. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3684) Support column range filter

[Jonathan Hsieh (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-3684:
--

Fix Version/s: 0.92.0

> Support column range filter 
> 
>
> Key: HBASE-3684
> URL: https://issues.apache.org/jira/browse/HBASE-3684
> Project: HBase
>  Issue Type: New Feature
>  Components: filters
>Reporter: Jerry Chen
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 3684.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Currently we have a ColumnPrefix filter which will seek to the proper column 
> prefix. We also need a column range filter to query a range of columns. The 
> proposed interface is the following: ColumnRangeFilter(final byte[] 
> minColumn, boolean minColumnInclusive,final byte[] maxColumn, boolean 
> maxColumnInclusive) 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151010#comment-13151010
 ] 

Hadoop QA commented on HBASE-4795:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12503832/Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 4 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -163 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 51 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/262//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/262//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/262//console

This message is automatically generated.

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: D459.1.patch, D459.2.patch, 
> Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3717) deprecate HTable isTableEnabled() methods in favor of HBaseAdmin methods

[Jonathan Hsieh (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-3717:
--

Fix Version/s: 0.90.3
   0.92.0

> deprecate HTable isTableEnabled() methods in favor of HBaseAdmin methods
> 
>
> Key: HBASE-3717
> URL: https://issues.apache.org/jira/browse/HBASE-3717
> Project: HBase
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 0.90.1
>Reporter: David Buttler
>Assignee: David Buttler
>Priority: Trivial
> Fix For: 0.90.3, 0.92.0
>
> Attachments: deprecate_HTable_isTableEnabled.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> the static methods on HTable.isTableEnabled() can lead to unintended 
> consequences if used naively without understanding potential side-effects.  
> Suggest deprecating these methods and pointing at the HBaseAdmin methods to 
> accomplish same task instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3705) Allow passing timestamp into importtsv

[Jonathan Hsieh (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-3705:
--

Fix Version/s: 0.92.0

> Allow passing timestamp into importtsv
> --
>
> Key: HBASE-3705
> URL: https://issues.apache.org/jira/browse/HBASE-3705
> Project: HBase
>  Issue Type: Improvement
>  Components: util
>Affects Versions: 0.90.1
>Reporter: Andy Sautins
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 3705.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> importtsv sets the timestamp for the imported records to the current time in 
> ImportTsv.TsvImporter.Mapper.setup.  It can be useful to be able to set the 
> specific timestamp used for the import.  This JIRA adds a 
> -Dimporttsv.timestamp parameter to the importtsv job.   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151000#comment-13151000
 ] 

Hadoop QA commented on HBASE-4795:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12503828/D459.2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -163 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 51 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
  org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/261//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/261//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/261//console

This message is automatically generated.

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: D459.1.patch, D459.2.patch, 
> Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4611) Add support for Phabricator/Differential as an alternative code review tool

[Marek Sapota (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150998#comment-13150998
 ] 

Marek Sapota commented on HBASE-4611:
-

secure.phabricator.com doesn't allow anonymous access, reviews.facebook.net 
that we use has anonymous access enabled.  Keep in mind that this is a new 
feature and not everything is "anonymous aware".  `arc patch` will work with an 
updated Arcanist, if you need any other particular workflow to work without a 
certificate and it doesn't right now, drop me an email, I can probably fix it.

> Add support for Phabricator/Differential as an alternative code review tool
> ---
>
> Key: HBASE-4611
> URL: https://issues.apache.org/jira/browse/HBASE-4611
> Project: HBase
>  Issue Type: Task
>Reporter: Jonathan Gray
>Assignee: Nicolas Spiegelberg
> Fix For: 0.92.0, 0.94.0
>
> Attachments: D153.1.patch, D165.1.patch, D165.2.patch, D171.1.patch, 
> D177.1.patch, D177.2.patch, D183.1.patch, D189.1.patch, D201.1.patch, 
> D207.1.patch, D21.1.patch, D21.1.patch, HBASE-4611.D423.1.patch, 
> HBASE-4611.D423.2.patch
>
>
> From http://phabricator.org/ : "Phabricator is a open source collection of 
> web applications which make it easier to write, review, and share source 
> code. It is currently available as an early release. Phabricator was 
> developed at Facebook."
> It's open source so pretty much anyone could host an instance of this 
> software.
> To begin with, there will be a public-facing instance located at 
> http://reviews.facebook.net (sponsored by Facebook and hosted by the OSUOSL 
> http://osuosl.org).
> We will use this JIRA to deal with adding (and ensuring) Apache-friendly 
> support that will allow us to do code reviews with Phabricator for HBase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4797) [availability] Give recovered.edits files better names, ones that include first and last sequence id so we can skip files with edits we know older than current region h

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150992#comment-13150992
 ] 

stack commented on HBASE-4797:
--

Oh... i suppose its a bit worse than I though.  I'm looking at a region that 
has nearly 6k recovered.edits files to replay.  The RegionServer is doing this 
per file:

{code}
2011-11-16 03:06:02,403 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Applied 0, skipped 33, firstSequenceidInLog=296860, maxSequenceidInLog=351600, 
path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0296860
2011-11-16 03:06:02,405 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
Replaying edits from 
hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0296914;
 minSequenceid=351600; 
path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0296914
2011-11-16 03:06:05,097 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:7003-0x133a5bab186271f Attempting to transition node 
69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to 
RS_ZK_REGION_OPENING
2011-11-16 03:06:05,278 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:7003-0x133a5bab186271f Successfully transitioned node 
69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to 
RS_ZK_REGION_OPENING
2011-11-16 03:06:05,278 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Applied 0, skipped 33, firstSequenceidInLog=296914, maxSequenceidInLog=351600, 
path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0296914
2011-11-16 03:06:05,279 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
Replaying edits from 
hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0296970;
 minSequenceid=351600; 
path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0296970
2011-11-16 03:06:05,952 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:7003-0x133a5bab186271f Attempting to transition node 
69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to 
RS_ZK_REGION_OPENING
2011-11-16 03:06:06,093 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:7003-0x133a5bab186271f Successfully transitioned node 
69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to 
RS_ZK_REGION_OPENING
2011-11-16 03:06:06,093 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Applied 0, skipped 44, firstSequenceidInLog=296970, maxSequenceidInLog=351600, 
path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0296970
2011-11-16 03:06:06,094 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
Replaying edits from 
hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0297041;
 minSequenceid=351600; 
path=hdfs://sv4r11s38:7000/hbase/TestTable/69ab6eb0e2feff1fda52d36d8fa75798/recovered.edits/0297041
2011-11-16 03:06:06,795 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:7003-0x133a5bab186271f Attempting to transition node 
69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to 
RS_ZK_REGION_OPENING
2011-11-16 03:06:06,810 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:7003-0x133a5bab186271f Successfully transitioned node 
69ab6eb0e2feff1fda52d36d8fa75798 from RS_ZK_REGION_OPENING to 
RS_ZK_REGION_OPENING
{code}

> [availability] Give recovered.edits files better names, ones that include 
> first and last sequence id so we can skip files with edits we know older than 
> current region has
> --
>
> Key: HBASE-4797
> URL: https://issues.apache.org/jira/browse/HBASE-4797
> Project: HBase
>  Issue Type: Bug
>  Components: performance
>Reporter: stack
>
> Testing 0.92, I crashed all servers out.  Another bug makes it so WALs are 
> not getting cleaned so I had 7000 regions to replay.  The distributed split 
> code did a nice job and cluster came back but interesting is that some hot 
> regions ended up having loads of recovered.edits files -- tens if not 
> hundreds -- to replay against the region (can we bulk load recovered.edits 
> instead of replaying them?).  Each recovered.edits file is taking about a 
> second to process (though only about 30 odd edits per file it seems).  The 
> region is unavailable during this time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4795:
-

   Resolution: Fixed
Fix Version/s: 0.94.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks for fixing this Mikhail

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: D459.1.patch, D459.2.patch, 
> Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4611) Add support for Phabricator/Differential as an alternative code review tool

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150986#comment-13150986
 ] 

stack commented on HBASE-4611:
--

Do I need a cert to talk to phabricator?

{code}
h-25-20:arcanist stack$ ./bin/arc patch D459
Usage Exception: YOU NEED TO INSTALL A CERTIFICATE TO LOGIN TO PHABRICATOR

You are trying to connect to 'https://secure.phabricator.com/api/' but do not 
have a certificate installed for this host. Run:

  $ arc install-certificate

...to install one.
{code}

> Add support for Phabricator/Differential as an alternative code review tool
> ---
>
> Key: HBASE-4611
> URL: https://issues.apache.org/jira/browse/HBASE-4611
> Project: HBase
>  Issue Type: Task
>Reporter: Jonathan Gray
>Assignee: Nicolas Spiegelberg
> Fix For: 0.92.0, 0.94.0
>
> Attachments: D153.1.patch, D165.1.patch, D165.2.patch, D171.1.patch, 
> D177.1.patch, D177.2.patch, D183.1.patch, D189.1.patch, D201.1.patch, 
> D207.1.patch, D21.1.patch, D21.1.patch, HBASE-4611.D423.1.patch, 
> HBASE-4611.D423.2.patch
>
>
> From http://phabricator.org/ : "Phabricator is a open source collection of 
> web applications which make it easier to write, review, and share source 
> code. It is currently available as an early release. Phabricator was 
> developed at Facebook."
> It's open source so pretty much anyone could host an instance of this 
> software.
> To begin with, there will be a public-facing instance located at 
> http://reviews.facebook.net (sponsored by Facebook and hosted by the OSUOSL 
> http://osuosl.org).
> We will use this JIRA to deal with adding (and ensuring) Apache-friendly 
> support that will allow us to do code reviews with Phabricator for HBase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4797) [availability] Give recovered.edits files better names, ones that include first and last sequence id so we can skip files with edits we know older than current region has

[stack (Created) (JIRA)]

[availability] Give recovered.edits files better names, ones that include first 
and last sequence id so we can skip files with edits we know older than current 
region has
--

 Key: HBASE-4797
 URL: https://issues.apache.org/jira/browse/HBASE-4797
 Project: HBase
  Issue Type: Bug
  Components: performance
Reporter: stack


Testing 0.92, I crashed all servers out.  Another bug makes it so WALs are not 
getting cleaned so I had 7000 regions to replay.  The distributed split code 
did a nice job and cluster came back but interesting is that some hot regions 
ended up having loads of recovered.edits files -- tens if not hundreds -- to 
replay against the region (can we bulk load recovered.edits instead of 
replaying them?).  Each recovered.edits file is taking about a second to 
process (though only about 30 odd edits per file it seems).  The region is 
unavailable during this time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4796) Race between SplitRegionHandlers for the same region kills the master

[Jean-Daniel Cryans (Created) (JIRA)]

Race between SplitRegionHandlers for the same region kills the master
-

 Key: HBASE-4796
 URL: https://issues.apache.org/jira/browse/HBASE-4796
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
 Fix For: 0.92.0, 0.94.0


I just saw that multiple SplitRegionHandlers can be created for the same region 
because of the RS tickling, but it becomes deadly when more than 1 are trying 
to delete the znode at the same time:

{quote}
2011-11-16 02:25:28,778 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Handling transition=RS_ZK_REGION_SPLIT, server=sv4r7s38,62023,1321410237387, 
region=f80b6a904048a99ce88d61420b8906d1
2011-11-16 02:25:28,780 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Handling transition=RS_ZK_REGION_SPLIT, server=sv4r7s38,62023,1321410237387, 
region=f80b6a904048a99ce88d61420b8906d1
2011-11-16 02:25:28,796 DEBUG 
org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT event 
for f80b6a904048a99ce88d61420b8906d1; deleting node
2011-11-16 02:25:28,798 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:62003-0x132f043bbde094b Deleting existing unassigned node for 
f80b6a904048a99ce88d61420b8906d1 that is in expected state RS_ZK_REGION_SPLIT
2011-11-16 02:25:28,804 DEBUG 
org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT event 
for f80b6a904048a99ce88d61420b8906d1; deleting node
2011-11-16 02:25:28,806 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:62003-0x132f043bbde094b Deleting existing unassigned node for 
f80b6a904048a99ce88d61420b8906d1 that is in expected state RS_ZK_REGION_SPLIT
2011-11-16 02:25:28,821 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:62003-0x132f043bbde094b Successfully deleted unassigned node for region 
f80b6a904048a99ce88d61420b8906d1 in expected state RS_ZK_REGION_SPLIT
2011-11-16 02:25:28,821 INFO 
org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
report); 
parent=TestTable,006304,1321409743253.f80b6a904048a99ce88d61420b8906d1. 
daughter 
a=TestTable,006304,1321410325564.e0f5d201683bcabe14426817224334b8.daughter 
b=TestTable,007054,1321410325564.1b82eeb5d230c47ccc51c08256134839.
2011-11-16 02:25:28,829 WARN 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node 
/hbase/unassigned/f80b6a904048a99ce88d61420b8906d1 already deleted, and this is 
not a retry
2011-11-16 02:25:28,830 FATAL org.apache.hadoop.hbase.master.HMaster: Error 
deleting SPLIT node in ZK for transition ZK node 
(f80b6a904048a99ce88d61420b8906d1)
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode 
for /hbase/unassigned/f80b6a904048a99ce88d61420b8906d1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:107)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:884)
at 
org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:506)
at 
org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:453)
at 
org.apache.hadoop.hbase.master.handler.SplitRegionHandler.process(SplitRegionHandler.java:95)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:168)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}

Stack and I came up with the solution that we need just manage that exception 
because handleSplitReport is an in-memory thing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Mikhail Bautin (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4795:
--

Status: Patch Available  (was: Open)

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch, D459.2.patch, 
> Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Mikhail Bautin (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4795:
--

Attachment: Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch

Patches that Phabricator uploads automatically do not apply cleanly using patch 
-p0  Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch, D459.2.patch, 
> Fix-TestHFileBlock-heap-size-test-on-a-32-bit-JVM.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Phabricator (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150973#comment-13150973
 ] 

Phabricator commented on HBASE-4795:


stack has accepted the revision "[jira] [HBASE-4795] Fix TestHFileBlock when 
running on a 32-bit JVM".

  Good on you Mikhail.  Patch looks good.  Put it up in JIRA and I'll commit.

  I started to look at this.  Doing this to the pom will make the test run w/ 
32bit jvm so you can confirm your fix (if that helps):

  h-25-20:hbase stack$ git diff pom.xml
  diff --git a/pom.xml b/pom.xml
  index 2847416..ec6a9fb 100644
  --- a/pom.xml
  +++ b/pom.xml
  @@ -569,6 +569,7 @@
   ${myParallel}
   false
   ${myThreadCount}
  +-d32
 
   ${unittest.include}
 



REVISION DETAIL
  https://reviews.facebook.net/D459


> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch, D459.2.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Mikhail Bautin (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4795:
--

Status: Open  (was: Patch Available)

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch, D459.2.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Mikhail Bautin (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4795:
--

Status: Patch Available  (was: Open)

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch, D459.2.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Mikhail Bautin (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4795:
--

Status: Open  (was: Patch Available)

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch, D459.2.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Phabricator (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4795:
---

Attachment: D459.2.patch

mbautin updated the revision "[jira] [HBASE-4795] Fix TestHFileBlock when 
running on a 32-bit JVM".
Reviewers: tedyu, JIRA

  Rebasing on upstream changes.

REVISION DETAIL
  https://reviews.facebook.net/D459

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java
  src/main/java/org/apache/hadoop/hbase/util/ClassSize.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java


> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch, D459.2.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150966#comment-13150966
 ] 

Hadoop QA commented on HBASE-4795:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12503827/D459.1.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/260//console

This message is automatically generated.

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Mikhail Bautin (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4795:
--

Priority: Minor  (was: Major)

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Mikhail Bautin (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4795:
--

Status: Patch Available  (was: Open)

> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: D459.1.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Phabricator (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4795:
---

Attachment: D459.1.patch

mbautin requested code review of "[jira] [HBASE-4795] Fix TestHFileBlock when 
running on a 32-bit JVM".
Reviewers: tedyu, JIRA

  This fixes TestHFileBlock broken in 
https://issues.apache.org/jira/browse/HBASE-4768 when running on a 32-bit JVM, 
where reference size is 4 bytes instead of 8 bytes.

TEST PLAN
  Run TestHFileBlock and TestSchemaConfigured in both 32-bit and 64-bit mode.

REVISION DETAIL
  https://reviews.facebook.net/D459

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java
  src/main/java/org/apache/hadoop/hbase/util/ClassSize.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/969/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.


> Fix TestHFileBlock when running on a 32-bit JVM
> ---
>
> Key: HBASE-4795
> URL: https://issues.apache.org/jira/browse/HBASE-4795
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: D459.1.patch
>
>
> Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
> TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4795) Fix TestHFileBlock when running on a 32-bit JVM

[Mikhail Bautin (Created) (JIRA)]

Fix TestHFileBlock when running on a 32-bit JVM
---

 Key: HBASE-4795
 URL: https://issues.apache.org/jira/browse/HBASE-4795
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin


Our Hudson test server seems to run a 32-bit JVM. This patch fixes 
TestHFileBlock to work correctly for both 64-bit and 32-bit JVM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[Jean-Daniel Cryans (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150948#comment-13150948
 ] 

Jean-Daniel Cryans commented on HBASE-4792:
---

It mirrors what's done RS-side, there's no limit there either. We could fix 
that too if we want.

The reason there's no sleep is because the RS is sleeping and we don't want to 
hit the same race another time.

> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
> parent region stuck offline
> --
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
> daughter 
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
>  b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Region 
> e5be6551c8584a6a1065466e520faf4e not found on server 
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619 
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it 
> doesn't check the boolean that's returned. In this case it was false. So for 
> the master the split was completed, but for the region server it's another 
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't really deleted, it thinks the master just haven't got 
> to process its region thus waits which leaves the region *unavailable*.
> We need to just retry the delete master-side ASAP since the RS will wait 
> 100ms between retries.
> A

[jira] [Commented] (HBASE-4793) HBase shell still using deprecated methods removed in HBASE-4436

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150949#comment-13150949
 ] 

Ted Yu commented on HBASE-4793:
---

Patch integrated to 0.92 and TRUNK.

Thanks for the review Stack.

> HBase shell still using deprecated methods removed in HBASE-4436
> 
>
> Key: HBASE-4793
> URL: https://issues.apache.org/jira/browse/HBASE-4793
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4793.txt
>
>
> The patch applied in HBASE-4622 (subtask of HBASE-4436) to remove deprecated 
> methods seems to have missed some usage of those methods by the HBase shell.  
> At least src/main/ruby/hbase/admin.rb is still using some of the removed 
> methods, breaking some shell commands:
> {noformat}
> hbase(main):007:0> alter 'privatetable', { NAME => 'f1', VERSIONS => 2}
> ERROR: wrong number of arguments (3 for 2)
> Backtrace: /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:344:in `alter'
>org/jruby/RubyArray.java:1572:in `each'
>/usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:317:in `alter'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:79:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:68:in 
> `format_simple_command'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:78:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:74:in 
> `translate_hbase_exceptions'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell.rb:110:in `command'
>(eval):2:in `alter'
> {noformat}
> This trace translates to the line:
> {code}
>   @admin.modifyColumn(table_name, column_name, descriptor)
> {code}
> which is calling one of the removed methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150944#comment-13150944
 ] 

Ted Yu commented on HBASE-4792:
---

A loop was introduced which doesn't have sleep().
Should we exit the loop after certain number of attempts (certain amount of 
time)?

Also:
{code}
   * @throws KeeperException.NoNodeException if node does not exist
   */
  public static boolean deleteNode(ZooKeeperWatcher zkw, String regionName,
  EventType expectedState)
{code}
We should handle KeeperException.NoNodeException differently from other 
KeeperException's.

> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
> parent region stuck offline
> --
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
> daughter 
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
>  b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Region 
> e5be6551c8584a6a1065466e520faf4e not found on server 
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619 
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it 
> doesn't check the boolean that's returned. In this case it was false. So for 
> the master the split was completed, but for the region server it's another 
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't really deleted, it thinks the master jus

[jira] [Commented] (HBASE-4793) HBase shell still using deprecated methods removed in HBASE-4436

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150940#comment-13150940
 ] 

stack commented on HBASE-4793:
--

Can fix others when we find them.

> HBase shell still using deprecated methods removed in HBASE-4436
> 
>
> Key: HBASE-4793
> URL: https://issues.apache.org/jira/browse/HBASE-4793
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4793.txt
>
>
> The patch applied in HBASE-4622 (subtask of HBASE-4436) to remove deprecated 
> methods seems to have missed some usage of those methods by the HBase shell.  
> At least src/main/ruby/hbase/admin.rb is still using some of the removed 
> methods, breaking some shell commands:
> {noformat}
> hbase(main):007:0> alter 'privatetable', { NAME => 'f1', VERSIONS => 2}
> ERROR: wrong number of arguments (3 for 2)
> Backtrace: /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:344:in `alter'
>org/jruby/RubyArray.java:1572:in `each'
>/usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:317:in `alter'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:79:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:68:in 
> `format_simple_command'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:78:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:74:in 
> `translate_hbase_exceptions'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell.rb:110:in `command'
>(eval):2:in `alter'
> {noformat}
> This trace translates to the line:
> {code}
>   @admin.modifyColumn(table_name, column_name, descriptor)
> {code}
> which is calling one of the removed methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4793) HBase shell still using deprecated methods removed in HBASE-4436

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150939#comment-13150939
 ] 

stack commented on HBASE-4793:
--

+1 on patch

> HBase shell still using deprecated methods removed in HBASE-4436
> 
>
> Key: HBASE-4793
> URL: https://issues.apache.org/jira/browse/HBASE-4793
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4793.txt
>
>
> The patch applied in HBASE-4622 (subtask of HBASE-4436) to remove deprecated 
> methods seems to have missed some usage of those methods by the HBase shell.  
> At least src/main/ruby/hbase/admin.rb is still using some of the removed 
> methods, breaking some shell commands:
> {noformat}
> hbase(main):007:0> alter 'privatetable', { NAME => 'f1', VERSIONS => 2}
> ERROR: wrong number of arguments (3 for 2)
> Backtrace: /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:344:in `alter'
>org/jruby/RubyArray.java:1572:in `each'
>/usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:317:in `alter'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:79:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:68:in 
> `format_simple_command'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:78:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:74:in 
> `translate_hbase_exceptions'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell.rb:110:in `command'
>(eval):2:in `alter'
> {noformat}
> This trace translates to the line:
> {code}
>   @admin.modifyColumn(table_name, column_name, descriptor)
> {code}
> which is calling one of the removed methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4793) HBase shell still using deprecated methods removed in HBASE-4436

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4793:
--

Attachment: 4793.txt

Patch fixes parameters for admin.modifyColumn()

There could be other places to be fixed.

> HBase shell still using deprecated methods removed in HBASE-4436
> 
>
> Key: HBASE-4793
> URL: https://issues.apache.org/jira/browse/HBASE-4793
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4793.txt
>
>
> The patch applied in HBASE-4622 (subtask of HBASE-4436) to remove deprecated 
> methods seems to have missed some usage of those methods by the HBase shell.  
> At least src/main/ruby/hbase/admin.rb is still using some of the removed 
> methods, breaking some shell commands:
> {noformat}
> hbase(main):007:0> alter 'privatetable', { NAME => 'f1', VERSIONS => 2}
> ERROR: wrong number of arguments (3 for 2)
> Backtrace: /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:344:in `alter'
>org/jruby/RubyArray.java:1572:in `each'
>/usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:317:in `alter'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:79:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:68:in 
> `format_simple_command'
>
> /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:78:in `command'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:74:in 
> `translate_hbase_exceptions'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
> `command_safe'
>/usr/lib/hbase/bin/../bin/../lib/ruby/shell.rb:110:in `command'
>(eval):2:in `alter'
> {noformat}
> This trace translates to the line:
> {code}
>   @admin.modifyColumn(table_name, column_name, descriptor)
> {code}
> which is calling one of the removed methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[Jean-Daniel Cryans (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans resolved HBASE-4792.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

Committed to branch and trunk, thanks for looking at my code Stack.

> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
> parent region stuck offline
> --
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
> daughter 
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
>  b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Region 
> e5be6551c8584a6a1065466e520faf4e not found on server 
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619 
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it 
> doesn't check the boolean that's returned. In this case it was false. So for 
> the master the split was completed, but for the region server it's another 
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't really deleted, it thinks the master just haven't got 
> to process its region thus waits which leaves the region *unavailable*.
> We need to just retry the delete master-side ASAP since the RS will wait 
> 100ms between retries.
> At the same time, it would be nice if ZKAssign.deleteNode always printed out 
> the name of the region in its messages because it took me a while t

[jira] [Commented] (HBASE-4794) Altering a tables that splits can hold the command for the CatalogJanitor sleep time

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150933#comment-13150933
 ] 

Ted Yu commented on HBASE-4794:
---

We should pass true to MetaReader.getTableRegions() in 
AssignmentManager.getReopenStatus()

> Altering a tables that splits can hold the command for the CatalogJanitor 
> sleep time
> 
>
> Key: HBASE-4794
> URL: https://issues.apache.org/jira/browse/HBASE-4794
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
>
> In AssignmentManager.getReopenStatus, it calls a version of 
> MetaReader.getTableRegions that sets excludeOfflinedSplitParents to false 
> meaning that the offline parents are returned. What this means is that if one 
> of them was already closed before the alter command was issued (and I believe 
> there are a few other cases) then the alter will hang until the 
> CatalogJanitor sweeps the parent .META. row.
> Since the CJ sleep time is 5 minutes, the worst case scenario is an alter 
> that takes almost 5 minutes.
> Here's an example:
> {quote}
> 925/948 regions updated.
> 920/943 regions updated.
> 913/934 regions updated.
> 912/928 regions updated.
> 912/928 regions updated.
> (5 minutes later)
> 912/928 regions updated.
> 912/928 regions updated.
> 905/918 regions updated.
> 897/906 regions updated.
> 891/892 regions updated.
> 891/891 regions updated.
> Done.
> {quote}
> I can confirm with the log that 37 parent regions were cleaned up.
> Also it's pretty nice to see how the number fluctuates up and down :)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4789:
-

Attachment: 4789.txt

Possible workaround.  Let me do some more testing before commit.  I can't find 
the hole in our accounting of oldest seqid.

> On split, parent region is sticking around in oldest sequenceid to region map 
> though not online; we don't cleanup WALs.
> ---
>
> Key: HBASE-4789
> URL: https://issues.apache.org/jira/browse/HBASE-4789
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4789.txt
>
>
> Here is log for a particular region:
> {code}
> 2011-11-15 05:46:31,382 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
> 2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to 
> transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to 
> RS_ZK_REG
> ION_SPLIT
> 2011-11-15 05:46:31,484 INFO 
> org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META 
> updated, and report to master. 
> Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d., 
> new regions: TestTab
> le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c., 
> TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split 
> took 1sec
> 2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
> Compacting 
> hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
> hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
>  keycount=717559, bloomtype=NONE, size=711.1m
> 2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
> Compacting 
> hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
> hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
>  keycount=416691, bloomtype=NONE, size=412.9m
> 2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
> Compacting 
> hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
> hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
>  keycount=717559, bloomtype=NONE, size=711.1m
> 2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
> Compacting 
> hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
> hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
>  keycount=416691, bloomtype=NONE, size=412.9m
> 2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is 
> 5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
> 2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s): 
> 8bbd7388262dc8cb1ce2cf4f04a7281d
> 2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller: 
> Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null, 
> requester=null
> 2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s): 
> 8bbd7388262dc8cb1ce2cf4f04a7281d
> 2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller: 
> Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null, 
> requester=null
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150927#comment-13150927
 ] 

stack commented on HBASE-4792:
--

Looks good to me.  Just logging changes and looking at returned boolean.  Too 
small for unit test.  +1 on commit for 0.92.

> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
> parent region stuck offline
> --
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
> daughter 
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
>  b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Region 
> e5be6551c8584a6a1065466e520faf4e not found on server 
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619 
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it 
> doesn't check the boolean that's returned. In this case it was false. So for 
> the master the split was completed, but for the region server it's another 
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't really deleted, it thinks the master just haven't got 
> to process its region thus waits which leaves the region *unavailable*.
> We need to just retry the delete master-side ASAP since the RS will wait 
> 100ms between retries.
> At the same time, it would be nice if ZKAssign.deleteNode always printed out 
> the name of the region in its mess

[jira] [Commented] (HBASE-4729) Race between online altering and splitting kills the master

[Jean-Daniel Cryans (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150924#comment-13150924
 ] 

Jean-Daniel Cryans commented on HBASE-4729:
---

I tested out the patch and I saw it working. Would be nice to have a unit test.

> Race between online altering and splitting kills the master
> ---
>
> Key: HBASE-4729
> URL: https://issues.apache.org/jira/browse/HBASE-4729
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 4729.txt
>
>
> I was running an online alter while regions were splitting, and suddenly the 
> master died and left my table half-altered (haven't restarted the master yet).
> What killed the master:
> {quote}
> 2011-11-02 17:06:44,428 FATAL org.apache.hadoop.hbase.master.HMaster: 
> Unexpected ZK exception creating node CLOSING
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
> NodeExists for /hbase/unassigned/f7e1783e65ea8d621a4bc96ad310f101
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:110)
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:459)
> at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:441)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:769)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:568)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1722)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1661)
> at org.apache.hadoop.hbase.master.BulkReOpen$1.run(BulkReOpen.java:69)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> A znode was created because the region server was splitting the region 4 
> seconds before:
> {quote}
> 2011-11-02 17:06:40,704 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of 
> region TestTable,0012469153,1320253135043.f7e1783e65ea8d621a4bc96ad310f101.
> 2011-11-02 17:06:40,704 DEBUG 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: 
> regionserver:62023-0x132f043bbde0710 Creating ephemeral node for 
> f7e1783e65ea8d621a4bc96ad310f101 in SPLITTING state
> 2011-11-02 17:06:40,751 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde0710 Attempting to transition node 
> f7e1783e65ea8d621a4bc96ad310f101 from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLITTING
> ...
> 2011-11-02 17:06:44,061 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde0710 Successfully transitioned node 
> f7e1783e65ea8d621a4bc96ad310f101 from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-02 17:06:44,061 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for f7e1783e65ea8d621a4bc96ad310f101
> {quote}
> Now that the master is dead the region server is spewing those last two lines 
> like mad.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4654) [replication] Add a check to make sure we don't replicate to ourselves

[Jean-Daniel Cryans (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150926#comment-13150926
 ] 

Jean-Daniel Cryans commented on HBASE-4654:
---

By comparing the master address in the znode it can be done in 0.90, for the 
others we can just use the ClusterId yeah.

> [replication] Add a check to make sure we don't replicate to ourselves
> --
>
> Key: HBASE-4654
> URL: https://issues.apache.org/jira/browse/HBASE-4654
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
> Fix For: 0.90.5
>
>
> It's currently possible to add a peer for replication and point it to the 
> local cluster, which I believe could very well happen for those like us that 
> use only one ZK ensemble per DC so that only the root znode changes when you 
> want to set up replication intra-DC.
> I don't think comparing just the cluster ID would be enough because you would 
> normally use a different one for another cluster and nothing will block you 
> from pointing elsewhere.
> Comparing the ZK ensemble address doesn't work either when you have multiple 
> DNS entries that point at the same place.
> I think this could be resolved by looking up the master address in the 
> relevant znode as it should be exactly the same thing in the case where you 
> have the same cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150921#comment-13150921
 ] 

Ted Yu commented on HBASE-4213:
---

I tried the following commands for table t1 which had one column f1 and they 
worked:
{code}
alter 't1', { NAME => 'f1', VERSIONS => 1 }
alter 't1', { NAME => 'f2', VERSIONS => 1 }
{code}

> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[Jean-Daniel Cryans (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans updated HBASE-4792:
--

Attachment: HBASE-4792-0.92.patch

Patch that does the fix I described and adds more info ZKAssign WARN logs.

It probably needs a unit test.

> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
> parent region stuck offline
> --
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
> daughter 
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
>  b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Region 
> e5be6551c8584a6a1065466e520faf4e not found on server 
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619 
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it 
> doesn't check the boolean that's returned. In this case it was false. So for 
> the master the split was completed, but for the region server it's another 
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't really deleted, it thinks the master just haven't got 
> to process its region thus waits which leaves the region *unavailable*.
> We need to just retry the delete master-side ASAP since the RS will wait 
> 100ms between retries.
> At the same time, it would be nice if ZKAssign.deleteNode always printed out 
> the name of the region in its messages beca

[jira] [Created] (HBASE-4794) Altering a tables that splits can hold the command for the CatalogJanitor sleep time

[Jean-Daniel Cryans (Created) (JIRA)]

Altering a tables that splits can hold the command for the CatalogJanitor sleep 
time


 Key: HBASE-4794
 URL: https://issues.apache.org/jira/browse/HBASE-4794
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Priority: Critical
 Fix For: 0.92.0, 0.94.0


In AssignmentManager.getReopenStatus, it calls a version of 
MetaReader.getTableRegions that sets excludeOfflinedSplitParents to false 
meaning that the offline parents are returned. What this means is that if one 
of them was already closed before the alter command was issued (and I believe 
there are a few other cases) then the alter will hang until the CatalogJanitor 
sweeps the parent .META. row.

Since the CJ sleep time is 5 minutes, the worst case scenario is an alter that 
takes almost 5 minutes.

Here's an example:

{quote}
925/948 regions updated.
920/943 regions updated.
913/934 regions updated.
912/928 regions updated.
912/928 regions updated.

(5 minutes later)

912/928 regions updated.
912/928 regions updated.
905/918 regions updated.
897/906 regions updated.
891/892 regions updated.
891/891 regions updated.
Done.
{quote}

I can confirm with the log that 37 parent regions were cleaned up.

Also it's pretty nice to see how the number fluctuates up and down :)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[Jean-Daniel Cryans (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans reassigned HBASE-4792:
-

Assignee: Jean-Daniel Cryans

> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
> parent region stuck offline
> --
>
> Key: HBASE-4792
> URL: https://issues.apache.org/jira/browse/HBASE-4792
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT 
> event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
> e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
> RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
> 2011-11-15 22:28:57,975 INFO 
> org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
> report); 
> parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
> daughter 
> a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter
>  b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
> region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Region 
> e5be6551c8584a6a1065466e520faf4e not found on server 
> sv4r28s44,62023,1321395865619; failed processing
> 2011-11-15 22:28:58,052 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region 
> e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619 
> but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it 
> doesn't check the boolean that's returned. In this case it was false. So for 
> the master the split was completed, but for the region server it's another 
> story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
> e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't really deleted, it thinks the master just haven't got 
> to process its region thus waits which leaves the region *unavailable*.
> We need to just retry the delete master-side ASAP since the RS will wait 
> 100ms between retries.
> At the same time, it would be nice if ZKAssign.deleteNode always printed out 
> the name of the region in its messages because it took me a while to see that 
> the delete didn't take affect while looking at a grep.

--
This message is automatically generated by JIRA.
If you think it was 

[jira] [Created] (HBASE-4793) HBase shell still using deprecated methods removed in HBASE-4436

[Gary Helmling (Created) (JIRA)]

HBase shell still using deprecated methods removed in HBASE-4436


 Key: HBASE-4793
 URL: https://issues.apache.org/jira/browse/HBASE-4793
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 0.92.0, 0.94.0
Reporter: Gary Helmling
Priority: Blocker
 Fix For: 0.92.0


The patch applied in HBASE-4622 (subtask of HBASE-4436) to remove deprecated 
methods seems to have missed some usage of those methods by the HBase shell.  
At least src/main/ruby/hbase/admin.rb is still using some of the removed 
methods, breaking some shell commands:

{noformat}
hbase(main):007:0> alter 'privatetable', { NAME => 'f1', VERSIONS => 2}

ERROR: wrong number of arguments (3 for 2)
Backtrace: /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:344:in `alter'
   org/jruby/RubyArray.java:1572:in `each'
   /usr/lib/hbase/bin/../bin/../lib/ruby/hbase/admin.rb:317:in `alter'
   /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:79:in 
`command'
   /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:68:in 
`format_simple_command'
   /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands/alter.rb:78:in 
`command'
   /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
`command_safe'
   /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:74:in 
`translate_hbase_exceptions'
   /usr/lib/hbase/bin/../bin/../lib/ruby/shell/commands.rb:31:in 
`command_safe'
   /usr/lib/hbase/bin/../bin/../lib/ruby/shell.rb:110:in `command'
   (eval):2:in `alter'
{noformat}

This trace translates to the line:
{code}
  @admin.modifyColumn(table_name, column_name, descriptor)
{code}

which is calling one of the removed methods.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150910#comment-13150910
 ] 

Lars Hofhansl commented on HBASE-4256:
--

You are right (just looked at the code :) ). Comes back to my question above 
then, what do we want to get out of this.
Personally, we have need here to have finer control over where exactly to start 
a query and where to stop it.

> Intra-row scanning (part deux)
> --
>
> Key: HBASE-4256
> URL: https://issues.apache.org/jira/browse/HBASE-4256
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Dave Revell
>Priority: Critical
> Fix For: 0.94.0
>
>
> Dave Revell was asking on IRC today if there's a way to scan ranges of 
> *qualifiers* within a row. That is, to be able to specify a *start qualifier* 
> and an *end qualifier* so that the Get or Scan seeks directly to the first 
> qualifier and stops at some point which can be predeterminate by a qualifier 
> or simply a batch configuration (already exists).
> This is particularly useful for large rows with time-based qualifiers.
> Dave also mentioned that another popular database has such a feature that 
> they call "column slices".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150906#comment-13150906
 ] 

Hudson commented on HBASE-4768:
---

Integrated in HBase-TRUNK #2444 (See 
[https://builds.apache.org/job/HBase-TRUNK/2444/])
HBASE-4768 Addendum removes unnecessary assertion

tedyu : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java


> Per-(table, columnFamily) metrics with configurable table name inclusion
> 
>
> Key: HBASE-4768
> URL: https://issues.apache.org/jira/browse/HBASE-4768
> Project: HBase
>  Issue Type: New Feature
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 4768.addendum, 4768.addendum2, D363.1.patch, 
> D363.2.patch, D363.3.patch, D363.4.patch, D363.5.patch
>
>
> As we kept adding more granular block read and block cache usage statistics, 
> a combinatorial explosion of various cases to monitor started to happen, 
> especially when we wanted both per-table/column family/block type statistics 
> and aggregate statistics on various subsets of these dimensions. Here, we 
> un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a 
> centralized class that knows how to update all kinds of per-table/CF/block 
> type counters. 
> Table name and column family configuration have been pushed to a base class, 
> SchemaConfigured. This is convenient as many of existing classes that have 
> these properties (HFile readers/writers, HFile blocks, etc.) did not have a 
> base class. Whether to collect per-(table, columnFamily) or per-columnFamily 
> only metrics can be configured with the hbase.metrics.showTableName 
> configuration key. We don't expect this configuration to change at runtime, 
> so we cache the setting statically and log a warning when an attempt is made 
> to flip it once already set. This way we don't have to pass configuration to 
> a lot more places, e.g. everywhere an HFile reader is instantiated.
> Thanks to Liyin for his initial version of per-table metrics patch and a lot 
> of valuable feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)

[Phabricator (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4218:
---

Attachment: D447.3.patch

mbautin updated the revision "[jira] [HBASE-4218] Delta encoding for keys in 
HFile".
Reviewers: JIRA, tedyu, stack, nspiegelberg, Kannan

  Fixing TestHFileBlockDeltaEncoder that was broken by per-table/CF metrics.

REVISION DETAIL
  https://reviews.facebook.net/D447

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  src/main/java/org/apache/hadoop/hbase/KeyValue.java
  src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/BitsetKeyDeltaEncoder.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/BufferedDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/CompressionState.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/CopyKeyDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncodedBlock.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoder.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderAlgorithms.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderToSmallBufferException.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DiffKeyDeltaEncoder.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/FastDiffDeltaEncoder.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/PrefixKeyDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/BlockDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/EmptyBlockDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
  
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java
  src/main/ruby/hbase/admin.rb
  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
  src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java
  src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
  src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
  
src/test/java/org/apache/hadoop/hbase/io/deltaencoder/RedundantKVGenerator.java
  
src/test/java/org/apache/hadoop/hbase/io/deltaencoder/TestBufferedDeltaEncoder.java
  src/test/java/org/apache/hadoop/hbase/io/deltaencoder/TestDeltaEncoders.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockDeltaEncoder.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/DeltaEncodingSeekPerformance.java
  src/test/java/org/apache/hadoop/hbase/regionserver/DeltaEncodingUtil.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java


> Delta Encoding of KeyValues  (aka prefix compression)
> -
>
> Key: HBASE-4218
> URL: https://issues.apache.org/jira/browse/HBASE-4218
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 0.94.0
>Reporter: Jacek Migdal
>  Labels: compression
> Attachments: D447.1.patch, D447.2.patch, D447.3.patch, 
> open-source.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually very 
> similar. Because of that, it is possible to design bette

[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150889#comment-13150889
 ] 

Hadoop QA commented on HBASE-4768:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12503799/4768.addendum2
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 javadoc.  The javadoc tool appears to have generated -163 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 51 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.TestDrainingServer
  org.apache.hadoop.hbase.io.hfile.TestHFileBlock

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/259//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/259//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/259//console

This message is automatically generated.

> Per-(table, columnFamily) metrics with configurable table name inclusion
> 
>
> Key: HBASE-4768
> URL: https://issues.apache.org/jira/browse/HBASE-4768
> Project: HBase
>  Issue Type: New Feature
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 4768.addendum, 4768.addendum2, D363.1.patch, 
> D363.2.patch, D363.3.patch, D363.4.patch, D363.5.patch
>
>
> As we kept adding more granular block read and block cache usage statistics, 
> a combinatorial explosion of various cases to monitor started to happen, 
> especially when we wanted both per-table/column family/block type statistics 
> and aggregate statistics on various subsets of these dimensions. Here, we 
> un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a 
> centralized class that knows how to update all kinds of per-table/CF/block 
> type counters. 
> Table name and column family configuration have been pushed to a base class, 
> SchemaConfigured. This is convenient as many of existing classes that have 
> these properties (HFile readers/writers, HFile blocks, etc.) did not have a 
> base class. Whether to collect per-(table, columnFamily) or per-columnFamily 
> only metrics can be configured with the hbase.metrics.showTableName 
> configuration key. We don't expect this configuration to change at runtime, 
> so we cache the setting statically and log a warning when an attempt is made 
> to flip it once already set. This way we don't have to pass configuration to 
> a lot more places, e.g. everywhere an HFile reader is instantiated.
> Thanks to Liyin for his initial version of per-table metrics patch and a lot 
> of valuable feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150888#comment-13150888
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--



bq.  On 2011-11-15 21:01:17, Michael Stack wrote:
bq.  > 
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java,
 line 80
bq.  > 
bq.  >
bq.  > You didn't want to change name of table?
bq.  
bq.  Gary Helmling wrote:
bq.  Doesn't seem currently possible to use ".ACL." without changing 
HTableDescriptor constructors to allow bypassing the isLegalTableName() check?  
Unless we define a static ACL_TABLEDESC variable in HTD so we can use the 
protected constructor (same as .META. and -ROOT-).  But doing that seems to 
violate the separation of concerns to me.
bq.  
bq.  Michael Stack wrote:
bq.  " But doing that seems to violate the separation of concerns to me."
bq.  
bq.  This is a 'catalog' table?  Non-user table?
bq.  
bq.  Andrew Purtell wrote:
bq.  The separation that Gary refers to is between security specific 
implementation detail and the core code.

OK.  I did not get that.  I can live w/ this table name.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/#review3276
---


On 2011-11-15 19:54:02, Gary Helmling wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2041/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:54:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).
bq.  
bq.  Key parts of the implementation are:
bq.  
bq.  * AccessControlLists - encapsulates storage of permission grants in a 
metadata table ("_acl_").  This differs from previous implementation where the 
".META." table was used to store permissions.
bq.  
bq.  * AccessController - 
bq.- implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
bq.- implements AccessControllerProtocol as a coprocessor endpoint to 
provide RPC methods for granting, revoking and listing permissions.
bq.  
bq.  * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries 
and updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.
bq.  
bq.  * Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands
bq.  
bq.  * Support for a new OWNER attribute in HTableDescriptor.  I could separate 
out this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.
bq.  
bq.  
bq.  This addresses bug HBASE-3025.
bq.  https://issues.apache.org/jira/browse/HBASE-3025
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestTableP

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150887#comment-13150887
 ] 

jirapos...@reviews.apache.org commented on HBASE-4213:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1786/
---

(Updated 2011-11-15 23:28:55.604278)


Review request for Todd Lipcon, Andrew Purtell and Subbu Iyer.


Changes
---

Updated with 4213-0.92.v2


Summary
---

bq. From Subbu:
here is the latest patch that support alter_instant, an instant schema change 
command that supports (Add, Modify, Delete column and Modify table) actions 
through ZK.

1. This pattern capitalizes on the fact that HRI's are now in HDFS and need not 
be sent over again from Master to RS cloud on every schema change event.

2. Offers real time instant schema change as we bypass the explicit bulk 
reassign (unassign + assign) of regions from master to RS.

3. Offers fault tolerant schema change support as schema changes now go through 
ZK. Secondary master taking over a failed schema change will be addressed 
through a separate JIRA.


This addresses bug HBASE-4213.
https://issues.apache.org/jira/browse/HBASE-4213


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/MasterServices.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 
1202381 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java
 1202381 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java
 1202381 
  /src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java 
1202381 
  
/src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java
 1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1202381 
  /src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java 
1202381 
  
/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
1202381 
  /src/main/resources/hbase-default.xml 1202381 
  /src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java 
PRE-CREATION 
  
/src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java
 PRE-CREATION 
  /src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1202381 
  /src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
1202381 

Diff: https://reviews.apache.org/r/1786/diff


Testing
---

Unit tests pass.


Thanks,

Ted



> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Insta

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150886#comment-13150886
 ] 

Ted Yu commented on HBASE-4213:
---

https://reviews.apache.org/r/1786/ has been updated with 4213-0.92.v2

> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150885#comment-13150885
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--



bq.  On 2011-11-15 21:01:17, Michael Stack wrote:
bq.  > 
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java,
 line 80
bq.  > 
bq.  >
bq.  > You didn't want to change name of table?
bq.  
bq.  Gary Helmling wrote:
bq.  Doesn't seem currently possible to use ".ACL." without changing 
HTableDescriptor constructors to allow bypassing the isLegalTableName() check?  
Unless we define a static ACL_TABLEDESC variable in HTD so we can use the 
protected constructor (same as .META. and -ROOT-).  But doing that seems to 
violate the separation of concerns to me.
bq.  
bq.  Michael Stack wrote:
bq.  " But doing that seems to violate the separation of concerns to me."
bq.  
bq.  This is a 'catalog' table?  Non-user table?

The separation that Gary refers to is between security specific implementation 
detail and the core code.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/#review3276
---


On 2011-11-15 19:54:02, Gary Helmling wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2041/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:54:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).
bq.  
bq.  Key parts of the implementation are:
bq.  
bq.  * AccessControlLists - encapsulates storage of permission grants in a 
metadata table ("_acl_").  This differs from previous implementation where the 
".META." table was used to store permissions.
bq.  
bq.  * AccessController - 
bq.- implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
bq.- implements AccessControllerProtocol as a coprocessor endpoint to 
provide RPC methods for granting, revoking and listing permissions.
bq.  
bq.  * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries 
and updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.
bq.  
bq.  * Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands
bq.  
bq.  * Support for a new OWNER attribute in HTableDescriptor.  I could separate 
out this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.
bq.  
bq.  
bq.  This addresses bug HBASE-3025.
bq.  https://issues.apache.org/jira/browse/HBASE-3025
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/acces

[jira] [Updated] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)

[Phabricator (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4218:
---

Attachment: D447.2.patch

mbautin updated the revision "[jira] [HBASE-4218] Delta encoding for keys in 
HFile".
Reviewers: JIRA, tedyu, stack, nspiegelberg, Kannan

  Rebased Jacek's patch on the recent changes from trunk and resolved 
conflicts. The code compiles but no testing done yet.

REVISION DETAIL
  https://reviews.facebook.net/D447

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  src/main/java/org/apache/hadoop/hbase/KeyValue.java
  src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/BitsetKeyDeltaEncoder.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/BufferedDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/CompressionState.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/CopyKeyDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncodedBlock.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoder.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderAlgorithms.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderToSmallBufferException.java
  src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DiffKeyDeltaEncoder.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/FastDiffDeltaEncoder.java
  
src/main/java/org/apache/hadoop/hbase/io/deltaencoder/PrefixKeyDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/BlockDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/EmptyBlockDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockDeltaEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
  src/main/ruby/hbase/admin.rb
  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
  src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java
  src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
  src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
  
src/test/java/org/apache/hadoop/hbase/io/deltaencoder/RedundantKVGenerator.java
  
src/test/java/org/apache/hadoop/hbase/io/deltaencoder/TestBufferedDeltaEncoder.java
  src/test/java/org/apache/hadoop/hbase/io/deltaencoder/TestDeltaEncoders.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockDeltaEncoder.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/DeltaEncodingSeekPerformance.java
  src/test/java/org/apache/hadoop/hbase/regionserver/DeltaEncodingUtil.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java


> Delta Encoding of KeyValues  (aka prefix compression)
> -
>
> Key: HBASE-4218
> URL: https://issues.apache.org/jira/browse/HBASE-4218
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 0.94.0
>Reporter: Jacek Migdal
>  Labels: compression
> Attachments: D447.1.patch, D447.2.patch, open-source.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually very 
> similar. Because of that, it is possible to design better compression than 
> general purpose algorithms,
> I

[jira] [Created] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline

[Jean-Daniel Cryans (Created) (JIRA)]

SplitRegionHandler doesn't care if it deletes the znode or not, leaves the 
parent region stuck offline
--

 Key: HBASE-4792
 URL: https://issues.apache.org/jira/browse/HBASE-4792
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Priority: Critical
 Fix For: 0.92.0, 0.94.0


Saw this on a little test cluster, really easy to trigger.

First the master log:

{quote}
2011-11-15 22:28:57,900 DEBUG 
org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handling SPLIT event 
for e5be6551c8584a6a1065466e520faf4e; deleting node
2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:62003-0x132f043bbde08c1 Deleting existing unassigned node for 
e5be6551c8584a6a1065466e520faf4e that is in expected state RS_ZK_REGION_SPLIT
2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:62003-0x132f043bbde08c1 Attempting to delete unassigned node in 
RS_ZK_REGION_SPLIT state but after verifying state, we got a version mismatch
2011-11-15 22:28:57,975 INFO 
org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
report); 
parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e. 
daughter 
a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter 
b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
...
2011-11-15 22:28:58,052 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Handling transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, 
region=e5be6551c8584a6a1065466e520faf4e
2011-11-15 22:28:58,052 WARN org.apache.hadoop.hbase.master.AssignmentManager: 
Region e5be6551c8584a6a1065466e520faf4e not found on server 
sv4r28s44,62023,1321395865619; failed processing
2011-11-15 22:28:58,052 WARN org.apache.hadoop.hbase.master.AssignmentManager: 
Received SPLIT for region e5be6551c8584a6a1065466e520faf4e from server 
sv4r28s44,62023,1321395865619 but it doesn't exist anymore, probably already 
processed its split
(repeated forever)
{quote}

The master processes the split but when it calls ZKAssign.deleteNode it doesn't 
check the boolean that's returned. In this case it was false. So for the master 
the split was completed, but for the region server it's another story:

{quote}
2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
RS_ZK_REGION_SPLIT
2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING to 
RS_ZK_REGION_SPLIT
2011-11-15 22:28:57,775 INFO 
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
master to process the split for e5be6551c8584a6a1065466e520faf4e
2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:62023-0x132f043bbde08d3 Attempting to transition node 
e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
regionserver:62023-0x132f043bbde08d3 Successfully transitioned node 
e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to RS_ZK_REGION_SPLIT
(printed forever)
{quote}

Since the znode isn't really deleted, it thinks the master just haven't got to 
process its region thus waits which leaves the region *unavailable*.

We need to just retry the delete master-side ASAP since the RS will wait 100ms 
between retries.

At the same time, it would be nice if ZKAssign.deleteNode always printed out 
the name of the region in its messages because it took me a while to see that 
the delete didn't take affect while looking at a grep.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150879#comment-13150879
 ] 

jirapos...@reviews.apache.org commented on HBASE-2418:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3286
---



pom.xml


This addresses the comment about shipping a 3.4-SNAPSHOT if it is not 
warranted. The POM stanza here is a dup from related changes in HBASE-3025 with 
the zookeeper.version property added.


- Andrew


On 2011-11-15 23:15:34, Andrew Purtell wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2837/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 23:15:34)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Eugene Koontz.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  These changes add support for protecting the state of HBase znodes on a 
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at 
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025 
(Coprocessor based access control).
bq.  
bq.  SASL authentication of ZooKeeper clients with the quorum is handled in the 
ZK client independently of HBase concerns. To enable strong ZK authentication, 
one must create a suitable JaaS configuration, for example:
bq.  
bq.Server {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  storeKey=true
bq.  useTicketCache=false
bq.  principal="zookeeper/$HOSTNAME";
bq.};
bq.Client {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  useTicketCache=false
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  principal="hbase/$HOSTNAME";
bq.};
bq.  
bq.  and then configure both the client and server processes to use it, for 
example in hbase-site.xml:
bq.  
bq.HBASE_OPTS="${HBASE_OPTS} 
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeHostFromPrincipal=true"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeRealmFromPrincipal=true"
bq.  
bq.  HBase will then secure all znodes but for a few world-readable read-only 
ones needed for clients to look up region locations. All internal cluster 
operations will be protected from unauthenticated ZK clients, or clients not 
authenticated to the HBase principal. Presumably the only ZK clients 
authenticated to the HBase principal will be those embedded in the master and 
regionservers.
bq.  
bq.  There is extraneous whitespace in code surrounding these changes.
bq.  
bq.  
bq.  This addresses bug HBASE-2418.
bq.  https://issues.apache.org/jira/browse/HBASE-2418
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.pom.xml c74ce25 
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java 
05abeb7 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
a75cf87 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9 
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2837/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  These changes are running in production at Trend Micro, using a snapshot 
build of ZooKeeper 3.4.0.
bq.  
bq.  New unit test TestZooKeeperACL passes 100 iterations. All test pass not 
otherwise currently failing on trunk.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Andrew
bq.  
bq.



> add support for ZooKeeper authentication
> 
>
> Key: HBASE-2418
> URL: https://issues.apache.org/jira/browse/HBASE-2418
> Project: HBase
>  Issue Type: Improvement
>  Components: master, regionserver
>Reporter: Patrick Hunt
>Assignee: Eugene Koontz
>Priority: Critical
>  Labels: security, zookeeper
>
> Some users may run a ZooKeeper cluster in "multi tenant mode" meaning that 
> more than one client service would
> like to share a single ZooKeeper service instance (cluster). In this case the 
> client services typically want to protect
> their data (ZK znodes) from access by other services (tenants) on the 
> cluster. Say you are running HBase and Solr 
> and Neo4j, or multiple HBase instances, etc... having 
> authentication/authorization on the znodes is important for bo

[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150876#comment-13150876
 ] 

jirapos...@reviews.apache.org commented on HBASE-2418:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/
---

(Updated 2011-11-15 23:15:34.845036)


Review request for hbase, Gary Helmling and Eugene Koontz.


Changes
---

Updated patch addresses review comments to date that have specific suggestions.


Summary
---

These changes add support for protecting the state of HBase znodes on a 
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at 
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025 
(Coprocessor based access control).

SASL authentication of ZooKeeper clients with the quorum is handled in the ZK 
client independently of HBase concerns. To enable strong ZK authentication, one 
must create a suitable JaaS configuration, for example:

  Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/hbase/conf/hbase.keytab"
storeKey=true
useTicketCache=false
principal="zookeeper/$HOSTNAME";
  };
  Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
keyTab="/etc/hbase/conf/hbase.keytab"
principal="hbase/$HOSTNAME";
  };

and then configure both the client and server processes to use it, for example 
in hbase-site.xml:

  HBASE_OPTS="${HBASE_OPTS} 
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf"
  HBASE_OPTS="${HBASE_OPTS} -Dzookeeper.kerberos.removeHostFromPrincipal=true"
  HBASE_OPTS="${HBASE_OPTS} -Dzookeeper.kerberos.removeRealmFromPrincipal=true"

HBase will then secure all znodes but for a few world-readable read-only ones 
needed for clients to look up region locations. All internal cluster operations 
will be protected from unauthenticated ZK clients, or clients not authenticated 
to the HBase principal. Presumably the only ZK clients authenticated to the 
HBase principal will be those embedded in the master and regionservers.

There is extraneous whitespace in code surrounding these changes.


This addresses bug HBASE-2418.
https://issues.apache.org/jira/browse/HBASE-2418


Diffs (updated)
-

  pom.xml c74ce25 
  src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java 
05abeb7 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java a75cf87 
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9 
  src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/2837/diff


Testing
---

These changes are running in production at Trend Micro, using a snapshot build 
of ZooKeeper 3.4.0.

New unit test TestZooKeeperACL passes 100 iterations. All test pass not 
otherwise currently failing on trunk.


Thanks,

Andrew



> add support for ZooKeeper authentication
> 
>
> Key: HBASE-2418
> URL: https://issues.apache.org/jira/browse/HBASE-2418
> Project: HBase
>  Issue Type: Improvement
>  Components: master, regionserver
>Reporter: Patrick Hunt
>Assignee: Eugene Koontz
>Priority: Critical
>  Labels: security, zookeeper
>
> Some users may run a ZooKeeper cluster in "multi tenant mode" meaning that 
> more than one client service would
> like to share a single ZooKeeper service instance (cluster). In this case the 
> client services typically want to protect
> their data (ZK znodes) from access by other services (tenants) on the 
> cluster. Say you are running HBase and Solr 
> and Neo4j, or multiple HBase instances, etc... having 
> authentication/authorization on the znodes is important for both 
> security and helping to ensure that services don't interact negatively (touch 
> each other's data).
> Today HBase does not have support for authentication or authorization. This 
> should be added to the HBase clients
> that are accessing the ZK cluster. In general it means calling addAuthInfo 
> once after a session is established:
> http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String,
>  byte[])
> with a user specific credential, often times this is a shared secret or 
> certificate. You may be able to statically configure this
> in some cases (config string or file to read from), however in my case in 
> particular you may need to access it programmatically,
> which adds complexity as the end user may need to load code into H

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150871#comment-13150871
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--



bq.  On 2011-11-15 21:01:17, Michael Stack wrote:
bq.  > src/main/resources/hbase-default.xml, line 132
bq.  > 
bq.  >
bq.  > This looks like its leakage from another issue altogether
bq.  
bq.  Gary Helmling wrote:
bq.  Ugh, no idea where this came from.  Thanks for catching, I'll strip it 
out.
bq.  
bq.  Gary Helmling wrote:
bq.  Weird, the section highlighted here is not actually in my patch.  Just 
a trunk change that reviewboard is showing as different from the previous patch 
uploaded?
bq.  
bq.  The actual hbase-default.xml change is:
bq.  
bq.  +zookeeper.znode.acl.parent
bq.  +acl
bq.  +Root ZNode for access control lists.
bq.  +  
bq.  +  
bq.  +  

No problem.  Long as you commit the right stuff (smile).


bq.  On 2011-11-15 21:01:17, Michael Stack wrote:
bq.  > 
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java,
 line 80
bq.  > 
bq.  >
bq.  > You didn't want to change name of table?
bq.  
bq.  Gary Helmling wrote:
bq.  Doesn't seem currently possible to use ".ACL." without changing 
HTableDescriptor constructors to allow bypassing the isLegalTableName() check?  
Unless we define a static ACL_TABLEDESC variable in HTD so we can use the 
protected constructor (same as .META. and -ROOT-).  But doing that seems to 
violate the separation of concerns to me.

" But doing that seems to violate the separation of concerns to me."

This is a 'catalog' table?  Non-user table?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/#review3276
---


On 2011-11-15 19:54:02, Gary Helmling wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2041/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:54:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).
bq.  
bq.  Key parts of the implementation are:
bq.  
bq.  * AccessControlLists - encapsulates storage of permission grants in a 
metadata table ("_acl_").  This differs from previous implementation where the 
".META." table was used to store permissions.
bq.  
bq.  * AccessController - 
bq.- implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
bq.- implements AccessControllerProtocol as a coprocessor endpoint to 
provide RPC methods for granting, revoking and listing permissions.
bq.  
bq.  * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries 
and updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.
bq.  
bq.  * Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands
bq.  
bq.  * Support for a new OWNER attribute in HTableDescriptor.  I could separate 
out this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.
bq.  
bq.  
bq.  This addresses bug HBASE-3025.
bq.  https://issues.apache.org/jira/browse/HBASE-3025
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
security/src/main/java/org

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150860#comment-13150860
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--



bq.  On 2011-11-15 21:01:17, Michael Stack wrote:
bq.  > src/main/resources/hbase-default.xml, line 132
bq.  > 
bq.  >
bq.  > This looks like its leakage from another issue altogether
bq.  
bq.  Gary Helmling wrote:
bq.  Ugh, no idea where this came from.  Thanks for catching, I'll strip it 
out.

Weird, the section highlighted here is not actually in my patch.  Just a trunk 
change that reviewboard is showing as different from the previous patch 
uploaded?

The actual hbase-default.xml change is:

+zookeeper.znode.acl.parent
+acl
+Root ZNode for access control lists.
+  
+  
+  


- Gary


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/#review3276
---


On 2011-11-15 19:54:02, Gary Helmling wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2041/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:54:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).
bq.  
bq.  Key parts of the implementation are:
bq.  
bq.  * AccessControlLists - encapsulates storage of permission grants in a 
metadata table ("_acl_").  This differs from previous implementation where the 
".META." table was used to store permissions.
bq.  
bq.  * AccessController - 
bq.- implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
bq.- implements AccessControllerProtocol as a coprocessor endpoint to 
provide RPC methods for granting, revoking and listing permissions.
bq.  
bq.  * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries 
and updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.
bq.  
bq.  * Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands
bq.  
bq.  * Support for a new OWNER attribute in HTableDescriptor.  I could separate 
out this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.
bq.  
bq.  
bq.  This addresses bug HBASE-3025.
bq.  https://issues.apache.org/jira/browse/HBASE-3025
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestZKPermissionsWatcher.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 99875b8 
bq.
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java 
8a40762 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150857#comment-13150857
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--



bq.  On 2011-11-15 21:01:17, Michael Stack wrote:
bq.  > 
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java,
 line 80
bq.  > 
bq.  >
bq.  > You didn't want to change name of table?

Doesn't seem currently possible to use ".ACL." without changing 
HTableDescriptor constructors to allow bypassing the isLegalTableName() check?  
Unless we define a static ACL_TABLEDESC variable in HTD so we can use the 
protected constructor (same as .META. and -ROOT-).  But doing that seems to 
violate the separation of concerns to me.


bq.  On 2011-11-15 21:01:17, Michael Stack wrote:
bq.  > src/main/resources/hbase-default.xml, line 132
bq.  > 
bq.  >
bq.  > This looks like its leakage from another issue altogether

Ugh, no idea where this came from.  Thanks for catching, I'll strip it out.


- Gary


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/#review3276
---


On 2011-11-15 19:54:02, Gary Helmling wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2041/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:54:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).
bq.  
bq.  Key parts of the implementation are:
bq.  
bq.  * AccessControlLists - encapsulates storage of permission grants in a 
metadata table ("_acl_").  This differs from previous implementation where the 
".META." table was used to store permissions.
bq.  
bq.  * AccessController - 
bq.- implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
bq.- implements AccessControllerProtocol as a coprocessor endpoint to 
provide RPC methods for granting, revoking and listing permissions.
bq.  
bq.  * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries 
and updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.
bq.  
bq.  * Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands
bq.  
bq.  * Support for a new OWNER attribute in HTableDescriptor.  I could separate 
out this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.
bq.  
bq.  
bq.  This addresses bug HBASE-3025.
bq.  https://issues.apache.org/jira/browse/HBASE-3025
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hb

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150854#comment-13150854
 ] 

Ted Yu commented on HBASE-4213:
---

It turns out that 4213-0.92.v2 can be cleanly applied to TRUNK as well.
{code}
  635  mt 
-Dtest=TestInstantSchemaChange#testConcurrentInstantSchemaChangeAndSplit 
  636  mt -Dtest=TestInstantSchemaChange#testInstantSchemaJanitor 
  637  mt -Dtest=TestInstantSchemaChange#testInstantSchemaChangeExclusions 
  638  mt 
-Dtest=TestInstantSchemaChange#testInstantSchemaChangeWhileRSOpenRegionFailure
{code}
The above tests passed individually on MacBook.

> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150850#comment-13150850
 ] 

jirapos...@reviews.apache.org commented on HBASE-2418:
--



bq.  On 2011-11-15 20:12:11, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java, line 676
bq.  > 
bq.  >
bq.  > Would suggest filing the jira and reference it here.
bq.  
bq.  Andrew Purtell wrote:
bq.  Eugene opened HBASE-4791. Will make a note.

JIRA created and linked to HBASE-2418: 
https://issues.apache.org/jira/browse/HBASE-4791


bq.  On 2011-11-15 20:12:11, Michael Stack wrote:
bq.  > src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java, 
line 77
bq.  > 
bq.  >
bq.  > This is an insecure cluster using a secure zk?
bq.  
bq.  Andrew Purtell wrote:
bq.  ZooKeeper auth is independent of RPC auth.

Both zookeeper server and client are using SASL authentication since there are 
both "Server" and "Client" sections in the jaas.conf file. So HBase and 
Zookeeper authenticate with each other in this test.


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3270
---


On 2011-11-15 19:43:37, Andrew Purtell wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2837/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:43:37)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Eugene Koontz.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  These changes add support for protecting the state of HBase znodes on a 
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at 
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025 
(Coprocessor based access control).
bq.  
bq.  SASL authentication of ZooKeeper clients with the quorum is handled in the 
ZK client independently of HBase concerns. To enable strong ZK authentication, 
one must create a suitable JaaS configuration, for example:
bq.  
bq.Server {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  storeKey=true
bq.  useTicketCache=false
bq.  principal="zookeeper/$HOSTNAME";
bq.};
bq.Client {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  useTicketCache=false
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  principal="hbase/$HOSTNAME";
bq.};
bq.  
bq.  and then configure both the client and server processes to use it, for 
example in hbase-site.xml:
bq.  
bq.HBASE_OPTS="${HBASE_OPTS} 
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeHostFromPrincipal=true"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeRealmFromPrincipal=true"
bq.  
bq.  HBase will then secure all znodes but for a few world-readable read-only 
ones needed for clients to look up region locations. All internal cluster 
operations will be protected from unauthenticated ZK clients, or clients not 
authenticated to the HBase principal. Presumably the only ZK clients 
authenticated to the HBase principal will be those embedded in the master and 
regionservers.
bq.  
bq.  There is extraneous whitespace in code surrounding these changes.
bq.  
bq.  
bq.  This addresses bug HBASE-2418.
bq.  https://issues.apache.org/jira/browse/HBASE-2418
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9 
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
a75cf87 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 
bq.pom.xml c74ce25 
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java 
05abeb7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2837/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  These changes are running in production at Trend Micro, using a snapshot 
build of ZooKeeper 3.4.0.
bq.  
bq.  New unit test TestZooKeeperACL passes 100 iterations. All test pass not 
otherwise currently failing on trunk.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Andrew
bq.  
bq.



> add support for ZooKeeper authentication
> 
>
> Key: HBASE-2418
> URL: https://issue

[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150831#comment-13150831
 ] 

jirapos...@reviews.apache.org commented on HBASE-2418:
--



bq.  On 2011-11-15 20:15:21, Eugene Koontz wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java, line 669
bq.  > 
bq.  >
bq.  > I agree that simply checking for a configuration file is 
insufficient. 
bq.  > 
bq.  > We should use the JAAS configuration class 
(http://download.oracle.com/javase/1.4.2/docs/api/javax/security/auth/login/Configuration.html)
 API to parse the supplied JAAS configuration and make sure that the "Client" 
section exists. We could also log some warnings or info based on what we find 
there (for example, if the "Client" section exists, but there's no principal, 
LOG.warn("No principal found in Client section").

Thanks for opening HBASE-4791. I think we can follow up on this there.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3272
---


On 2011-11-15 19:43:37, Andrew Purtell wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2837/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:43:37)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Eugene Koontz.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  These changes add support for protecting the state of HBase znodes on a 
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at 
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025 
(Coprocessor based access control).
bq.  
bq.  SASL authentication of ZooKeeper clients with the quorum is handled in the 
ZK client independently of HBase concerns. To enable strong ZK authentication, 
one must create a suitable JaaS configuration, for example:
bq.  
bq.Server {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  storeKey=true
bq.  useTicketCache=false
bq.  principal="zookeeper/$HOSTNAME";
bq.};
bq.Client {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  useTicketCache=false
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  principal="hbase/$HOSTNAME";
bq.};
bq.  
bq.  and then configure both the client and server processes to use it, for 
example in hbase-site.xml:
bq.  
bq.HBASE_OPTS="${HBASE_OPTS} 
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeHostFromPrincipal=true"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeRealmFromPrincipal=true"
bq.  
bq.  HBase will then secure all znodes but for a few world-readable read-only 
ones needed for clients to look up region locations. All internal cluster 
operations will be protected from unauthenticated ZK clients, or clients not 
authenticated to the HBase principal. Presumably the only ZK clients 
authenticated to the HBase principal will be those embedded in the master and 
regionservers.
bq.  
bq.  There is extraneous whitespace in code surrounding these changes.
bq.  
bq.  
bq.  This addresses bug HBASE-2418.
bq.  https://issues.apache.org/jira/browse/HBASE-2418
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9 
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
a75cf87 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 
bq.pom.xml c74ce25 
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java 
05abeb7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2837/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  These changes are running in production at Trend Micro, using a snapshot 
build of ZooKeeper 3.4.0.
bq.  
bq.  New unit test TestZooKeeperACL passes 100 iterations. All test pass not 
otherwise currently failing on trunk.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Andrew
bq.  
bq.



> add support for ZooKeeper authentication
> 
>
> Key: HBASE-2418
> URL: https://issues.apache.org/jira/browse/HBASE-2418
> Project: HBase
>  Issue Type: Improvement
>  Components: master, regionserver
>Reporter: Patrick Hunt
>Assignee: Eugen

[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150828#comment-13150828
 ] 

jirapos...@reviews.apache.org commented on HBASE-2418:
--



bq.  On 2011-11-15 20:12:11, Michael Stack wrote:
bq.  > pom.xml, line 794
bq.  > 
bq.  >
bq.  > What are the implications of shipping a 3.4 zk snapshot with 0.92 
hbase?  Will a 3.4 client be able to talk to a 3.3.3. ensemble?

bq.  Will a 3.4 client be able to talk to a 3.3.3. ensemble?

No, but we can make this change conditional on enabling the Maven '-P security' 
profile. I should have done that already.


bq.  On 2011-11-15 20:12:11, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java, line 668
bq.  > 
bq.  >
bq.  > So, how do we guarantee that our session w/ zk is secure if the 
hbase install is configured secure?  What is in place to prevent our connecting 
insecure to a zk ensemble if the hbase is supposed to be secure?

ZooKeeper auth is independent of RPC auth.

Connecting insecurely to a secure ZK ensemble is totally acceptable from a 
non-HBase point of view. It allows backwards compatibility over in ZK land.

Even when running secure HBase, most clients would have no trouble even if 
connecting insecurely to ZK; only the Master and RegionServers would want to 
authenticate and set ACLs accordingly.

We could add another check elsewhere in the Master and RegionServer (via ZKUtil 
presumably) if HBase security is enabled to test that ACLs are set up, but this 
won't let someone run with an insecure ZooKeeper version, maybe 3.3.3 or 
whatever. Maybe someone will want that. I think it's a user concern.


bq.  On 2011-11-15 20:12:11, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java, line 676
bq.  > 
bq.  >
bq.  > Would suggest filing the jira and reference it here.

Eugene opened HBASE-4791. Will make a note.


bq.  On 2011-11-15 20:12:11, Michael Stack wrote:
bq.  > src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java, 
line 77
bq.  > 
bq.  >
bq.  > This is an insecure cluster using a secure zk?

ZooKeeper auth is independent of RPC auth.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3270
---


On 2011-11-15 19:43:37, Andrew Purtell wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2837/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:43:37)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Eugene Koontz.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  These changes add support for protecting the state of HBase znodes on a 
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at 
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025 
(Coprocessor based access control).
bq.  
bq.  SASL authentication of ZooKeeper clients with the quorum is handled in the 
ZK client independently of HBase concerns. To enable strong ZK authentication, 
one must create a suitable JaaS configuration, for example:
bq.  
bq.Server {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  storeKey=true
bq.  useTicketCache=false
bq.  principal="zookeeper/$HOSTNAME";
bq.};
bq.Client {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  useTicketCache=false
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  principal="hbase/$HOSTNAME";
bq.};
bq.  
bq.  and then configure both the client and server processes to use it, for 
example in hbase-site.xml:
bq.  
bq.HBASE_OPTS="${HBASE_OPTS} 
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeHostFromPrincipal=true"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeRealmFromPrincipal=true"
bq.  
bq.  HBase will then secure all znodes but for a few world-readable read-only 
ones needed for clients to look up region locations. All internal cluster 
operations will be protected from unauthenticated ZK clients, or clients not 
authenticated to the HBase principal. Presumably the only ZK clients 
authenticated to the HBase principal will be those embedded in the master and 

[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150820#comment-13150820
 ] 

Hadoop QA commented on HBASE-4213:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12503785/4213-0.92.v2
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 12 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 58 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.io.hfile.TestHFileBlock
  org.apache.hadoop.hbase.master.TestDistributedLogSplitting
  org.apache.hadoop.hbase.client.TestInstantSchemaChange
  org.apache.hadoop.hbase.regionserver.wal.TestLogRolling

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/258//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/258//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/258//console

This message is automatically generated.

> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4768:
--

Attachment: 4768.addendum2

Second attempt to fixing assertion error: 
https://builds.apache.org/job/PreCommit-HBASE-Build/257//testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testBlockHeapSize/

> Per-(table, columnFamily) metrics with configurable table name inclusion
> 
>
> Key: HBASE-4768
> URL: https://issues.apache.org/jira/browse/HBASE-4768
> Project: HBase
>  Issue Type: New Feature
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 4768.addendum, 4768.addendum2, D363.1.patch, 
> D363.2.patch, D363.3.patch, D363.4.patch, D363.5.patch
>
>
> As we kept adding more granular block read and block cache usage statistics, 
> a combinatorial explosion of various cases to monitor started to happen, 
> especially when we wanted both per-table/column family/block type statistics 
> and aggregate statistics on various subsets of these dimensions. Here, we 
> un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a 
> centralized class that knows how to update all kinds of per-table/CF/block 
> type counters. 
> Table name and column family configuration have been pushed to a base class, 
> SchemaConfigured. This is convenient as many of existing classes that have 
> these properties (HFile readers/writers, HFile blocks, etc.) did not have a 
> base class. Whether to collect per-(table, columnFamily) or per-columnFamily 
> only metrics can be configured with the hbase.metrics.showTableName 
> configuration key. We don't expect this configuration to change at runtime, 
> so we cache the setting statically and log a warning when an attempt is made 
> to flip it once already set. This way we don't have to pass configuration to 
> a lot more places, e.g. everywhere an HFile reader is instantiated.
> Thanks to Liyin for his initial version of per-table metrics patch and a lot 
> of valuable feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4768:
--

Status: Patch Available  (was: Open)

Patch testing addendum v2.

> Per-(table, columnFamily) metrics with configurable table name inclusion
> 
>
> Key: HBASE-4768
> URL: https://issues.apache.org/jira/browse/HBASE-4768
> Project: HBase
>  Issue Type: New Feature
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 4768.addendum, 4768.addendum2, D363.1.patch, 
> D363.2.patch, D363.3.patch, D363.4.patch, D363.5.patch
>
>
> As we kept adding more granular block read and block cache usage statistics, 
> a combinatorial explosion of various cases to monitor started to happen, 
> especially when we wanted both per-table/column family/block type statistics 
> and aggregate statistics on various subsets of these dimensions. Here, we 
> un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a 
> centralized class that knows how to update all kinds of per-table/CF/block 
> type counters. 
> Table name and column family configuration have been pushed to a base class, 
> SchemaConfigured. This is convenient as many of existing classes that have 
> these properties (HFile readers/writers, HFile blocks, etc.) did not have a 
> base class. Whether to collect per-(table, columnFamily) or per-columnFamily 
> only metrics can be configured with the hbase.metrics.showTableName 
> configuration key. We don't expect this configuration to change at runtime, 
> so we cache the setting statically and log a warning when an attempt is made 
> to flip it once already set. This way we don't have to pass configuration to 
> a lot more places, e.g. everywhere an HFile reader is instantiated.
> Thanks to Liyin for his initial version of per-table metrics patch and a lot 
> of valuable feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4768:
--

Status: Open  (was: Patch Available)

> Per-(table, columnFamily) metrics with configurable table name inclusion
> 
>
> Key: HBASE-4768
> URL: https://issues.apache.org/jira/browse/HBASE-4768
> Project: HBase
>  Issue Type: New Feature
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 4768.addendum, D363.1.patch, D363.2.patch, D363.3.patch, 
> D363.4.patch, D363.5.patch
>
>
> As we kept adding more granular block read and block cache usage statistics, 
> a combinatorial explosion of various cases to monitor started to happen, 
> especially when we wanted both per-table/column family/block type statistics 
> and aggregate statistics on various subsets of these dimensions. Here, we 
> un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a 
> centralized class that knows how to update all kinds of per-table/CF/block 
> type counters. 
> Table name and column family configuration have been pushed to a base class, 
> SchemaConfigured. This is convenient as many of existing classes that have 
> these properties (HFile readers/writers, HFile blocks, etc.) did not have a 
> base class. Whether to collect per-(table, columnFamily) or per-columnFamily 
> only metrics can be configured with the hbase.metrics.showTableName 
> configuration key. We don't expect this configuration to change at runtime, 
> so we cache the setting statically and log a warning when an attempt is made 
> to flip it once already set. This way we don't have to pass configuration to 
> a lot more places, e.g. everywhere an HFile reader is instantiated.
> Thanks to Liyin for his initial version of per-table metrics patch and a lot 
> of valuable feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)

[Jerry Chen (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150799#comment-13150799
 ] 

Jerry Chen commented on HBASE-4256:
---

For ColumnRangeFilter, I believe we seek to the beginning of the start column. 

> Intra-row scanning (part deux)
> --
>
> Key: HBASE-4256
> URL: https://issues.apache.org/jira/browse/HBASE-4256
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Dave Revell
>Priority: Critical
> Fix For: 0.94.0
>
>
> Dave Revell was asking on IRC today if there's a way to scan ranges of 
> *qualifiers* within a row. That is, to be able to specify a *start qualifier* 
> and an *end qualifier* so that the Get or Scan seeks directly to the first 
> qualifier and stops at some point which can be predeterminate by a qualifier 
> or simply a batch configuration (already exists).
> This is particularly useful for large rows with time-based qualifiers.
> Dave also mentioned that another popular database has such a feature that 
> they call "column slices".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150793#comment-13150793
 ] 

Lars Hofhansl commented on HBASE-4256:
--

I think the idea is the speed improvement. It is already possible to specify 
which columns/versions you want in a scan, but the the scan always has to start 
at the beginning of the row, which might be inefficient for very wide rows.


> Intra-row scanning (part deux)
> --
>
> Key: HBASE-4256
> URL: https://issues.apache.org/jira/browse/HBASE-4256
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Dave Revell
>Priority: Critical
> Fix For: 0.94.0
>
>
> Dave Revell was asking on IRC today if there's a way to scan ranges of 
> *qualifiers* within a row. That is, to be able to specify a *start qualifier* 
> and an *end qualifier* so that the Get or Scan seeks directly to the first 
> qualifier and stops at some point which can be predeterminate by a qualifier 
> or simply a batch configuration (already exists).
> This is particularly useful for large rows with time-based qualifiers.
> Dave also mentioned that another popular database has such a feature that 
> they call "column slices".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4729) Race between online altering and splitting kills the master

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150780#comment-13150780
 ] 

stack commented on HBASE-4729:
--

The above test fails don't look related (TDLS is probably too many files, THFB 
is about to be fixed, and TR just failed w/o this patch on TRUNK).

> Race between online altering and splitting kills the master
> ---
>
> Key: HBASE-4729
> URL: https://issues.apache.org/jira/browse/HBASE-4729
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 4729.txt
>
>
> I was running an online alter while regions were splitting, and suddenly the 
> master died and left my table half-altered (haven't restarted the master yet).
> What killed the master:
> {quote}
> 2011-11-02 17:06:44,428 FATAL org.apache.hadoop.hbase.master.HMaster: 
> Unexpected ZK exception creating node CLOSING
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
> NodeExists for /hbase/unassigned/f7e1783e65ea8d621a4bc96ad310f101
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:110)
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:459)
> at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:441)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:769)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:568)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1722)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1661)
> at org.apache.hadoop.hbase.master.BulkReOpen$1.run(BulkReOpen.java:69)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> A znode was created because the region server was splitting the region 4 
> seconds before:
> {quote}
> 2011-11-02 17:06:40,704 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of 
> region TestTable,0012469153,1320253135043.f7e1783e65ea8d621a4bc96ad310f101.
> 2011-11-02 17:06:40,704 DEBUG 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: 
> regionserver:62023-0x132f043bbde0710 Creating ephemeral node for 
> f7e1783e65ea8d621a4bc96ad310f101 in SPLITTING state
> 2011-11-02 17:06:40,751 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde0710 Attempting to transition node 
> f7e1783e65ea8d621a4bc96ad310f101 from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLITTING
> ...
> 2011-11-02 17:06:44,061 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde0710 Successfully transitioned node 
> f7e1783e65ea8d621a4bc96ad310f101 from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-02 17:06:44,061 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the 
> master to process the split for f7e1783e65ea8d621a4bc96ad310f101
> {quote}
> Now that the master is dead the region server is spewing those last two lines 
> like mad.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150776#comment-13150776
 ] 

Hadoop QA commented on HBASE-4768:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12503778/4768.addendum
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -163 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 51 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.io.hfile.TestHFileBlock

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/257//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/257//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/257//console

This message is automatically generated.

> Per-(table, columnFamily) metrics with configurable table name inclusion
> 
>
> Key: HBASE-4768
> URL: https://issues.apache.org/jira/browse/HBASE-4768
> Project: HBase
>  Issue Type: New Feature
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 4768.addendum, D363.1.patch, D363.2.patch, D363.3.patch, 
> D363.4.patch, D363.5.patch
>
>
> As we kept adding more granular block read and block cache usage statistics, 
> a combinatorial explosion of various cases to monitor started to happen, 
> especially when we wanted both per-table/column family/block type statistics 
> and aggregate statistics on various subsets of these dimensions. Here, we 
> un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a 
> centralized class that knows how to update all kinds of per-table/CF/block 
> type counters. 
> Table name and column family configuration have been pushed to a base class, 
> SchemaConfigured. This is convenient as many of existing classes that have 
> these properties (HFile readers/writers, HFile blocks, etc.) did not have a 
> base class. Whether to collect per-(table, columnFamily) or per-columnFamily 
> only metrics can be configured with the hbase.metrics.showTableName 
> configuration key. We don't expect this configuration to change at runtime, 
> so we cache the setting statically and log a warning when an attempt is made 
> to flip it once already set. This way we don't have to pass configuration to 
> a lot more places, e.g. everywhere an HFile reader is instantiated.
> Thanks to Liyin for his initial version of per-table metrics patch and a lot 
> of valuable feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150773#comment-13150773
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/#review3276
---

Ship it!


Patch looks good to go.  Some fixups to do on commit below.


security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java


You didn't want to change name of table?



security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java


Thanks



security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java


This is easier to understand.



security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java


Remove this on commit



security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java


Remove this on commit



src/main/resources/hbase-default.xml


This looks like its leakage from another issue altogether


- Michael


On 2011-11-15 19:54:02, Gary Helmling wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2041/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:54:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).
bq.  
bq.  Key parts of the implementation are:
bq.  
bq.  * AccessControlLists - encapsulates storage of permission grants in a 
metadata table ("_acl_").  This differs from previous implementation where the 
".META." table was used to store permissions.
bq.  
bq.  * AccessController - 
bq.- implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
bq.- implements AccessControllerProtocol as a coprocessor endpoint to 
provide RPC methods for granting, revoking and listing permissions.
bq.  
bq.  * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries 
and updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.
bq.  
bq.  * Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands
bq.  
bq.  * Support for a new OWNER attribute in HTableDescriptor.  I could separate 
out this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.
bq.  
bq.  
bq.  This addresses bug HBASE-3025.
bq.  https://issues.apache.org/jira/browse/HBASE-3025
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security

[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150772#comment-13150772
 ] 

jirapos...@reviews.apache.org commented on HBASE-2418:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3277
---



src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java


Maybe add something about final test, since we are summarizing the other 
tests: 

"Finally, we check the ACLs of a node outside of the /hbase hierarchy and 
verify that its ACL is simply 'hbase:Perms.ALL'."


- Eugene


On 2011-11-15 19:43:37, Andrew Purtell wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2837/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:43:37)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Eugene Koontz.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  These changes add support for protecting the state of HBase znodes on a 
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at 
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025 
(Coprocessor based access control).
bq.  
bq.  SASL authentication of ZooKeeper clients with the quorum is handled in the 
ZK client independently of HBase concerns. To enable strong ZK authentication, 
one must create a suitable JaaS configuration, for example:
bq.  
bq.Server {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  storeKey=true
bq.  useTicketCache=false
bq.  principal="zookeeper/$HOSTNAME";
bq.};
bq.Client {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  useTicketCache=false
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  principal="hbase/$HOSTNAME";
bq.};
bq.  
bq.  and then configure both the client and server processes to use it, for 
example in hbase-site.xml:
bq.  
bq.HBASE_OPTS="${HBASE_OPTS} 
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeHostFromPrincipal=true"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeRealmFromPrincipal=true"
bq.  
bq.  HBase will then secure all znodes but for a few world-readable read-only 
ones needed for clients to look up region locations. All internal cluster 
operations will be protected from unauthenticated ZK clients, or clients not 
authenticated to the HBase principal. Presumably the only ZK clients 
authenticated to the HBase principal will be those embedded in the master and 
regionservers.
bq.  
bq.  There is extraneous whitespace in code surrounding these changes.
bq.  
bq.  
bq.  This addresses bug HBASE-2418.
bq.  https://issues.apache.org/jira/browse/HBASE-2418
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9 
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
a75cf87 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 
bq.pom.xml c74ce25 
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java 
05abeb7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2837/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  These changes are running in production at Trend Micro, using a snapshot 
build of ZooKeeper 3.4.0.
bq.  
bq.  New unit test TestZooKeeperACL passes 100 iterations. All test pass not 
otherwise currently failing on trunk.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Andrew
bq.  
bq.



> add support for ZooKeeper authentication
> 
>
> Key: HBASE-2418
> URL: https://issues.apache.org/jira/browse/HBASE-2418
> Project: HBase
>  Issue Type: Improvement
>  Components: master, regionserver
>Reporter: Patrick Hunt
>Assignee: Eugene Koontz
>Priority: Critical
>  Labels: security, zookeeper
>
> Some users may run a ZooKeeper cluster in "multi tenant mode" meaning that 
> more than one client service would
> like to share a single ZooKeeper service instance (cluster). In this case the 
> client services typically want to protect
> their data (ZK znodes) from access by other services (tenants) on the 
> cluster. Say you are running HBase and Solr 
> and Neo4j, or multiple HBase in

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)

[Jerry Chen (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150770#comment-13150770
 ] 

Jerry Chen commented on HBASE-4256:
---

Should we consider implementing in filters? We already have a ColumnRangeFilter 
which does something similar to what we are discussing here. 

> Intra-row scanning (part deux)
> --
>
> Key: HBASE-4256
> URL: https://issues.apache.org/jira/browse/HBASE-4256
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Dave Revell
>Priority: Critical
> Fix For: 0.94.0
>
>
> Dave Revell was asking on IRC today if there's a way to scan ranges of 
> *qualifiers* within a row. That is, to be able to specify a *start qualifier* 
> and an *end qualifier* so that the Get or Scan seeks directly to the first 
> qualifier and stops at some point which can be predeterminate by a qualifier 
> or simply a batch configuration (already exists).
> This is particularly useful for large rows with time-based qualifiers.
> Dave also mentioned that another popular database has such a feature that 
> they call "column slices".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150769#comment-13150769
 ] 

Ted Yu commented on HBASE-4768:
---

Applied addendum to TRUNK.

> Per-(table, columnFamily) metrics with configurable table name inclusion
> 
>
> Key: HBASE-4768
> URL: https://issues.apache.org/jira/browse/HBASE-4768
> Project: HBase
>  Issue Type: New Feature
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 4768.addendum, D363.1.patch, D363.2.patch, D363.3.patch, 
> D363.4.patch, D363.5.patch
>
>
> As we kept adding more granular block read and block cache usage statistics, 
> a combinatorial explosion of various cases to monitor started to happen, 
> especially when we wanted both per-table/column family/block type statistics 
> and aggregate statistics on various subsets of these dimensions. Here, we 
> un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a 
> centralized class that knows how to update all kinds of per-table/CF/block 
> type counters. 
> Table name and column family configuration have been pushed to a base class, 
> SchemaConfigured. This is convenient as many of existing classes that have 
> these properties (HFile readers/writers, HFile blocks, etc.) did not have a 
> base class. Whether to collect per-(table, columnFamily) or per-columnFamily 
> only metrics can be configured with the hbase.metrics.showTableName 
> configuration key. We don't expect this configuration to change at runtime, 
> so we cache the setting statically and log a warning when an attempt is made 
> to flip it once already set. This way we don't have to pass configuration to 
> a lot more places, e.g. everywhere an HFile reader is instantiated.
> Thanks to Liyin for his initial version of per-table metrics patch and a lot 
> of valuable feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4790) Occasional TestDistributedLogSplitting failure

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150762#comment-13150762
 ] 

Hudson commented on HBASE-4790:
---

Integrated in HBase-TRUNK #2443 (See 
[https://builds.apache.org/job/HBase-TRUNK/2443/])
HBASE-4790  Occasional TestDistributedLogSplitting failure (Jinchao)

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java


> Occasional TestDistributedLogSplitting failure
> --
>
> Key: HBASE-4790
> URL: https://issues.apache.org/jira/browse/HBASE-4790
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.92.0
>Reporter: gaojinchao
>Assignee: gaojinchao
>Priority: Minor
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 4790.txt, HBASE-4790_Trunk.patch
>
>
> looks this link:
> https://builds.apache.org/job/PreCommit-HBASE-Build/253//testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testRecoveredEdits/
> // it said that regions is 0.
> 2011-11-15 03:53:11,215 INFO  [Thread-2335] 
> master.TestDistributedLogSplitting(211): #regions = 0
> 2011-11-15 03:53:11,215 DEBUG 
> [RegionServer:0;asf001.sp2.ygridcore.net,36721,1321329179789.logSyncer] 
> wal.HLog$LogSyncer(1192): 
> RegionServer:0;asf001.sp2.ygridcore.net,36721,1321329179789.logSyncer 
> interrupted while waiting for sync requests
> 2011-11-15 03:53:11,215 INFO  
> [RegionServer:0;asf001.sp2.ygridcore.net,36721,1321329179789.logSyncer] 
> wal.HLog$LogSyncer(1194): 
> RegionServer:0;asf001.sp2.ygridcore.net,36721,1321329179789.logSyncer exiting
> 2011-11-15 03:53:11,215 DEBUG [Thread-2335] wal.HLog(967): closing hlog 
> writer in 
> hdfs://localhost:46229/user/jenkins/.logs/asf001.sp2.ygridcore.net,36721,1321329179789
> 2011-11-15 03:53:11,637 DEBUG [Thread-2335] master.SplitLogManager(233): 
> Scheduling batch of logs to split

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4784) Void return types not handled correctly for CoprocessorProtocol methods

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150763#comment-13150763
 ] 

Hudson commented on HBASE-4784:
---

Integrated in HBase-TRUNK #2443 (See 
[https://builds.apache.org/job/HBase-TRUNK/2443/])
HBASE-4784  Handle void return types in CoprocessorProtocol methods

garyh : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/coprocessor/ExecResult.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java


> Void return types not handled correctly for CoprocessorProtocol methods
> ---
>
> Key: HBASE-4784
> URL: https://issues.apache.org/jira/browse/HBASE-4784
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-4784.patch
>
>
> If a CoprocessorProtocol derived interface defines a method with a void 
> return type, the method cannot be called using HTable.coprocessorExec().  
> Instead ExecResult will throw an IOException on the client trying to do a 
> Class.forName() on "void".
> Looking at ExecResult, it appears that the valueType field (which causes the 
> error) is no longer even used, so I'd suggest we just get rid of it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150759#comment-13150759
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--



bq.  On 2011-11-13 19:47:04, Michael Stack wrote:
bq.  > 
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java,
 line 238
bq.  > 
bq.  >
bq.  > These are expensive calls now that tableinfo has been removed from 
HRI (IIRC); I don't think there caching going on.
bq.  
bq.  Gary Helmling wrote:
bq.  I think the HRegionInfo version is the expensive one (doing the HDFS 
read).  For HRegion, this is just returning the HTD instance, so seems like it 
should be okay...

OK


bq.  On 2011-11-13 19:47:04, Michael Stack wrote:
bq.  > 
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java, 
line 41
bq.  > 
bq.  >
bq.  > static?
bq.  
bq.  Gary Helmling wrote:
bq.  Nested enums are implicitly static

Thanks.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/#review3191
---


On 2011-11-15 19:54:02, Gary Helmling wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2041/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:54:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).
bq.  
bq.  Key parts of the implementation are:
bq.  
bq.  * AccessControlLists - encapsulates storage of permission grants in a 
metadata table ("_acl_").  This differs from previous implementation where the 
".META." table was used to store permissions.
bq.  
bq.  * AccessController - 
bq.- implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
bq.- implements AccessControllerProtocol as a coprocessor endpoint to 
provide RPC methods for granting, revoking and listing permissions.
bq.  
bq.  * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries 
and updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.
bq.  
bq.  * Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands
bq.  
bq.  * Support for a new OWNER attribute in HTableDescriptor.  I could separate 
out this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.
bq.  
bq.  
bq.  This addresses bug HBASE-3025.
bq.  https://issues.apache.org/jira/browse/HBASE-3025
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestZKPermissionsWatcher.java
 P

[jira] [Created] (HBASE-4791) Allow Secure Zookeeper JAAS configuration to be programmatically set (rather than only by reading JAAS configuration file)

[Eugene Koontz (Created) (JIRA)]

Allow Secure Zookeeper JAAS configuration to be programmatically set (rather 
than only by reading JAAS configuration file)
--

 Key: HBASE-4791
 URL: https://issues.apache.org/jira/browse/HBASE-4791
 Project: HBase
  Issue Type: Bug
Reporter: Eugene Koontz


In the currently proposed fix for HBASE-2418, there must be a JAAS file 
specified in System.setProperty("java.security.auth.login.config"). 

However, it might be preferable to construct a JAAS configuration 
programmatically, as is done with secure Hadoop (see 
https://github.com/apache/hadoop-common/blob/a48eceb62c9b5c1a5d71ee2945d9eea2ed62527b/src/java/org/apache/hadoop/security/UserGroupInformation.java#L175).

This would have the benefit of avoiding a usage of a system property setting, 
and allow instead an HBase-local configuration setting.   



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150758#comment-13150758
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--



bq.  On 2011-11-13 06:04:14, Michael Stack wrote:
bq.  > 
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java,
 line 475
bq.  > 
bq.  >
bq.  > Don't have to flush or close the BAOS?  It probably does flush/close 
when you call toByteArray?  Over in Writables#getBytes we do something similar 
to this.
bq.  
bq.  Andrew Purtell wrote:
bq.  Closing a ByteArrayOutputStream has no effect according to 
http://download.oracle.com/javase/6/docs/api/java/io/ByteArrayOutputStream.html

Thanks


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/#review3187
---


On 2011-11-15 19:54:02, Gary Helmling wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2041/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:54:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).
bq.  
bq.  Key parts of the implementation are:
bq.  
bq.  * AccessControlLists - encapsulates storage of permission grants in a 
metadata table ("_acl_").  This differs from previous implementation where the 
".META." table was used to store permissions.
bq.  
bq.  * AccessController - 
bq.- implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
bq.- implements AccessControllerProtocol as a coprocessor endpoint to 
provide RPC methods for granting, revoking and listing permissions.
bq.  
bq.  * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries 
and updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.
bq.  
bq.  * Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands
bq.  
bq.  * Support for a new OWNER attribute in HTableDescriptor.  I could separate 
out this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.
bq.  
bq.  
bq.  This addresses bug HBASE-3025.
bq.  https://issues.apache.org/jira/browse/HBASE-3025
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
bq.
security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
 PRE-CREATION 
bq.
security/src/test/java/org/apache/hadoop/hbase/security/access/TestZKPermissionsWatcher.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 99875b8 
bq.
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java 
8a40762 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 
bq.src/main/resources/hbase-default.xml 6f98f5d 
bq.s

[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[Mikhail Bautin (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150754#comment-13150754
 ] 

Mikhail Bautin commented on HBASE-4768:
---

@Ted: thanks for clarifying and for taking care of this! Sorry for breaking the 
unit test. Now I remember that this issue also came up when porting HFile v2.

+1 on the addendum


> Per-(table, columnFamily) metrics with configurable table name inclusion
> 
>
> Key: HBASE-4768
> URL: https://issues.apache.org/jira/browse/HBASE-4768
> Project: HBase
>  Issue Type: New Feature
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 4768.addendum, D363.1.patch, D363.2.patch, D363.3.patch, 
> D363.4.patch, D363.5.patch
>
>
> As we kept adding more granular block read and block cache usage statistics, 
> a combinatorial explosion of various cases to monitor started to happen, 
> especially when we wanted both per-table/column family/block type statistics 
> and aggregate statistics on various subsets of these dimensions. Here, we 
> un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a 
> centralized class that knows how to update all kinds of per-table/CF/block 
> type counters. 
> Table name and column family configuration have been pushed to a base class, 
> SchemaConfigured. This is convenient as many of existing classes that have 
> these properties (HFile readers/writers, HFile blocks, etc.) did not have a 
> base class. Whether to collect per-(table, columnFamily) or per-columnFamily 
> only metrics can be configured with the hbase.metrics.showTableName 
> configuration key. We don't expect this configuration to change at runtime, 
> so we cache the setting statically and log a warning when an attempt is made 
> to flip it once already set. This way we don't have to pass configuration to 
> a lot more places, e.g. everywhere an HFile reader is instantiated.
> Thanks to Liyin for his initial version of per-table metrics patch and a lot 
> of valuable feedback.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150742#comment-13150742
 ] 

jirapos...@reviews.apache.org commented on HBASE-2418:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3272
---



src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java


I agree that simply checking for a configuration file is insufficient. 

We should use the JAAS configuration class 
(http://download.oracle.com/javase/1.4.2/docs/api/javax/security/auth/login/Configuration.html)
 API to parse the supplied JAAS configuration and make sure that the "Client" 
section exists. We could also log some warnings or info based on what we find 
there (for example, if the "Client" section exists, but there's no principal, 
LOG.warn("No principal found in Client section").



src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java


s/Its/It's/



src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java


Note that this fix is also in the patch for HBASE-3861: 
https://issues.apache.org/jira/secure/attachment/12478359/HBASE-3861.patch


- Eugene


On 2011-11-15 19:43:37, Andrew Purtell wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2837/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:43:37)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Eugene Koontz.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  These changes add support for protecting the state of HBase znodes on a 
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at 
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025 
(Coprocessor based access control).
bq.  
bq.  SASL authentication of ZooKeeper clients with the quorum is handled in the 
ZK client independently of HBase concerns. To enable strong ZK authentication, 
one must create a suitable JaaS configuration, for example:
bq.  
bq.Server {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  storeKey=true
bq.  useTicketCache=false
bq.  principal="zookeeper/$HOSTNAME";
bq.};
bq.Client {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  useTicketCache=false
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  principal="hbase/$HOSTNAME";
bq.};
bq.  
bq.  and then configure both the client and server processes to use it, for 
example in hbase-site.xml:
bq.  
bq.HBASE_OPTS="${HBASE_OPTS} 
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeHostFromPrincipal=true"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeRealmFromPrincipal=true"
bq.  
bq.  HBase will then secure all znodes but for a few world-readable read-only 
ones needed for clients to look up region locations. All internal cluster 
operations will be protected from unauthenticated ZK clients, or clients not 
authenticated to the HBase principal. Presumably the only ZK clients 
authenticated to the HBase principal will be those embedded in the master and 
regionservers.
bq.  
bq.  There is extraneous whitespace in code surrounding these changes.
bq.  
bq.  
bq.  This addresses bug HBASE-2418.
bq.  https://issues.apache.org/jira/browse/HBASE-2418
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9 
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
a75cf87 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 
bq.pom.xml c74ce25 
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java 
05abeb7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2837/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  These changes are running in production at Trend Micro, using a snapshot 
build of ZooKeeper 3.4.0.
bq.  
bq.  New unit test TestZooKeeperACL passes 100 iterations. All test pass not 
otherwise currently failing on trunk.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Andrew
bq.  
bq.



> add support for ZooKeeper authentication
> 
>
> Key: HBASE-2418
> URL: https://issues.apache.org/j

[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150733#comment-13150733
 ] 

jirapos...@reviews.apache.org commented on HBASE-2418:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3270
---


Patch looks good.  There's a few questions below.  You fellas are running this 
at TM?


pom.xml


What are the implications of shipping a 3.4 zk snapshot with 0.92 hbase?  
Will a 3.4 client be able to talk to a 3.3.3. ensemble?



src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java


So, how do we guarantee that our session w/ zk is secure if the hbase 
install is configured secure?  What is in place to prevent our connecting 
insecure to a zk ensemble if the hbase is supposed to be secure?



src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java


Would suggest filing the jira and reference it here.



src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java


Good



src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java


This line is no longer needed.



src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java


This is an insecure cluster using a secure zk?


- Michael


On 2011-11-15 19:43:37, Andrew Purtell wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2837/
bq.  ---
bq.  
bq.  (Updated 2011-11-15 19:43:37)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Eugene Koontz.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  These changes add support for protecting the state of HBase znodes on a 
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at 
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025 
(Coprocessor based access control).
bq.  
bq.  SASL authentication of ZooKeeper clients with the quorum is handled in the 
ZK client independently of HBase concerns. To enable strong ZK authentication, 
one must create a suitable JaaS configuration, for example:
bq.  
bq.Server {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  storeKey=true
bq.  useTicketCache=false
bq.  principal="zookeeper/$HOSTNAME";
bq.};
bq.Client {
bq.  com.sun.security.auth.module.Krb5LoginModule required
bq.  useKeyTab=true
bq.  useTicketCache=false
bq.  keyTab="/etc/hbase/conf/hbase.keytab"
bq.  principal="hbase/$HOSTNAME";
bq.};
bq.  
bq.  and then configure both the client and server processes to use it, for 
example in hbase-site.xml:
bq.  
bq.HBASE_OPTS="${HBASE_OPTS} 
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeHostFromPrincipal=true"
bq.HBASE_OPTS="${HBASE_OPTS} 
-Dzookeeper.kerberos.removeRealmFromPrincipal=true"
bq.  
bq.  HBase will then secure all znodes but for a few world-readable read-only 
ones needed for clients to look up region locations. All internal cluster 
operations will be protected from unauthenticated ZK clients, or clients not 
authenticated to the HBase principal. Presumably the only ZK clients 
authenticated to the HBase principal will be those embedded in the master and 
regionservers.
bq.  
bq.  There is extraneous whitespace in code surrounding these changes.
bq.  
bq.  
bq.  This addresses bug HBASE-2418.
bq.  https://issues.apache.org/jira/browse/HBASE-2418
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9 
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
a75cf87 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53 
bq.pom.xml c74ce25 
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java 
05abeb7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2837/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  These changes are running in production at Trend Micro, using a snapshot 
build of ZooKeeper 3.4.0.
bq.  
bq.  New unit test TestZooKeeperACL passes 100 iterations. All test pass not 
otherwise currently failing on trunk.
bq.  
bq.  
bq.  Tha

[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4213:
--

Attachment: 4213-0.92.v2

Version 2 of patch for 0.92 branch.

> Support for fault tolerant, instant schema updates with out master's 
> intervention (i.e with out enable/disable and bulk assign/unassign) through 
> ZK.
> 
>
> Key: HBASE-4213
> URL: https://issues.apache.org/jira/browse/HBASE-4213
> Project: HBase
>  Issue Type: Improvement
>Reporter: Subbu M Iyer
>Assignee: Subbu M Iyer
> Fix For: 0.92.0
>
> Attachments: 4213-0.92.txt, 4213-0.92.v2, 
> 4213-101211-Support_instant_schema_changes_through_ZK.patch, 
> 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 
> 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 
> 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 
> 4213-V10-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V5-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V7-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V8-Support_instant_schema_changes_through_ZK.patch, 
> 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 
> 4213.v6, HBASE-4213-Instant_schema_change.patch, 
> HBASE-4213_Instant_schema_change_-Version_2_.patch, 
> HBASE_Instant_schema_change-version_3_.patch
>
>
> This Jira is a slight variation in approach to what is being done as part of 
> https://issues.apache.org/jira/browse/HBASE-1730
> Support instant schema updates such as Modify Table, Add Column, Modify 
> Column operations:
> 1. With out enable/disabling the table.
> 2. With out bulk unassign/assign of regions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150723#comment-13150723
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/
---

(Updated 2011-11-15 19:54:02.538497)


Review request for hbase.


Changes
---

Updated patch addressing review comments.


Summary
---

This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).

Key parts of the implementation are:

* AccessControlLists - encapsulates storage of permission grants in a metadata 
table ("_acl_").  This differs from previous implementation where the ".META." 
table was used to store permissions.

* AccessController - 
  - implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
  - implements AccessControllerProtocol as a coprocessor endpoint to provide 
RPC methods for granting, revoking and listing permissions.

* ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries and 
updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.

* Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands

* Support for a new OWNER attribute in HTableDescriptor.  I could separate out 
this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.


This addresses bug HBASE-3025.
https://issues.apache.org/jira/browse/HBASE-3025


Diffs (updated)
-

  
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java 
PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/access/TestZKPermissionsWatcher.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 99875b8 
  src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java 
8a40762 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 
  src/main/resources/hbase-default.xml 6f98f5d 
  src/main/ruby/hbase.rb 4d27191 
  src/main/ruby/hbase/admin.rb 17cc891 
  src/main/ruby/hbase/hbase.rb beb2450 
  src/main/ruby/hbase/security.rb PRE-CREATION 
  src/main/ruby/shell.rb 9a47600 
  src/main/ruby/shell/commands.rb a352c2e 
  src/main/ruby/shell/commands/grant.rb PRE-CREATION 
  src/main/ruby/shell/commands/revoke.rb PRE-CREATION 
  src/main/ruby/shell/commands/user_permission.rb PRE-CREATION 

Diff: https://reviews.apache.org/r/2041/diff


Testing
---


Thanks,

Gary



> Coprocessor based simple access control
> ---
>
> Key: HBASE-3025
> URL: https://issues.apache.org/jira/browse/HBASE-3025
> Project: HBase
>  Issue Type: Sub-task
>  Components: coprocessors
>Reporter: Andrew Purtell
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-3025.1.patch, HBASE-3025.2011-02-01.patch
>
>
> Thanks for the clarification Jeff which reminds me to edit this issue.
> Goals of this issue
> # Client access to HBase is authe

[jira] [Commented] (HBASE-4729) Race between online altering and splitting kills the master

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150722#comment-13150722
 ] 

Hadoop QA commented on HBASE-4729:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12503776/4729.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 javadoc.  The javadoc tool appears to have generated -163 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 51 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
  org.apache.hadoop.hbase.replication.TestReplication
  org.apache.hadoop.hbase.io.hfile.TestHFileBlock

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/256//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/256//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/256//console

This message is automatically generated.

> Race between online altering and splitting kills the master
> ---
>
> Key: HBASE-4729
> URL: https://issues.apache.org/jira/browse/HBASE-4729
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 4729.txt
>
>
> I was running an online alter while regions were splitting, and suddenly the 
> master died and left my table half-altered (haven't restarted the master yet).
> What killed the master:
> {quote}
> 2011-11-02 17:06:44,428 FATAL org.apache.hadoop.hbase.master.HMaster: 
> Unexpected ZK exception creating node CLOSING
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
> NodeExists for /hbase/unassigned/f7e1783e65ea8d621a4bc96ad310f101
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:110)
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:459)
> at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:441)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:769)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:568)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1722)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1661)
> at org.apache.hadoop.hbase.master.BulkReOpen$1.run(BulkReOpen.java:69)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> A znode was created because the region server was splitting the region 4 
> seconds before:
> {quote}
> 2011-11-02 17:06:40,704 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of 
> region TestTable,0012469153,1320253135043.f7e1783e65ea8d621a4bc96ad310f101.
> 2011-11-02 17:06:40,704 DEBUG 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: 
> regionserver:62023-0x132f043bbde0710 Creating ephemeral node for 
> f7e1783e65ea8d621a4bc96ad310f101 in SPLITTING state
> 2011-11-02 17:06:40,751 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde0710 Attempting to transition node 
> f7e1783e65ea8d621a4bc96ad310f101 from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLITTING
> ...
> 2011-11-02 17:06:44,061 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> regionserver:62023-0x132f043bbde0710 Successfully transitioned node 
> f7e1783e65ea8d621a4bc96ad310f101 from RS_ZK_REGION_SPLITTING to 
> RS_ZK_REGION_SPLIT
> 2011-11-02 17:06:44,061 INFO 
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on t