date:20111114


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4780:
---

Attachment: 4780_trunk.patch

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4780:
---

Status: Patch Available  (was: Open)

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4739) Master dying while going to close a region can leave it in transition forever

2011-11-14 Thread gaojinchao (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149532#comment-13149532
 ] 

gaojinchao commented on HBASE-4739:
---

Please review this patch.


 Master dying while going to close a region can leave it in transition forever
 -

 Key: HBASE-4739
 URL: https://issues.apache.org/jira/browse/HBASE-4739
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.92.0, 0.94.0, 0.90.5

 Attachments: HBASE-4739_Trunk.patch


 I saw this in the aftermath of HBASE-4729 on a 0.92 refreshed yesterday, when 
 the master died it had just created the RIT znode for a region but didn't 
 tell the RS to close it yet.
 When the master restarted it saw the znode and started printing this:
 {quote}
 2011-11-03 00:02:49,130 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed 
 out:  TestTable,0007560564,1320253568406.f76899564cabe7e9857c3aeb526ec9dc. 
 state=CLOSING, ts=1320253605285, server=sv4r11s38,62003,1320195046948
 2011-11-03 00:02:49,130 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Region has been CLOSING for 
 too long, this should eventually complete or the server will expire, doing 
 nothing
 {quote}
 It's never going to happen, and it's blocking balancing.
 I'm marking this as minor since I believe this situation is pretty rare 
 unless you hit other bugs while trying out stuff to root bugs out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown

[
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149611#comment-13149611
]

Hadoop QA commented on HBASE-4780:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503593/4780_trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

-1 javadoc. The javadoc tool appears to have generated -163 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 50 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestScannerTimeout
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.TestFullLogReconstruction
org.apache.hadoop.hbase.TestZooKeeper
org.apache.hadoop.hbase.master.TestRollingRestart
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
org.apache.hadoop.hbase.master.TestDistributedLogSplitting

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/241//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/241//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/241//console

This message is automatically generated.

Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and
ServerManager#letRegionServersShutdown

Key: HBASE-4780
URL: https://issues.apache.org/jira/browse/HBASE-4780
Project: HBase
Issue Type: Improvement
Components: master, regionserver, test
Affects Versions: 0.94.0
Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 4780_trunk.patch

HRegionServer#waitOnAllRegionsToClose sleep 1s
Lower waiting time
benefits: around 0.8s at shutdown
ServerManager.letRegionServersShutdown() waits 1 second = responsible for
0.5 seconds in a HBase cluster stops
Can be replaced by a notification
Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4780:
---

Status: Open  (was: Patch Available)

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch, 4780_trunk.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4780:
---

Status: Patch Available  (was: Open)

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch, 4780_trunk.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4780:
---

Attachment: 4780_trunk.patch

let give it another go

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch, 4780_trunk.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4739) Master dying while going to close a region can leave it in transition forever

2011-11-14 Thread Ted Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4739:
--

Status: Patch Available  (was: Open)

 Master dying while going to close a region can leave it in transition forever
 -

 Key: HBASE-4739
 URL: https://issues.apache.org/jira/browse/HBASE-4739
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.92.0, 0.94.0, 0.90.5

 Attachments: HBASE-4739_Trunk.patch


 I saw this in the aftermath of HBASE-4729 on a 0.92 refreshed yesterday, when 
 the master died it had just created the RIT znode for a region but didn't 
 tell the RS to close it yet.
 When the master restarted it saw the znode and started printing this:
 {quote}
 2011-11-03 00:02:49,130 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed 
 out:  TestTable,0007560564,1320253568406.f76899564cabe7e9857c3aeb526ec9dc. 
 state=CLOSING, ts=1320253605285, server=sv4r11s38,62003,1320195046948
 2011-11-03 00:02:49,130 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Region has been CLOSING for 
 too long, this should eventually complete or the server will expire, doing 
 nothing
 {quote}
 It's never going to happen, and it's blocking balancing.
 I'm marking this as minor since I believe this situation is pretty rare 
 unless you hit other bugs while trying out stuff to root bugs out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4739) Master dying while going to close a region can leave it in transition forever


[ 
https://issues.apache.org/jira/browse/HBASE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149660#comment-13149660
 ] 

Ted Yu commented on HBASE-4739:
---

Why was the check for zk node existence in actOnTimeOut() at line 2531 removed ?

Would sendRegionClose() be a better name for the new method, 
handleRegionClose() ?

 Master dying while going to close a region can leave it in transition forever
 -

 Key: HBASE-4739
 URL: https://issues.apache.org/jira/browse/HBASE-4739
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.92.0, 0.94.0, 0.90.5

 Attachments: HBASE-4739_Trunk.patch


 I saw this in the aftermath of HBASE-4729 on a 0.92 refreshed yesterday, when 
 the master died it had just created the RIT znode for a region but didn't 
 tell the RS to close it yet.
 When the master restarted it saw the znode and started printing this:
 {quote}
 2011-11-03 00:02:49,130 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed 
 out:  TestTable,0007560564,1320253568406.f76899564cabe7e9857c3aeb526ec9dc. 
 state=CLOSING, ts=1320253605285, server=sv4r11s38,62003,1320195046948
 2011-11-03 00:02:49,130 INFO 
 org.apache.hadoop.hbase.master.AssignmentManager: Region has been CLOSING for 
 too long, this should eventually complete or the server will expire, doing 
 nothing
 {quote}
 It's never going to happen, and it's blocking balancing.
 I'm marking this as minor since I believe this situation is pretty rare 
 unless you hit other bugs while trying out stuff to root bugs out.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown

[
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149696#comment-13149696
]

Hadoop QA commented on HBASE-4780:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503610/4780_trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 javadoc. The javadoc tool appears to have generated -163 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 50 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestRollingRestart
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.client.TestScannerTimeout
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
org.apache.hadoop.hbase.TestFullLogReconstruction
org.apache.hadoop.hbase.TestZooKeeper

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/242//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/242//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/242//console

This message is automatically generated.

Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and
ServerManager#letRegionServersShutdown

[jira] [Commented] (HBASE-4739) Master dying while going to close a region can leave it in transition forever

[
https://issues.apache.org/jira/browse/HBASE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149727#comment-13149727
]

Hadoop QA commented on HBASE-4739:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12503590/HBASE-4739_Trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -163 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 50 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/243//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/243//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/243//console

This message is automatically generated.

Master dying while going to close a region can leave it in transition forever
-

Key: HBASE-4739
URL: https://issues.apache.org/jira/browse/HBASE-4739
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Minor
Fix For: 0.92.0, 0.94.0, 0.90.5

Attachments: HBASE-4739_Trunk.patch

I saw this in the aftermath of HBASE-4729 on a 0.92 refreshed yesterday, when
the master died it had just created the RIT znode for a region but didn't
tell the RS to close it yet.
When the master restarted it saw the znode and started printing this:
{quote}
2011-11-03 00:02:49,130 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: TestTable,0007560564,1320253568406.f76899564cabe7e9857c3aeb526ec9dc.
state=CLOSING, ts=1320253605285, server=sv4r11s38,62003,1320195046948
2011-11-03 00:02:49,130 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been CLOSING for
too long, this should eventually complete or the server will expire, doing
nothing
{quote}
It's never going to happen, and it's blocking balancing.
I'm marking this as minor since I believe this situation is pretty rare
unless you hit other bugs while trying out stuff to root bugs out.

[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine

2011-11-14 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149750#comment-13149750
 ] 

Ted Yu commented on HBASE-2742:
---

For ZKSecretWatcher.refreshNodes(), can we remove the zk node corresponding to 
path ?


 Provide strong authentication with a secure RPC engine
 --

 Key: HBASE-2742
 URL: https://issues.apache.org/jira/browse/HBASE-2742
 Project: HBase
  Issue Type: Improvement
  Components: ipc
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.92.0


 The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off 
 of Hadoop RPC classes, with some performance tweaks added.  Those 
 optimizations have come at a cost in keeping up with Hadoop RPC changes 
 however, both bug fixes and improvements/new features.  
 In particular, this impacts how we implement security features in HBase (see 
 HBASE-1697 and HBASE-2016).  The secure Hadoop implementation (HADOOP-4487) 
 relies heavily on RPC changes to support client authentication via kerberos 
 and securing and mutual authentication of client/server connections via SASL. 
  Making use of the built-in Hadoop RPC classes will gain us these pieces for 
 free in a secure HBase.
 So, I'm proposing that we drop the HBase forked version of RPC and convert to 
 direct use of Hadoop RPC, while working to contribute important fixes back 
 upstream to Hadoop core.  Based on a review of the HBase RPC changes, the key 
 divergences seem to be:
 HBaseClient:
  - added use of TCP keepalive (HBASE-1754)
  - made connection retries and sleep configurable (HBASE-1815)
  - prevent NPE if socket == null due to creation failure (HBASE-2443)
 HBaseRPC:
  - mapping of method names - codes (removed in HBASE-2219)
 HBaseServer:
  - use of TCP keep alives (HBASE-1754)
  - OOME in server does not trigger abort (HBASE-1198)
 HbaseObjectWritable:
  - allows List serialization
  - includes it's own class - code mapping (HBASE-328)
 Proposed process is:
 1. open issues with patches on Hadoop core for important fixes/adjustments 
 from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a 
 pluggable ObjectWritable implementation in RPC.Invocation to allow use of 
 HbaseObjectWritable).
 2. ship a Hadoop version with RPC patches applied -- ideally we should avoid 
 another copy-n-paste code fork, subject to ability to isolate changes from 
 impacting Hadoop internal RPC wire formats
 3. if all Hadoop core patches are applied we can drop back to a plain vanilla 
 Hadoop version
 I realize there are many different opinions on how to proceed with HBase RPC, 
 so I'm hoping this issue will kick off a discussion on what the best approach 
 might be.  My own motivation is maximizing re-use of the authentication and 
 connection security work that's already gone into Hadoop core.  I'll put 
 together a set of patches around #1 and #2, but obviously we need some 
 consensus around this to move forward.  If I'm missing other differences 
 between HBase and Hadoop RPC, please list as well.  Discuss!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2742) Provide strong authentication with a secure RPC engine

2011-11-14 Thread Ted Yu (Issue Comment Edited) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149750#comment-13149750
 ] 

Ted Yu edited comment on HBASE-2742 at 11/14/11 5:18 PM:
-

For ZKSecretWatcher.refreshNodes(), can we remove the zk node corresponding to 
path whose deserialization resulted in IOE ?


  was (Author: yuzhih...@gmail.com):
For ZKSecretWatcher.refreshNodes(), can we remove the zk node corresponding 
to path ?

  
 Provide strong authentication with a secure RPC engine
 --

 Key: HBASE-2742
 URL: https://issues.apache.org/jira/browse/HBASE-2742
 Project: HBase
  Issue Type: Improvement
  Components: ipc
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.92.0


 The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off 
 of Hadoop RPC classes, with some performance tweaks added.  Those 
 optimizations have come at a cost in keeping up with Hadoop RPC changes 
 however, both bug fixes and improvements/new features.  
 In particular, this impacts how we implement security features in HBase (see 
 HBASE-1697 and HBASE-2016).  The secure Hadoop implementation (HADOOP-4487) 
 relies heavily on RPC changes to support client authentication via kerberos 
 and securing and mutual authentication of client/server connections via SASL. 
  Making use of the built-in Hadoop RPC classes will gain us these pieces for 
 free in a secure HBase.
 So, I'm proposing that we drop the HBase forked version of RPC and convert to 
 direct use of Hadoop RPC, while working to contribute important fixes back 
 upstream to Hadoop core.  Based on a review of the HBase RPC changes, the key 
 divergences seem to be:
 HBaseClient:
  - added use of TCP keepalive (HBASE-1754)
  - made connection retries and sleep configurable (HBASE-1815)
  - prevent NPE if socket == null due to creation failure (HBASE-2443)
 HBaseRPC:
  - mapping of method names - codes (removed in HBASE-2219)
 HBaseServer:
  - use of TCP keep alives (HBASE-1754)
  - OOME in server does not trigger abort (HBASE-1198)
 HbaseObjectWritable:
  - allows List serialization
  - includes it's own class - code mapping (HBASE-328)
 Proposed process is:
 1. open issues with patches on Hadoop core for important fixes/adjustments 
 from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a 
 pluggable ObjectWritable implementation in RPC.Invocation to allow use of 
 HbaseObjectWritable).
 2. ship a Hadoop version with RPC patches applied -- ideally we should avoid 
 another copy-n-paste code fork, subject to ability to isolate changes from 
 impacting Hadoop internal RPC wire formats
 3. if all Hadoop core patches are applied we can drop back to a plain vanilla 
 Hadoop version
 I realize there are many different opinions on how to proceed with HBase RPC, 
 so I'm hoping this issue will kick off a discussion on what the best approach 
 might be.  My own motivation is maximizing re-use of the authentication and 
 connection security work that's already gone into Hadoop core.  I'll put 
 together a set of patches around #1 and #2, but obviously we need some 
 consensus around this to move forward.  If I'm missing other differences 
 between HBase and Hadoop RPC, please list as well.  Discuss!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4628) Enhance Table Create Presplit Functionality within the HBase Shell

2011-11-14 Thread Phabricator (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149789#comment-13149789
 ] 

Phabricator commented on HBASE-4628:


cgist has commented on the revision HBASE-4628 [jira] Enhance Table Create 
Presplit Functionality within the HBase Shell.

  This diff was only intended for 89-fb. There is a separate diff for apache 
trunk D429.

REVISION DETAIL
  https://reviews.facebook.net/D417


 Enhance Table Create Presplit Functionality within the HBase Shell
 --

 Key: HBASE-4628
 URL: https://issues.apache.org/jira/browse/HBASE-4628
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
 Attachments: HBASE-4628.D411.1.patch, HBASE-4628.D417.1.patch, 
 HBASE-4628.D417.2.patch, HBASE-4628.D429.1.patch


 Currently, we allow the user to presplit in the HBase shell by explicitly 
 listing the startkey of all the region shards that they want.  Instead, we 
 should provide the RegionSplitter functionality of choosing a split 
 algorithm, followed by the number of splits that they want.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4760) Add Developer Debug Options to HBase Config


[ 
https://issues.apache.org/jira/browse/HBASE-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149799#comment-13149799
 ] 

Nicolas Spiegelberg commented on HBASE-4760:


@stack : sorry, that's HBASE-4761.  To fix the recursion issue, please see 
HBASE-4760 ;)

 Add Developer Debug Options to HBase Config
 ---

 Key: HBASE-4760
 URL: https://issues.apache.org/jira/browse/HBASE-4760
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Minor
 Fix For: 0.94.0


 Add in optional HBase configuration options that core developers will 
 commonly use: an option to enable JDWP debugging  an option to use a 
 separate logfile for GC information.  (Part of the effort to move 89-fb 
 features over to trunk)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


 [ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4764:
---

Attachment: 4764_trunk.patch

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


 [ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4764:
---

Status: Patch Available  (was: Open)

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4780:
---

Status: Open  (was: Patch Available)

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch, 4780_trunk.patch, 4780_trunk.v2.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4780:
---

Attachment: 4780_trunk.v2.patch

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch, 4780_trunk.patch, 4780_trunk.v2.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4704) A JRuby script for identifying active master

2011-11-14 Thread Ted Yu (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-4704.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

 A JRuby script for identifying active master
 

 Key: HBASE-4704
 URL: https://issues.apache.org/jira/browse/HBASE-4704
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Trivial
 Fix For: 0.94.0

 Attachments: D117.1.patch, HBASE-4704.D117.2.patch, 
 HBASE-4704.D117.3.patch


 This simple script reads the HBase master ZK node and outputs the hostname of 
 the active master. This is needed so that operational scripts can decide 
 where the primary master is running. I am also including a one-line 
 hbase-jruby script so we can make our jruby scripts proper UNIX executables 
 by including an #!/usr/bin/env hbase-jruby at the top.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4704) A JRuby script for identifying active master

2011-11-14 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149822#comment-13149822
 ] 

Ted Yu commented on HBASE-4704:
---

Integrated to TRUNK.

Thanks for the patch Mikhail.

Thanks for the review Michael.

 A JRuby script for identifying active master
 

 Key: HBASE-4704
 URL: https://issues.apache.org/jira/browse/HBASE-4704
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Trivial
 Fix For: 0.94.0

 Attachments: D117.1.patch, HBASE-4704.D117.2.patch, 
 HBASE-4704.D117.3.patch


 This simple script reads the HBase master ZK node and outputs the hostname of 
 the active master. This is needed so that operational scripts can decide 
 where the primary master is running. I am also including a one-line 
 hbase-jruby script so we can make our jruby scripts proper UNIX executables 
 by including an #!/usr/bin/env hbase-jruby at the top.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown

2011-11-14 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149857#comment-13149857
 ] 

Lars Hofhansl commented on HBASE-4780:
--

v2 looks good to me

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch, 4780_trunk.patch, 4780_trunk.v2.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine


[ 
https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149859#comment-13149859
 ] 

jirapos...@reviews.apache.org commented on HBASE-2742:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1991/
---

(Updated 2011-11-14 20:03:17.125142)


Review request for hbase.


Changes
---

This update addresses the current review comments.

The only comment not addressed is Ted's suggestion in ZKSecretWatcher to 
attempt to remove the znode for any secret key that could not be deserialized.  
I need to look at the implications of this for the leader secret manager, as 
well as for potential upgrade scenarios.


Summary
---

This patch creates a new secure RPC engine for HBase, which provides Kerberos 
based authentication of clients, and a token-based authentication mechanism for 
mapreduce jobs.  Primary components of the patch are:

- a new maven profile for secure Hadoop/HBase: hadoop-0.20S
  - Secure Hadoop dependent classes are separated under a pseudo-module in the 
security/ directory.  These source and test directories are only including if 
building the secure Hadoop profile
  - Currently the security classes get packaged with the regular HBase build 
artifacts.  We need a way to at least override project.version, so we can 
append something like a -security suffix indicating the additional security 
components.
  - The pseudo-module here is really a half-step forward.  It enables the 
security code to be optionally included in the build for now, and sets up the 
structure for a security module.  But we still will want to pursue full 
modularization (see HBASE-4336), which will allow packing the security code in 
a separate build artifact.

- a new RPC engine providing kerberos and token-based authentication: 
org.apache.hadoop.hbase.ipc.SecureRpcEngine
  - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
  - The implementation classes extend the existing HBaseClient and HBaseServer 
to share as much of the RPC code as possible.  The main override is of the 
connection classes to allow control over the SASL negotiation of secure 
connections

- existing RPC changes
  - The existing HBaseClient and HBaseServer have been modified to make 
subclassing possible
  - All references to Hadoop UserGroupInformation have been replaced with 
org.apache.hadoop.hbase.security.User to insulate from future dependencies on 
specific Hadoop versions

- a coprocessor endpoint for obtaining new authentication tokens: 
TokenProvider, and supporting classes for token generation and synchronization 
(incorporating HBASE-3615)
  - implementation is under 
security/src/main/java/org/apache/hadoop/hbase/security/token/
  - Secret keys for token generation and verification are synchronized 
throughout the cluster in zookeeper, under /hbase/tokenauth/keys


To enable secure RPC, add the following configuration to hbase-site.xml:

  property
   namehadoop.security.authorization/name
   valuetrue/value
  /property
  property
   namehadoop.security.authentication/name
   valuekerberos/value
  /property
  property
   namehbase.rpc.engine/name
   valueorg.apache.hadoop.hbase.ipc.SecureRpcEngine/value
  /property
  property
   namehbase.coprocessor.region.classes/name
   valueorg.apache.hadoop.hbase.security.token.TokenProvider/value
  /property

In addition, the master and regionserver processes must be configured for 
kerberos authentication using the properties:

 * hbase.(master|regionserver).keytab.file
 * hbase.(master|regionserver).kerberos.principal
 * hbase.(master|regionserver).kerberos.https.principal


This addresses bug HBASE-2742.
https://issues.apache.org/jira/browse/HBASE-2742


Diffs (updated)
-

  conf/hbase-policy.xml PRE-CREATION 
  pom.xml 2847416 
  security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java 
PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java 
PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java 
PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java 
PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java 
PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java 
PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java

[jira] [Commented] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown


[ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149860#comment-13149860
 ] 

nkeywal commented on HBASE-4780:


Yep, I re run the tests to be sure.

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch, 4780_trunk.patch, 4780_trunk.v2.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4780:
---

Status: Patch Available  (was: Open)

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4780_trunk.patch, 4780_trunk.patch, 4780_trunk.v2.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4781) Pom update to use the new versions of surefire junit.

2011-11-14 Thread nkeywal (Created) (JIRA)

Pom update to use the new versions of surefire  junit.
---

 Key: HBASE-4781
 URL: https://issues.apache.org/jira/browse/HBASE-4781
 Project: HBase
  Issue Type: Improvement
  Components: build
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor


Upgrading surefire  junit requires some .pom modifications. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4781) Pom update to use the new versions of surefire junit.


 [ 
https://issues.apache.org/jira/browse/HBASE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4781:
---

Attachment: 4781_trunk_pom.patch

Used for testing surefire  junit private version. Not to be commited 
immediatly.

mvn test for small  medium test. Generates two reports.

mvn verify or mvn test -P runAllTests for all tests. The former does not 
run the large tests if the small or medium failed.

 Pom update to use the new versions of surefire  junit.
 ---

 Key: HBASE-4781
 URL: https://issues.apache.org/jira/browse/HBASE-4781
 Project: HBase
  Issue Type: Improvement
  Components: build
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4781_trunk_pom.patch


 Upgrading surefire  junit requires some .pom modifications. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)

2011-11-14 Thread Dave Revell (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149878#comment-13149878
 ] 

Dave Revell commented on HBASE-4256:


I've started work on this and I'm having trouble deciding on the best 
interface. I can see a couple ways that it might work:

1. Add Scan.setStartQualifier() and Scan.setEndQualifier(). What if the user 
also uses addColumn(), would we fetch those columns also?

2. Add Scan.addColumnRange(byte[] start, byte[] end). Could the user specify 
multiple (possibly overlapping) ranges? Again, would this conflict with 
addColumn()?

Any input would be appreciated.

 Intra-row scanning (part deux)
 --

 Key: HBASE-4256
 URL: https://issues.apache.org/jira/browse/HBASE-4256
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Dave Revell
Priority: Critical
 Fix For: 0.94.0


 Dave Revell was asking on IRC today if there's a way to scan ranges of 
 *qualifiers* within a row. That is, to be able to specify a *start qualifier* 
 and an *end qualifier* so that the Get or Scan seeks directly to the first 
 qualifier and stops at some point which can be predeterminate by a qualifier 
 or simply a batch configuration (already exists).
 This is particularly useful for large rows with time-based qualifiers.
 Dave also mentioned that another popular database has such a feature that 
 they call column slices.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest

[
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149903#comment-13149903
]

Hadoop QA commented on HBASE-4764:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503651/4764_trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 10 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -163 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 50 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/244//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/244//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/244//console

This message is automatically generated.

naming errors for TestHLogUtils and SoftValueSortedMapTest
--

Key: HBASE-4764
URL: https://issues.apache.org/jira/browse/HBASE-4764
Project: HBase
Issue Type: Improvement
Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 4764_trunk.patch

SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's
not called. I don't know if it's used.
TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses
the script looking for the tests. It would seems a better thing to rename it.
Is there anything special to do to keep the history attached to this file?

[jira] [Commented] (HBASE-4704) A JRuby script for identifying active master

2011-11-14 Thread Hudson (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149912#comment-13149912
 ] 

Hudson commented on HBASE-4704:
---

Integrated in HBase-TRUNK #2437 (See 
[https://builds.apache.org/job/HBase-TRUNK/2437/])
HBASE-4704 Differential Revision: #117 A JRuby script for identifying 
active master

tedyu : 
Files : 
* /hbase/trunk/bin/get-active-master.rb
* /hbase/trunk/bin/hbase-jruby


 A JRuby script for identifying active master
 

 Key: HBASE-4704
 URL: https://issues.apache.org/jira/browse/HBASE-4704
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Trivial
 Fix For: 0.94.0

 Attachments: D117.1.patch, HBASE-4704.D117.2.patch, 
 HBASE-4704.D117.3.patch


 This simple script reads the HBase master ZK node and outputs the hostname of 
 the active master. This is needed so that operational scripts can decide 
 where the primary master is running. I am also including a one-line 
 hbase-jruby script so we can make our jruby scripts proper UNIX executables 
 by including an #!/usr/bin/env hbase-jruby at the top.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


[ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149913#comment-13149913
 ] 

Ted Yu commented on HBASE-4764:
---

TestMasterFailover hung.

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


[ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149924#comment-13149924
 ] 

nkeywal commented on HBASE-4764:


It's just a renaming of a class that impacted only client.TestAdmin? I checked 
again the patch, it seems ok

This said; I saw the tests randomly blocked in my env as well but not when I 
tested this patch. May be there is something else (I hope there is something 
else!). 

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


[ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149928#comment-13149928
 ] 

nkeywal commented on HBASE-4764:


@ted: If the process is still alive, can you jstack it? Dead processes can be 
identified by their ppid equals to 1, but it's still possible to jstack them 
with their pid.

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest

2011-11-14 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149929#comment-13149929
 ] 

Ted Yu commented on HBASE-4764:
---

I don't have access to the build machine.

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4704) A JRuby script for identifying active master


[ 
https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149974#comment-13149974
 ] 

Nicolas Spiegelberg commented on HBASE-4704:


instead of creating a new file, we'd have less files by adding a 'jruby' 
shortcut to bin/hbase and invoke 'bin/hbase jruby'

 A JRuby script for identifying active master
 

 Key: HBASE-4704
 URL: https://issues.apache.org/jira/browse/HBASE-4704
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Trivial
 Fix For: 0.94.0

 Attachments: D117.1.patch, HBASE-4704.D117.2.patch, 
 HBASE-4704.D117.3.patch


 This simple script reads the HBase master ZK node and outputs the hostname of 
 the active master. This is needed so that operational scripts can decide 
 where the primary master is running. I am also including a one-line 
 hbase-jruby script so we can make our jruby scripts proper UNIX executables 
 by including an #!/usr/bin/env hbase-jruby at the top.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4778) Don't ignore corrupt StoreFiles when opening a region

2011-11-14 Thread Nicolas Spiegelberg (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149977#comment-13149977
]

Nicolas Spiegelberg commented on HBASE-4778:

@Ted: In this particular case, I think we ended up finding that HDFS wasn't
reading truncated HFiles properly. Overall though, the idea is to prioritize
data integrity consistency over availability. We shouldn't be silently
opening regions with missing data. We should instead understand why the data
is missing. If someone wants to add a 'data.loss.acceptable' flag and allow
this to happen, then that's fine.

Don't ignore corrupt StoreFiles when opening a region
-

Key: HBASE-4778
URL: https://issues.apache.org/jira/browse/HBASE-4778
Project: HBase
Issue Type: Bug
Reporter: Nicolas Spiegelberg
Assignee: Mikhail Bautin
Fix For: 0.94.0

Attachments: HBASE-4778.patch

We used to ignore StoreFiles that failed to open, which led to a situation
when only a subset of regions was opened, and HBase did not return results to
clients for the affected set of keys. This change makes sure we propagate
IOExceptions coming from an attempt to open a StoreFile all the way up to
HRegionServer.openRegion, where it will lead to a failure to open the whole
region. This way we can avoid returning corrupt data to the application.

[jira] [Commented] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown

[
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149976#comment-13149976
]

Hadoop QA commented on HBASE-4780:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503652/4780_trunk.v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 javadoc. The javadoc tool appears to have generated -163 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 51 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/245//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/245//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/245//console

This message is automatically generated.

Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and
ServerManager#letRegionServersShutdown

[jira] [Commented] (HBASE-4778) Don't ignore corrupt StoreFiles when opening a region


[ 
https://issues.apache.org/jira/browse/HBASE-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149980#comment-13149980
 ] 

Ted Yu commented on HBASE-4778:
---

I agree.

 Don't ignore corrupt StoreFiles when opening a region
 -

 Key: HBASE-4778
 URL: https://issues.apache.org/jira/browse/HBASE-4778
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Assignee: Mikhail Bautin
 Fix For: 0.94.0

 Attachments: HBASE-4778.patch


 We used to ignore StoreFiles that failed to open, which led to a situation 
 when only a subset of regions was opened, and HBase did not return results to 
 clients for the affected set of keys. This change makes sure we propagate 
 IOExceptions coming from an attempt to open a StoreFile all the way up to 
 HRegionServer.openRegion, where it will lead to a failure to open the whole 
 region. This way we can avoid returning corrupt data to the application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


[ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150001#comment-13150001
 ] 

nkeywal commented on HBASE-4764:


For example, after a look at the code where the above sample blocks; the 
current implementation is:
{noformat}
while (!this.stopped  (notimeout || remaining  0)  this.data == null) {
  if (notimeout) {
wait();
continue;
  }
  wait(remaining);
  remaining = timeout - (System.currentTimeMillis() - startTime);
}
{noformat}

This means that if the notification is sent before we actually started waiting, 
we will wait forever. Probability is quite low and I don't think that's the 
root cause here, but who knows?. An implementation like this one would be more 
secure:
{noformat}
long previousLogTime = 0;
while (!this.stopped  (notimeout || remaining  0)  this.data == null) {
  if (System.currentTimeMillis()  previousLogTime + 1000) {
LOG.info(Waiting for node to be available );
previousLogTime = System.currentTimeMillis();
  }

  wait(200);
  remaining = timeout - (System.currentTimeMillis() - startTime);
}
{noformat}


The is no notifications in getData(), don't know if it's voluntary.

The recursive call to start in ZooKeeperNodeTracker#start() is not really 
necessary.

But to conclude, we don't even know if it's a stack like this one on the build 
machine.


Can we put a script in the cron tab? Something that would run every 10 minutes 
and look for dead processes? We could also hijack a maven task to run a script 
when it's launched and make it send a mail.

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4778) Don't ignore corrupt StoreFiles when opening a region

2011-11-14 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150002#comment-13150002
 ] 

Ted Yu commented on HBASE-4778:
---

+1 on patch.

 Don't ignore corrupt StoreFiles when opening a region
 -

 Key: HBASE-4778
 URL: https://issues.apache.org/jira/browse/HBASE-4778
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Assignee: Mikhail Bautin
 Fix For: 0.94.0

 Attachments: HBASE-4778.patch


 We used to ignore StoreFiles that failed to open, which led to a situation 
 when only a subset of regions was opened, and HBase did not return results to 
 clients for the affected set of keys. This change makes sure we propagate 
 IOExceptions coming from an attempt to open a StoreFile all the way up to 
 HRegionServer.openRegion, where it will lead to a failure to open the whole 
 region. This way we can avoid returning corrupt data to the application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4778) Don't ignore corrupt StoreFiles when opening a region

2011-11-14 Thread Nicolas Spiegelberg (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-4778:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

 Don't ignore corrupt StoreFiles when opening a region
 -

 Key: HBASE-4778
 URL: https://issues.apache.org/jira/browse/HBASE-4778
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Assignee: Mikhail Bautin
 Fix For: 0.94.0

 Attachments: HBASE-4778.patch


 We used to ignore StoreFiles that failed to open, which led to a situation 
 when only a subset of regions was opened, and HBase did not return results to 
 clients for the affected set of keys. This change makes sure we propagate 
 IOExceptions coming from an attempt to open a StoreFile all the way up to 
 HRegionServer.openRegion, where it will lead to a failure to open the whole 
 region. This way we can avoid returning corrupt data to the application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4771) Script to List HRegion Availability


[ 
https://issues.apache.org/jira/browse/HBASE-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150015#comment-13150015
 ] 

Nicolas Spiegelberg commented on HBASE-4771:


sounds good.  I'm going to have Karthik commit

 Script to List HRegion Availability
 ---

 Key: HBASE-4771
 URL: https://issues.apache.org/jira/browse/HBASE-4771
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Minor
 Attachments: D399.1.patch, D399.2.patch


 One critical feature necessary to completely automate production pushes is 
 the ability to recognize when HBase has successfully made all regions 
 available for reads/writes (step prior to HBCK).  Created script to display 
 region availability status and stall until all onlined regions are actually 
 available.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine

2011-11-14 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150018#comment-13150018
]

jirapos...@reviews.apache.org commented on HBASE-2742:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1991/#review3235
---

security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
https://reviews.apache.org/r/1991/#comment7204

Should we use org.apache.hadoop.hbase.ipc.SecureClient ?

security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
https://reviews.apache.org/r/1991/#comment7203

Should read 'when running as master'

security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
https://reviews.apache.org/r/1991/#comment7202

This log should be guarded by LOG.isDebugEnabled()

src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
https://reviews.apache.org/r/1991/#comment7231

Please add curly braces around the log statement.

src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java
https://reviews.apache.org/r/1991/#comment7227

Can we rewrite using a loop instead of recursion ?

- Ted

On 2011-11-14 20:03:17, Gary Helmling wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1991/
bq. ---
bq.
bq. (Updated 2011-11-14 20:03:17)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. This patch creates a new secure RPC engine for HBase, which provides
Kerberos based authentication of clients, and a token-based authentication
mechanism for mapreduce jobs. Primary components of the patch are:
bq.
bq. - a new maven profile for secure Hadoop/HBase: hadoop-0.20S
bq.- Secure Hadoop dependent classes are separated under a pseudo-module in
the security/ directory. These source and test directories are only including
if building the secure Hadoop profile
bq.- Currently the security classes get packaged with the regular HBase
build artifacts. We need a way to at least override project.version, so we can
append something like a -security suffix indicating the additional security
components.
bq.- The pseudo-module here is really a half-step forward. It enables the
security code to be optionally included in the build for now, and sets up the
structure for a security module. But we still will want to pursue full
modularization (see HBASE-4336), which will allow packing the security code in
a separate build artifact.
bq.
bq. - a new RPC engine providing kerberos and token-based authentication:
org.apache.hadoop.hbase.ipc.SecureRpcEngine
bq.- implementation under
security/src/main/java/org/apache/hadoop/hbase/ipc/
bq.- The implementation classes extend the existing HBaseClient and
HBaseServer to share as much of the RPC code as possible. The main override is
of the connection classes to allow control over the SASL negotiation of secure
connections
bq.
bq. - existing RPC changes
bq.- The existing HBaseClient and HBaseServer have been modified to make
subclassing possible
bq.- All references to Hadoop UserGroupInformation have been replaced with
org.apache.hadoop.hbase.security.User to insulate from future dependencies on
specific Hadoop versions
bq.
bq. - a coprocessor endpoint for obtaining new authentication tokens:
TokenProvider, and supporting classes for token generation and synchronization
(incorporating HBASE-3615)
bq.- implementation is under
security/src/main/java/org/apache/hadoop/hbase/security/token/
bq.- Secret keys for token generation and verification are synchronized
throughout the cluster in zookeeper, under /hbase/tokenauth/keys
bq.
bq.
bq. To enable secure RPC, add the following configuration to hbase-site.xml:
bq.
bq.property
bq. namehadoop.security.authorization/name
bq. valuetrue/value
bq./property
bq.property
bq. namehadoop.security.authentication/name
bq. valuekerberos/value
bq./property
bq.property
bq. namehbase.rpc.engine/name
bq. valueorg.apache.hadoop.hbase.ipc.SecureRpcEngine/value
bq./property
bq.property
bq. namehbase.coprocessor.region.classes/name
bq. valueorg.apache.hadoop.hbase.security.token.TokenProvider/value
bq./property
bq.
bq. In addition, the master and regionserver processes must be configured for
kerberos authentication using the properties:
bq.
bq. * hbase.(master|regionserver).keytab.file
bq. * hbase.(master|regionserver).kerberos.principal
bq. *

[jira] [Updated] (HBASE-4776) HLog.closed should be checked inside of updateLock

2011-11-14 Thread stack (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4776:
-

Fix Version/s: 0.92.0

Committed to 0.92 too.

 HLog.closed should be checked inside of updateLock
 --

 Key: HBASE-4776
 URL: https://issues.apache.org/jira/browse/HBASE-4776
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Assignee: Prakash Khemani
 Fix For: 0.92.0, 0.94.0

 Attachments: HBASE-4776.patch


 Concurrency issue: HLog.closed is set inside the updateLock but not checked 
 inside the lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.


 [ 
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4718:
--

Status: Open  (was: Patch Available)

 Backport HBASE-4552 to 0.90 branch.
 ---

 Key: HBASE-4718
 URL: https://issues.apache.org/jira/browse/HBASE-4718
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 4718-v2.90, hbase-4718.0.90.patch, 
 hbase-4718.v3.includes-hbase-3316.patch


 In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about 
 whether and how to backport HBASE-4552 to the 0.90 branch.  This is a 
 potentially compatibility breaking so several approaches hav ebeen suggested.
 1) provide patch but do not integrate
 2) integrate patch that extends and deprecates old api without removing old 
 api.  It has been argued that  clients are supposed to use 
 LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.


 [ 
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4718:
--

Attachment: (was: hbase-4552.v4.patch)

 Backport HBASE-4552 to 0.90 branch.
 ---

 Key: HBASE-4718
 URL: https://issues.apache.org/jira/browse/HBASE-4718
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 4718-v2.90, hbase-4718.0.90.patch, 
 hbase-4718.v3.includes-hbase-3316.patch


 In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about 
 whether and how to backport HBASE-4552 to the 0.90 branch.  This is a 
 potentially compatibility breaking so several approaches hav ebeen suggested.
 1) provide patch but do not integrate
 2) integrate patch that extends and deprecates old api without removing old 
 api.  It has been argued that  clients are supposed to use 
 LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.


 [ 
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4718:
--

Attachment: hbase-4552.v4.patch

New version of patch that includes backport of HBASE-4552, HBASE-4716, 
HBASE-3316 and HBASE-4740.

 Backport HBASE-4552 to 0.90 branch.
 ---

 Key: HBASE-4718
 URL: https://issues.apache.org/jira/browse/HBASE-4718
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 4718-v2.90, hbase-4718.0.90.patch, 
 hbase-4718.v3.includes-hbase-3316.patch


 In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about 
 whether and how to backport HBASE-4552 to the 0.90 branch.  This is a 
 potentially compatibility breaking so several approaches hav ebeen suggested.
 1) provide patch but do not integrate
 2) integrate patch that extends and deprecates old api without removing old 
 api.  It has been argued that  clients are supposed to use 
 LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4778) Don't ignore corrupt StoreFiles when opening a region

2011-11-14 Thread stack (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4778:
-

Fix Version/s: 0.92.0

Committed to 0.92.

 Don't ignore corrupt StoreFiles when opening a region
 -

 Key: HBASE-4778
 URL: https://issues.apache.org/jira/browse/HBASE-4778
 Project: HBase
  Issue Type: Bug
Reporter: Nicolas Spiegelberg
Assignee: Mikhail Bautin
 Fix For: 0.92.0, 0.94.0

 Attachments: HBASE-4778.patch


 We used to ignore StoreFiles that failed to open, which led to a situation 
 when only a subset of regions was opened, and HBase did not return results to 
 clients for the affected set of keys. This change makes sure we propagate 
 IOExceptions coming from an attempt to open a StoreFile all the way up to 
 HRegionServer.openRegion, where it will lead to a failure to open the whole 
 region. This way we can avoid returning corrupt data to the application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.


 [ 
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4718:
--

Status: Patch Available  (was: Open)

 Backport HBASE-4552 to 0.90 branch.
 ---

 Key: HBASE-4718
 URL: https://issues.apache.org/jira/browse/HBASE-4718
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 4718-v2.90, hbase-4718.0.90.patch, 
 hbase-4718.v3.includes-hbase-3316.patch, hbase-4718.v4.patch


 In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about 
 whether and how to backport HBASE-4552 to the 0.90 branch.  This is a 
 potentially compatibility breaking so several approaches hav ebeen suggested.
 1) provide patch but do not integrate
 2) integrate patch that extends and deprecates old api without removing old 
 api.  It has been argued that  clients are supposed to use 
 LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.


 [ 
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4718:
--

Attachment: hbase-4718.v4.patch

 Backport HBASE-4552 to 0.90 branch.
 ---

 Key: HBASE-4718
 URL: https://issues.apache.org/jira/browse/HBASE-4718
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 4718-v2.90, hbase-4718.0.90.patch, 
 hbase-4718.v3.includes-hbase-3316.patch, hbase-4718.v4.patch


 In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about 
 whether and how to backport HBASE-4552 to the 0.90 branch.  This is a 
 potentially compatibility breaking so several approaches hav ebeen suggested.
 1) provide patch but do not integrate
 2) integrate patch that extends and deprecates old api without removing old 
 api.  It has been argued that  clients are supposed to use 
 LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


[ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150031#comment-13150031
 ] 

stack commented on HBASE-4764:
--

@nkeywal We don't have access to the apache build boxes.  My blanket i want 
access was rebuked with a we have lots of build machines and giving you 
access to them all is untenable (Fair enough I suppose)

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

2011-11-14 Thread Mikhail Bautin (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mikhail Bautin updated HBASE-4768:
--

Release Note: Trying to get the most recent patch tested on Hudson.
Status: Patch Available (was: Open)

Per-(table, columnFamily) metrics with configurable table name inclusion

Key: HBASE-4768
URL: https://issues.apache.org/jira/browse/HBASE-4768
Project: HBase
Issue Type: New Feature
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D363.1.patch, D363.2.patch, D363.3.patch, D363.4.patch

As we kept adding more granular block read and block cache usage statistics,
a combinatorial explosion of various cases to monitor started to happen,
especially when we wanted both per-table/column family/block type statistics
and aggregate statistics on various subsets of these dimensions. Here, we
un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a
centralized class that knows how to update all kinds of per-table/CF/block
type counters.
Table name and column family configuration have been pushed to a base class,
SchemaConfigured. This is convenient as many of existing classes that have
these properties (HFile readers/writers, HFile blocks, etc.) did not have a
base class. Whether to collect per-(table, columnFamily) or per-columnFamily
only metrics can be configured with the hbase.metrics.showTableName
configuration key. We don't expect this configuration to change at runtime,
so we cache the setting statically and log a warning when an attempt is made
to flip it once already set. This way we don't have to pass configuration to
a lot more places, e.g. everywhere an HFile reader is instantiated.
Thanks to Liyin for his initial version of per-table metrics patch and a lot
of valuable feedback.

[jira] [Updated] (HBASE-4780) Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and ServerManager#letRegionServersShutdown

2011-11-14 Thread stack (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4780:
-

   Resolution: Fixed
Fix Version/s: 0.94.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to TRUNK.  Thanks for patch N.

 Lower mini cluster shutdown time in HRegionServer#waitOnAllRegionsToClose and 
 ServerManager#letRegionServersShutdown
 

 Key: HBASE-4780
 URL: https://issues.apache.org/jira/browse/HBASE-4780
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver, test
Affects Versions: 0.94.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.94.0

 Attachments: 4780_trunk.patch, 4780_trunk.patch, 4780_trunk.v2.patch


 HRegionServer#waitOnAllRegionsToClose sleep 1s 
  Lower waiting time
  benefits: around 0.8s at shutdown
 ServerManager.letRegionServersShutdown() waits 1 second = responsible for 
 0.5 seconds in a HBase cluster stops
   Can be replaced by a notification
   Benefits: around 0.5s at shutdown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4704) A JRuby script for identifying active master


[ 
https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150035#comment-13150035
 ] 

stack commented on HBASE-4704:
--

@Nicolas So, one script that does it all? You'd pass jruby a bunch of command 
options first to choose the operation and then that operations params?  Or are 
you thinking 'jruby' is short for 'org.Main.jruby'?

 A JRuby script for identifying active master
 

 Key: HBASE-4704
 URL: https://issues.apache.org/jira/browse/HBASE-4704
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Trivial
 Fix For: 0.94.0

 Attachments: D117.1.patch, HBASE-4704.D117.2.patch, 
 HBASE-4704.D117.3.patch


 This simple script reads the HBase master ZK node and outputs the hostname of 
 the active master. This is needed so that operational scripts can decide 
 where the primary master is running. I am also including a one-line 
 hbase-jruby script so we can make our jruby scripts proper UNIX executables 
 by including an #!/usr/bin/env hbase-jruby at the top.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


[ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150041#comment-13150041
 ] 

nkeywal commented on HBASE-4764:


Ok, may be we can handle this with a post build task that would execute a
script?
However, we need to be careful not to break anything, but it should work I
think.

On Mon, Nov 14, 2011 at 11:47 PM, stack (Commented) (JIRA)



 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.

[
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150046#comment-13150046
]

Hadoop QA commented on HBASE-4718:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503685/hbase-4718.v4.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 10 new or modified tests.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/247//console

This message is automatically generated.

Backport HBASE-4552 to 0.90 branch.
---

Key: HBASE-4718
URL: https://issues.apache.org/jira/browse/HBASE-4718
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.90.5

Attachments: 4718-v2.90, hbase-4718.0.90.patch,
hbase-4718.v3.includes-hbase-3316.patch, hbase-4718.v4.patch

In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about
whether and how to backport HBASE-4552 to the 0.90 branch. This is a
potentially compatibility breaking so several approaches hav ebeen suggested.
1) provide patch but do not integrate
2) integrate patch that extends and deprecates old api without removing old
api. It has been argued that clients are supposed to use
LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

[jira] [Updated] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.

2011-11-14 Thread Ted Yu (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ted Yu updated HBASE-4718:
--

Fix Version/s: 0.90.5
Release Note:
HRegion.bulkLoadHFile() has been removed in 0.92 and TRUNK. It is deprecated in
0.90.5
HRegion.bulkLoadHFiles() is introduced for multi-family bulk load.
Default value for hbase.bulkload.retries.number is 0, meaning bulk load would
never give up.

Backport HBASE-4552 to 0.90 branch.
---

Key: HBASE-4718
URL: https://issues.apache.org/jira/browse/HBASE-4718
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.90.5

Attachments: 4718-v2.90, hbase-4718.0.90.patch,
hbase-4718.v3.includes-hbase-3316.patch, hbase-4718.v4.patch

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)


[ 
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150048#comment-13150048
 ] 

stack commented on HBASE-4256:
--

On 1. I think it depends.

addColumn can specify a (a) cf only, (b) a cf+qualifier, or a list of (c) 
cf+qualifiers.

If (a), then the setStartQualifier/setEndQualifier would return items from 
within a CF between the specified qualifiers (do you need an 
'inclusive/exclusive' flag or you just going to have them be both inclusive of 
specified qualifier?)

On (b), i'd think that we'd just return the qualifier only IFF it was within 
the setStartQualifier/setEndQualifier bounds.

For (c), we'd return members of the list that are inside the 
setStartQualifier/setEndQualifier bounds.

I think I prefer 2. api to 1. api.  It maps well to TimeRange.  Its a bit more 
of a pain specifying 'start x to the end of the row' with api 2. but its 
palatable enough.

Ditto on how api 2. plays with addColumn.

 Intra-row scanning (part deux)
 --

 Key: HBASE-4256
 URL: https://issues.apache.org/jira/browse/HBASE-4256
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Dave Revell
Priority: Critical
 Fix For: 0.94.0


 Dave Revell was asking on IRC today if there's a way to scan ranges of 
 *qualifiers* within a row. That is, to be able to specify a *start qualifier* 
 and an *end qualifier* so that the Get or Scan seeks directly to the first 
 qualifier and stops at some point which can be predeterminate by a qualifier 
 or simply a batch configuration (already exists).
 This is particularly useful for large rows with time-based qualifiers.
 Dave also mentioned that another popular database has such a feature that 
 they call column slices.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4761) Add Developer Debug Options to HBase Config


[ 
https://issues.apache.org/jira/browse/HBASE-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150056#comment-13150056
 ] 

stack commented on HBASE-4761:
--

+1 for 0.92 too.

 Add Developer Debug Options to HBase Config
 ---

 Key: HBASE-4761
 URL: https://issues.apache.org/jira/browse/HBASE-4761
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Minor
 Attachments: HBASE-4761.patch


 Add in optional HBase configuration options that core developers will 
 commonly use: an option to enable JDWP debugging  an option to use a 
 separate logfile for GC information.  (Part of the effort to move 89-fb 
 features over to trunk)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4761) Add Developer Debug Options to HBase Config


[ 
https://issues.apache.org/jira/browse/HBASE-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150055#comment-13150055
 ] 

stack commented on HBASE-4761:
--

+1

 Add Developer Debug Options to HBase Config
 ---

 Key: HBASE-4761
 URL: https://issues.apache.org/jira/browse/HBASE-4761
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Minor
 Attachments: HBASE-4761.patch


 Add in optional HBase configuration options that core developers will 
 commonly use: an option to enable JDWP debugging  an option to use a 
 separate logfile for GC information.  (Part of the effort to move 89-fb 
 features over to trunk)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.

2011-11-14 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150062#comment-13150062
 ] 

Jonathan Hsieh commented on HBASE-4718:
---

Patch *failed* to apply..

 Backport HBASE-4552 to 0.90 branch.
 ---

 Key: HBASE-4718
 URL: https://issues.apache.org/jira/browse/HBASE-4718
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.5

 Attachments: 4718-v2.90, hbase-4718.0.90.patch, 
 hbase-4718.v3.includes-hbase-3316.patch, hbase-4718.v4.patch


 In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about 
 whether and how to backport HBASE-4552 to the 0.90 branch.  This is a 
 potentially compatibility breaking so several approaches hav ebeen suggested.
 1) provide patch but do not integrate
 2) integrate patch that extends and deprecates old api without removing old 
 api.  It has been argued that  clients are supposed to use 
 LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.

2011-11-14 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150061#comment-13150061
 ] 

Jonathan Hsieh commented on HBASE-4718:
---

Patch filed to apply since this was targeted to the 0.90 branch instead of the 
trunk/0.92 branch.  Attached output of selected unit tests run.


{code}
---
 T E S T S
---
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 146.315 sec
Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.407 sec

Results :

Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
{code}

 Backport HBASE-4552 to 0.90 branch.
 ---

 Key: HBASE-4718
 URL: https://issues.apache.org/jira/browse/HBASE-4718
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.5

 Attachments: 4718-v2.90, hbase-4718.0.90.patch, 
 hbase-4718.v3.includes-hbase-3316.patch, hbase-4718.v4.patch


 In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about 
 whether and how to backport HBASE-4552 to the 0.90 branch.  This is a 
 potentially compatibility breaking so several approaches hav ebeen suggested.
 1) provide patch but do not integrate
 2) integrate patch that extends and deprecates old api without removing old 
 api.  It has been argued that  clients are supposed to use 
 LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)

2011-11-14 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150069#comment-13150069
 ] 

Lars Hofhansl commented on HBASE-4256:
--

On the other hand API 1 maps better to setStartRow, setStopRow.
+1 on everything else Stack said.

Would scanning within a row imply start and end row set? I think it should. To 
the same row?
(Or are we saying for all rows returned by the scan returns scan only within 
these column boundaries?)
If so, maybe we can combine startRow and startQualifier, ditto for stopRow and 
endQualifier.

Or we can start at RowX, QualifierA and scan to RowY, QualifierB? I.e. for all 
rows between RowX and RowY we'd return all column...


 Intra-row scanning (part deux)
 --

 Key: HBASE-4256
 URL: https://issues.apache.org/jira/browse/HBASE-4256
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Dave Revell
Priority: Critical
 Fix For: 0.94.0


 Dave Revell was asking on IRC today if there's a way to scan ranges of 
 *qualifiers* within a row. That is, to be able to specify a *start qualifier* 
 and an *end qualifier* so that the Get or Scan seeks directly to the first 
 qualifier and stops at some point which can be predeterminate by a qualifier 
 or simply a batch configuration (already exists).
 This is particularly useful for large rows with time-based qualifiers.
 Dave also mentioned that another popular database has such a feature that 
 they call column slices.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.

2011-11-14 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150078#comment-13150078
 ] 

Ted Yu commented on HBASE-4718:
---

TestLoadIncrementalHFilesSplitRecovery and TestHRegionServerBulkLoad also 
passed.
Running whole test suite.

 Backport HBASE-4552 to 0.90 branch.
 ---

 Key: HBASE-4718
 URL: https://issues.apache.org/jira/browse/HBASE-4718
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.90.5

 Attachments: 4718-v2.90, hbase-4718.0.90.patch, 
 hbase-4718.v3.includes-hbase-3316.patch, hbase-4718.v4.patch


 In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about 
 whether and how to backport HBASE-4552 to the 0.90 branch.  This is a 
 potentially compatibility breaking so several approaches hav ebeen suggested.
 1) provide patch but do not integrate
 2) integrate patch that extends and deprecates old api without removing old 
 api.  It has been argued that  clients are supposed to use 
 LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4761) Add Developer Debug Options to HBase Config


[ 
https://issues.apache.org/jira/browse/HBASE-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150082#comment-13150082
 ] 

Nicolas Spiegelberg commented on HBASE-4761:


Great.  Will have Kannan handle this commit

 Add Developer Debug Options to HBase Config
 ---

 Key: HBASE-4761
 URL: https://issues.apache.org/jira/browse/HBASE-4761
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Minor
 Attachments: HBASE-4761.patch


 Add in optional HBase configuration options that core developers will 
 commonly use: an option to enable JDWP debugging  an option to use a 
 separate logfile for GC information.  (Part of the effort to move 89-fb 
 features over to trunk)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)


[ 
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150088#comment-13150088
 ] 

stack commented on HBASE-4256:
--

I like Lars observation of how setStartQualifier maps to setStartRow; i'd say 
that would tip me in favor of api 1 (mapping setStartRow/setEndRow seems like a 
better fit than mapping time range).

 Intra-row scanning (part deux)
 --

 Key: HBASE-4256
 URL: https://issues.apache.org/jira/browse/HBASE-4256
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Dave Revell
Priority: Critical
 Fix For: 0.94.0


 Dave Revell was asking on IRC today if there's a way to scan ranges of 
 *qualifiers* within a row. That is, to be able to specify a *start qualifier* 
 and an *end qualifier* so that the Get or Scan seeks directly to the first 
 qualifier and stops at some point which can be predeterminate by a qualifier 
 or simply a batch configuration (already exists).
 This is particularly useful for large rows with time-based qualifiers.
 Dave also mentioned that another popular database has such a feature that 
 they call column slices.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)

2011-11-14 Thread Lars Hofhansl (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150091#comment-13150091
]

Lars Hofhansl commented on HBASE-4256:
--

What about the other points? Are we seeing this as:
1. A way to scan subset of many rows (would be inefficient as we need to reseek
at the beginning of every new row)?
or
2. A scan strictly within a single row? (should enforce start/endRow set to the
same value in this case.)
or
3. More control where to start and stop a scan? (leave the scan logic mostly
the way it is, and allow a way to seek to a column of a row, and end at a
different column in another row).

#1 makes no sense (to me anyway, but maybe there's something I didn't see).
#3 is a superset of #2. As described above, in this case we could start at
RowX:QualA and scan to RowY:QualB, all rows between RowX and RowY would scan
all Quals. Can be combined easily with timerange scans.

I'd be in favor of #3. Or #2 if it turns out to be less work.

Intra-row scanning (part deux)
--

Key: HBASE-4256
URL: https://issues.apache.org/jira/browse/HBASE-4256
Project: HBase
Issue Type: New Feature
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: Dave Revell
Priority: Critical
Fix For: 0.94.0

Dave Revell was asking on IRC today if there's a way to scan ranges of
*qualifiers* within a row. That is, to be able to specify a *start qualifier*
and an *end qualifier* so that the Get or Scan seeks directly to the first
qualifier and stops at some point which can be predeterminate by a qualifier
or simply a batch configuration (already exists).
This is particularly useful for large rows with time-based qualifiers.
Dave also mentioned that another popular database has such a feature that
they call column slices.

[jira] [Updated] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest


 [ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4764:
---

Status: Open  (was: Patch Available)

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4782) Consistency Check Utility for META Schemas

2011-11-14 Thread Nicolas Spiegelberg (Created) (JIRA)

Consistency Check Utility for META Schemas
--

 Key: HBASE-4782
 URL: https://issues.apache.org/jira/browse/HBASE-4782
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0


Adding a script to check if table descriptors for each region in META for a 
given table are consistent.  The script compares the table descriptors in META 
for all regions of a particular table to the table's descriptor (which is just 
the descriptor for the first region).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest

2011-11-14 Thread Nicolas Spiegelberg (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4764:
---

Attachment: 4764_trunk.patch

same patch try again

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch, 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4782) Consistency Check Utility for META Schemas


[ 
https://issues.apache.org/jira/browse/HBASE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150101#comment-13150101
 ] 

Nicolas Spiegelberg commented on HBASE-4782:


Part of 89-fb to trunk port.  See r1181944 

 Consistency Check Utility for META Schemas
 --

 Key: HBASE-4782
 URL: https://issues.apache.org/jira/browse/HBASE-4782
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0


 Adding a script to check if table descriptors for each region in META for a 
 given table are consistent.  The script compares the table descriptors in 
 META for all regions of a particular table to the table's descriptor (which 
 is just the descriptor for the first region).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4764) naming errors for TestHLogUtils and SoftValueSortedMapTest

2011-11-14 Thread Nicolas Spiegelberg (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4764:
---

Status: Patch Available  (was: Open)

 naming errors for TestHLogUtils and SoftValueSortedMapTest
 --

 Key: HBASE-4764
 URL: https://issues.apache.org/jira/browse/HBASE-4764
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Attachments: 4764_trunk.patch, 4764_trunk.patch


 SoftValueSortedMapTest it's a test, but not a junit one, I tend to think it's 
 not called. I don't know if it's used.
 TestHLogUtils has a wrong name: it's not a test, but an helper. It confuses 
 the script looking for the tests. It would seems a better thing to rename it. 
 Is there anything special to do to keep the history attached to this file?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4782) Consistency Check Utility for META Schemas

2011-11-14 Thread Nicolas Spiegelberg (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg reassigned HBASE-4782:
--

Assignee: Madhuwanti Vaidya  (was: Nicolas Spiegelberg)

script designed by Madhu

 Consistency Check Utility for META Schemas
 --

 Key: HBASE-4782
 URL: https://issues.apache.org/jira/browse/HBASE-4782
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Madhuwanti Vaidya
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4782.patch


 Adding a script to check if table descriptors for each region in META for a 
 given table are consistent.  The script compares the table descriptors in 
 META for all regions of a particular table to the table's descriptor (which 
 is just the descriptor for the first region).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4782) Consistency Check Utility for META Schemas

2011-11-14 Thread Nicolas Spiegelberg (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-4782:
---

Attachment: HBASE-4782.patch

 Consistency Check Utility for META Schemas
 --

 Key: HBASE-4782
 URL: https://issues.apache.org/jira/browse/HBASE-4782
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4782.patch


 Adding a script to check if table descriptors for each region in META for a 
 given table are consistent.  The script compares the table descriptors in 
 META for all regions of a particular table to the table's descriptor (which 
 is just the descriptor for the first region).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4782) Consistency Check Utility for META Schemas

2011-11-14 Thread Nicolas Spiegelberg (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-4782:
---

Status: Patch Available  (was: Open)

 Consistency Check Utility for META Schemas
 --

 Key: HBASE-4782
 URL: https://issues.apache.org/jira/browse/HBASE-4782
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Madhuwanti Vaidya
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4782.patch


 Adding a script to check if table descriptors for each region in META for a 
 given table are consistent.  The script compares the table descriptors in 
 META for all regions of a particular table to the table's descriptor (which 
 is just the descriptor for the first region).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4783) Improve RowCounter to count rows in a specific key range.

2011-11-14 Thread Nicolas Spiegelberg (Created) (JIRA)

Improve RowCounter to count rows in a specific key range.
-

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Priority: Trivial


Currently RowCounter in MR package is a very simple map only job that does a 
full scan of a table. Enhance the utility to let the user specify a key range 
and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4783) Improve RowCounter to count rows in a specific key range.


[ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150106#comment-13150106
 ] 

Nicolas Spiegelberg commented on HBASE-4783:


Part of 89-fb to trunk port.  See r#.  Usage: Specify as arguments 
--range=foo,bar to the program, which would count between [foo, bar) that 
has any column.

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Priority: Trivial

 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4783) Improve RowCounter to count rows in a specific key range.

2011-11-14 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150107#comment-13150107
 ] 

Nicolas Spiegelberg commented on HBASE-4783:


This is necessary for the incremental snapshot feature that we are developing.

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Priority: Trivial

 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4782) Consistency Check Utility for META Schemas

2011-11-14 Thread Todd Lipcon (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150111#comment-13150111
 ] 

Todd Lipcon commented on HBASE-4782:


this should probably be part of hbck.. I think sprinkling ruby scripts in the 
bin/ directory just leaves us a bunch of untested cruft which usually falls out 
of date.

 Consistency Check Utility for META Schemas
 --

 Key: HBASE-4782
 URL: https://issues.apache.org/jira/browse/HBASE-4782
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Madhuwanti Vaidya
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4782.patch


 Adding a script to check if table descriptors for each region in META for a 
 given table are consistent.  The script compares the table descriptors in 
 META for all regions of a particular table to the table's descriptor (which 
 is just the descriptor for the first region).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4783) Improve RowCounter to count rows in a specific key range.

2011-11-14 Thread Nicolas Spiegelberg (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg reassigned HBASE-4783:
--

Assignee: Nicolas Spiegelberg

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4783.patch


 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work stopped] (HBASE-4783) Improve RowCounter to count rows in a specific key range.

2011-11-14 Thread Nicolas Spiegelberg (Work stopped) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-4783 stopped by Nicolas Spiegelberg.

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4783.patch


 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4783) Improve RowCounter to count rows in a specific key range.

2011-11-14 Thread Nicolas Spiegelberg (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-4783:
---

Attachment: HBASE-4783.patch

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4783.patch


 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (HBASE-4783) Improve RowCounter to count rows in a specific key range.

2011-11-14 Thread Nicolas Spiegelberg (Work started) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-4783 started by Nicolas Spiegelberg.

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4783.patch


 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4783) Improve RowCounter to count rows in a specific key range.

2011-11-14 Thread Nicolas Spiegelberg (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-4783:
---

Fix Version/s: 0.94.0
   Status: Patch Available  (was: Open)

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4783.patch


 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4784) Void return types not handled correctly for CoprocessorProtocol methods

2011-11-14 Thread Gary Helmling (Created) (JIRA)

Void return types not handled correctly for CoprocessorProtocol methods
---

 Key: HBASE-4784
 URL: https://issues.apache.org/jira/browse/HBASE-4784
 Project: HBase
  Issue Type: Bug
  Components: coprocessors
Affects Versions: 0.92.0, 0.94.0
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.92.0


If a CoprocessorProtocol derived interface defines a method with a void return 
type, the method cannot be called using HTable.coprocessorExec().  Instead 
ExecResult will throw an IOException on the client trying to do a 
Class.forName() on void.

Looking at ExecResult, it appears that the valueType field (which causes the 
error) is no longer even used, so I'd suggest we just get rid of it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion

[
https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150119#comment-13150119
]

Hadoop QA commented on HBASE-4768:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503521/D363.4.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 37 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -163 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 51 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestHFileBlock

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/246//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/246//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/246//console

This message is automatically generated.

Per-(table, columnFamily) metrics with configurable table name inclusion

[jira] [Commented] (HBASE-4783) Improve RowCounter to count rows in a specific key range.


[ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150123#comment-13150123
 ] 

Ted Yu commented on HBASE-4783:
---

Minor comment,
I think the order of the following two statements should be exchanged:
{code}
+for (int i = 1; i  args.length; i++) {
+  final String rangeSwitch = --range=;
{code}

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4783.patch


 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4783) Improve RowCounter to count rows in a specific key range.


[ 
https://issues.apache.org/jira/browse/HBASE-4783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150122#comment-13150122
 ] 

Hadoop QA commented on HBASE-4783:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12503697/HBASE-4783.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/250//console

This message is automatically generated.

 Improve RowCounter to count rows in a specific key range.
 -

 Key: HBASE-4783
 URL: https://issues.apache.org/jira/browse/HBASE-4783
 Project: HBase
  Issue Type: Improvement
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
Priority: Trivial
 Fix For: 0.94.0

 Attachments: HBASE-4783.patch


 Currently RowCounter in MR package is a very simple map only job that does a 
 full scan of a table. Enhance the utility to let the user specify a key range 
 and count the number of rows in this range. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.