[jira] [Commented] (HBASE-4468) Wrong resource name in an error massage: webapps instead of hbase-webapps

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113906#comment-13113906
 ] 

Hudson commented on HBASE-4468:
---

Integrated in HBase-TRUNK #2247 (See 
[https://builds.apache.org/job/HBase-TRUNK/2247/])
HBASE-4468 Wrong resource name in an error massage: webapps instead of 
hbase-webapps

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/InfoServer.java


> Wrong resource name in an error massage: webapps instead of hbase-webapps
> -
>
> Key: HBASE-4468
> URL: https://issues.apache.org/jira/browse/HBASE-4468
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Trivial
> Fix For: 0.92.0
>
> Attachments: 20110923_4468_InfoServer.patch
>
>
> org.apache.hadoop.hbase.util.InfoServer loads a resource in 'hbase-webapps' 
> but displays a message about 'webapps' when it does not find it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113904#comment-13113904
 ] 

Hudson commented on HBASE-4472:
---

Integrated in HBase-TRUNK #2247 (See 
[https://builds.apache.org/job/HBase-TRUNK/2247/])
HBASE-4472  MiniHBaseCluster.shutdown() doesn't work if no active master

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java


> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch, jstack.txt
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4280) [replication] ReplicationSink can deadlock itself via handlers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113905#comment-13113905
 ] 

Hudson commented on HBASE-4280:
---

Integrated in HBase-TRUNK #2247 (See 
[https://builds.apache.org/job/HBase-TRUNK/2247/])
HBASE-4280  [replication] ReplicationSink can deadlock itself via handlers

jdcryans : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


> [replication] ReplicationSink can deadlock itself via handlers
> --
>
> Key: HBASE-4280
> URL: https://issues.apache.org/jira/browse/HBASE-4280
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 0.90.5
>
> Attachments: HBASE-4280-0.90.patch
>
>
> I've experienced this problem a few times, ReplicationSink calls are received 
> through the normal handlers and potentially can call itself which, in certain 
> situations, call fill up all the handlers. For example, 10 handlers that are 
> all replication calls are all trying to talk to the local server at the same 
> time.
> HRS.replicateLogEntries should have @QosPriority(priority=HIGH_QOS) to use 
> the other set of handlers.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113903#comment-13113903
 ] 

Hudson commented on HBASE-4434:
---

Integrated in HBase-TRUNK #2247 (See 
[https://builds.apache.org/job/HBase-TRUNK/2247/])
HBASE-4434 seek optimization: don't do eager HFile Scanner next() unless 
the next KV is needed

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java


> seek optimization: don't do eager HFile Scanner next() unless the next KV is 
> needed
> ---
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Assignee: Kannan Muthukkaruppan
> Fix For: 0.92.0
>
> Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the 
> current KV, it also does a HFileScanner level next() ahead of time even if 
> the next KV is never actually required. This inefficiency can potentially 
> result in additional disk seeks and sub-optimal use of the block cache 
> (primarily for cases where the KVs are large and each occupies an HFile block 
> of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4449) LoadIncrementalHFiles should be able to handle CFs with blooms

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113901#comment-13113901
 ] 

Hudson commented on HBASE-4449:
---

Integrated in HBase-TRUNK #2247 (See 
[https://builds.apache.org/job/HBase-TRUNK/2247/])
Added comment for clarity while reading code to review HBASE-4449

nspiegelberg : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java


> LoadIncrementalHFiles should be able to handle CFs with blooms
> --
>
> Key: HBASE-4449
> URL: https://issues.apache.org/jira/browse/HBASE-4449
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Dave Revell
>Assignee: Dave Revell
> Fix For: 0.90.5
>
> Attachments: HBASE-4449-trunk-testsonly.patch, HBASE-4449-v2.patch, 
> HBASE-4449.patch
>
>
> When LoadIncrementalHFiles loads a store file that crosses region boundaries, 
> it will split the file at the boundary to create two store files. If the 
> store file is for a column family that has a bloom filter, then a 
> "java.lang.ArithmeticException: / by zero" will be raised because 
> ByteBloomFilter() is called with maxKeys of 0.
> The included patch assumes that the number of keys in each split child will 
> be equal to the number of keys in the parent's bloom filter (instead of 0). 
> This is an overestimate, but it's safe and easy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4131) Make the Replication Service pluggable via a standard interface definition

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113902#comment-13113902
 ] 

Hudson commented on HBASE-4131:
---

Integrated in HBase-TRUNK #2247 (See 
[https://builds.apache.org/job/HBase-TRUNK/2247/])
HBASE-4131 Make the Replication Service pluggable via a standard interface 
definition; BACKED IT OUT -- WAS CAUSING TestReplication failures

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java


> Make the Replication Service pluggable via a standard interface definition
> --
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, 
> replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to 
> sync data from from one hbase cluster to another. It would be nice to make it 
> a pluggable interface so that other cross-data-center replication services 
> can be used in conjuction with HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4132) Extend the WALActionsListener API to accomodate log archival

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113895#comment-13113895
 ] 

Ted Yu edited comment on HBASE-4132 at 9/24/11 5:25 AM:


Here is the addendum.
TestLogRolling#testLogRollOnPipelineRestart passed on my MacBook.
Let me see if the addendum helps on Jenkins.

  was (Author: yuzhih...@gmail.com):
Here is the addendum.
It seems that number of times preLogRolled is called is not fixed.
  
> Extend the WALActionsListener API to accomodate log archival
> 
>
> Key: HBASE-4132
> URL: https://issues.apache.org/jira/browse/HBASE-4132
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4132.addendum, walArchive.txt, walArchive2.txt, 
> walArchive3.txt
>
>
> The WALObserver interface exposes the log roll events. It would be nice to 
> extend it to accomodate log archival events as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4132) Extend the WALActionsListener API to accomodate log archival

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4132:
--

Attachment: 4132.addendum

Here is the addendum.
It seems that number of times preLogRolled is called is not fixed.

> Extend the WALActionsListener API to accomodate log archival
> 
>
> Key: HBASE-4132
> URL: https://issues.apache.org/jira/browse/HBASE-4132
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4132.addendum, walArchive.txt, walArchive2.txt, 
> walArchive3.txt
>
>
> The WALObserver interface exposes the log roll events. It would be nice to 
> extend it to accomodate log archival events as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4478) Improve AssignmentManager.handleRegion so that it can process certain ZK state in the case of RS offline

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113894#comment-13113894
 ] 

ramkrishna.s.vasudevan commented on HBASE-4478:
---

Even i had faced some problems due to this.  +1 for this JIRA

> Improve AssignmentManager.handleRegion so that it can process certain ZK 
> state in the case of RS offline
> 
>
> Key: HBASE-4478
> URL: https://issues.apache.org/jira/browse/HBASE-4478
> Project: HBase
>  Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
>
> Currently AssignmentManager.handleRegion skips processing of ZK event change 
> if the RS is offline. It relies on TimeoutMonitor and ServerShutdownHandler 
> to process RIT.
>   // Verify this is a known server
>   if (!serverManager.isServerOnline(sn) &&
>   !this.master.getServerName().equals(sn)) {
> LOG.warn("Attempted to handle region transition for server but " +
>   "server is not online: " + Bytes.toString(data.getRegionName()));
> return;
>   }
> For certain states like OPENED, OPENING, FAILED_OPEN, CLOSED, it can continue 
> the progressing even if the RS is offline.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4132) Extend the WALActionsListener API to accomodate log archival

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113893#comment-13113893
 ] 

Ted Yu commented on HBASE-4132:
---

Running TestLogRolling#testLogRollOnPipelineRestart in debugger, I got:
{code}
java.lang.AssertionError: preLogRolledCalled has size of 3
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:311)
{code}

The first two calls were:
{code}
TestLogRolling$1.preLogRoll(Path, Path) line: 245
HLog.rollWriter(boolean) line: 564  
LogRoller.run() line: 93
{code}
The third one was:
{code}
TestLogRolling$1.preLogRoll(Path, Path) line: 245   
HLog.rollWriter(boolean) line: 564  
TestLogRolling.testLogRollOnPipelineRestart() line: 310 
{code}

> Extend the WALActionsListener API to accomodate log archival
> 
>
> Key: HBASE-4132
> URL: https://issues.apache.org/jira/browse/HBASE-4132
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: walArchive.txt, walArchive2.txt, walArchive3.txt
>
>
> The WALObserver interface exposes the log roll events. It would be nice to 
> extend it to accomodate log archival events as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4478) Improve AssignmentManager.handleRegion so that it can process certain ZK state in the case of RS offline

[Ming Ma (JIRA)]

Improve AssignmentManager.handleRegion so that it can process certain ZK state 
in the case of RS offline


 Key: HBASE-4478
 URL: https://issues.apache.org/jira/browse/HBASE-4478
 Project: HBase
  Issue Type: Bug
Reporter: Ming Ma
Assignee: Ming Ma


Currently AssignmentManager.handleRegion skips processing of ZK event change if 
the RS is offline. It relies on TimeoutMonitor and ServerShutdownHandler to 
process RIT.

  // Verify this is a known server
  if (!serverManager.isServerOnline(sn) &&
  !this.master.getServerName().equals(sn)) {
LOG.warn("Attempted to handle region transition for server but " +
  "server is not online: " + Bytes.toString(data.getRegionName()));
return;
  }

For certain states like OPENED, OPENING, FAILED_OPEN, CLOSED, it can continue 
the progressing even if the RS is offline.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4014) Coprocessors: Flag the presence of coprocessors in logged exceptions

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113888#comment-13113888
 ] 

Ted Yu commented on HBASE-4014:
---

>From Eugene:
Every test passed except TestLogRolling:
http://pastebin.com/wcsRnnhW

TestLogRolling.testLogRollOnPipelineRestart failed in TRUNK build 2246.

So there was no regression.

+1 on latest patch.

> Coprocessors: Flag the presence of coprocessors in logged exceptions
> 
>
> Key: HBASE-4014
> URL: https://issues.apache.org/jira/browse/HBASE-4014
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Reporter: Andrew Purtell
>Assignee: Eugene Koontz
> Fix For: 0.92.0
>
> Attachments: HBASE-4014.patch, HBASE-4014.patch, HBASE-4014.patch, 
> HBASE-4014.patch, HBASE-4014.patch
>
>
> For some initial triage of bug reports for core versus for deployments with 
> loaded coprocessors, we need something like the Linux kernel's taint flag, 
> and list of linked in modules that show up in the output of every OOPS, to 
> appear above or below exceptions that appear in the logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4455) Rolling restart RSs scenario, -ROOT-, .META. regions are lost in AssignmentManager

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113887#comment-13113887
 ] 

jirapos...@reviews.apache.org commented on HBASE-4455:
--



bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > Great stuff!  I have some questions throughout but seems like this will 
make everything more resilient to root/meta servers failing.  Is the general 
approach to always verify / always check rather than relying on cached 
locations or values?
bq.  > 
bq.  > Have you thought about any ways that we could add some better unit tests 
around this stuff?  There's a TestRollingRestart that is obviously not good 
enough :)

The repro of such bug depends on timing of events. Initially I thought perhaps 
we can inject timeout into various places in the code. At this point, it is 
easier to just do the testing on a small cluster and eventually the bug will 
appear. Perhaps something we can work on later.


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java,
 line 309
bq.  > 
bq.  >
bq.  > why log the cached META server here?  didn't we just verify that it 
was not valid?

Fixed.


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java,
 line 310
bq.  > 
bq.  >
bq.  > why log the cached meta location here?  it might be confusing since 
it doesn't log that we just found this meta location was invalid

Fixed.


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java,
 line 2532
bq.  > 
bq.  >
bq.  > add another * here, so: /**
bq.  > 
bq.  > that ensure this gets picked up as javadoc

Fixed.


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java,
 line 2558
bq.  > 
bq.  >
bq.  > this looks like a random debug statement, what does matchZK, sn: 
server mean?

Fixed.


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java,
 lines 62-69
bq.  > 
bq.  >
bq.  > why this change?  should this be rolled into the ZKNodeTracker 
rather than overriding the getData() behavior?

Fixed.


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java,
 line 90
bq.  > 
bq.  >
bq.  > it seems like you're covering up for bugs in the underlying 
ZKNodeTracker... can we fix that instead?  or if it's a matter of returning a 
cached value or not, can we just add a boolean flag for refresh/nocache?

Fixed.


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java,
 line 294
bq.  > 
bq.  >
bq.  > so we always verify the connection now?

Before the fix, all callers set it to "true". So there is no behavior change.


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java,
 line 368
bq.  > 
bq.  >
bq.  > why do we have two hard-coded timeouts in this area of code? :)
bq.  > 
bq.  > this code seems to always sleep 500ms at a time unless you set 
timeout=0 and then it loops every 50ms?  that doesn't seem right... i could set 
timeout to 100ms and it would sleep 500ms.  sleeping 50ms every time would be 
better but there's probably a solution with less overhead (doing remote read 
queries every 50ms in a loop)
bq.  > 
bq.  > could we just notifyAll() on metaAvailable whenever we relocate root?

Choose the min(500ms, timeout) at this point, given we might do more code 
cleanup around RootRegionTracker later. 


bq.  On 2011-09-23 08:17:29, Jonathan Gray wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java,
 line 215
bq

[jira] [Commented] (HBASE-4455) Rolling restart RSs scenario, -ROOT-, .META. regions are lost in AssignmentManager

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113886#comment-13113886
 ] 

jirapos...@reviews.apache.org commented on HBASE-4455:
--



bq.  On 2011-09-23 05:44:11, Michael Stack wrote:
bq.  > Really good stuff.  I just need a bit of it explained to me below 
because I'm being a little slow.  I think you've also nailed HBASE-3809 with 
this patch Ming (What you think?)  Good stuff.

That sounds right. HBASE-4245 could be the same as well.


bq.  On 2011-09-23 05:44:11, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaNodeTracker.java,
 line 62
bq.  > 
bq.  >
bq.  > Whats this?

Fixed.


bq.  On 2011-09-23 05:44:11, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java,
 line 299
bq.  > 
bq.  >
bq.  > Why this change? If we did not ask to refresh, why not return what 
we found?

No callers call the function with refresh==false, took it out for now.


bq.  On 2011-09-23 05:44:11, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java,
 line 222
bq.  > 
bq.  >
bq.  > Why make this public?

Fixed.


bq.  On 2011-09-23 05:44:11, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java,
 line 535
bq.  > 
bq.  >
bq.  > nit: No need of the parens

Fixed.


bq.  On 2011-09-23 05:44:11, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java,
 line 215
bq.  > 
bq.  >
bq.  > Oh, here I see the don't split logs flag.
bq.  > 
bq.  > Funny.
bq.  > 
bq.  > We used to do something like this in the old days.
bq.  > 
bq.  > So, we come back in again, we don't split logs, but we are still a 
server that was carrying root or meta -- we reassign again?  I don't get how it 
works here.

If the server carries -ROOT- or .META, it will first do the log splitting and 
then submit another ServerShutDownHandler request without 
shouldSplitHLog==false.


- Ming


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2007/#review2035
---


On 2011-09-24 01:50:02, Ming Ma wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2007/
bq.  ---
bq.  
bq.  (Updated 2011-09-24 01:50:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  1. Add more logging.
bq.  2. Clean up CatalogTracker. waitForMeta waits for "timeout" value. When 
waitForMetaServerConnectionDefault is called by MetaNodeTracker, the timeout 
value is large. So it doesn't retry in case .ROOT. is updated; add the proper 
implementation for CatalogTracker.verifyMetaRegionLocation
bq.  4. Check for the latest -ROOT- and .META. region location during the 
handling of server shutdown.
bq.  5. Right after assigning the -ROOT- or .META. in ServerShutdownHandler, 
don't block and wait for .META. availability. Resubmit another 
ServerShutdownHandler for regular regions.
bq.  
bq.  
bq.  This addresses bug HBASE-4455.
bq.  https://issues.apache.org/jira/browse/HBASE-4455
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hb

[jira] [Commented] (HBASE-4455) Rolling restart RSs scenario, -ROOT-, .META. regions are lost in AssignmentManager

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113884#comment-13113884
 ] 

jirapos...@reviews.apache.org commented on HBASE-4455:
--



bq.  On 2011-09-23 03:02:39, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java,
 line 2555
bq.  > 
bq.  >
bq.  > addressFromZK != null can be omitted - it is the condition of if 
block.

Fixed.


- Ming


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2007/#review2033
---


On 2011-09-24 01:50:02, Ming Ma wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2007/
bq.  ---
bq.  
bq.  (Updated 2011-09-24 01:50:02)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  1. Add more logging.
bq.  2. Clean up CatalogTracker. waitForMeta waits for "timeout" value. When 
waitForMetaServerConnectionDefault is called by MetaNodeTracker, the timeout 
value is large. So it doesn't retry in case .ROOT. is updated; add the proper 
implementation for CatalogTracker.verifyMetaRegionLocation
bq.  4. Check for the latest -ROOT- and .META. region location during the 
handling of server shutdown.
bq.  5. Right after assigning the -ROOT- or .META. in ServerShutdownHandler, 
don't block and wait for .META. availability. Resubmit another 
ServerShutdownHandler for regular regions.
bq.  
bq.  
bq.  This addresses bug HBASE-4455.
bq.  https://issues.apache.org/jira/browse/HBASE-4455
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java
 1172205 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
 1172205 
bq.  
bq.  Diff: https://reviews.apache.org/r/2007/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Keep Master up all the time, do rolling restart of RSs like this - stop 
RS1, wait for 2 seconds, stop RS2, start RS1, wait for 2 seconds, stop RS3, 
start RS2, wait for 2 seconds, etc. The program can run for couple hours until 
it stops. -ROOT- and .META. are available during that time.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Ming
bq.  
bq.



> Rolling restart RSs scenario, -ROOT-, .META. regions are lost in 
> AssignmentManager
> --
>
> Key: HBASE-4455
> URL: https://issues.apache.org/jira/browse/HBASE-4455
> Project: HBase
>  Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 0.92.0
>
>
> Keep Master up all the time, 

[jira] [Commented] (HBASE-4449) LoadIncrementalHFiles should be able to handle CFs with blooms

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113882#comment-13113882
 ] 

Ted Yu commented on HBASE-4449:
---

See my comment @ 20/Sep/11 21:54

> LoadIncrementalHFiles should be able to handle CFs with blooms
> --
>
> Key: HBASE-4449
> URL: https://issues.apache.org/jira/browse/HBASE-4449
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Dave Revell
>Assignee: Dave Revell
> Fix For: 0.90.5
>
> Attachments: HBASE-4449-trunk-testsonly.patch, HBASE-4449-v2.patch, 
> HBASE-4449.patch
>
>
> When LoadIncrementalHFiles loads a store file that crosses region boundaries, 
> it will split the file at the boundary to create two store files. If the 
> store file is for a column family that has a bloom filter, then a 
> "java.lang.ArithmeticException: / by zero" will be raised because 
> ByteBloomFilter() is called with maxKeys of 0.
> The included patch assumes that the number of keys in each split child will 
> be equal to the number of keys in the parent's bloom filter (instead of 0). 
> This is an overestimate, but it's safe and easy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113877#comment-13113877
 ] 

Hudson commented on HBASE-4434:
---

Integrated in HBase-0.92 #18 (See 
[https://builds.apache.org/job/HBase-0.92/18/])
HBASE-4434: Don't do HFile Scanner next() unless the next KV is needed

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java


> seek optimization: don't do eager HFile Scanner next() unless the next KV is 
> needed
> ---
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Assignee: Kannan Muthukkaruppan
> Fix For: 0.92.0
>
> Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the 
> current KV, it also does a HFileScanner level next() ahead of time even if 
> the next KV is never actually required. This inefficiency can potentially 
> result in additional disk seeks and sub-optimal use of the block cache 
> (primarily for cases where the KVs are large and each occupies an HFile block 
> of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4280) [replication] ReplicationSink can deadlock itself via handlers

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113879#comment-13113879
 ] 

Hudson commented on HBASE-4280:
---

Integrated in HBase-0.92 #18 (See 
[https://builds.apache.org/job/HBase-0.92/18/])
HBASE-4280  [replication] ReplicationSink can deadlock itself via handlers

jdcryans : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


> [replication] ReplicationSink can deadlock itself via handlers
> --
>
> Key: HBASE-4280
> URL: https://issues.apache.org/jira/browse/HBASE-4280
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 0.90.5
>
> Attachments: HBASE-4280-0.90.patch
>
>
> I've experienced this problem a few times, ReplicationSink calls are received 
> through the normal handlers and potentially can call itself which, in certain 
> situations, call fill up all the handlers. For example, 10 handlers that are 
> all replication calls are all trying to talk to the local server at the same 
> time.
> HRS.replicateLogEntries should have @QosPriority(priority=HIGH_QOS) to use 
> the other set of handlers.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4450) test for number of blocks read: to serve as baseline for expected blocks read and for catching regressions

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113876#comment-13113876
 ] 

Hudson commented on HBASE-4450:
---

Integrated in HBase-0.92 #18 (See 
[https://builds.apache.org/job/HBase-0.92/18/])
HBASE-4450 test for number of blocks read: to serve as baseline for 
expected blocks read and for catching regressions

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java


> test for number of blocks read: to serve as baseline for expected blocks read 
> and for catching regressions
> --
>
> Key: HBASE-4450
> URL: https://issues.apache.org/jira/browse/HBASE-4450
> Project: HBase
>  Issue Type: Test
>Reporter: Kannan Muthukkaruppan
>Assignee: Kannan Muthukkaruppan
> Attachments: HBASE-4450.txt
>
>
> Add a simple test for number of blocks read. The tests intent is to serve as 
> baseline for expected blocks read and for catching regressions. As 
> optimizations for HBase-4433 or Hbase-4434 are committed, the test would need 
> to be updated to adjust the counts for expected blocks read in various cases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113878#comment-13113878
 ] 

Hudson commented on HBASE-4472:
---

Integrated in HBase-0.92 #18 (See 
[https://builds.apache.org/job/HBase-0.92/18/])
HBASE-4472  MiniHBaseCluster.shutdown() doesn't work if no active master

tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java


> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch, jstack.txt
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4468) Wrong resource name in an error massage: webapps instead of hbase-webapps

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113880#comment-13113880
 ] 

Hudson commented on HBASE-4468:
---

Integrated in HBase-0.92 #18 (See 
[https://builds.apache.org/job/HBase-0.92/18/])
HBASE-4468 Wrong resource name in an error massage: webapps instead of 
hbase-webapps

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/InfoServer.java


> Wrong resource name in an error massage: webapps instead of hbase-webapps
> -
>
> Key: HBASE-4468
> URL: https://issues.apache.org/jira/browse/HBASE-4468
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Trivial
> Fix For: 0.92.0
>
> Attachments: 20110923_4468_InfoServer.patch
>
>
> org.apache.hadoop.hbase.util.InfoServer loads a resource in 'hbase-webapps' 
> but displays a message about 'webapps' when it does not find it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4449) LoadIncrementalHFiles should be able to handle CFs with blooms

[Jonathan Gray (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113872#comment-13113872
 ] 

Jonathan Gray commented on HBASE-4449:
--

It looks like the test change was committed but not the change to 
LoadIncrementalHFiles?

> LoadIncrementalHFiles should be able to handle CFs with blooms
> --
>
> Key: HBASE-4449
> URL: https://issues.apache.org/jira/browse/HBASE-4449
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Dave Revell
>Assignee: Dave Revell
> Fix For: 0.90.5
>
> Attachments: HBASE-4449-trunk-testsonly.patch, HBASE-4449-v2.patch, 
> HBASE-4449.patch
>
>
> When LoadIncrementalHFiles loads a store file that crosses region boundaries, 
> it will split the file at the boundary to create two store files. If the 
> store file is for a column family that has a bloom filter, then a 
> "java.lang.ArithmeticException: / by zero" will be raised because 
> ByteBloomFilter() is called with maxKeys of 0.
> The included patch assumes that the number of keys in each split child will 
> be equal to the number of keys in the parent's bloom filter (instead of 0). 
> This is an overestimate, but it's safe and easy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4449) LoadIncrementalHFiles should be able to handle CFs with blooms

[Jonathan Gray (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113870#comment-13113870
 ] 

Jonathan Gray commented on HBASE-4449:
--

Is this done now?

> LoadIncrementalHFiles should be able to handle CFs with blooms
> --
>
> Key: HBASE-4449
> URL: https://issues.apache.org/jira/browse/HBASE-4449
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Dave Revell
>Assignee: Dave Revell
> Fix For: 0.90.5
>
> Attachments: HBASE-4449-trunk-testsonly.patch, HBASE-4449-v2.patch, 
> HBASE-4449.patch
>
>
> When LoadIncrementalHFiles loads a store file that crosses region boundaries, 
> it will split the file at the boundary to create two store files. If the 
> store file is for a column family that has a bloom filter, then a 
> "java.lang.ArithmeticException: / by zero" will be raised because 
> ByteBloomFilter() is called with maxKeys of 0.
> The included patch assumes that the number of keys in each split child will 
> be equal to the number of keys in the parent's bloom filter (instead of 0). 
> This is an overestimate, but it's safe and easy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4455) Rolling restart RSs scenario, -ROOT-, .META. regions are lost in AssignmentManager

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113869#comment-13113869
 ] 

jirapos...@reviews.apache.org commented on HBASE-4455:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2007/
---

(Updated 2011-09-24 01:50:02.986731)


Review request for hbase.


Changes
---

Thanks folks for the review. I have fixes most of them. The below comments 
explain why the rests aren't fixed and some general questions asked.

1. I wondered if RootRegionTracker and MetaNodeTracker are really needed. 
Instead of waiting for ZK notification, checking with ZK directly should be ok. 
This won't have much impact on performance given most of the time where there 
isn't much regions movement. For now, we can keep RootRegionTracker and 
MetaNodeTracker. We can open a separate jira if that is needed.
2. Took out "refresh" parameter in CatalogTracker.getMetaServerConnection. All 
the callers of this function call the function will "true". So at this point, I 
just took it out.
3. Per Jonathan suggestions, modified ZookeeperNodeTracker to add refresh flag 
to getData, blockUntilAvailable.
4. About the question from Stack, yes, it looks like the same as 3809. it could 
be the same as 4245.
5. I put a more detailed description in ServerShutdownHandler.java about why we 
need to resubmit another ServerShutdownHandler request back to thread pool if 
the server carries -ROOT- or .META.
6. Regarding Jonathan's suggestion about relying on notifyAll() from -ROOT- 
inside waitForMeta, I just fixed the timeout value issue instead, in case later 
we decide RootRegionTracker isn't that useful.
7. Regarding Stack's HLogSplitting question, if the shutdown server carries 
-ROOT- or .META., it will first do HLogSplitting, and then resubmit another 
ServerShutdownHandler request for the same server which doesn't do 
HLogSplitting.


Summary
---

1. Add more logging.
2. Clean up CatalogTracker. waitForMeta waits for "timeout" value. When 
waitForMetaServerConnectionDefault is called by MetaNodeTracker, the timeout 
value is large. So it doesn't retry in case .ROOT. is updated; add the proper 
implementation for CatalogTracker.verifyMetaRegionLocation
4. Check for the latest -ROOT- and .META. region location during the handling 
of server shutdown.
5. Right after assigning the -ROOT- or .META. in ServerShutdownHandler, don't 
block and wait for .META. availability. Resubmit another ServerShutdownHandler 
for regular regions.


This addresses bug HBASE-4455.
https://issues.apache.org/jira/browse/HBASE-4455


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java
 1172205 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
 1172205 

Diff: https://reviews.apache.org/r/2007/diff


Testing
---

Keep Master up all the time, do rolling restart of RSs like this - stop RS1, 
wait for 2 seconds, stop RS2, sta

[jira] [Commented] (HBASE-4132) Extend the WALActionsListener API to accomodate log archival

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113866#comment-13113866
 ] 

Lars Hofhansl commented on HBASE-4132:
--

Any plan to also add this to org.apache.hadoop.hbase.coprocessor.WALObserver?
I'm happy to prepare a patch for that, maybe in a different jira.


> Extend the WALActionsListener API to accomodate log archival
> 
>
> Key: HBASE-4132
> URL: https://issues.apache.org/jira/browse/HBASE-4132
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: walArchive.txt, walArchive2.txt, walArchive3.txt
>
>
> The WALObserver interface exposes the log roll events. It would be nice to 
> extend it to accomodate log archival events as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4439) Move ClientScanner out of HTable

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4439:
-

Attachment: 4439-v1.txt

ClientScanner also copies the passed Scan now. Not pretty, but since this is 
going to do at least one roundtrip to HBase it should not be a problem.

iterator() and next(int) moved to an abstract helper class (that one makes not 
assumption about transport, RPC, etc).

> Move ClientScanner out of HTable
> 
>
> Key: HBASE-4439
> URL: https://issues.apache.org/jira/browse/HBASE-4439
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4439-v1.txt, 4439.txt
>
>
> See HBASE-1935 for motivation.
> ClientScanner should be able to exist outside of HTable.
> While we're at it, we can also add an abstract client scanner to easy 
> development of new client side scanners (such as parallel scanners, or per 
> region scanners).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it

[Roman Shaposhnik (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113859#comment-13113859
 ] 

Roman Shaposhnik commented on HBASE-4209:
-

@stack

> At least log the exception.
> What is the exception we are suppressing?  Should we at least log it here too?

Basically, the only root cause of possible exceptions here is the code in 
suppressHdfsShutdownHook() (which is quite fragile and overly aggressive in
throwing exceptions if things don't look quite right). Now, if you look at 
how this code is executed in a non-standalone case -- HBase does bail on 
exceptions 
thrown from there. We can adopt the same approach in a standalone and unit 
testing
case, but I wasn't sure it was the right thing to do. After all, NOT bailing out
(and lets say simply logging these things) will be no worse than what we 
currently
have (not calling shutdown hooks at all). On the other hand -- if we do bail out
aggressively we get a better chance of catching incompatibilities between 
suppressHdfsShutdownHook() logic and future hadoop releases during unit tests.

So perhaps -- bailing out would make the most sense of all.

> The HBase hbase-daemon.sh SIGKILLs master when stopping it
> --
>
> Key: HBASE-4209
> URL: https://issues.apache.org/jira/browse/HBASE-4209
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Attachments: HBASE-4209.patch.txt
>
>
> There's a bit of code in hbase-daemon.sh that makes HBase master being 
> SIGKILLed when stopping it rather than trying SIGTERM (like it does for other 
> daemons). When HBase is executed in a standalone mode (and the only daemon 
> you need to run is master) that causes newly created tables to go missing as 
> unflushed data is thrown out. If there was not a good reason to kill master 
> with SIGKILL perhaps we can take that special case out and rely on SIGTERM.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4439) Move ClientScanner out of HTable

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113858#comment-13113858
 ] 

Lars Hofhansl commented on HBASE-4439:
--

Hmm... The new ClientScanner still modifies the passed Scan, so somebody using 
ClientScanner directly would still get Scan modified. In order to avoid this, 
Scan would have to be copied twice (since HTable also needs to copy it, because 
it sets its caching on it, at least until we can remove {get|set}ScannerCaching 
from HTable).


> Move ClientScanner out of HTable
> 
>
> Key: HBASE-4439
> URL: https://issues.apache.org/jira/browse/HBASE-4439
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: 4439.txt
>
>
> See HBASE-1935 for motivation.
> ClientScanner should be able to exist outside of HTable.
> While we're at it, we can also add an abstract client scanner to easy 
> development of new client side scanners (such as parallel scanners, or per 
> region scanners).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4477) Ability for an application to store metadata into the transaction log

[dhruba borthakur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dhruba borthakur updated HBASE-4477:


Attachment: hlogMetadata1.txt

The initial design I am considering here:

1. The Put call already has a map of arbitrary attributes (keys/values). The 
application will set a special key named "_hbaseWalMetadata". The value of this 
attribute is what needs to be stored in the HLog. From this perspective, the 
format of the Put call does not need any change.

2. The regionserver will store a metakv in the same Waledit record as the 
transaction. This kv will have  column family name as ""METAFAMILY" and column 
name as "METACOL".

3. Code already exists to ignore kvs with column family as ""METAFAMILY" from 
the log splitting process. The reason is because we already write such records 
to the HLog (see completeCacheFlush)


> Ability for an application to store metadata into the transaction log
> -
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
>  Issue Type: Improvement
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each 
> transaction in its transaction logs. This JIRA is to have a similar feature 
> request for HBASE.
> The use case is as follows: An application on one data center A stores a blob 
> of data along with each transaction. A replication software picks up these 
> blobs from the transaction logs in A and hands it to another instance of the 
> same application running on a remote data center B. The application in B is 
> responsible for applying this to the remote Hbase cluster (and also handle 
> conflict resolution if any).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4344) Persist memstoreTS to disk

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113854#comment-13113854
 ] 

Lars Hofhansl commented on HBASE-4344:
--

+1
This is an important fix.
Nice team work on this one :)

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v2.txt, 4344-v4.txt, 
> 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4131) Make the Replication Service pluggable via a standard interface definition

[dhruba borthakur (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113850#comment-13113850
 ] 

dhruba borthakur commented on HBASE-4131:
-

Stack: thanks for taking the time to review this patch. I had not yet run the 
full test suite and was expecting the submitPatch request to run it (similar to 
Hadoop). But extremely sorry that it got committed much earlier.

I am working on this one now and will verify and post what I find.

> Make the Replication Service pluggable via a standard interface definition
> --
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, 
> replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to 
> sync data from from one hbase cluster to another. It would be nice to make it 
> a pluggable interface so that other cross-data-center replication services 
> can be used in conjuction with HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4477) Ability for an application to store metadata into the transaction log

[dhruba borthakur (JIRA)]

Ability for an application to store metadata into the transaction log
-

 Key: HBASE-4477
 URL: https://issues.apache.org/jira/browse/HBASE-4477
 Project: HBase
  Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur


mySQL allows an application to store an arbitrary blob along with each 
transaction in its transaction logs. This JIRA is to have a similar feature 
request for HBASE.

The use case is as follows: An application on one data center A stores a blob 
of data along with each transaction. A replication software picks up these 
blobs from the transaction logs in A and hands it to another instance of the 
same application running on a remote data center B. The application in B is 
responsible for applying this to the remote Hbase cluster (and also handle 
conflict resolution if any).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4463) Run more aggressive compactions during off peak hours

[Karthik Ranganathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113847#comment-13113847
 ] 

Karthik Ranganathan commented on HBASE-4463:


@Stack - we can find the exact amount of data we are writing to the dfs (only 
hfile blocks will contribute to this during compactions). So adding a threshold 
like this is not too hard... but there could be disk iops pressure (instead of 
network bandwidth) and detecting that would be hard. So we would still need to 
set off-peak time.

I was trying to come up with a more generic solution but that involves setting 
up a feedback loop inside the regionserver - keep track of max, min and average 
latencies over the last k days (would have to store this in META or some other 
location as it needs to persist beyond restarts). Need to remove any spikes in 
the values. When we run an aggressive compaction, we need to make sure the 
latencies are still acceptable, otherwise dont run aggressive compactions. This 
is much harder to get right though.

> Run more aggressive compactions during off peak hours
> -
>
> Key: HBASE-4463
> URL: https://issues.apache.org/jira/browse/HBASE-4463
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Karthik Ranganathan
>Assignee: Karthik Ranganathan
>
> The number of iops on the disk and the top of the rack bandwidth utilization 
> at off peak hours is much lower than at peak hours depending on the 
> application usage pattern. We can utilize this knowledge to improve the 
> performance of the HBase cluster by increasing the compact selection ratio to 
> a much larger value during off-peak hours than otherwise - increasing 
> hbase.hstore.compaction.ratio (1.2 default) to 
> hbase.hstore.compaction.ratio.offpeak (5 default). This will help reduce the 
> average number of files per store.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4344) Persist memstoreTS to disk

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113846#comment-13113846
 ] 

Ted Yu commented on HBASE-4344:
---

Based on a stable TRUNK tree, all tests passed, including TestAcidGuarantees.
More testing is planned before committing, possibly on Monday.

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v2.txt, 4344-v4.txt, 
> 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4476) Compactions must fail if column tracker gets columns out of order

[Mikhail Bautin (JIRA)]

Compactions must fail if column tracker gets columns out of order
-

 Key: HBASE-4476
 URL: https://issues.apache.org/jira/browse/HBASE-4476
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
 Fix For: 0.92.0, 0.94.0


We found this in ScanWildcardColumnTracker:

// new col < oldcol
// if (cmp < 0) {
// WARNING: This means that very likely an edit for some other family
// was incorrectly stored into the store for this one. Continue, but
// complain.
LOG.error("ScanWildcardColumnTracker.checkColumn ran " +
"into a column actually smaller than the previous column: " +

This went under the radar in our dark launch cluster when a column family name 
was misspelled first, but then was "renamed" by renaming directories in the 
HBase storage directory tree. We ended up with inconsistent data, but 
compactions still succeeded most of the time, likely discarding part of input 
data.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113839#comment-13113839
 ] 

Jonathan Hsieh commented on HBASE-4377:
---

I have a very *hacky* version that I've successfully recently used to rebuild a 
.META. table with over 10k regions. It can be found here:

https://github.com/jmhsieh/hbase/tree/hbase-4377

I've also hacked the hack to backport it onto an 0.90.x branch.

To run it build hbase and then use the following command line

{code}
bin/hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -base 
~/pathToHbase/hbase -details
{code}

The program will fail telling the user about any problems it encounters. It 
only succeed if all the info gathered from .regioninfo's  is clean after going 
through the regionsplit calculator.  

This code will take some time to clean up.  

I would like to do some refactoring of the current hbck and create a 
o.a.h.hbase.util.hbck or o.a.h.hbase.hbck package.  Any preferences or concerns 
there?

> [hbck] Offline rebuild .META. from fs data only.
> 
>
> Key: HBASE-4377
> URL: https://issues.apache.org/jira/browse/HBASE-4377
> Project: HBase
>  Issue Type: New Feature
>Reporter: Jonathan Hsieh
>
> In a worst case situation, it may be helpful to have an offline .META. 
> rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
> from scratch.  Users could move bad regions out until there is a clean 
> rebuild.  
> It would likely fill in region split holes.  Follow on work could given 
> options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4475) When running an embedded ThriftServer, use User.runAs() to allow it to run as a separate principal from the embedding region server

[Gary Helmling (JIRA)]

When running an embedded ThriftServer, use User.runAs() to allow it to run as a 
separate principal from the embedding region server
---

 Key: HBASE-4475
 URL: https://issues.apache.org/jira/browse/HBASE-4475
 Project: HBase
  Issue Type: Improvement
  Components: security, thrift
Reporter: Gary Helmling


As discussed over in HBASE-4460, the current approach to ThriftServer 
authentication (provided in HBASE-4099) will not work in an embedded context, 
since the region server will already does a login for the process.

We could make the embedded thrift server still run as a separate user, though, 
by doing something like the following:

* add a {{User.loginAndReturnUser()}} variant that delegates to 
{{UserGroupInformation.loginUserFromKeytabAndReturnUGI()}}, then returns a 
wrapping {{User}} instance
* call this method on startup for the embedded thrift server to get the thrift 
user instance
* use {{User.runAs()}} to execute the body of {{HRegionThriftServer.run()}} as 
the logged in thrift user


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4474) Use variable length format to store the memstoreTS

[Ted Yu (JIRA)]

Use variable length format to store the memstoreTS
--

 Key: HBASE-4474
 URL: https://issues.apache.org/jira/browse/HBASE-4474
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
 Fix For: 0.92.0


HBASE-4344 introduced memstoreTS for KeyValues.

The following suggestion was from Kannan:
We should consider using variable length format to store the memstoreTS on 
disk. Also, at the start of the flush, we can probably prune most of these 
timestamps to 0 since only the ones that are higher than the current read point 
for all active scanners need to be maintained at the fine grain level. So like 
often times, for a majority of the KVs, we might be able to just write a 0. And 
hence, storing in varying width format would be an even bigger win.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4387) Error while syncing: DFSOutputStream is closed

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113832#comment-13113832
 ] 

Gary Helmling commented on HBASE-4387:
--

The TestLogRolling failure is on the assertion:
{code}
assertTrue(preLogRolledCalled.size() == 1);
{code}

So I would suspect it's more due to HBASE-4132.  Unless there's some 
interaction between this change and that.

> Error while syncing: DFSOutputStream is closed
> --
>
> Key: HBASE-4387
> URL: https://issues.apache.org/jira/browse/HBASE-4387
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: 4387.txt, errors-with-context.txt
>
>
> In a billion-row load on ~25 servers, I see "error while syncing" reasonable 
> often with the error "DFSOutputStream is closed" around a roll. We have some 
> race where a roll at the same time as heavy inserts causes a problem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4387) Error while syncing: DFSOutputStream is closed

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113829#comment-13113829
 ] 

Lars Hofhansl commented on HBASE-4387:
--

Don't see how it could. I'll be on computer soon to check it out.




> Error while syncing: DFSOutputStream is closed
> --
>
> Key: HBASE-4387
> URL: https://issues.apache.org/jira/browse/HBASE-4387
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: 4387.txt, errors-with-context.txt
>
>
> In a billion-row load on ~25 servers, I see "error while syncing" reasonable 
> often with the error "DFSOutputStream is closed" around a roll. We have some 
> race where a roll at the same time as heavy inserts causes a problem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113828#comment-13113828
 ] 

jirapos...@reviews.apache.org commented on HBASE-4415:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1925/
---

(Updated 2011-09-23 23:17:00.742946)


Review request for hbase.


Changes
---

Patch for finding location of hadoop jar files.


Summary
---

Create a post installation script to streamline configuration tasks for HBase.

usage: /usr/sbin/hbase-setup-conf.sh 

  Optional parameters:
--hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
location
--hadoop-home=/usr   Set Hadoop directory location
--hadoop-namenode=localhost  Set Hadoop namenode hostname
--hadoop-replication=3   Set HDFS replication
--hbase-home=/usrSet HBase directory location
--hbase-conf=/etc/hbase  Set HBase configuration directory 
location
--hbase-log=/var/log/hbase   Set HBase log directory location
--hbase-pid=/var/run/hbase   Set HBase pid directory location
--hbase-user=hbase   Set HBase user
--java-home=/usr/java/defaultSet JAVA_HOME directory location
--kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
--kerberos-principal-id=_HOSTSet Kerberos principal ID 
--keytab-dir=/etc/security/keytabs   Set keytab directory
--regionservers=localhostSet regionservers hostnames
--zookeeper-home=/usrSet ZooKeeper directory location
--zookeeper-quorum=localhost Set ZooKeeper Quorum
--zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location


This addresses bug HBASE-4415.
https://issues.apache.org/jira/browse/HBASE-4415


Diffs (updated)
-

  /src/assembly/all.xml 1175049 
  /src/docbkx/getting_started.xml 1175049 
  /src/packages/hbase-setup-conf.sh PRE-CREATION 
  /src/packages/templates/conf/hbase-env.sh PRE-CREATION 
  /src/packages/templates/conf/hbase-site.xml PRE-CREATION 

Diff: https://reviews.apache.org/r/1925/diff


Testing
---


Thanks,

Eric



> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --
>
> Key: HBASE-4415
> URL: https://issues.apache.org/jira/browse/HBASE-4415
> Project: HBase
>  Issue Type: New Feature
>  Components: scripts
> Environment: Java 6, Linux
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, 
> HBASE-4415-3.patch, HBASE-4415-4.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring 
> HBase environment and configuration.  By using the same pattern of 
> *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
> script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh 
>   Optional parameters:
> --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
> location
> --hadoop-home=/usr   Set Hadoop directory location
> --hadoop-namenode=localhost  Set Hadoop namenode hostname
> --hadoop-replication=3   Set HDFS replication
> --hbase-home=/usrSet HBase directory location
> --hbase-conf=/etc/hbase  Set HBase configuration 
> directory location
> --hbase-log=/var/log/hbase   Set HBase log directory location
> --hbase-pid=/var/run/hbase   Set HBase pid directory location
> --hbase-user=hbase   Set HBase user
> --java-home=/usr/java/defaultSet JAVA_HOME directory location
> --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
> --kerberos-principal-id=_HOSTSet Kerberos principal ID 
> --keytab-dir=/etc/security/keytabs   Set keytab directory
> --regionservers=localhostSet regionservers hostnames
> --zookeeper-home=/usrSet ZooKeeper directory location
> --zookeeper-quorum=localhost Set ZooKeeper Quorum
> --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4446) Rolling restart RSs scenario, regions could stay in OPENING state

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113825#comment-13113825
 ] 

Hudson commented on HBASE-4446:
---

Integrated in HBase-0.92 #17 (See 
[https://builds.apache.org/job/HBase-0.92/17/])
HBASE-4446 Rolling restart RSs scenario, regions could stay in OPENING state

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


> Rolling restart RSs scenario, regions could stay in OPENING state
> -
>
> Key: HBASE-4446
> URL: https://issues.apache.org/jira/browse/HBASE-4446
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 0.92.0
>
> Attachments: HBASE-4446-trunk.patch
>
>
> Keep Master up all the time, do rolling restart of RSs like this - stop RS1, 
> wait for 2 seconds, stop RS2, start RS1, wait for 2 seconds, stop RS3, start 
> RS2, wait for 2 seconds, etc. Region sometimes can just stay in OPENING state 
> even after timeoutmonitor period.
> 2011-09-19 08:10:33,131 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: While timing out a region 
> in state OPENING, found ZK node in unexpected state: RS_ZK_REGION_FAILED_OPEN
> The issue - RS was shutdown when a region is being opened, it was 
> transitioned to RS_ZK_REGION_FAILED_OPEN in ZK. In timeoutmonitor, it didn't 
> take care of RS_ZK_REGION_FAILED_OPEN.
> processOpeningState
> ...
>else if (dataInZNode.getEventType() != EventType.RS_ZK_REGION_OPENING &&
> LOG.warn("While timing out a region in state OPENING, "
> + "found ZK node in unexpected state: "
> + dataInZNode.getEventType());
> return;
>   }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4387) Error while syncing: DFSOutputStream is closed

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113823#comment-13113823
 ] 

Hudson commented on HBASE-4387:
---

Integrated in HBase-0.92 #17 (See 
[https://builds.apache.org/job/HBase-0.92/17/])
HBASE-4387 Error while syncing: DFSOutputStream is closed

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java


> Error while syncing: DFSOutputStream is closed
> --
>
> Key: HBASE-4387
> URL: https://issues.apache.org/jira/browse/HBASE-4387
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: 4387.txt, errors-with-context.txt
>
>
> In a billion-row load on ~25 servers, I see "error while syncing" reasonable 
> often with the error "DFSOutputStream is closed" around a roll. We have some 
> race where a roll at the same time as heavy inserts causes a problem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4295) rowcounter does not return the correct number of rows in certain circumstances

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113824#comment-13113824
 ] 

Hudson commented on HBASE-4295:
---

Integrated in HBase-0.92 #17 (See 
[https://builds.apache.org/job/HBase-0.92/17/])
HBASE-4295 rowcounter does not return the correct number of rows in certain 
circumstances

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapred/RowCounter.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java


> rowcounter does not return the correct number of rows in certain circumstances
> --
>
> Key: HBASE-4295
> URL: https://issues.apache.org/jira/browse/HBASE-4295
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Wing Yew Poon
>Assignee: Dave Revell
> Fix For: 0.90.5
>
> Attachments: HBASE-4295-v1.patch
>
>
> When you run
> {noformat}
> hadoop jar hbase.jar rowcounter 
> {noformat}
> the org.apache.hadoop.hbase.mapreduce.RowCounter class is run.
> The RowCounterMapper class in the RowCounter mapreduce job contains the 
> following:
> {noformat}
> @Override
> public void map(ImmutableBytesWritable row, Result values,
>   Context context)
> throws IOException {
>   for (KeyValue value: values.list()) {
> if (value.getValue().length > 0) {
>   context.getCounter(Counters.ROWS).increment(1);
>   break;
> }
>   }
> }
> {noformat}
> The intention is to go through the column values in the row, and increment 
> the ROWS counter if some value is non-empty. However, values.list() always 
> has size 1. This is because the createSubmittableJob static method uses a 
> Scan as follows:
> {noformat}
> Scan scan = new Scan();
> scan.setFilter(new FirstKeyOnlyFilter());
> {noformat}
> So the input map splits always contain just the first KV. If the column 
> corresponding to that first KV is empty, even though other columns are 
> non-empty, that row is skipped.
> This way, rowcounter can return an incorrect result.
> One way to reproduce this is to create an hbase table with two columns, say 
> f1:q1 and f2:q2. Create some (say 2) rows with empty f1:q1 but non-empty 
> f2:q2, and some (say 3) rows with empty f2:q2 and non-empty f1:q1.
> Then run rowcounter (specifying only the table but not any columns). The 
> count will be either 2 short or 3 short.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4387) Error while syncing: DFSOutputStream is closed

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113817#comment-13113817
 ] 

stack commented on HBASE-4387:
--

Does this cause TestLogRolling to fail?  
https://builds.apache.org/job/HBase-TRUNK/2246/testReport/org.apache.hadoop.hbase.regionserver.wal/TestLogRolling/testLogRollOnPipelineRestart/

I need to look at it



> Error while syncing: DFSOutputStream is closed
> --
>
> Key: HBASE-4387
> URL: https://issues.apache.org/jira/browse/HBASE-4387
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: 4387.txt, errors-with-context.txt
>
>
> In a billion-row load on ~25 servers, I see "error while syncing" reasonable 
> often with the error "DFSOutputStream is closed" around a roll. We have some 
> race where a roll at the same time as heavy inserts causes a problem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4280) [replication] ReplicationSink can deadlock itself via handlers

[Jean-Daniel Cryans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans resolved HBASE-4280.
---

  Resolution: Fixed
Hadoop Flags: [Reviewed]

Committed to 0.90, 0.92, trunk. Thanks for your comments Stack!

> [replication] ReplicationSink can deadlock itself via handlers
> --
>
> Key: HBASE-4280
> URL: https://issues.apache.org/jira/browse/HBASE-4280
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 0.90.5
>
> Attachments: HBASE-4280-0.90.patch
>
>
> I've experienced this problem a few times, ReplicationSink calls are received 
> through the normal handlers and potentially can call itself which, in certain 
> situations, call fill up all the handlers. For example, 10 handlers that are 
> all replication calls are all trying to talk to the local server at the same 
> time.
> HRS.replicateLogEntries should have @QosPriority(priority=HIGH_QOS) to use 
> the other set of handlers.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4344:
--

Attachment: (was: 4344-v11.txt)

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v2.txt, 4344-v4.txt, 
> 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4344:
--

Attachment: 4344-v11.txt

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v2.txt, 4344-v4.txt, 
> 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4472:
--

Fix Version/s: 0.92.0

> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch, jstack.txt
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-4472:
-

Assignee: Gary Helmling

> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch, jstack.txt
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-4472.
---

  Resolution: Fixed
Hadoop Flags: [Reviewed]

> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch, jstack.txt
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113806#comment-13113806
 ] 

Ted Yu commented on HBASE-4472:
---

I applied Gary's patch to 0.92 and TRUNK.

Thanks Gary and Eugene.

> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch, jstack.txt
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

[Eric Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated HBASE-4415:
-

Attachment: HBASE-4415-4.patch

Fixed location to find Hadoop jar file for both rpm and tarball deployment.

> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --
>
> Key: HBASE-4415
> URL: https://issues.apache.org/jira/browse/HBASE-4415
> Project: HBase
>  Issue Type: New Feature
>  Components: scripts
> Environment: Java 6, Linux
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, 
> HBASE-4415-3.patch, HBASE-4415-4.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring 
> HBase environment and configuration.  By using the same pattern of 
> *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
> script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh 
>   Optional parameters:
> --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
> location
> --hadoop-home=/usr   Set Hadoop directory location
> --hadoop-namenode=localhost  Set Hadoop namenode hostname
> --hadoop-replication=3   Set HDFS replication
> --hbase-home=/usrSet HBase directory location
> --hbase-conf=/etc/hbase  Set HBase configuration 
> directory location
> --hbase-log=/var/log/hbase   Set HBase log directory location
> --hbase-pid=/var/run/hbase   Set HBase pid directory location
> --hbase-user=hbase   Set HBase user
> --java-home=/usr/java/defaultSet JAVA_HOME directory location
> --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
> --kerberos-principal-id=_HOSTSet Kerberos principal ID 
> --keytab-dir=/etc/security/keytabs   Set keytab directory
> --regionservers=localhostSet regionservers hostnames
> --zookeeper-home=/usrSet ZooKeeper directory location
> --zookeeper-quorum=localhost Set ZooKeeper Quorum
> --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HBASE-4131) Make the Replication Service pluggable via a standard interface definition

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-4131:
--


Backed out this patch.  Was causing TestReplication failures.  See 
https://builds.apache.org/job/HBase-TRUNK/2246/testReport/

> Make the Replication Service pluggable via a standard interface definition
> --
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, 
> replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to 
> sync data from from one hbase cluster to another. It would be nice to make it 
> a pluggable interface so that other cross-data-center replication services 
> can be used in conjuction with HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4131) Make the Replication Service pluggable via a standard interface definition

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4131:
-

Attachment: 4131-backedout.txt

Here is what I backed out.

> Make the Replication Service pluggable via a standard interface definition
> --
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, 
> replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to 
> sync data from from one hbase cluster to another. It would be nice to make it 
> a pluggable interface so that other cross-data-center replication services 
> can be used in conjuction with HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4473) NPE when executors are down but events are still coming in

[Jean-Daniel Cryans (JIRA)]

NPE when executors are down but events are still coming in
--

 Key: HBASE-4473
 URL: https://issues.apache.org/jira/browse/HBASE-4473
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Priority: Minor
 Fix For: 0.90.5


Minor annoyance when shutting down a cluster and the master is still receiving 
events from Zookeeper:
{quote}
2011-09-22 23:53:01,552 DEBUG 
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
master:6-0x3292d87deb004f Received InterruptedException, doing nothing here
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:726)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:938)
at 
org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:407)
at 
org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteOpenedNode(ZKAssign.java:284)
at 
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler.process(OpenedRegionHandler.java:88)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
...
2011-09-22 23:53:01,558 DEBUG org.apache.hadoop.hbase.executor.ExecutorService: 
Executor service [MASTER_OPEN_REGION-sv2borg170:6] not found in {}
2011-09-22 23:53:01,558 ERROR org.apache.zookeeper.ClientCnxn: Error while 
calling watcher
java.lang.NullPointerException
at 
org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:220)
at 
org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:447)
at 
org.apache.hadoop.hbase.master.AssignmentManager.nodeDataChanged(AssignmentManager.java:546)
at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
{quote}

It's annoying because it then spams you with a bunch of NPEs that have nothing 
to do with the reason the Master is shutting down. Googling I saw someone also 
had that issue in June: http://pastebin.com/5Tqrj0nq

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4344:
--

Attachment: 4344-v11.txt

It enables the tests from TestAcidGuarantees which pass.
I also introduced a constant, KEY_VALUE_VER_WITH_MEMSTORE, for my earlier 
comment.

Will run through test suite once TRUNK build stabilizes.


> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v2.txt, 4344-v4.txt, 
> 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4457) Starting region server on non-default info port is resulting in broken URL's in master UI

[Praveen Patibandla (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113793#comment-13113793
 ] 

Praveen Patibandla commented on HBASE-4457:
---

Thanks Stack, I was initially thinking of replacing the HSL with new data 
structure.   

> Starting region server on non-default info port is resulting in broken URL's 
> in master UI
> -
>
> Key: HBASE-4457
> URL: https://issues.apache.org/jira/browse/HBASE-4457
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Praveen Patibandla
>Priority: Minor
>  Labels: newbie
> Fix For: 0.92.0
>
> Attachments: 4457-V1.patch, 4457.patch
>
>
> When  "hbase.regionserver.info.port" is set to non-default port, Master UI 
> has broken URL's in the region server table because it's hard coded to 
> default port.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

[Giridharan Kesavan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113791#comment-13113791
 ] 

Giridharan Kesavan commented on HBASE-4415:
---

hbase-setup-conf.sh script check for $HADOOP_HOME/bin/hadoop and looks for 
$HADOOP_HOME/hadoop-core*.jar 

when HADOOP_HOME is set to /usr the first condition passes and the second 
condition fails for not being able to find the hadoop-core jar as the core jar 
is installed by hadoop rpm at /usr/share/hadoop/hadoop-core-*.jar


> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --
>
> Key: HBASE-4415
> URL: https://issues.apache.org/jira/browse/HBASE-4415
> Project: HBase
>  Issue Type: New Feature
>  Components: scripts
> Environment: Java 6, Linux
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, 
> HBASE-4415-3.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring 
> HBase environment and configuration.  By using the same pattern of 
> *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
> script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh 
>   Optional parameters:
> --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
> location
> --hadoop-home=/usr   Set Hadoop directory location
> --hadoop-namenode=localhost  Set Hadoop namenode hostname
> --hadoop-replication=3   Set HDFS replication
> --hbase-home=/usrSet HBase directory location
> --hbase-conf=/etc/hbase  Set HBase configuration 
> directory location
> --hbase-log=/var/log/hbase   Set HBase log directory location
> --hbase-pid=/var/run/hbase   Set HBase pid directory location
> --hbase-user=hbase   Set HBase user
> --java-home=/usr/java/defaultSet JAVA_HOME directory location
> --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
> --kerberos-principal-id=_HOSTSet Kerberos principal ID 
> --keytab-dir=/etc/security/keytabs   Set keytab directory
> --regionservers=localhostSet regionservers hostnames
> --zookeeper-home=/usrSet ZooKeeper directory location
> --zookeeper-quorum=localhost Set ZooKeeper Quorum
> --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Eugene Koontz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113784#comment-13113784
 ] 

Eugene Koontz commented on HBASE-4472:
--

I think Gary's patch is better. It's simpler - with mine, you have the master 
modifying things outside itself (i.e. a zookeeper node) before aborting.

> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch, jstack.txt
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4132) Extend the WALActionsListener API to accomodate log archival

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113777#comment-13113777
 ] 

Hudson commented on HBASE-4132:
---

Integrated in HBase-TRUNK #2246 (See 
[https://builds.apache.org/job/HBase-TRUNK/2246/])
HBASE-4132 Extend the WALActionsListener API to accomodate log archival

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java


> Extend the WALActionsListener API to accomodate log archival
> 
>
> Key: HBASE-4132
> URL: https://issues.apache.org/jira/browse/HBASE-4132
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: walArchive.txt, walArchive2.txt, walArchive3.txt
>
>
> The WALObserver interface exposes the log roll events. It would be nice to 
> extend it to accomodate log archival events as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4295) rowcounter does not return the correct number of rows in certain circumstances

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113776#comment-13113776
 ] 

Hudson commented on HBASE-4295:
---

Integrated in HBase-TRUNK #2246 (See 
[https://builds.apache.org/job/HBase-TRUNK/2246/])
HBASE-4295 rowcounter does not return the correct number of rows in certain 
circumstances

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapred/RowCounter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java


> rowcounter does not return the correct number of rows in certain circumstances
> --
>
> Key: HBASE-4295
> URL: https://issues.apache.org/jira/browse/HBASE-4295
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Wing Yew Poon
>Assignee: Dave Revell
> Fix For: 0.90.5
>
> Attachments: HBASE-4295-v1.patch
>
>
> When you run
> {noformat}
> hadoop jar hbase.jar rowcounter 
> {noformat}
> the org.apache.hadoop.hbase.mapreduce.RowCounter class is run.
> The RowCounterMapper class in the RowCounter mapreduce job contains the 
> following:
> {noformat}
> @Override
> public void map(ImmutableBytesWritable row, Result values,
>   Context context)
> throws IOException {
>   for (KeyValue value: values.list()) {
> if (value.getValue().length > 0) {
>   context.getCounter(Counters.ROWS).increment(1);
>   break;
> }
>   }
> }
> {noformat}
> The intention is to go through the column values in the row, and increment 
> the ROWS counter if some value is non-empty. However, values.list() always 
> has size 1. This is because the createSubmittableJob static method uses a 
> Scan as follows:
> {noformat}
> Scan scan = new Scan();
> scan.setFilter(new FirstKeyOnlyFilter());
> {noformat}
> So the input map splits always contain just the first KV. If the column 
> corresponding to that first KV is empty, even though other columns are 
> non-empty, that row is skipped.
> This way, rowcounter can return an incorrect result.
> One way to reproduce this is to create an hbase table with two columns, say 
> f1:q1 and f2:q2. Create some (say 2) rows with empty f1:q1 but non-empty 
> f2:q2, and some (say 3) rows with empty f2:q2 and non-empty f1:q1.
> Then run rowcounter (specifying only the table but not any columns). The 
> count will be either 2 short or 3 short.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4131) Make the Replication Service pluggable via a standard interface definition

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113774#comment-13113774
 ] 

Hudson commented on HBASE-4131:
---

Integrated in HBase-TRUNK #2246 (See 
[https://builds.apache.org/job/HBase-TRUNK/2246/])
HBASE-4131 Make the Replication Service pluggable via a standard interface 
definition

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java


> Make the Replication Service pluggable via a standard interface definition
> --
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: replicationInterface1.txt, replicationInterface2.txt, 
> replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to 
> sync data from from one hbase cluster to another. It would be nice to make it 
> a pluggable interface so that other cross-data-center replication services 
> can be used in conjuction with HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4387) Error while syncing: DFSOutputStream is closed

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113775#comment-13113775
 ] 

Hudson commented on HBASE-4387:
---

Integrated in HBase-TRUNK #2246 (See 
[https://builds.apache.org/job/HBase-TRUNK/2246/])
HBASE-4387 Error while syncing: DFSOutputStream is closed
HBASE-4387 Error while syncing: DFSOutputStream is closed

stack : 
Files : 
* /hbase/trunk/CHANGES.txt

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java


> Error while syncing: DFSOutputStream is closed
> --
>
> Key: HBASE-4387
> URL: https://issues.apache.org/jira/browse/HBASE-4387
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: 4387.txt, errors-with-context.txt
>
>
> In a billion-row load on ~25 servers, I see "error while syncing" reasonable 
> often with the error "DFSOutputStream is closed" around a roll. We have some 
> race where a roll at the same time as heavy inserts causes a problem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4461) Expose getRowOrBefore via Thrift

[Jonathan Gray (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113768#comment-13113768
 ] 

Jonathan Gray commented on HBASE-4461:
--

Man, I remember when i could buy your vote for $2.00!

> Expose getRowOrBefore via Thrift
> 
>
> Key: HBASE-4461
> URL: https://issues.apache.org/jira/browse/HBASE-4461
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Attachments: HBASE-4461-v2.patch
>
>
> In order for fat Thrift-based clients to locate region locations they need to 
> utilize the getRowOrBefore method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4344) Persist memstoreTS to disk

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113766#comment-13113766
 ] 

Ted Yu commented on HBASE-4344:
---

@Amit:
{code}
+appendFileInfo(KEY_VALUE_VERSION, Bytes.toBytes(1));
{code}
I think we should introduce a constant for the 1 above.

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: 4344-v10.txt, 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 
> 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4461) Expose getRowOrBefore via Thrift

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113767#comment-13113767
 ] 

stack commented on HBASE-4461:
--

I'd say commit to TRUNK then maybe run a vote or something if you need it in 
0.92 (My vote will cost you $5.00)

> Expose getRowOrBefore via Thrift
> 
>
> Key: HBASE-4461
> URL: https://issues.apache.org/jira/browse/HBASE-4461
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Attachments: HBASE-4461-v2.patch
>
>
> In order for fat Thrift-based clients to locate region locations they need to 
> utilize the getRowOrBefore method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4471) HTable.close() should shut down executor pool

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113765#comment-13113765
 ] 

stack commented on HBASE-4471:
--

Not doing pool shutdown like a bug to me (If you look in 0.92, we seem to call 
pool shutdown -- where are you looking?)

> HTable.close() should shut down executor pool
> -
>
> Key: HBASE-4471
> URL: https://issues.apache.org/jira/browse/HBASE-4471
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.3
>Reporter: Josh Rosenblum
>
> Right now, it looks like HTable.close() is primarily concerned with flushing 
> commits. I understand the intended semantics of close to be that clients 
> should not attempt to call any other methods on that HTable instance after 
> close is called. If that's true, then close() might leave around some 
> relatively heavy resources after close() is called that can serve no further 
> purpose. In particular, the executor this.pool may have a number of threads 
> outstanding for some period of time (a minute with the default keepAliveTime 
> of 60). With the default number of threads == the number of regionservers and 
> with each thread having a 1mb stack by default on 64-bit jvms, this can be a 
> considerable amount of memory left around (in addition to any other resources 
> consumed by each thread). Is there any reason for close() not to also call 
> this.pool.shutdown() after it calls flushCommits()?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4457) Starting region server on non-default info port is resulting in broken URL's in master UI

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113764#comment-13113764
 ] 

stack commented on HBASE-4457:
--

@Praveen Yeah.. so I'd think there'd be a setHSL on the combined datastructure 
into which you could add latest HSL gotten from RS.

> Starting region server on non-default info port is resulting in broken URL's 
> in master UI
> -
>
> Key: HBASE-4457
> URL: https://issues.apache.org/jira/browse/HBASE-4457
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Praveen Patibandla
>Priority: Minor
>  Labels: newbie
> Fix For: 0.92.0
>
> Attachments: 4457-V1.patch, 4457.patch
>
>
> When  "hbase.regionserver.info.port" is set to non-default port, Master UI 
> has broken URL's in the region server table because it's hard coded to 
> default port.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4344) Persist memstoreTS to disk

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113763#comment-13113763
 ] 

Ted Yu commented on HBASE-4344:
---

@Stack:
See my comment @ 23/Sep/11 13:51
Version 9 combines the changes from HBASE-4345.

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: 4344-v10.txt, 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 
> 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4418) Show all the hbase configuration in the web ui

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113762#comment-13113762
 ] 

stack commented on HBASE-4418:
--

@Liyin I don't believe there is one.  It came for free when we moved to later 
hadoop.  We don't have it exposed in our UI IIRC.  We need to add a link along 
the top.

> Show all the hbase configuration in the web ui
> --
>
> Key: HBASE-4418
> URL: https://issues.apache.org/jira/browse/HBASE-4418
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>
> The motivation is to show ALL the HBase configuration, which takes effect in 
> the run time, in a global place.
> So we can easily know which configuration takes effect and what the value is.
> The configuration shows all the HBase and DFS configuration entry in the 
> configuration file and also includes all the HBase default setting in the 
> code, which is not the config file.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4462) Properly treating SocketTimeoutException

[Douglas Campbell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113749#comment-13113749
 ] 

Douglas Campbell commented on HBASE-4462:
-

+1 on no retry for socket timeout.  retrying after already timing out opens the 
possibility of timing out again which could double the delay with no reward.

Another thought is to allow retries but ensure older scan iterators are 
aborted, try to notify client threads holding them to abort, and only allow the 
newest scan iterator to stay alive.  

Without knowing the code, this may be difficult but it get's what you want i.e. 
only one iterator over a single object, a configurable number of retries, and 
region server is not locked up with different threads over a single scanner 
object. 


> Properly treating SocketTimeoutException
> 
>
> Key: HBASE-4462
> URL: https://issues.apache.org/jira/browse/HBASE-4462
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
> Fix For: 0.92.0
>
>
> SocketTimeoutException is currently treated like any IOE inside of 
> HCM.getRegionServerWithRetries and I think this is a problem. This method 
> should only do retries in cases where we are pretty sure the operation will 
> complete, but with STE we already waited for (by default) 60 seconds and 
> nothing happened.
> I found this while debugging Douglas Campbell's problem on the mailing list 
> where it seemed like he was using the same scanner from multiple threads, but 
> actually it was just the same client doing retries while the first run didn't 
> even finish yet (that's another problem). You could see the first scanner, 
> then up to two other handlers waiting for it to finish in order to run 
> (because of the synchronization on RegionScanner).
> So what should we do? We could treat STE as a DoNotRetryException and let the 
> client deal with it, or we could retry only once.
> There's also the option of having a different behavior for get/put/icv/scan, 
> the issue with operations that modify a cell is that you don't know if the 
> operation completed or not (same when a RS dies hard after completing let's 
> say a Put but just before returning to the client).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113742#comment-13113742
 ] 

jirapos...@reviews.apache.org commented on HBASE-4415:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1925/
---

(Updated 2011-09-23 20:22:38.162933)


Review request for hbase.


Changes
---

Updated the ability to use different keytab for HBase master and region server.


Summary
---

Create a post installation script to streamline configuration tasks for HBase.

usage: /usr/sbin/hbase-setup-conf.sh 

  Optional parameters:
--hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
location
--hadoop-home=/usr   Set Hadoop directory location
--hadoop-namenode=localhost  Set Hadoop namenode hostname
--hadoop-replication=3   Set HDFS replication
--hbase-home=/usrSet HBase directory location
--hbase-conf=/etc/hbase  Set HBase configuration directory 
location
--hbase-log=/var/log/hbase   Set HBase log directory location
--hbase-pid=/var/run/hbase   Set HBase pid directory location
--hbase-user=hbase   Set HBase user
--java-home=/usr/java/defaultSet JAVA_HOME directory location
--kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
--kerberos-principal-id=_HOSTSet Kerberos principal ID 
--keytab-dir=/etc/security/keytabs   Set keytab directory
--regionservers=localhostSet regionservers hostnames
--zookeeper-home=/usrSet ZooKeeper directory location
--zookeeper-quorum=localhost Set ZooKeeper Quorum
--zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location


This addresses bug HBASE-4415.
https://issues.apache.org/jira/browse/HBASE-4415


Diffs (updated)
-

  /src/assembly/all.xml 1172936 
  /src/docbkx/getting_started.xml 1172936 
  /src/packages/hbase-setup-conf.sh PRE-CREATION 
  /src/packages/templates/conf/hbase-env.sh PRE-CREATION 
  /src/packages/templates/conf/hbase-site.xml PRE-CREATION 

Diff: https://reviews.apache.org/r/1925/diff


Testing
---


Thanks,

Eric



> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --
>
> Key: HBASE-4415
> URL: https://issues.apache.org/jira/browse/HBASE-4415
> Project: HBase
>  Issue Type: New Feature
>  Components: scripts
> Environment: Java 6, Linux
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, 
> HBASE-4415-3.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring 
> HBase environment and configuration.  By using the same pattern of 
> *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
> script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh 
>   Optional parameters:
> --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
> location
> --hadoop-home=/usr   Set Hadoop directory location
> --hadoop-namenode=localhost  Set Hadoop namenode hostname
> --hadoop-replication=3   Set HDFS replication
> --hbase-home=/usrSet HBase directory location
> --hbase-conf=/etc/hbase  Set HBase configuration 
> directory location
> --hbase-log=/var/log/hbase   Set HBase log directory location
> --hbase-pid=/var/run/hbase   Set HBase pid directory location
> --hbase-user=hbase   Set HBase user
> --java-home=/usr/java/defaultSet JAVA_HOME directory location
> --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
> --kerberos-principal-id=_HOSTSet Kerberos principal ID 
> --keytab-dir=/etc/security/keytabs   Set keytab directory
> --regionservers=localhostSet regionservers hostnames
> --zookeeper-home=/usrSet ZooKeeper directory location
> --zookeeper-quorum=localhost Set ZooKeeper Quorum
> --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

[Eric Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113737#comment-13113737
 ] 

Eric Yang commented on HBASE-4415:
--

Updated the setup script to use unique key tabs for HBase master and region 
servers.

> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --
>
> Key: HBASE-4415
> URL: https://issues.apache.org/jira/browse/HBASE-4415
> Project: HBase
>  Issue Type: New Feature
>  Components: scripts
> Environment: Java 6, Linux
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, 
> HBASE-4415-3.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring 
> HBase environment and configuration.  By using the same pattern of 
> *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
> script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh 
>   Optional parameters:
> --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
> location
> --hadoop-home=/usr   Set Hadoop directory location
> --hadoop-namenode=localhost  Set Hadoop namenode hostname
> --hadoop-replication=3   Set HDFS replication
> --hbase-home=/usrSet HBase directory location
> --hbase-conf=/etc/hbase  Set HBase configuration 
> directory location
> --hbase-log=/var/log/hbase   Set HBase log directory location
> --hbase-pid=/var/run/hbase   Set HBase pid directory location
> --hbase-user=hbase   Set HBase user
> --java-home=/usr/java/defaultSet JAVA_HOME directory location
> --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
> --kerberos-principal-id=_HOSTSet Kerberos principal ID 
> --keytab-dir=/etc/security/keytabs   Set keytab directory
> --regionservers=localhostSet regionservers hostnames
> --zookeeper-home=/usrSet ZooKeeper directory location
> --zookeeper-quorum=localhost Set ZooKeeper Quorum
> --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

[Eric Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated HBASE-4415:
-

Attachment: HBASE-4415-3.patch

> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --
>
> Key: HBASE-4415
> URL: https://issues.apache.org/jira/browse/HBASE-4415
> Project: HBase
>  Issue Type: New Feature
>  Components: scripts
> Environment: Java 6, Linux
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, 
> HBASE-4415-3.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring 
> HBase environment and configuration.  By using the same pattern of 
> *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
> script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh 
>   Optional parameters:
> --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
> location
> --hadoop-home=/usr   Set Hadoop directory location
> --hadoop-namenode=localhost  Set Hadoop namenode hostname
> --hadoop-replication=3   Set HDFS replication
> --hbase-home=/usrSet HBase directory location
> --hbase-conf=/etc/hbase  Set HBase configuration 
> directory location
> --hbase-log=/var/log/hbase   Set HBase log directory location
> --hbase-pid=/var/run/hbase   Set HBase pid directory location
> --hbase-user=hbase   Set HBase user
> --java-home=/usr/java/defaultSet JAVA_HOME directory location
> --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
> --kerberos-principal-id=_HOSTSet Kerberos principal ID 
> --keytab-dir=/etc/security/keytabs   Set keytab directory
> --regionservers=localhostSet regionservers hostnames
> --zookeeper-home=/usrSet ZooKeeper directory location
> --zookeeper-quorum=localhost Set ZooKeeper Quorum
> --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Eugene Koontz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koontz updated HBASE-4472:
-

Attachment: jstack.txt

jstack of test during hang: note the TIMED_WAITING in HRegionServer.getMaster().

> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch, jstack.txt
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4415) Add configuration script for setup HBase (hbase-setup-conf.sh)

[Giridharan Kesavan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113727#comment-13113727
 ] 

Giridharan Kesavan commented on HBASE-4415:
---

can we have the option to set diff keytabs for hbasemaster and region servers?

> Add configuration script for setup HBase (hbase-setup-conf.sh)
> --
>
> Key: HBASE-4415
> URL: https://issues.apache.org/jira/browse/HBASE-4415
> Project: HBase
>  Issue Type: New Feature
>  Components: scripts
> Environment: Java 6, Linux
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: HBASE-4415-1.patch, HBASE-4415-2.patch, HBASE-4415.patch
>
>
> The goal of this jura is to provide a installation script for configuring 
> HBase environment and configuration.  By using the same pattern of 
> *-setup-conf.sh for all Hadoop related projects.  For HBase, the usage of the 
> script looks like this:
> {noformat}
> usage: ./hbase-setup-conf.sh 
>   Optional parameters:
> --hadoop-conf=/etc/hadoopSet Hadoop configuration directory 
> location
> --hadoop-home=/usr   Set Hadoop directory location
> --hadoop-namenode=localhost  Set Hadoop namenode hostname
> --hadoop-replication=3   Set HDFS replication
> --hbase-home=/usrSet HBase directory location
> --hbase-conf=/etc/hbase  Set HBase configuration 
> directory location
> --hbase-log=/var/log/hbase   Set HBase log directory location
> --hbase-pid=/var/run/hbase   Set HBase pid directory location
> --hbase-user=hbase   Set HBase user
> --java-home=/usr/java/defaultSet JAVA_HOME directory location
> --kerberos-realm=KERBEROS.EXAMPLE.COMSet Kerberos realm
> --kerberos-principal-id=_HOSTSet Kerberos principal ID 
> --keytab-dir=/etc/security/keytabs   Set keytab directory
> --regionservers=localhostSet regionservers hostnames
> --zookeeper-home=/usrSet ZooKeeper directory location
> --zookeeper-quorum=localhost Set ZooKeeper Quorum
> --zookeeper-snapshot=/var/lib/zookeeper  Set ZooKeeper snapshot location
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Eugene Koontz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113724#comment-13113724
 ] 

Eugene Koontz commented on HBASE-4472:
--

HBASE-4472.delete-zk-cluster-node-on-master-abort.patch is an alternative patch 
to fix the same bug: Master will remove the cluster-status ZK node 
(/hbase/shutdown by default) when it aborts. This will prevent the 
HRegionServer from hanging in HRegionServer.getMaster().

> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113723#comment-13113723
 ] 

stack commented on HBASE-4470:
--

Related to HBASE-3446.  Need retry.  The HBASE-3446 fix is not backportable 
though, at least not the version being worked on there.

> ServerNotRunningException coming out of assignRootAndMeta kills the Master
> --
>
> Key: HBASE-4470
> URL: https://issues.apache.org/jira/browse/HBASE-4470
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Priority: Blocker
> Fix For: 0.90.5
>
>
> I'm surprised we still have issues like that and I didn't get a hit while 
> googling so forgive me if there's already a jira about it.
> When the master starts it verifies the locations of root and meta before 
> assigning them, if the server is started but not running you'll get this:
> {quote}
> 2011-09-23 04:47:44,859 WARN 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
> RemoteException connecting to RS
> org.apache.hadoop.ipc.RemoteException: 
> org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running 
> yet
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at 
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> at 
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
> at 
> org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
> at 
> org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
> at 
> org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
> at 
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
> at 
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
> {quote}
> I hit that 3-4 times this week while debugging something else. The worst is 
> that when you restart the master it sees that as a failover, but none of the 
> regions are assigned so it takes an eternity to get back fully online.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Eugene Koontz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koontz updated HBASE-4472:
-

Attachment: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch

Call this.clusterStatusTracker.setClusterDown() during HMaster.abort().


> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
> Attachments: HBASE-4472.delete-zk-cluster-node-on-master-abort.patch, 
> HBASE-4472.patch
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4434.
--

   Resolution: Fixed
Fix Version/s: 0.92.0
 Hadoop Flags: [Reviewed]

Committed to 0.92 branch and to TRUNK.  Thanks for the sweet patch Kannan.

> seek optimization: don't do eager HFile Scanner next() unless the next KV is 
> needed
> ---
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Assignee: Kannan Muthukkaruppan
> Fix For: 0.92.0
>
> Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the 
> current KV, it also does a HFileScanner level next() ahead of time even if 
> the next KV is never actually required. This inefficiency can potentially 
> result in additional disk seeks and sub-optimal use of the block cache 
> (primarily for cases where the KVs are large and each occupies an HFile block 
> of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4450) test for number of blocks read: to serve as baseline for expected blocks read and for catching regressions

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113714#comment-13113714
 ] 

stack commented on HBASE-4450:
--

Committed to 0.92 branch too.

> test for number of blocks read: to serve as baseline for expected blocks read 
> and for catching regressions
> --
>
> Key: HBASE-4450
> URL: https://issues.apache.org/jira/browse/HBASE-4450
> Project: HBase
>  Issue Type: Test
>Reporter: Kannan Muthukkaruppan
>Assignee: Kannan Muthukkaruppan
> Attachments: HBASE-4450.txt
>
>
> Add a simple test for number of blocks read. The tests intent is to serve as 
> baseline for expected blocks read and for catching regressions. As 
> optimizations for HBase-4433 or Hbase-4434 are committed, the test would need 
> to be updated to adjust the counts for expected blocks read in various cases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4458) HBase should give actionable information when a region is compressed with a codec that is not available.

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113715#comment-13113715
 ] 

Jonathan Hsieh commented on HBASE-4458:
---

>From reading the description it looks like HBASE-2515 worked.  However, in 
>this particular scenario, *none* of the regions servers had LZO installed.  
>The goal was to move to another codec and somehow during the migration some 
>regions were missed.  

The symptom we saw was that regions failed to be deployed. Admittedly, we 
didn't initially look at the logs.  We tried assigning from the shell, but I 
don't recall seeing any class not found exceptions when the shell command 
failed.  We ended up figuring out the problem by using HFile on the data in the 
hdfs region data and getting the ClassNotFound there.

This brings up four potential places for an error message -- in the shell, in 
hbck, in the web ui, as well as in the logs.




> HBase should give actionable information when a region is compressed with a 
> codec that is not available.
> 
>
> Key: HBASE-4458
> URL: https://issues.apache.org/jira/browse/HBASE-4458
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Jonathan Hsieh
>
> A cluster that previously used LZO codec was upgraded with the intent of 
> moving away from the codec to another.  Several regions failed to deploy 
> because the LZO codec was no longer present.  However, there was little 
> indication that this as the problem.
> Ideally, the master web ui or hbck would detect these problems and provide 
> why it fails to deploy and also provide an actionable error message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4450) test for number of blocks read: to serve as baseline for expected blocks read and for catching regressions

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4450:
-

Attachment: HBASE-4450.txt

>From rb

> test for number of blocks read: to serve as baseline for expected blocks read 
> and for catching regressions
> --
>
> Key: HBASE-4450
> URL: https://issues.apache.org/jira/browse/HBASE-4450
> Project: HBase
>  Issue Type: Test
>Reporter: Kannan Muthukkaruppan
>Assignee: Kannan Muthukkaruppan
> Attachments: HBASE-4450.txt
>
>
> Add a simple test for number of blocks read. The tests intent is to serve as 
> baseline for expected blocks read and for catching regressions. As 
> optimizations for HBase-4433 or Hbase-4434 are committed, the test would need 
> to be updated to adjust the counts for expected blocks read in various cases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4434:
-

Attachment: HBASE-4434.txt

Copied from rb.

> seek optimization: don't do eager HFile Scanner next() unless the next KV is 
> needed
> ---
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Assignee: Kannan Muthukkaruppan
> Attachments: HBASE-4434.txt
>
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the 
> current KV, it also does a HFileScanner level next() ahead of time even if 
> the next KV is never actually required. This inefficiency can potentially 
> result in additional disk seeks and sub-optimal use of the block cache 
> (primarily for cases where the KVs are large and each occupies an HFile block 
> of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4434) seek optimization: don't do eager HFile Scanner next() unless the next KV is needed

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113701#comment-13113701
 ] 

jirapos...@reviews.apache.org commented on HBASE-4434:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2034/#review2056
---

Ship it!


- Michael


On 2011-09-23 05:13:36, Kannan Muthukkaruppan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2034/
bq.  ---
bq.  
bq.  (Updated 2011-09-23 05:13:36)
bq.  
bq.  
bq.  Review request for Michael Stack, Jonathan Gray and Mikhail Bautin.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The increase in blocks read in one sub-test case is due to HBASE-4466, 
which should be fixed soon. It is primarily an accounting issue, not an extra 
cache miss.
bq.  
bq.  
bq.  This addresses bug HBASE-4434.
bq.  https://issues.apache.org/jira/browse/HBASE-4434
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
 1174514 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
 1174514 
bq.  
bq.  Diff: https://reviews.apache.org/r/2034/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran TestBlocksRead. Will run the entire suite as well. On our 89 based 
internal branch, the tests passed without any issues.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Kannan
bq.  
bq.



> seek optimization: don't do eager HFile Scanner next() unless the next KV is 
> needed
> ---
>
> Key: HBASE-4434
> URL: https://issues.apache.org/jira/browse/HBASE-4434
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Assignee: Kannan Muthukkaruppan
>
> When a seek/reseek is done on StoreFileScanner, in addition to setting the 
> current KV, it also does a HFileScanner level next() ahead of time even if 
> the next KV is never actually required. This inefficiency can potentially 
> result in additional disk seeks and sub-optimal use of the block cache 
> (primarily for cases where the KVs are large and each occupies an HFile block 
> of its own).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling updated HBASE-4472:
-

Attachment: HBASE-4472.patch

Adds an explicit call to HRegionServer.stop() in JVMClusterUtil.shutdown().

> MiniHBaseCluster.shutdown() doesn't work if no active master
> 
>
> Key: HBASE-4472
> URL: https://issues.apache.org/jira/browse/HBASE-4472
> Project: HBase
>  Issue Type: Bug
>Reporter: Gary Helmling
> Attachments: HBASE-4472.patch
>
>
> Running tests over in HBASE-4014 brought up this issue.  If the active master 
> in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
> will just hang in JVMClusterUtil.shutdown(), waiting to join each of the 
> region server threads.
> Seems like we should explicitly stop each region server instead of just 
> relying on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4472) MiniHBaseCluster.shutdown() doesn't work if no active master

[Gary Helmling (JIRA)]

MiniHBaseCluster.shutdown() doesn't work if no active master


 Key: HBASE-4472
 URL: https://issues.apache.org/jira/browse/HBASE-4472
 Project: HBase
  Issue Type: Bug
Reporter: Gary Helmling
 Attachments: HBASE-4472.patch

Running tests over in HBASE-4014 brought up this issue.  If the active master 
in a MiniHBaseCluster has aborted, then calling MiniHBaseCluster.shutdown() 
will just hang in JVMClusterUtil.shutdown(), waiting to join each of the region 
server threads.

Seems like we should explicitly stop each region server instead of just relying 
on an active master instance deleting the cluster status znode?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4467) Handle inconsistencies in Hadoop libraries naming in hbase script

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113688#comment-13113688
 ] 

stack commented on HBASE-4467:
--

+1 on patch.  If it works for you commit LarsG.

> Handle inconsistencies in Hadoop libraries naming in hbase script
> -
>
> Key: HBASE-4467
> URL: https://issues.apache.org/jira/browse/HBASE-4467
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Lars George
>Assignee: Lars George
>Priority: Trivial
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-4467.patch
>
>
> When using an Hadoop tarball that has a library naming of "hadoop-x.y.z-core" 
> as opposed to "hadoop-core-x.y.z" then the hbase script throws errors.
> {noformat}
> $ bin/start-hbase.sh 
> ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No such file 
> or directory
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/util/PlatformName
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.util.PlatformName
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No such file 
> or directory
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/util/PlatformName
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.util.PlatformName
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> localhost: starting zookeeper, logging to 
> /projects/opensource/hbase-trunk-rw//logs/hbase-larsgeorge-zookeeper-de1-app-mbp-2.out
> localhost: /projects/opensource/hadoop-0.20.2-append
> localhost: ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No 
> such file or directory
> localhost: Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/util/PlatformName
> localhost: Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.util.PlatformName
> localhost:at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> localhost:at java.security.AccessController.doPrivileged(Native Method)
> localhost:at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> localhost:at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> localhost:at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> localhost:at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> starting master, logging to 
> /projects/opensource/hbase-trunk-rw/bin/../logs/hbase-larsgeorge-master-de1-app-mbp-2.out
> /projects/opensource/hadoop-0.20.2-append
> ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No such file 
> or directory
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/util/PlatformName
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.util.PlatformName
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> localhost: starting regionserver, logging to 
> /projects/opensource/hbase-trunk-rw//logs/hbase-larsgeorge-regionserver-de1-app-mbp-2.out
> localhost: /projects/opensource/hadoop-0.20.2-append
> localhost: ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No 
> such file or directory
> localhost: Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/util/PlatformName
> localhost: Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.util.PlatformName
> localhost:at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> localhost:at java.security.AccessController.doPrivileged(Native Method)
> localhost:at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> localhost:at java.lang.Cl

[jira] [Resolved] (HBASE-4468) Wrong resource name in an error massage: webapps instead of hbase-webapps

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4468.
--

   Resolution: Fixed
Fix Version/s: 0.92.0
 Hadoop Flags: [Reviewed]

Thanks for the patch nkeywal

> Wrong resource name in an error massage: webapps instead of hbase-webapps
> -
>
> Key: HBASE-4468
> URL: https://issues.apache.org/jira/browse/HBASE-4468
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Trivial
> Fix For: 0.92.0
>
> Attachments: 20110923_4468_InfoServer.patch
>
>
> org.apache.hadoop.hbase.util.InfoServer loads a resource in 'hbase-webapps' 
> but displays a message about 'webapps' when it does not find it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4468) Wrong resource name in an error massage: webapps instead of hbase-webapps

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113685#comment-13113685
 ] 

stack commented on HBASE-4468:
--

Applied TRUNK and 0.92 branch.

> Wrong resource name in an error massage: webapps instead of hbase-webapps
> -
>
> Key: HBASE-4468
> URL: https://issues.apache.org/jira/browse/HBASE-4468
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Trivial
> Fix For: 0.92.0
>
> Attachments: 20110923_4468_InfoServer.patch
>
>
> org.apache.hadoop.hbase.util.InfoServer loads a resource in 'hbase-webapps' 
> but displays a message about 'webapps' when it does not find it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4296) Deprecate HTable[Interface].getRowOrBefore(...)

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113676#comment-13113676
 ] 

stack commented on HBASE-4296:
--

We can hold off till 2600 is done.

I don't think we can call it scanMeta or do you mean scanMetaGetRowOrBefore?  
The latter we could do.

I'd think that when 2600 goes in, this method just won't make sense any more -- 
clients that find regions using this will just be broke... since the .META. 
format will have changed (endkeys rather than startkeys).  Ain't sure what to 
do about that.  It'll be at a major version transition.

> Deprecate HTable[Interface].getRowOrBefore(...)
> ---
>
> Key: HBASE-4296
> URL: https://issues.apache.org/jira/browse/HBASE-4296
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Trivial
> Fix For: 0.92.0
>
> Attachments: 4296.txt
>
>
> HTable's getRowOrBefore(...) internally calls into Store.getRowKeyAtOrBefore. 
> That method was created to allow our scanning of .META. (see HBASE-2600).
> Store.getRowKeyAtOrBefore(...) lists a bunch of requirements for this to be 
> performant that a user of HTable will not be aware of.
> I propose deprecating this in the public interface in 0.92 and removing it 
> from the public interface in 0.94. If we don't get to HBASE-2600 in 0.94 it 
> will still remain as internal interface for scanning meta.
> Comments?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4131) Make the Replication Service pluggable via a standard interface definition

[Jonathan Gray (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113674#comment-13113674
 ] 

Jonathan Gray commented on HBASE-4131:
--

Thanks stack!

> Make the Replication Service pluggable via a standard interface definition
> --
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: replicationInterface1.txt, replicationInterface2.txt, 
> replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to 
> sync data from from one hbase cluster to another. It would be nice to make it 
> a pluggable interface so that other cross-data-center replication services 
> can be used in conjuction with HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4446) Rolling restart RSs scenario, regions could stay in OPENING state

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113673#comment-13113673
 ] 

Hudson commented on HBASE-4446:
---

Integrated in HBase-TRUNK #2245 (See 
[https://builds.apache.org/job/HBase-TRUNK/2245/])
HBASE-4446 Rolling restart RSs scenario, regions could stay in OPENING state

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


> Rolling restart RSs scenario, regions could stay in OPENING state
> -
>
> Key: HBASE-4446
> URL: https://issues.apache.org/jira/browse/HBASE-4446
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 0.92.0
>
> Attachments: HBASE-4446-trunk.patch
>
>
> Keep Master up all the time, do rolling restart of RSs like this - stop RS1, 
> wait for 2 seconds, stop RS2, start RS1, wait for 2 seconds, stop RS3, start 
> RS2, wait for 2 seconds, etc. Region sometimes can just stay in OPENING state 
> even after timeoutmonitor period.
> 2011-09-19 08:10:33,131 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: While timing out a region 
> in state OPENING, found ZK node in unexpected state: RS_ZK_REGION_FAILED_OPEN
> The issue - RS was shutdown when a region is being opened, it was 
> transitioned to RS_ZK_REGION_FAILED_OPEN in ZK. In timeoutmonitor, it didn't 
> take care of RS_ZK_REGION_FAILED_OPEN.
> processOpeningState
> ...
>else if (dataInZNode.getEventType() != EventType.RS_ZK_REGION_OPENING &&
> LOG.warn("While timing out a region in state OPENING, "
> + "found ZK node in unexpected state: "
> + dataInZNode.getEventType());
> return;
>   }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4131) Make the Replication Service pluggable via a standard interface definition

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4131:
-

   Resolution: Fixed
Fix Version/s: 0.94.0
 Hadoop Flags: [Reviewed]
   Status: Resolved  (was: Patch Available)

Committed to TRUNK

> Make the Replication Service pluggable via a standard interface definition
> --
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: replicationInterface1.txt, replicationInterface2.txt, 
> replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to 
> sync data from from one hbase cluster to another. It would be nice to make it 
> a pluggable interface so that other cross-data-center replication services 
> can be used in conjuction with HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4461) Expose getRowOrBefore via Thrift

[Jonathan Gray (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113671#comment-13113671
 ] 

Jonathan Gray commented on HBASE-4461:
--

and I'm saving up my new features to force in 92 to try and get the 
HLog/Delayable stuff in ;)

> Expose getRowOrBefore via Thrift
> 
>
> Key: HBASE-4461
> URL: https://issues.apache.org/jira/browse/HBASE-4461
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Attachments: HBASE-4461-v2.patch
>
>
> In order for fat Thrift-based clients to locate region locations they need to 
> utilize the getRowOrBefore method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4461) Expose getRowOrBefore via Thrift

[Jonathan Gray (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113669#comment-13113669
 ] 

Jonathan Gray commented on HBASE-4461:
--

Well my plan is to use it internally on 0.92 (we are porting all the changes 
necessary for our fat C++ client from our internal 90 branch).  But wherever 
you think it should go is fine.

> Expose getRowOrBefore via Thrift
> 
>
> Key: HBASE-4461
> URL: https://issues.apache.org/jira/browse/HBASE-4461
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Attachments: HBASE-4461-v2.patch
>
>
> In order for fat Thrift-based clients to locate region locations they need to 
> utilize the getRowOrBefore method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3025) Coprocessor based simple access control

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113670#comment-13113670
 ] 

jirapos...@reviews.apache.org commented on HBASE-3025:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2041/
---

Review request for hbase.


Summary
---

This patch implements access control list based authorization of HBase 
operations.  The patch depends on the currently posted patch for HBASE-2742 
(secure RPC engine).

Key parts of the implementation are:

* AccessControlLists - encapsulates storage of permission grants in a metadata 
table ("_acl_").  This differs from previous implementation where the ".META." 
table was used to store permissions.

* AccessController - 
  - implements MasterObserver and RegionObserver, performing authorization 
checks in each of the preXXX() hooks.  If authorization fails, an 
AccessDeniedException is thrown.
  - implements AccessControllerProtocol as a coprocessor endpoint to provide 
RPC methods for granting, revoking and listing permissions.

* ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries and 
updates throughout the cluster nodes using ZK.  ACL entries are stored in 
per-table znodes as /hbase/acl/tablename.

* Additional ruby shell scripts providing the "grant", "revoke" and 
"user_permission" commands

* Support for a new OWNER attribute in HTableDescriptor.  I could separate out 
this change into a new JIRA for discussion, but I don't see it as currently 
useful outside of security.  Alternately, I could handle the OWNER attribute 
completely in AccessController without changing HTD, but that would make 
interaction via hbase shell a bit uglier.


This addresses bug HBASE-3025.
https://issues.apache.org/jira/browse/HBASE-3025


Diffs
-

  
security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControlFilter.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControlLists.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControllerProtocol.java
 PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/security/rbac/Permission.java 
PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/rbac/TableAuthManager.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/rbac/TablePermission.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/rbac/UserPermission.java
 PRE-CREATION 
  
security/src/main/java/org/apache/hadoop/hbase/security/rbac/ZKPermissionWatcher.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/rbac/SecureTestUtil.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/rbac/TestAccessControlFilter.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/rbac/TestAccessController.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/rbac/TestTablePermissions.java
 PRE-CREATION 
  
security/src/test/java/org/apache/hadoop/hbase/security/rbac/TestZKPermissionsWatcher.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 46a1a3d 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5 
  src/main/resources/hbase-default.xml 2c8f44b 
  src/main/ruby/hbase.rb 4d27191 
  src/main/ruby/hbase/admin.rb b244ffe 
  src/main/ruby/hbase/hbase.rb beb2450 
  src/main/ruby/hbase/security.rb PRE-CREATION 
  src/main/ruby/shell.rb 9a47600 
  src/main/ruby/shell/commands.rb a352c2e 
  src/main/ruby/shell/commands/grant.rb PRE-CREATION 
  src/main/ruby/shell/commands/revoke.rb PRE-CREATION 
  src/main/ruby/shell/commands/table_permission.rb PRE-CREATION 
  src/main/ruby/shell/commands/user_permission.rb PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java 4d7ee22 

Diff: https://reviews.apache.org/r/2041/diff


Testing
---


Thanks,

Gary



> Coprocessor based simple access control
> ---
>
> Key: HBASE-3025
> URL: https://issues.apache.org/jira/browse/HBASE-3025
> Project: HBase
>  Issue Type: Sub-task
>  Components: coprocessors
>Reporter: Andrew Purtell
> Attachments: HBASE-3025.1.patch, HBASE-3025.2011-02-01.patch
>
>
> Thanks for the clarification Jeff which reminds me to edit this issue.
> Goals of this issue
> # Client access to HBase is authenticated
> # User data is private unless access has been granted
> # Access to data can be granted at a table or per column family basis. 
> Non-Goals of this issue
> Th

[jira] [Commented] (HBASE-4461) Expose getRowOrBefore via Thrift

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113666#comment-13113666
 ] 

stack commented on HBASE-4461:
--

+1

This for TRUNK only?

> Expose getRowOrBefore via Thrift
> 
>
> Key: HBASE-4461
> URL: https://issues.apache.org/jira/browse/HBASE-4461
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Attachments: HBASE-4461-v2.patch
>
>
> In order for fat Thrift-based clients to locate region locations they need to 
> utilize the getRowOrBefore method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4463) Run more aggressive compactions during off peak hours

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113665#comment-13113665
 ] 

stack commented on HBASE-4463:
--

@Dhruba We need this too.  How would we do it?

> Run more aggressive compactions during off peak hours
> -
>
> Key: HBASE-4463
> URL: https://issues.apache.org/jira/browse/HBASE-4463
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Karthik Ranganathan
>Assignee: Karthik Ranganathan
>
> The number of iops on the disk and the top of the rack bandwidth utilization 
> at off peak hours is much lower than at peak hours depending on the 
> application usage pattern. We can utilize this knowledge to improve the 
> performance of the HBase cluster by increasing the compact selection ratio to 
> a much larger value during off-peak hours than otherwise - increasing 
> hbase.hstore.compaction.ratio (1.2 default) to 
> hbase.hstore.compaction.ratio.offpeak (5 default). This will help reduce the 
> average number of files per store.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira