[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119923#comment-13119923
 ] 

Lars Hofhansl commented on HBASE-4536:
--

Started to look at the code.

Scanning now has three different scenarios:
# Normal scan. Under all circumstanced filter deleted rows (unless it's a past 
scan, of course). Do not return delete markers.
# Minor compaction. If min versions is not set, follow current behavior (filter 
deleted rows, keep delete markers). If min versions is set, keep deleted rows, 
treat delete markers as normal rows (but do not count them as versions). I.e. 
delete markers are subject to maxversions, expiration, etc.
# Major compaction. If min versions is not set, follow current behavior (filter 
deleted rows, remove delete markers, like #1). If min versions is set, keep 
deleted rows, treat delete markers as normal rows (again, do not count them as 
versions).

In all three cases rows past max versions or expired rows past min versions 
will be filtered out.

Jon could reuse scan type #2 or #3 for his needs, I think.


> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119903#comment-13119903
 ] 

jirapos...@reviews.apache.org commented on HBASE-4422:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2089/#review2305
---


Tests run: 1018, Failures: 0, Errors: 0, Skipped: 21

woot

- Jonathan


On 2011-10-04 02:50:58, Jonathan Gray wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2089/
bq.  ---
bq.  
bq.  (Updated 2011-10-04 02:50:58)
bq.  
bq.  
bq.  Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Creates a new CacheConfig class and moves almost everything block cache 
related into this single class.  Adding new configuration params and booleans 
and such should be much better.
bq.  
bq.  All tests are NOT passing yet, still working on it, but wanted to have 
something up today.  Basically "code complete" but broken :)
bq.  
bq.  
bq.  This addresses bug HBASE-4422.
bq.  https://issues.apache.org/jira/browse/HBASE-4422
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 
PRE-CREATION 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1178676 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 
1178676 
bq.
/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178676 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1178676 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 
1178676 
bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1178676 
bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 
1178676 
bq.
/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1178676 
bq.
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 
1178676 
bq.
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
1178676 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 
1178676 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
 1178676 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 
1178676 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1178676 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
 1178676 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 
1178676 
bq.  
bq.  Diff: https://reviews.apache.org/r/2089/diff
b

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119901#comment-13119901
 ] 

Ted Yu commented on HBASE-4528:
---

How about possible IOE from the following call at line 1809:
{code}
  txid = this.log.appendNoSync(regionInfo, this.htableDescriptor.getName(),
{code}

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119895#comment-13119895
 ] 

jirapos...@reviews.apache.org commented on HBASE-4534:
--



bq.  On 2011-10-04 00:56:39, Lars Hofhansl wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java, line 
100
bq.  > 
bq.  >
bq.  > On many JVMs (including hotspot) this creates a memory barrier.
bq.  > This might actually cause a measurable slowdown on multi-core 
machines.
bq.  > (And maybe I'm just too paranoid).
bq.  >
bq.  
bq.  Mikhail Bautin wrote:
bq.  I could shut off these counters in production, but we already 
increment AtomicLongs all over the place to collect performance counters. So if 
this is a concern, this is probably best addressed in a separate feature. I 
don't yet know what the solution would be (some custom high-performance 
non-locking counters?) We have also added a lot of per-table/column family 
metrics in our internal branch, and that did not seem to hurt performance at 
all. A more important point, however, is that if we can cut disk IO, a small 
amount additional computation is probably OK. We are already seeing a huge 
reduction in block cache misses after pushing lazy seek to production.
bq.  
bq.  Please let me know if you still want me to get rid of these increments 
in production.
bq.  
bq.  Todd Lipcon wrote:
bq.  We already have a dependency on high-scale-lib, which includes 
efficient striped counters. But I agree we shouldn't worry about it in a 
particular patch, and rather do it throughout if we see a degredation

Yeah... I am probably overly concerned because Seek and reseek are called 
pretty frequently. Let's not worry about it.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2155/#review2297
---


On 2011-10-04 01:20:14, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2155/
bq.  ---
bq.  
bq.  (Updated 2011-10-04 01:20:14)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
all-column, single-column, and multi-column). Also all combinations of Bloom 
filters and compression (NONE vs GZIP) are tested. The unit test flushes 
multiple StoreFiles with disjoint timestamp ranges and runs various types of 
queries against them. Currently we are not testing overlapping timestamp ranges.
bq.  
bq.  
bq.  This addresses bug HBASE-4534.
bq.  https://issues.apache.org/jira/browse/HBASE-4534
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
3ec1962 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
ede42b2 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java a828846 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java 
a0d8abf 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java 
e86c1c6 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2155/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All 1026 unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[dhruba borthakur (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119894#comment-13119894
 ] 

dhruba borthakur commented on HBASE-4528:
-

from my understanding of the code in HLog.sync(), if it throws an exception 
then the entire regionserver is supposed to exit, is it not? If that is the 
case, then is there any point in catching the exception from log.sync()? 

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Akash Ashok (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash Ashok updated HBASE-4531:
---

Status: Patch Available  (was: Reopened)

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Akash Ashok
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531-v2.patch, 
> HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Akash Ashok (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119883#comment-13119883
 ] 

Akash Ashok commented on HBASE-4531:


@Stack: The build is failing again. Realized we might have to do some changes 
in xmls for them to work with maven-site-plugin 2.1 . 

HBase-4531.patch instead of HBase-4531-v2.patch would do the trick. That would 
use maven-site-plugin 2.0.1 and maven-surefire-reports plugin 2.7.2.

Just ran the whole command "mvn -e clean 
-Dmaven.test.redirectTestOutputToFile=true site assembly:assembly" and its 
running fine.

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Akash Ashok
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531-v2.patch, 
> HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Akash Ashok (Reopened) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash Ashok reopened HBASE-4531:



> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Akash Ashok
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531-v2.patch, 
> HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119882#comment-13119882
 ] 

jirapos...@reviews.apache.org commented on HBASE-4422:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2089/
---

(Updated 2011-10-04 02:50:58.274414)


Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi.


Changes
---

Minor fixes to some tests.  I have all tests passing now!


Summary
---

Creates a new CacheConfig class and moves almost everything block cache related 
into this single class.  Adding new configuration params and booleans and such 
should be much better.

All tests are NOT passing yet, still working on it, but wanted to have 
something up today.  Basically "code complete" but broken :)


This addresses bug HBASE-4422.
https://issues.apache.org/jira/browse/HBASE-4422


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1178676 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1178676 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
1178676 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 
1178676 
  /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
1178676 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1178676 
  /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1178676 
  /src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1178676 
  /src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1178676 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1178676 
  /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 
1178676 
  
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 
1178676 
  
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
 1178676 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 
1178676 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1178676 
  
/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
 1178676 
  /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 
1178676 

Diff: https://reviews.apache.org/r/2089/diff


Testing
---

Still working through some tests that aren't passing.


Thanks,

Jonathan



> Move block cache parameters and references into single CacheConf class
> --
>
> Key: HBASE-4422
> URL: https://issues.apache.org/jira/browse/HBASE-4422
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Fix For: 0.92.0
>
>
> From StoreFile down to HFile, we currently us

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119869#comment-13119869
 ] 

Jonathan Gray commented on HBASE-4534:
--

LUCENE-3408 is just a wrapper around two counter implementations.  One is 
thread-safe and uses an AtomicLong, the other is not and uses a long.  It looks 
like they were just trying to improve performance when the counter was being 
used in a single thread.

+1 that we should deal with changing AtomicLong to something else in another 
jira.

> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119864#comment-13119864
 ] 

jirapos...@reviews.apache.org commented on HBASE-4534:
--



bq.  On 2011-10-04 00:56:39, Lars Hofhansl wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java, line 
100
bq.  > 
bq.  >
bq.  > On many JVMs (including hotspot) this creates a memory barrier.
bq.  > This might actually cause a measurable slowdown on multi-core 
machines.
bq.  > (And maybe I'm just too paranoid).
bq.  >
bq.  
bq.  Mikhail Bautin wrote:
bq.  I could shut off these counters in production, but we already 
increment AtomicLongs all over the place to collect performance counters. So if 
this is a concern, this is probably best addressed in a separate feature. I 
don't yet know what the solution would be (some custom high-performance 
non-locking counters?) We have also added a lot of per-table/column family 
metrics in our internal branch, and that did not seem to hurt performance at 
all. A more important point, however, is that if we can cut disk IO, a small 
amount additional computation is probably OK. We are already seeing a huge 
reduction in block cache misses after pushing lazy seek to production.
bq.  
bq.  Please let me know if you still want me to get rid of these increments 
in production.

We already have a dependency on high-scale-lib, which includes efficient 
striped counters. But I agree we shouldn't worry about it in a particular 
patch, and rather do it throughout if we see a degredation


- Todd


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2155/#review2297
---


On 2011-10-04 01:20:14, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2155/
bq.  ---
bq.  
bq.  (Updated 2011-10-04 01:20:14)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
all-column, single-column, and multi-column). Also all combinations of Bloom 
filters and compression (NONE vs GZIP) are tested. The unit test flushes 
multiple StoreFiles with disjoint timestamp ranges and runs various types of 
queries against them. Currently we are not testing overlapping timestamp ranges.
bq.  
bq.  
bq.  This addresses bug HBASE-4534.
bq.  https://issues.apache.org/jira/browse/HBASE-4534
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
3ec1962 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
ede42b2 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java a828846 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java 
a0d8abf 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java 
e86c1c6 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2155/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All 1026 unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119862#comment-13119862
 ] 

Ted Yu commented on HBASE-4534:
---

We can use high-scale-lib
We already use it in some of HBase code.

> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[Mikhail Bautin (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119853#comment-13119853
 ] 

Mikhail Bautin commented on HBASE-4534:
---

Regarding Lars's comment on AtomicLong performance: 
https://issues.apache.org/jira/browse/LUCENE-3408 is a similar issue in Lucene. 
We need to dig more to find out what their solution is, i.e. exactly how 
Lucene's Counter is better than AtomicLong.

> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119843#comment-13119843
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/
---

(Updated 2011-10-04 01:21:46.559878)


Review request for hbase and Mingjie Lai.


Changes
---

Address Andrew Purtell's review of patch version 5: 
-replace "coprocessor(s)name(s)" with "coprocessor(s)".
-HServerLoad and RegionLoad use Set rather than 
Set to represent coprocessors.


Summary
---

Proposed fix for HBASE-4070. 


This addresses bug HBASE-4070.
https://issues.apache.org/jira/browse/HBASE-4070


Diffs (updated)
-

  src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 
  src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
  src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 
  src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3840279 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/2029/diff


Testing
---

Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.


Thanks,

Eugene



> [Coprocessors] Improve region server metrics to report loaded coprocessors to 
> master
> 
>
> Key: HBASE-4070
> URL: https://issues.apache.org/jira/browse/HBASE-4070
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Mingjie Lai
>Assignee: Eugene Koontz
> Attachments: HBASE-4070.patch, HBASE-4070.patch, HBASE-4070.patch, 
> master-web-ui.jpg, rs-status-web-ui.jpg
>
>
> HBASE-3512 is about listing loaded cp classes at shell. To make it more 
> generic, we need a way to report this piece of information from region to 
> master (or just at region server level). So later on, we can display the 
> loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119842#comment-13119842
 ] 

jirapos...@reviews.apache.org commented on HBASE-4534:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2155/
---

(Updated 2011-10-04 01:20:14.425653)


Review request for hbase.


Changes
---

All unit tests passed.


Summary
---

A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
all-column, single-column, and multi-column). Also all combinations of Bloom 
filters and compression (NONE vs GZIP) are tested. The unit test flushes 
multiple StoreFiles with disjoint timestamp ranges and runs various types of 
queries against them. Currently we are not testing overlapping timestamp ranges.


This addresses bug HBASE-4534.
https://issues.apache.org/jira/browse/HBASE-4534


Diffs
-

  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
3ec1962 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ede42b2 
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java a828846 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java 
a0d8abf 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java 
e86c1c6 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/2155/diff


Testing (updated)
---

All 1026 unit tests passed.


Thanks,

Mikhail



> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119839#comment-13119839
 ] 

jirapos...@reviews.apache.org commented on HBASE-4534:
--



bq.  On 2011-10-04 00:56:39, Lars Hofhansl wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java, line 
100
bq.  > 
bq.  >
bq.  > On many JVMs (including hotspot) this creates a memory barrier.
bq.  > This might actually cause a measurable slowdown on multi-core 
machines.
bq.  > (And maybe I'm just too paranoid).
bq.  >

I could shut off these counters in production, but we already increment 
AtomicLongs all over the place to collect performance counters. So if this is a 
concern, this is probably best addressed in a separate feature. I don't yet 
know what the solution would be (some custom high-performance non-locking 
counters?) We have also added a lot of per-table/column family metrics in our 
internal branch, and that did not seem to hurt performance at all. A more 
important point, however, is that if we can cut disk IO, a small amount 
additional computation is probably OK. We are already seeing a huge reduction 
in block cache misses after pushing lazy seek to production.

Please let me know if you still want me to get rid of these increments in 
production.


- Mikhail


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2155/#review2297
---


On 2011-10-03 21:59:42, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2155/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 21:59:42)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
all-column, single-column, and multi-column). Also all combinations of Bloom 
filters and compression (NONE vs GZIP) are tested. The unit test flushes 
multiple StoreFiles with disjoint timestamp ranges and runs various types of 
queries against them. Currently we are not testing overlapping timestamp ranges.
bq.  
bq.  
bq.  This addresses bug HBASE-4534.
bq.  https://issues.apache.org/jira/browse/HBASE-4534
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
3ec1962 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
ede42b2 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java a828846 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java 
a0d8abf 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java 
e86c1c6 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2155/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Re-running the test suite. Please do not commit yet.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119822#comment-13119822
 ] 

jirapos...@reviews.apache.org commented on HBASE-4534:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2155/#review2297
---


Cool stuff... See below on potential performance concern.


src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java


On many JVMs (including hotspot) this creates a memory barrier.
This might actually cause a measurable slowdown on multi-core machines.
(And maybe I'm just too paranoid).



- Lars


On 2011-10-03 21:59:42, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2155/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 21:59:42)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
all-column, single-column, and multi-column). Also all combinations of Bloom 
filters and compression (NONE vs GZIP) are tested. The unit test flushes 
multiple StoreFiles with disjoint timestamp ranges and runs various types of 
queries against them. Currently we are not testing overlapping timestamp ranges.
bq.  
bq.  
bq.  This addresses bug HBASE-4534.
bq.  https://issues.apache.org/jira/browse/HBASE-4534
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
3ec1962 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
ede42b2 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java a828846 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java 
a0d8abf 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java 
e86c1c6 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2155/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Re-running the test suite. Please do not commit yet.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119787#comment-13119787
 ] 

Doug Meil commented on HBASE-4533:
--

No problem!

> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch, 
> ops_mgt_HBASE_4533_v2.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119780#comment-13119780
 ] 

Lars Hofhansl commented on HBASE-4536:
--

@Andrew: Agree. There's no point really in setting up #minimumVersion if they 
get blown away by a delete marker... Could even argue it's a bug in the parent.

@Jon: Replication already replicates deletes. And (luckily :) ) it has no 
optimization to skip versions once it found a delete marker. So I think that 
should already work.
Or are you referring to some other method of multi-master replication. In that 
case we'd have invent a new way to let a client retrieve delete markers. I 
think that'd be a different jira.


> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4535) hbase-env.sh in hbase rpm does not set HBASE_CONF_DIR

[Ramya Sunil (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119770#comment-13119770
 ] 

Ramya Sunil commented on HBASE-4535:


+1. Tested it and works fine.

> hbase-env.sh in hbase rpm does not set HBASE_CONF_DIR
> -
>
> Key: HBASE-4535
> URL: https://issues.apache.org/jira/browse/HBASE-4535
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.90.3
>Reporter: Ramya Sunil
>Assignee: Eric Yang
> Attachments: HBASE-4535.patch
>
>
> After a hbase rpm install, hbase-env.sh does not define HBASE_CONF_DIR. This 
> needs to be fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119762#comment-13119762
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/#review2295
---


Getting close Eugene. Suggest a cleaner naming of some methods. Also question 
if passing in a set of coprocessor environments is necessary when a set or 
array of String can be built just as easily before calling the HSL constructor.


src/main/java/org/apache/hadoop/hbase/ClusterStatus.java


masterCoprocessors



src/main/java/org/apache/hadoop/hbase/ClusterStatus.java


Likewise




src/main/java/org/apache/hadoop/hbase/ClusterStatus.java


getMasterCoprocessors()



src/main/java/org/apache/hadoop/hbase/HServerLoad.java


Why do you need this if you have the set of coprocessor environments? 

Turn the question around, why do you need the coprocessor environments if 
you have this set of strings? Build this set and pass it to the HSL constructor.



src/main/java/org/apache/hadoop/hbase/HServerLoad.java


getCoprocessors()



src/main/java/org/apache/hadoop/hbase/HServerLoad.java


See above



src/main/java/org/apache/hadoop/hbase/HServerLoad.java


See above. Construct the set of strings or an array of strings and pass it 
in.



src/main/java/org/apache/hadoop/hbase/HServerLoad.java


getCoprocessors()



src/main/java/org/apache/hadoop/hbase/HServerLoad.java


See above



src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java


getMasterCoprocessors()



src/main/java/org/apache/hadoop/hbase/master/HMaster.java


getCoprocessors()



- Andrew


On 2011-10-03 22:32:46, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2029/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 22:32:46)
bq.  
bq.  
bq.  Review request for hbase and Mingjie Lai.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Proposed fix for HBASE-4070. 
bq.  
bq.  
bq.  This addresses bug HBASE-4070.
bq.  https://issues.apache.org/jira/browse/HBASE-4070
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon 
abeb850 
bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
be6fceb 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
96b763b 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2029/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Eugene
bq.  
bq.



> [Coprocessors] Improve region server metrics to report loaded coprocessors to 
> master
> 
>
> Key: HBASE-4070
> URL: https://issues.apache.org/jira/browse/HBASE-4070
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Mingjie Lai
>Assignee: Eugene Koontz
> Attachments: HBASE-4070.patch, HBASE-4070.patch, HBASE-4070.patch, 
> master-web-ui.jpg, rs-status-web-ui.jpg
>
>
> HBASE-3512 is about listing loaded cp classes at shell. To make it more 
> generic, we need a way to report this piece of information from region to 
> master (or just at region server level). So later on, we can

[jira] [Commented] (HBASE-4527) Fix versioning such that every update is unique

[Todd Lipcon (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119753#comment-13119753
 ] 

Todd Lipcon commented on HBASE-4527:


I don't like exposing the internal TS to the user.. I think when users want to 
implement optimistic concurrency control they should be including their own 
version number as a true column with real data.

> Fix versioning such that every update is unique
> ---
>
> Key: HBASE-4527
> URL: https://issues.apache.org/jira/browse/HBASE-4527
> Project: HBase
>  Issue Type: Wish
>Reporter: stack
>
> I wanted to use checkAndPut but there is a case where the check will not fail 
> though the cell has been updated: if a cell is update with exactly the value 
> it had before, we'll not know its been changed.  hbase-4507 did a checkAndPut 
> where you could pass a timestamp as part of the check so we'd check the cell 
> value AND that the timestamp was the same.
> This would work in most regards but one; an update is done in the same 
> millisecond.  This is generally impossible but in a distributed system where 
> clocks drift and a region can be moved to a server whose clock is retarded, 
> it is within the realm of possibilities that it could happen.  So we should 
> deal.
> One thought is that the version is made for sure unique.  We could make the 
> timestamp wider still so probability of the edits arriving within the same 
> microsecond -- or whatever it is that a double gives you -- would require us 
> to run through a couple of billion universe expand/contract cycles or we 
> could have a monotonically increasing sequence id per millisecond.
> There could be some overlap between this issue and the persisting of rwcc to 
> the filesystem (though not currently as rwcc is implemented).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119754#comment-13119754
 ] 

Jonathan Gray commented on HBASE-4536:
--

We were also discussion today that there are situations (especially in 
multi-master situations) where you want to retain the delete markers for some 
period of time as well.

I would think this would require both a family-level setting (a new one or the 
existing one) and also a read-time option, correct?  As of now, deletes are 
never returned to the client.  You'd have to return them in this case otherwise 
the user would have no idea what is actually there?  I'm not sure it's fair to 
ask a user to understand how our delete tombstones work :)

> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Andrew Purtell (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119749#comment-13119749
 ] 

Andrew Purtell commented on HBASE-4536:
---

bq. 2. Folds this into the parent change. I.e. keep minimum-number-of-versions 
of versions even past the delete marker

This would produce fewer opportunities for user surprise.

> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119748#comment-13119748
 ] 

Jonathan Gray commented on HBASE-4532:
--

Whoo!  +1

> Avoid top row seek by dedicated bloom filter for delete family
> --
>
> Key: HBASE-4532
> URL: https://issues.apache.org/jira/browse/HBASE-4532
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
>
> HBASE-4469 avoids the top row seek operation if row-col bloom filter is 
> enabled. 
> This jira tries to avoid top row seek for all the cases by creating a 
> dedicated bloom filter only for delete family.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4527) Fix versioning such that every update is unique

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119756#comment-13119756
 ] 

Jonathan Gray commented on HBASE-4527:
--

Agreed.

> Fix versioning such that every update is unique
> ---
>
> Key: HBASE-4527
> URL: https://issues.apache.org/jira/browse/HBASE-4527
> Project: HBase
>  Issue Type: Wish
>Reporter: stack
>
> I wanted to use checkAndPut but there is a case where the check will not fail 
> though the cell has been updated: if a cell is update with exactly the value 
> it had before, we'll not know its been changed.  hbase-4507 did a checkAndPut 
> where you could pass a timestamp as part of the check so we'd check the cell 
> value AND that the timestamp was the same.
> This would work in most regards but one; an update is done in the same 
> millisecond.  This is generally impossible but in a distributed system where 
> clocks drift and a region can be moved to a server whose clock is retarded, 
> it is within the realm of possibilities that it could happen.  So we should 
> deal.
> One thought is that the version is made for sure unique.  We could make the 
> timestamp wider still so probability of the edits arriving within the same 
> microsecond -- or whatever it is that a double gives you -- would require us 
> to run through a couple of billion universe expand/contract cycles or we 
> could have a monotonically increasing sequence id per millisecond.
> There could be some overlap between this issue and the persisting of rwcc to 
> the filesystem (though not currently as rwcc is implemented).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119750#comment-13119750
 ] 

Jonathan Gray commented on HBASE-4533:
--

Doug, do you think you could add [book] to the front of your commits or 
something?  I'm doing a lot of repository management and that'd be super 
helpful :)

> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch, 
> ops_mgt_HBASE_4533_v2.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4245) Cluster hangs if RS serving root fails during startup sequence

[Todd Lipcon (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HBASE-4245.


Resolution: Duplicate

> Cluster hangs if RS serving root fails during startup sequence
> --
>
> Key: HBASE-4245
> URL: https://issues.apache.org/jira/browse/HBASE-4245
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.3
>Reporter: Todd Lipcon
>Assignee: Ming Ma
>
> On a large-ish cluster, the following sequence of events was seen to happen:
> - master started, ROOT and META were both unassigned
> - ROOT is assigned to rs01
> - META is assigned to rs02
> - Upon open of META, it writes its location into ROOT on rs01
> - rs01 crashes while appending to its HLog due to some other bug
> - rs02 fails the region open sequence
> - master notices that rs01 has crashed, and enqueues a ServerShutdownHandler
> - ServerShutdownHandler blocks on CatalogTracker.waitForMeta() since ROOT and 
> META are not assigned yet
> - master times out assignment of META, but never succeeds because ROOT 
> location is still marked as rs01
> This causes the cluster to never start up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4527) Fix versioning such that every update is unique

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119747#comment-13119747
 ] 

Jonathan Gray commented on HBASE-4527:
--

Yeah, seems like we could utilize the memstoreTS once we solve those issues and 
then we don't care about the timestamp being the same.  But we'd then have to 
expose RWCC to the API because checkAndPut would need to specify it?  ugh

> Fix versioning such that every update is unique
> ---
>
> Key: HBASE-4527
> URL: https://issues.apache.org/jira/browse/HBASE-4527
> Project: HBase
>  Issue Type: Wish
>Reporter: stack
>
> I wanted to use checkAndPut but there is a case where the check will not fail 
> though the cell has been updated: if a cell is update with exactly the value 
> it had before, we'll not know its been changed.  hbase-4507 did a checkAndPut 
> where you could pass a timestamp as part of the check so we'd check the cell 
> value AND that the timestamp was the same.
> This would work in most regards but one; an update is done in the same 
> millisecond.  This is generally impossible but in a distributed system where 
> clocks drift and a region can be moved to a server whose clock is retarded, 
> it is within the realm of possibilities that it could happen.  So we should 
> deal.
> One thought is that the version is made for sure unique.  We could make the 
> timestamp wider still so probability of the edits arriving within the same 
> microsecond -- or whatever it is that a double gives you -- would require us 
> to run through a couple of billion universe expand/contract cycles or we 
> could have a monotonically increasing sequence id per millisecond.
> There could be some overlap between this issue and the persisting of rwcc to 
> the filesystem (though not currently as rwcc is implemented).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4535) hbase-env.sh in hbase rpm does not set HBASE_CONF_DIR

[Eric Yang (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated HBASE-4535:
-

Attachment: HBASE-4535.patch

Set HBASE_CONF_DIR in hbase-env.sh.

> hbase-env.sh in hbase rpm does not set HBASE_CONF_DIR
> -
>
> Key: HBASE-4535
> URL: https://issues.apache.org/jira/browse/HBASE-4535
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.90.3
>Reporter: Ramya Sunil
> Attachments: HBASE-4535.patch
>
>
> After a hbase rpm install, hbase-env.sh does not define HBASE_CONF_DIR. This 
> needs to be fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4535) hbase-env.sh in hbase rpm does not set HBASE_CONF_DIR

[Eric Yang (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated HBASE-4535:
-

Assignee: Eric Yang
  Status: Patch Available  (was: Open)

> hbase-env.sh in hbase rpm does not set HBASE_CONF_DIR
> -
>
> Key: HBASE-4535
> URL: https://issues.apache.org/jira/browse/HBASE-4535
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.90.3
>Reporter: Ramya Sunil
>Assignee: Eric Yang
> Attachments: HBASE-4535.patch
>
>
> After a hbase rpm install, hbase-env.sh does not define HBASE_CONF_DIR. This 
> needs to be fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4524) support for more than one region in .META. table

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119745#comment-13119745
 ] 

Jonathan Gray commented on HBASE-4524:
--

+1 removing root and putting META location into zk.

-1 on thinking about splittable META right now.

+1 thinking about mirroring META in ZK, adding new locations/redirects in 
NSREs, and other optimizations and availability improvements

> support for more than one region in .META. table
> 
>
> Key: HBASE-4524
> URL: https://issues.apache.org/jira/browse/HBASE-4524
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Ming Ma
>
> It seems there are some assumptions in the code that .META. table only has 
> one region FIRST_META_REGIONINFO in the following areas:
> 1) .META. table update with user region info.
> 2) .META. regions assignment.
> 3) .META. table split handling.
> Perhaps we don't have such requirement until we scale to really large number 
> of regions like 1M.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4334) HRegion.get never validates row

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119741#comment-13119741
 ] 

Hudson commented on HBASE-4334:
---

Integrated in HBase-0.92 #41 (See 
[https://builds.apache.org/job/HBase-0.92/41/])
HBASE-4334 HRegion.get never validates row (Lars H)

tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java


> HRegion.get never validates row
> ---
>
> Key: HBASE-4334
> URL: https://issues.apache.org/jira/browse/HBASE-4334
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: Joe Pallas
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4334-v2.txt, 4334-v3.txt, 4334.txt
>
>
> If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
> may send a request to the wrong region.  Paths through put, delete, 
> incrementColumnValue, and checkAndMutate all call checkRow either directly or 
> indirectly (through getLock).  But get apparently does not.  This can result 
> in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4473) NPE when executors are down but events are still coming in

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119713#comment-13119713
 ] 

Hudson commented on HBASE-4473:
---

Integrated in HBase-0.92 #40 (See 
[https://builds.apache.org/job/HBase-0.92/40/])
HBASE-4473  NPE when executors are down but events are still coming in

jdcryans : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/executor/TestExecutorService.java


> NPE when executors are down but events are still coming in
> --
>
> Key: HBASE-4473
> URL: https://issues.apache.org/jira/browse/HBASE-4473
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Minor
> Fix For: 0.90.5
>
> Attachments: HBASE-4473-0.90.patch, HBASE-4473-v2.patch, 
> HBASE-4473.patch
>
>
> Minor annoyance when shutting down a cluster and the master is still 
> receiving events from Zookeeper:
> {quote}
> 2011-09-22 23:53:01,552 DEBUG 
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> master:6-0x3292d87deb004f Received InterruptedException, doing nothing 
> here
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:726)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:938)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:407)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteOpenedNode(ZKAssign.java:284)
> at 
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler.process(OpenedRegionHandler.java:88)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ...
> 2011-09-22 23:53:01,558 DEBUG 
> org.apache.hadoop.hbase.executor.ExecutorService: Executor service 
> [MASTER_OPEN_REGION-sv2borg170:6] not found in {}
> 2011-09-22 23:53:01,558 ERROR org.apache.zookeeper.ClientCnxn: Error while 
> calling watcher
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:220)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:447)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.nodeDataChanged(AssignmentManager.java:546)
> at 
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at 
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
> at 
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
> {quote}
> It's annoying because it then spams you with a bunch of NPEs that have 
> nothing to do with the reason the Master is shutting down. Googling I saw 
> someone also had that issue in June: http://pastebin.com/5Tqrj0nq

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119714#comment-13119714
 ] 

Hudson commented on HBASE-4531:
---

Integrated in HBase-0.92 #40 (See 
[https://builds.apache.org/job/HBase-0.92/40/])
HBASE-4531 hbase-4454 failsafe broke mvn site; back it out or fix

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/pom.xml


> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Akash Ashok
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531-v2.patch, 
> HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4454) Add failsafe plugin to build and rename integration tests

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119715#comment-13119715
 ] 

Hudson commented on HBASE-4454:
---

Integrated in HBase-0.92 #40 (See 
[https://builds.apache.org/job/HBase-0.92/40/])
HBASE-4531 hbase-4454 failsafe broke mvn site; back it out or fix


> Add failsafe plugin to build and rename integration tests
> -
>
> Key: HBASE-4454
> URL: https://issues.apache.org/jira/browse/HBASE-4454
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jesse Yates
> Fix For: 0.92.0
>
> Attachments: mvn_HBASE-4454.patch
>
>
> Add the maven-failsafe-plugin to the build process so we can run integration 
> tests with "mvn verify". This will also involve a renaming of integration 
> tests to conform to a new integration test regex.
> This is a stopgap measure while we until break them out into their own module.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4334) HRegion.get never validates row

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119710#comment-13119710
 ] 

Ted Yu commented on HBASE-4334:
---

Integrated to 0.92 and TRUNK.

Thanks for the patch Lars.

Thanks for the review Michael.

> HRegion.get never validates row
> ---
>
> Key: HBASE-4334
> URL: https://issues.apache.org/jira/browse/HBASE-4334
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: Joe Pallas
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4334-v2.txt, 4334-v3.txt, 4334.txt
>
>
> If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
> may send a request to the wrong region.  Paths through put, delete, 
> incrementColumnValue, and checkAndMutate all call checkRow either directly or 
> indirectly (through getLock).  But get apparently does not.  This can result 
> in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119727#comment-13119727
 ] 

Jonathan Hsieh commented on HBASE-4377:
---

When backporting to 0.90, the TestOfflineMetaRebuild test case would fail out 
due to out of file handles exceptions.  I dug for a while and found that the 
static HConnections cached connections that are not flushed between tests.  
Even after avoiding that there are other resources (maybe pooling on hdfs 
client or zk client connections?) that cause the open file handles count to 
increase significantly after every test case.  

To avoid this problem, I'm going to split out the each rebuild tests into own 
test case so that each can be executed in a new process and avoid the out of 
file handles problem.  I'll do this for trunk and for the 0.90 backport.

> [hbck] Offline rebuild .META. from fs data only.
> 
>
> Key: HBASE-4377
> URL: https://issues.apache.org/jira/browse/HBASE-4377
> Project: HBase
>  Issue Type: New Feature
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch, 
> hbase-4377-trunk.v2.patch
>
>
> In a worst case situation, it may be helpful to have an offline .META. 
> rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
> from scratch.  Users could move bad regions out until there is a clean 
> rebuild.  
> It would likely fill in region split holes.  Follow on work could given 
> options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

[Eugene Koontz (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koontz updated HBASE-4070:
-

Attachment: HBASE-4070.patch

Identical to https://reviews.apache.org/r/2029/diff/5/

> [Coprocessors] Improve region server metrics to report loaded coprocessors to 
> master
> 
>
> Key: HBASE-4070
> URL: https://issues.apache.org/jira/browse/HBASE-4070
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Mingjie Lai
>Assignee: Eugene Koontz
> Attachments: HBASE-4070.patch, HBASE-4070.patch, HBASE-4070.patch, 
> master-web-ui.jpg, rs-status-web-ui.jpg
>
>
> HBASE-3512 is about listing loaded cp classes at shell. To make it more 
> generic, we need a way to report this piece of information from region to 
> master (or just at region server level). So later on, we can display the 
> loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3512) Coprocessors: Shell support for listing currently loaded coprocessor set

[Eugene Koontz (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koontz updated HBASE-3512:
-

Attachment: HBASE-3512.patch

patch against trunk.

> Coprocessors: Shell support for listing currently loaded coprocessor set
> 
>
> Key: HBASE-3512
> URL: https://issues.apache.org/jira/browse/HBASE-3512
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Reporter: Andrew Purtell
>Assignee: Eugene Koontz
> Fix For: 0.92.0
>
> Attachments: HBASE-3512-only.patch, HBASE-3512-only.patch, 
> HBASE-3512.patch, HBASE-3512.patch, hbase-shell-session.txt
>
>
> Add support to the shell for listing the coprocessors loaded globally on the 
> regionserver and those loaded on a per-table basis.
> Perhaps by extending the 'status' command.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3512) Coprocessors: Shell support for listing currently loaded coprocessor set

[Eugene Koontz (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koontz updated HBASE-3512:
-

Attachment: HBASE-3512-only.patch

patch against trunk+HBASE-4070 patch.

> Coprocessors: Shell support for listing currently loaded coprocessor set
> 
>
> Key: HBASE-3512
> URL: https://issues.apache.org/jira/browse/HBASE-3512
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Reporter: Andrew Purtell
>Assignee: Eugene Koontz
> Fix For: 0.92.0
>
> Attachments: HBASE-3512-only.patch, HBASE-3512-only.patch, 
> HBASE-3512.patch, HBASE-3512.patch, hbase-shell-session.txt
>
>
> Add support to the shell for listing the coprocessors loaded globally on the 
> regionserver and those loaded on a per-table basis.
> Perhaps by extending the 'status' command.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119698#comment-13119698
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/
---

(Updated 2011-10-03 22:32:46.964479)


Review request for hbase and Mingjie Lai.


Changes
---

use String[] as the return type to report loaded coprocessors.


Summary
---

Proposed fix for HBASE-4070. 


This addresses bug HBASE-4070.
https://issues.apache.org/jira/browse/HBASE-4070


Diffs (updated)
-

  src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 
  src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
  src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 
  src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/2029/diff


Testing
---

Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.


Thanks,

Eugene



> [Coprocessors] Improve region server metrics to report loaded coprocessors to 
> master
> 
>
> Key: HBASE-4070
> URL: https://issues.apache.org/jira/browse/HBASE-4070
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Mingjie Lai
>Assignee: Eugene Koontz
> Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, 
> rs-status-web-ui.jpg
>
>
> HBASE-3512 is about listing loaded cp classes at shell. To make it more 
> generic, we need a way to report this piece of information from region to 
> master (or just at region server level). So later on, we can display the 
> loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119677#comment-13119677
 ] 

jirapos...@reviews.apache.org commented on HBASE-4422:
--



bq.  On 2011-10-03 20:39:25, Lars Hofhansl wrote:
bq.  > /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java, line 64
bq.  > 
bq.  >
bq.  > That'd be pretty awesome :)

I have it implemented and tested on an internal branch.  It's not too bad once 
CacheConfig is in.

Also, some pretty awesome Delta Encoding stuff should be coming in the pipeline 
soon!  Can't wait


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2089/#review2274
---


On 2011-10-03 18:19:34, Jonathan Gray wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2089/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 18:19:34)
bq.  
bq.  
bq.  Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Creates a new CacheConfig class and moves almost everything block cache 
related into this single class.  Adding new configuration params and booleans 
and such should be much better.
bq.  
bq.  All tests are NOT passing yet, still working on it, but wanted to have 
something up today.  Basically "code complete" but broken :)
bq.  
bq.  
bq.  This addresses bug HBASE-4422.
bq.  https://issues.apache.org/jira/browse/HBASE-4422
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1178474 
bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1178474 
bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 
1178474 
bq.
/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178474 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 
PRE-CREATION 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
 1178474 
bq.
/src/test/java/org/apache/had

[jira] [Commented] (HBASE-4520) Better handling of Bloom filter type discrepancy between HFile and CF config

[Mikhail Bautin (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119668#comment-13119668
 ] 

Mikhail Bautin commented on HBASE-4520:
---

I believe there are no new unit test failures with the patch. The only test 
that fails is HMasterRPCException which also fails without the patch. Can 
someone please commit this one?

> Better handling of Bloom filter type discrepancy between HFile and CF config
> 
>
> Key: HBASE-4520
> URL: https://issues.apache.org/jira/browse/HBASE-4520
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Fix For: 0.94.0
>
>
> Modify StoreFile to make it clear where Bloom filter type settings come from. 
> We have two sources of truth: (1) HFile; and (2) CF configuration. (1) takes 
> precedence in the reader, and (2) takes precedence in the writer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4536) Allow CF to retain deleted rows

[Lars Hofhansl (Created) (JIRA)]

Allow CF to retain deleted rows
---

 Key: HBASE-4536
 URL: https://issues.apache.org/jira/browse/HBASE-4536
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Affects Versions: 0.92.0
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.92.0, 0.94.0


Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
of versions.
However, if a client deletes a row all version older than the delete tomb stone 
will be remove at the next major compaction (and even at memstore flush - see 
HBASE-4241).
There should be a way to retain those version to guard against software error.

I see two options here:
1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
versions even past the delete marker.

#1 would allow for more flexibility. #2 comes somewhat naturally with parent 
(from a user viewpoint)

Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4535) hbase-env.sh in hbase rpm does not set HBASE_CONF_DIR

[Ramya Sunil (Created) (JIRA)]

hbase-env.sh in hbase rpm does not set HBASE_CONF_DIR
-

 Key: HBASE-4535
 URL: https://issues.apache.org/jira/browse/HBASE-4535
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 0.90.3
Reporter: Ramya Sunil


After a hbase rpm install, hbase-env.sh does not define HBASE_CONF_DIR. This 
needs to be fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119667#comment-13119667
 ] 

jirapos...@reviews.apache.org commented on HBASE-4534:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2155/
---

Review request for hbase.


Summary
---

A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
all-column, single-column, and multi-column). Also all combinations of Bloom 
filters and compression (NONE vs GZIP) are tested. The unit test flushes 
multiple StoreFiles with disjoint timestamp ranges and runs various types of 
queries against them. Currently we are not testing overlapping timestamp ranges.


This addresses bug HBASE-4534.
https://issues.apache.org/jira/browse/HBASE-4534


Diffs
-

  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
3ec1962 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ede42b2 
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java a828846 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java 
a0d8abf 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java 
e86c1c6 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/2155/diff


Testing
---

Re-running the test suite. Please do not commit yet.


Thanks,

Mikhail



> A new unit test for lazy seek and StoreScanner in general
> -
>
> Key: HBASE-4534
> URL: https://issues.apache.org/jira/browse/HBASE-4534
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
> all-column, single-column, and multi-column). Also all combinations of Bloom 
> filters and compression (NONE vs GZIP) are tested. The unit test flushes 
> multiple StoreFiles with disjoint timestamp ranges and runs various types of 
> queries against them. Currently we are not testing overlapping timestamp 
> ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4386) NPE in TaskMonitor

[Todd Lipcon (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-4386:
---

Status: Patch Available  (was: Open)

> NPE in TaskMonitor
> --
>
> Key: HBASE-4386
> URL: https://issues.apache.org/jira/browse/HBASE-4386
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: hbase-4386.txt
>
>
> Saw the following hitting /rs-status
> INTERNAL_SERVER_ERRORCaused 
> by:java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.purgeExpiredTasks(TaskMonitor.java:97)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.getTasks(TaskMonitor.java:127)
> at 
> org.apache.hbase.tmpl.common.TaskMonitorTmplImpl.renderNoFlush(TaskMonitorTmplImpl.java:50)
> at 
> org.apache.hbase.tmpl.common.TaskMonitorTmpl.renderNoFlush(TaskMonitorTmpl.java:170)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:70)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:176)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:167)
> at 
> org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4386) NPE in TaskMonitor

[Todd Lipcon (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-4386:
---

Attachment: hbase-4386.txt

> NPE in TaskMonitor
> --
>
> Key: HBASE-4386
> URL: https://issues.apache.org/jira/browse/HBASE-4386
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: hbase-4386.txt
>
>
> Saw the following hitting /rs-status
> INTERNAL_SERVER_ERRORCaused 
> by:java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.purgeExpiredTasks(TaskMonitor.java:97)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.getTasks(TaskMonitor.java:127)
> at 
> org.apache.hbase.tmpl.common.TaskMonitorTmplImpl.renderNoFlush(TaskMonitorTmplImpl.java:50)
> at 
> org.apache.hbase.tmpl.common.TaskMonitorTmpl.renderNoFlush(TaskMonitorTmpl.java:170)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:70)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:176)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:167)
> at 
> org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119650#comment-13119650
 ] 

Ted Yu commented on HBASE-4482:
---

+1 on this simplification.

> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[Todd Lipcon (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119649#comment-13119649
 ] 

Todd Lipcon commented on HBASE-4482:


I was thinking about this a bit over the weekend. Though the current 
composition of SingleSizeCaches into SlabCache is pretty, it's causing a lot of 
these bugs regarding coordination between the two.

I'd like to consider a simplification here - what if we got rid of 
SingleSizeCache, and manually kept the LRU lists per size? Basically, dial back 
the concurrency a bit in favor of a simpler design, then if we see bottlenecks 
or lock contention, we can address it.

> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general

[Mikhail Bautin (Created) (JIRA)]

A new unit test for lazy seek and StoreScanner in general
-

 Key: HBASE-4534
 URL: https://issues.apache.org/jira/browse/HBASE-4534
 Project: HBase
  Issue Type: Test
Affects Versions: 0.94.0
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin


A randomized unit test for Gets/Scans (all-row, single-row, multi-row, 
all-column, single-column, and multi-column). Also all combinations of Bloom 
filters and compression (NONE vs GZIP) are tested. The unit test flushes 
multiple StoreFiles with disjoint timestamp ranges and runs various types of 
queries against them. Currently we are not testing overlapping timestamp ranges.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4241) Optimize flushing of the Store cache for max versions and (new) min versions

[Liyin Tang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119629#comment-13119629
 ] 

Liyin Tang commented on HBASE-4241:
---

Hi Lars, Thanks for your patch. I am trying to back port this feature for 
hbase-89
I have a quick question:) 

why we use CollectionBackedScanner but not reuse memstore scanner?

Thanks a lot

> Optimize flushing of the Store cache for max versions and (new) min versions
> 
>
> Key: HBASE-4241
> URL: https://issues.apache.org/jira/browse/HBASE-4241
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4241-v2.txt, 4241-v8.txt, 4241.txt
>
>
> As discussed with with Jon, there is room for improvement in how the memstore 
> is flushed to disk.
> Currently only expired KVs are pruned before flushing, but we can also prune 
> versions if we find at least maxVersions versions in the memstore.
> The same holds for the new minversion feature: If we find at least minVersion 
> versions in the store we can remove all further versions that are expired.
> Generally we should use the same mechanism here that is used for Compaction. 
> I.e. StoreScanner. We only need to add a scanner to Memstore that can scan 
> along the current snapshot.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[Li Pi (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119615#comment-13119615
 ] 

Li Pi commented on HBASE-4482:
--

Uh - thats not quite fixed yet, theres another race condition this
patch causes, after getting rid of one

Somehow, somewhere, the cache invariant is getting broken. Eventually
I'll figure out why.

On Mon, Oct 3, 2011 at 1:00 PM, jirapos...@reviews.apache.org


> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4533:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

now it's fixed

> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch, 
> ops_mgt_HBASE_4533_v2.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4533:
-

Status: Patch Available  (was: Reopened)

> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch, 
> ops_mgt_HBASE_4533_v2.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4533:
-

Attachment: ops_mgt_HBASE_4533_v2.xml.patch

There was one xml-id that was incorrect in my tweak, calling foul on myself.  
Reopening and re-attaching 2nd patch.

> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch, 
> ops_mgt_HBASE_4533_v2.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4533:
-

Attachment: ops_mgt_HBASE_4533.xml.patch

> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4402) Retaining locality after restart broken

[Todd Lipcon (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119596#comment-13119596
 ] 

Todd Lipcon commented on HBASE-4402:


Since we now have locality metrics on regions, it seems like we can already see 
that information elsewhere. The log output here is helpful since you can 
quickly see which servers didn't properly rejoin the cluster.

> Retaining locality after restart broken
> ---
>
> Key: HBASE-4402
> URL: https://issues.apache.org/jira/browse/HBASE-4402
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4402.txt, hbase-4402.txt
>
>
> In DefaultLoadBalancer, we implement the "retain assignment" function like so:
> {code}
>   if (sn != null && servers.contains(sn)) {
> assignments.get(sn).add(region.getKey());
> {code}
> but this will never work since after a cluster restart, all servers have a 
> new ServerName with a new startcode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4533:
-

Status: Patch Available  (was: Open)

> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Reopened) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil reopened HBASE-4533:
--


> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4533:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> ops_mgt.xml - tweaks to backup section
> --
>
> Key: HBASE-4533
> URL: https://issues.apache.org/jira/browse/HBASE-4533
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4533.xml.patch
>
>
> Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4533) ops_mgt.xml - tweaks to backup section

[Doug Meil (Created) (JIRA)]

ops_mgt.xml - tweaks to backup section
--

 Key: HBASE-4533
 URL: https://issues.apache.org/jira/browse/HBASE-4533
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor


Minor tweaks to backup section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4334) HRegion.get never validates row

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119586#comment-13119586
 ] 

Lars Hofhansl commented on HBASE-4334:
--

Probably... Maybe let it bake in 0.92/trunk for a bit?

> HRegion.get never validates row
> ---
>
> Key: HBASE-4334
> URL: https://issues.apache.org/jira/browse/HBASE-4334
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: Joe Pallas
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4334-v2.txt, 4334-v3.txt, 4334.txt
>
>
> If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
> may send a request to the wrong region.  Paths through put, delete, 
> incrementColumnValue, and checkAndMutate all call checkRow either directly or 
> indirectly (through getLock).  But get apparently does not.  This can result 
> in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119580#comment-13119580
 ] 

jirapos...@reviews.apache.org commented on HBASE-4422:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2089/#review2274
---


Looks good from here... I'll do some testing with the patch applied.


/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java


That'd be pretty awesome :)


- Lars


On 2011-10-03 18:19:34, Jonathan Gray wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2089/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 18:19:34)
bq.  
bq.  
bq.  Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Creates a new CacheConfig class and moves almost everything block cache 
related into this single class.  Adding new configuration params and booleans 
and such should be much better.
bq.  
bq.  All tests are NOT passing yet, still working on it, but wanted to have 
something up today.  Basically "code complete" but broken :)
bq.  
bq.  
bq.  This addresses bug HBASE-4422.
bq.  https://issues.apache.org/jira/browse/HBASE-4422
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1178474 
bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1178474 
bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 
1178474 
bq.
/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178474 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java 
PRE-CREATION 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1178474 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
 1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 
1178474 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1178474 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
 117

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119572#comment-13119572
 ] 

Ted Yu commented on HBASE-4528:
---

I was referring to possible exception coming out of log.sync() call.

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[dhruba borthakur (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119569#comment-13119569
 ] 

dhruba borthakur commented on HBASE-4528:
-

Hi Ted,

> Should we guard the rwcc call in the finally block ?

The method rwcc.completeMemstoreInsert() does not throw an exceptions, so the 
case you refer to above should not arise, isn't it?

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119553#comment-13119553
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/#review2283
---


This looks good but for the issues the boys raised above.  You going to fix Li 
or should I try and fix on commit?

- Michael


On 2011-09-26 19:17:34, Li Pi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2051/
bq.  ---
bq.  
bq.  (Updated 2011-09-26 19:17:34)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Bugfix, kills a race condition.
bq.  
bq.  Ignore r1, thats the wrong patch.
bq.  
bq.  
bq.  This addresses bug HBASE-4482.
bq.  https://issues.apache.org/jira/browse/HBASE-4482
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 
fe8b95a 
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
0c06f4f 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
fd9e7ef 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 
0814f41 
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 
bq.  
bq.  Diff: https://reviews.apache.org/r/2051/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Looped tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Li
bq.  
bq.



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4448) HBaseTestingUtilityFactory - pattern for re-using HBaseTestingUtility instances across unit tests

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119526#comment-13119526
 ] 

stack commented on HBASE-4448:
--

bq. ...This gets us faster running tests - you don't teardown and create a new 
jvm each time and things will more easily actually run in parallel (I think on 
the backend it just fires up a bunch of threads in the same jvm and starts 
ticking off tests)

Sure.  I just think this will take a bunch of effort to do this and even then, 
going by my experience with running the surefire in parallel, I think it'll be 
a while before that works properly... so I don't see this happening soon.  I 
was hoping for some speedups before that.

bq. ...but I think we can also parallelize the work among several build servers 

I was hoping to speed up tests for all users, not just apache build

bq. I just want to make sure we don't start slamming things together just so we 
cut down on creating clusters

I should have added the qualification that we aggregate together tests that 
properly cohere (as you say later on the above).

bq. I would rather a slower build and more confidence in the test coverage.

Agreed

> HBaseTestingUtilityFactory - pattern for re-using HBaseTestingUtility 
> instances across unit tests
> -
>
> Key: HBASE-4448
> URL: https://issues.apache.org/jira/browse/HBASE-4448
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: HBaseTestingUtilityFactory.java, 
> hbase_hbaseTestingUtility_uses_2011_09_22.xlsx, java_HBASE_4448.patch, 
> java_HBASE_4448_v2.patch
>
>
> Setting up and tearing down HBaseTestingUtility instances in unit tests is 
> very expensive.  On my MacBook it takes about 10 seconds to set up a 
> MiniCluster, and 7 seconds to tear it down.  When multiplied by the number of 
> test classes that use this facility, that's a lot of time in the build.
> This factory assumes that the JVM is being re-used across test classes in the 
> build, otherwise this pattern won't work. 
> I don't think this is appropriate for every use, but I think it can be 
> applicable in a great many cases - especially where developers just want a 
> simple MiniCluster with 1 slave.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4526) special case for stopping master in hbase-daemon.sh is no longer required

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119515#comment-13119515
 ] 

Hudson commented on HBASE-4526:
---

Integrated in HBase-0.92 #39 (See 
[https://builds.apache.org/job/HBase-0.92/39/])
HBASE-4526 special case for stopping master in hbase-daemon.sh is no longer 
required

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/bin/hbase-daemon.sh


> special case for stopping master in hbase-daemon.sh is no longer required
> -
>
> Key: HBASE-4526
> URL: https://issues.apache.org/jira/browse/HBASE-4526
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 0.90.4
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4526.patch.txt
>
>
> Now that HBASE-4209 is finally done (many thanks to stack for help and 
> encouragement!) I don't think there's any reason to keep this bit of code in 
> hbase-daemon.sh:
> {noformat}
> if [ "$command" = "master" ]; then
>   echo "`date` Killing $command" >> $loglog
>   kill -9 `cat $pid` > /dev/null 2>&1
> else
>   echo "`date` Killing $command" >> $loglog
>   kill `cat $pid` > /dev/null 2>&1
> fi
> {noformat}
> I suggest we remove the special case completely, since I don't think it 
> serves any useful purpose (patch attached).
> Now, as an additional precautionary measure. We can try to follow up with a 
> SIGKILL once a certain timeout expires.
> Let me know if you think it is necessary to do so and I'll update the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4473) NPE when executors are down but events are still coming in

[Jean-Daniel Cryans (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans resolved HBASE-4473.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

Committed to 0.90, 0.92 and trunk.

> NPE when executors are down but events are still coming in
> --
>
> Key: HBASE-4473
> URL: https://issues.apache.org/jira/browse/HBASE-4473
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Minor
> Fix For: 0.90.5
>
> Attachments: HBASE-4473-0.90.patch, HBASE-4473-v2.patch, 
> HBASE-4473.patch
>
>
> Minor annoyance when shutting down a cluster and the master is still 
> receiving events from Zookeeper:
> {quote}
> 2011-09-22 23:53:01,552 DEBUG 
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> master:6-0x3292d87deb004f Received InterruptedException, doing nothing 
> here
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:726)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:938)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:407)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteOpenedNode(ZKAssign.java:284)
> at 
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler.process(OpenedRegionHandler.java:88)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ...
> 2011-09-22 23:53:01,558 DEBUG 
> org.apache.hadoop.hbase.executor.ExecutorService: Executor service 
> [MASTER_OPEN_REGION-sv2borg170:6] not found in {}
> 2011-09-22 23:53:01,558 ERROR org.apache.zookeeper.ClientCnxn: Error while 
> calling watcher
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:220)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:447)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.nodeDataChanged(AssignmentManager.java:546)
> at 
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at 
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
> at 
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
> {quote}
> It's annoying because it then spams you with a bunch of NPEs that have 
> nothing to do with the reason the Master is shutting down. Googling I saw 
> someone also had that issue in June: http://pastebin.com/5Tqrj0nq

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4473) NPE when executors are down but events are still coming in

[Jean-Daniel Cryans (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans updated HBASE-4473:
--

Attachment: HBASE-4473-v2.patch

Attaching the patch I committed with Stack's comment.

> NPE when executors are down but events are still coming in
> --
>
> Key: HBASE-4473
> URL: https://issues.apache.org/jira/browse/HBASE-4473
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Minor
> Fix For: 0.90.5
>
> Attachments: HBASE-4473-0.90.patch, HBASE-4473-v2.patch, 
> HBASE-4473.patch
>
>
> Minor annoyance when shutting down a cluster and the master is still 
> receiving events from Zookeeper:
> {quote}
> 2011-09-22 23:53:01,552 DEBUG 
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> master:6-0x3292d87deb004f Received InterruptedException, doing nothing 
> here
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1317)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:726)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:938)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:407)
> at 
> org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteOpenedNode(ZKAssign.java:284)
> at 
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler.process(OpenedRegionHandler.java:88)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ...
> 2011-09-22 23:53:01,558 DEBUG 
> org.apache.hadoop.hbase.executor.ExecutorService: Executor service 
> [MASTER_OPEN_REGION-sv2borg170:6] not found in {}
> 2011-09-22 23:53:01,558 ERROR org.apache.zookeeper.ClientCnxn: Error while 
> calling watcher
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.executor.ExecutorService.submit(ExecutorService.java:220)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:447)
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.nodeDataChanged(AssignmentManager.java:546)
> at 
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:281)
> at 
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
> at 
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
> {quote}
> It's annoying because it then spams you with a bunch of NPEs that have 
> nothing to do with the reason the Master is shutting down. Googling I saw 
> someone also had that issue in June: http://pastebin.com/5Tqrj0nq

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4334) HRegion.get never validates row

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4334:
-

Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

+1 on patch

> HRegion.get never validates row
> ---
>
> Key: HBASE-4334
> URL: https://issues.apache.org/jira/browse/HBASE-4334
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: Joe Pallas
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4334-v2.txt, 4334-v3.txt, 4334.txt
>
>
> If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
> may send a request to the wrong region.  Paths through put, delete, 
> incrementColumnValue, and checkAndMutate all call checkRow either directly or 
> indirectly (through getLock).  But get apparently does not.  This can result 
> in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119492#comment-13119492
 ] 

stack commented on HBASE-4532:
--

I like this idea.  Nice one Liyin.

> Avoid top row seek by dedicated bloom filter for delete family
> --
>
> Key: HBASE-4532
> URL: https://issues.apache.org/jira/browse/HBASE-4532
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
>
> HBASE-4469 avoids the top row seek operation if row-col bloom filter is 
> enabled. 
> This jira tries to avoid top row seek for all the cases by creating a 
> dedicated bloom filter only for delete family.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4334) HRegion.get never validates row

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119491#comment-13119491
 ] 

stack commented on HBASE-4334:
--

Should this go into 0.90 too?

> HRegion.get never validates row
> ---
>
> Key: HBASE-4334
> URL: https://issues.apache.org/jira/browse/HBASE-4334
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: Joe Pallas
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4334-v2.txt, 4334-v3.txt, 4334.txt
>
>
> If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
> may send a request to the wrong region.  Paths through put, delete, 
> incrementColumnValue, and checkAndMutate all call checkRow either directly or 
> indirectly (through getLock).  But get apparently does not.  This can result 
> in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class

[Mikhail Bautin (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119479#comment-13119479
 ] 

Mikhail Bautin commented on HBASE-4422:
---

For some reason the most recent version of the patch does not apply for me (one 
hunk gets rejected in StoreFile.java). The way I am trying to apply the patch is

curl https://reviews.apache.org/r/2089/diff/4/raw/ | patch -p0

and I get http://pastebin.com/8aZ5r9Pf. My current revision is 1176613.


> Move block cache parameters and references into single CacheConf class
> --
>
> Key: HBASE-4422
> URL: https://issues.apache.org/jira/browse/HBASE-4422
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Fix For: 0.92.0
>
>
> From StoreFile down to HFile, we currently use a boolean argument for each of 
> the various block cache configuration parameters that exist.  The number of 
> parameters is going to continue to increase as we look at compressed cache, 
> delta encoding, and more specific L1/L2 configuration.  Every new config 
> currently requires changing many constructors because it introduces a new 
> boolean.
> We should move everything into a single class so that modifications are much 
> less disruptive.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4531:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I also tried it.  Committed 0.92 branch and trunk.  Thank you for the fix Akash.

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Akash Ashok
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531-v2.patch, 
> HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[stack (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reassigned HBASE-4531:


Assignee: Akash Ashok

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Akash Ashok
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531-v2.patch, 
> HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Akash Ashok (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119474#comment-13119474
 ] 

Akash Ashok commented on HBASE-4531:


Sorry attached the same file twice. It said attaching file failed the first 
time but then it had already attached

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531-v2.patch, 
> HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Akash Ashok (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash Ashok updated HBASE-4531:
---

Attachment: HBase-4531-v2.patch

Pluggin Configuration added to MavenSitePlugin 2.1. Downgrading 
surefire-reports-plugin not needed

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531-v2.patch, 
> HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Akash Ashok (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash Ashok updated HBASE-4531:
---

Attachment: HBase-4531-v2.patch

Plugin configuration was missing. Instead of downgrading the 
surefire-reports-plugin version this is better

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531-v2.patch, HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119466#comment-13119466
 ] 

Jonathan Hsieh commented on HBASE-4377:
---

Since the review for trunk had relatively minor issues, I'm going to work on 
re-backporting this to the 0.90 branch.

> [hbck] Offline rebuild .META. from fs data only.
> 
>
> Key: HBASE-4377
> URL: https://issues.apache.org/jira/browse/HBASE-4377
> Project: HBase
>  Issue Type: New Feature
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch, 
> hbase-4377-trunk.v2.patch
>
>
> In a worst case situation, it may be helpful to have an offline .META. 
> rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
> from scratch.  Users could move bad regions out until there is a clean 
> rebuild.  
> It would likely fill in region split holes.  Follow on work could given 
> options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119465#comment-13119465
 ] 

jirapos...@reviews.apache.org commented on HBASE-4422:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2089/
---

(Updated 2011-10-03 18:19:34.483689)


Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi.


Changes
---

Rebased following the commit of HBASE-4496.  Appreciate if people can download 
the patch and run the test suite and report back what new stuff is failing for 
you, getting some inconsistent results.


Summary
---

Creates a new CacheConfig class and moves almost everything block cache related 
into this single class.  Adding new configuration params and booleans and such 
should be much better.

All tests are NOT passing yet, still working on it, but wanted to have 
something up today.  Basically "code complete" but broken :)


This addresses bug HBASE-4422.
https://issues.apache.org/jira/browse/HBASE-4422


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1178474 
  /src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1178474 
  /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 
1178474 
  /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
1178474 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1178474 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 
1178474 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1178474 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1178474 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1178474 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1178474 
  /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 
1178474 
  
/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 
1178474 
  
/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
 1178474 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 
1178474 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1178474 
  
/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
 1178474 
  /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 
1178474 

Diff: https://reviews.apache.org/r/2089/diff


Testing
---

Still working through some tests that aren't passing.


Thanks,

Jonathan



> Move block cache parameters and references into single CacheConf class
> --
>
> Key: HBASE-4422
> URL: https://issues.apache.org/jira/browse/HBASE-4422
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>

[jira] [Updated] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[Jonathan Hsieh (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4377:
--

Attachment: hbase-4377-trunk.v2.patch

trunk version applies on 0.92 and trunk.

> [hbck] Offline rebuild .META. from fs data only.
> 
>
> Key: HBASE-4377
> URL: https://issues.apache.org/jira/browse/HBASE-4377
> Project: HBase
>  Issue Type: New Feature
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch, 
> hbase-4377-trunk.v2.patch
>
>
> In a worst case situation, it may be helpful to have an offline .META. 
> rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
> from scratch.  Users could move bad regions out until there is a clean 
> rebuild.  
> It would likely fill in region split holes.  Follow on work could given 
> options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119461#comment-13119461
 ] 

jirapos...@reviews.apache.org commented on HBASE-4377:
--



bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 
376
bq.  > 
bq.  >
bq.  > b is not needed here, same with question mark.
bq.  
bq.  jmhsieh wrote:
bq.  k

javadoc form for @param is to list the parameter name, so it should be there.  
agree that no question mark should be there.  i think the javadoc-y phrasing 
would be something like "whether to enable in-memory caching or not"


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2126/#review2231
---


On 2011-09-30 00:02:16, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2126/
bq.  ---
bq.  
bq.  (Updated 2011-09-30 00:02:16)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Andrew Purtell.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  commit fbf82c17be6b3ecca5a981f5270cf93aac26e479
bq.  Author: Jonathan Hsieh 
bq.  Date:   Wed Sep 28 10:18:11 2011 -0700
bq.  
bq.  HBASE-4377 [hbck] Offline rebuild .META. from fs data only
bq.  
bq.  
bq.  This patch rebuilds a new .META. table by reading all the .regioninfo 
files in the hbase main directory.  It depends on the yet to be committed 
HBASE-4515 (either my verison or Gary's version), HBASE-4509, and HBASE-4506.  
bq.  
bq.  Some follow on work includes backporting to 0.90, auto-patching true 
holes, and adding documentation.
bq.  
bq.  
bq.  This addresses bug HBASE-4377.
bq.  https://issues.apache.org/jira/browse/HBASE-4377
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 
b9c850d 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 8465724 
bq.src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fae0881 
bq.src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java 
PRE-CREATION 
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuild.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2126/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  An earlier version of this code (backported to 0.90) was used to diagnose 
and repair a cluster that had 2700 inconsistencies due to failed splits (the 
cluster was underprovisioned memory-wise, and on restart, the some regions 
would start splitting and then die due to oome's).  This was not actually used 
on a live cluster -- it was used to reconstruct a .META. from .regioninfo's 
laid out in hbase's directory structure.
bq.  
bq.  Note also that this is not an automatic fix -- whenever any problems are 
found, this bails out but dumps info on holes, suggests some fixes, and 
displays sets of overlapping regions.  It is up to the user to merge regions, 
to create .regioninfo files to plug hole, and to do any potential data loosing 
operations.
bq.  
bq.  The tests demonstrate current expected behavior -- rebuild meta if things 
line up, and fail without making modifications if holes or overlaps exist.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



> [hbck] Offline rebuild .META. from fs data only.
> 
>
> Key: HBASE-4377
> URL: https://issues.apache.org/jira/browse/HBASE-4377
> Project: HBase
>  Issue Type: New Feature
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch
>
>
> In a worst case situation, it may be helpful to have an offline .META. 
> rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
> from scratch.  Users could move bad regions out until there is a clean 
> rebuild.  
> It would likely fill in region split holes.  Follow on work could given 
> options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4334) HRegion.get never validates row

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119459#comment-13119459
 ] 

Lars Hofhansl commented on HBASE-4334:
--

Wanna commit this Ted?

> HRegion.get never validates row
> ---
>
> Key: HBASE-4334
> URL: https://issues.apache.org/jira/browse/HBASE-4334
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: Joe Pallas
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4334-v2.txt, 4334-v3.txt, 4334.txt
>
>
> If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
> may send a request to the wrong region.  Paths through put, delete, 
> incrementColumnValue, and checkAndMutate all call checkRow either directly or 
> indirectly (through getLock).  But get apparently does not.  This can result 
> in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4334) HRegion.get never validates row

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119458#comment-13119458
 ] 

Lars Hofhansl commented on HBASE-4334:
--

There's a thread on the mailing about a similar issue:
we are experiencing a strange behavior in some tests we are currently 
performing. What we are seeing is that scans on a table that is being written 
to at the same time sometimes end prematurely, with no error. This seems to be 
heavily dependent on the write pattern.

We've been able to reproduce the issue with the standard hbase tools:

in a terminal run:

OLDV=0; OLDT=0; while true; do NEWV=`hbase shell count_testtable | head -1 | 
cut -d' ' -f1`; NEWT=`date +%s`; echo $NEWV " -> " $(((NEWV - OLDV) / (NEWT - 
OLDT))) "msg/s"; OLDV=$NEWV; OLDT=$NEWT; done

where the contents of the file count_testtable are:

count 'TestTable', INTERVAL => 1, CACHE => 1
exit

This counts the rows in the TestTable repeatedly showing the number of rows and 
the delta rows per second inserted. In the hbase shell count is implemented as 
a full scan with a filter on the row key.

Meanwhile, in another terminal do:

hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=2 
randomWrite 5

and when over:

hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=2 
sequentialWrite 5

On the scan terminal we are seeing results similar to:

0  ->  0 msg/s
45552  ->  5694 msg/s <=== randomWrite starts
63284  ->  2955 msg/s
63284  ->  0 msg/s  <=== randomWrite ends
58829  ->  -636 msg/s <=== sequentialWrite starts
88764  ->  3741 msg/s
10  ->  802 msg/s
10  ->  0 msg/s <=== sequentialWrite ends

As you can see in the fifth row the count is lower than expected (resulting on 
a negative inserts/sec).

You may need to try a couple of times or tweak the number of rows to insert to 
see the problem.


> HRegion.get never validates row
> ---
>
> Key: HBASE-4334
> URL: https://issues.apache.org/jira/browse/HBASE-4334
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: Joe Pallas
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4334-v2.txt, 4334-v3.txt, 4334.txt
>
>
> If a client gets confused (possibly by a hole in .META., see HBASE-4333), it 
> may send a request to the wrong region.  Paths through put, delete, 
> incrementColumnValue, and checkAndMutate all call checkRow either directly or 
> indirectly (through getLock).  But get apparently does not.  This can result 
> in returning an incorrect empty result instead of a WrongRegionException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119460#comment-13119460
 ] 

jirapos...@reviews.apache.org commented on HBASE-4377:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2126/
---

(Updated 2011-10-03 18:16:51.000353)


Review request for hbase, Michael Stack and Andrew Purtell.


Changes
---

Addressed review comments.


Summary
---

commit fbf82c17be6b3ecca5a981f5270cf93aac26e479
Author: Jonathan Hsieh 
Date:   Wed Sep 28 10:18:11 2011 -0700

HBASE-4377 [hbck] Offline rebuild .META. from fs data only


This patch rebuilds a new .META. table by reading all the .regioninfo files in 
the hbase main directory.  It depends on the yet to be committed HBASE-4515 
(either my verison or Gary's version), HBASE-4509, and HBASE-4506.  

Some follow on work includes backporting to 0.90, auto-patching true holes, and 
adding documentation.


This addresses bug HBASE-4377.
https://issues.apache.org/jira/browse/HBASE-4377


Diffs (updated)
-

  src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java f5be448 
  src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b9c850d 
  src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 154ac32 
  src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuild.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/2126/diff


Testing
---

An earlier version of this code (backported to 0.90) was used to diagnose and 
repair a cluster that had 2700 inconsistencies due to failed splits (the 
cluster was underprovisioned memory-wise, and on restart, the some regions 
would start splitting and then die due to oome's).  This was not actually used 
on a live cluster -- it was used to reconstruct a .META. from .regioninfo's 
laid out in hbase's directory structure.

Note also that this is not an automatic fix -- whenever any problems are found, 
this bails out but dumps info on holes, suggests some fixes, and displays sets 
of overlapping regions.  It is up to the user to merge regions, to create 
.regioninfo files to plug hole, and to do any potential data loosing operations.

The tests demonstrate current expected behavior -- rebuild meta if things line 
up, and fail without making modifications if holes or overlaps exist.


Thanks,

jmhsieh



> [hbck] Offline rebuild .META. from fs data only.
> 
>
> Key: HBASE-4377
> URL: https://issues.apache.org/jira/browse/HBASE-4377
> Project: HBase
>  Issue Type: New Feature
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch
>
>
> In a worst case situation, it may be helpful to have an offline .META. 
> rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
> from scratch.  Users could move bad regions out until there is a clean 
> rebuild.  
> It would likely fill in region split holes.  Follow on work could given 
> options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family

[Liyin Tang (Created) (JIRA)]

Avoid top row seek by dedicated bloom filter for delete family
--

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang


HBASE-4469 avoids the top row seek operation if row-col bloom filter is 
enabled. 
This jira tries to avoid top row seek for all the cases by creating a dedicated 
bloom filter only for delete family.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119451#comment-13119451
 ] 

jirapos...@reviews.apache.org commented on HBASE-4377:
--



bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > How long did it take to scan the cluster with 2700 inconsistencies ?
bq.  > I see certain places in the code where more parallelism can be achieved 
if practical use of this feature takes long time.

The cluster that had 12k total regions after clenaup. It took 2m to run (this 
was localdisk accesses).  I didn't feel that the runtime was something to be 
concerned about.  And I honestly hope this code doesn't get used too often!

We could use the same WorkItem trick to speed up the code but my feeling is 
that straightforward and correct is the right first step.  


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 
374
bq.  > 
bq.  >
bq.  > Better replace root with -ROOT-

done.


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 
376
bq.  > 
bq.  >
bq.  > b is not needed here, same with question mark.

k


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 
391
bq.  > 
bq.  >
bq.  > Please remove b and question mark.

k


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 307
bq.  > 
bq.  >
bq.  > The .META. region is open upon return.
bq.  > I think we should document this.

changed "live" to "open"


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 288
bq.  > 
bq.  >
bq.  > It would be nice to log the path for the underlying region.
bq.  > Otherwise what purpose does this catch/rethrow serve ?

nice catch.  Updated to include table name and path.


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 309
bq.  > 
bq.  >
bq.  > Looking at the usage below, maybe createNewRootAndMeta would be a 
better name.

done


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 352
bq.  > 
bq.  >
bq.  > This log doesn't match the check above.
bq.  > If we only produce Put for the first HbckInfo, we'd better declare 
that in the log.

updated error message and change behavior so that it bails out.  In this 
particular case, the invariant is checked before this method is called, but 
I'll just make it more explicit.


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 356
bq.  > 
bq.  >
bq.  > This would produce exception if his.size() == 0.

problem avoided with update.


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 378
bq.  > 
bq.  >
bq.  > Do you plan to do this in the next patch or in another JIRA ?
bq.  > I haven't looked at the other JIRAs you mentioned, pardon me.

I'll file it as a follow-on jira.


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 428
bq.  > 
bq.  >
bq.  > Is there something we can do in case we get IOE from this call ?

added error logging and an attempt to revert.


bq.  On 2011-09-30 21:27:16, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 377
bq.  > 
bq.  >
bq.  > Better use boolean for return value to indicate success/failure.

done.


- jmhsieh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2126/#review2231
---


On 2011-09-30 00:02:16, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:

[jira] [Updated] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Akash Ashok (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash Ashok updated HBASE-4531:
---

Attachment: HBase-4531.patch

Changing maven-surefire-report plugin to last known stable version

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119436#comment-13119436
 ] 

jirapos...@reviews.apache.org commented on HBASE-4528:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2141/#review2270
---

Ship it!


- Lars


On 2011-10-03 05:54:07, Dhruba Borthakur wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2141/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 05:54:07)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The changes the multiPut operation so that the sync to the wal occurs 
outside the rowlock.
bq.  
bq.  This enhancement is done only to HRegion.mut(Put[]) because this is the 
only method that gets invoked from an application. The HRegion.put(Put) is used 
only by unit tests and should possibly be deprecated.
bq.  
bq.  I have attached a unit test. I have not yet run all unit tests, but early 
feedback on this patch will be very helpful.
bq.  
bq.  
bq.  This addresses bug HBASE-4528.
bq.  https://issues.apache.org/jira/browse/HBASE-4528
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178298 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2141/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Not yet run the full suite of unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Dhruba
bq.  
bq.



> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[Akash Ashok (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akash Ashok updated HBASE-4531:
---

Status: Patch Available  (was: Open)

> hbase-4454 failsafe broke mvn site; back it out or fix
> --
>
> Key: HBASE-4531
> URL: https://issues.apache.org/jira/browse/HBASE-4531
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBase-4531.patch
>
>
> mvn site is broke in head of trunk.  If I back out the last pom change it 
> works again:
> {code}
> 
> r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line
> HBASE-4454 Add failsafe plugin to build and rename integration tests
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4526) special case for stopping master in hbase-daemon.sh is no longer required

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4526:
-

   Resolution: Fixed
Fix Version/s: 0.92.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to 0.92 branch and trunk.  Thank you for the patch Roman.

> special case for stopping master in hbase-daemon.sh is no longer required
> -
>
> Key: HBASE-4526
> URL: https://issues.apache.org/jira/browse/HBASE-4526
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 0.90.4
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4526.patch.txt
>
>
> Now that HBASE-4209 is finally done (many thanks to stack for help and 
> encouragement!) I don't think there's any reason to keep this bit of code in 
> hbase-daemon.sh:
> {noformat}
> if [ "$command" = "master" ]; then
>   echo "`date` Killing $command" >> $loglog
>   kill -9 `cat $pid` > /dev/null 2>&1
> else
>   echo "`date` Killing $command" >> $loglog
>   kill `cat $pid` > /dev/null 2>&1
> fi
> {noformat}
> I suggest we remove the special case completely, since I don't think it 
> serves any useful purpose (patch attached).
> Now, as an additional precautionary measure. We can try to follow up with a 
> SIGKILL once a certain timeout expires.
> Let me know if you think it is necessary to do so and I'll update the patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119406#comment-13119406
 ] 

jirapos...@reviews.apache.org commented on HBASE-3446:
--



bq.  On 2011-10-03 01:13:26, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 
149
bq.  > 
bq.  >
bq.  > This is no longer true - see the fourth parameter below.

I can fix on commit


bq.  On 2011-10-03 01:13:26, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 
152
bq.  > 
bq.  >
bq.  > Out of date javadoc.

I can fix on commit.


I ran all tests and TestAdmin and TestMergeTool failed but running them 
individually, they passed.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2065/#review2258
---


On 2011-10-02 00:01:07, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2065/
bq.  ---
bq.  
bq.  (Updated 2011-10-02 00:01:07)
bq.  
bq.  
bq.  Review request for hbase and Jonathan Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Make the Meta* operations against meta retry.  We do it by using HTable 
instances.
bq.  (HTable calls HConnection.getRegionServerWithRetries for get, put, scan 
etc).
bq.  In 0.89, we had special RetryableMetaOperation class that was a
bq.  subclass of Callable which reproduced the guts of 
HConnection.getRegionServerWithRetries
bq.  with its retry loop.  Now we just use HTable instead (Costs some on setup 
but
bq.  otherwise, we avoid duplicating code).  Upped the retries on serverside 
too.
bq.  
bq.  Had problem with CatalogJanitor.  MetaReader and MetaEditor were relying
bq.  heavily on CT methods getting proxy connections to meta and root servers.
bq.  CT needs to be cut back.  This patch closes down access on (unused) public
bq.  methods and removes being able to get an HRegionInterface on meta and root
bq.  -- this stuff is used internally to CT only now; use MetaEditor or
bq.  MetaReader if you want to update or read catalog tables.  Opening new issue
bq.  to cutback CT use over the code base.
bq.  
bq.  A little off topic but couldn't help it since was in MetaReader and 
MetaEditor
bq.  trying to clean them up, I ended up moving meta migration code out to its
bq.  own class rather than have it in all inside in MetaEditor.
bq.  
bq.  Here is some detail to help reviews.
bq.  
bq.  M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java
bq.Clean up.  Shutdown access on some of these unused methods.  Don't
bq.let out HRegionInterface instances in particular since we are going
bq.away from raw HRI use to instead use a connection with retries:
bq.i.e. HTable.
bq.  
bq.Comments on state of this class. Javadoc edits.
bq.getZooKeeperWatcher on HConnection is deprecated so don't use it
bq.in constructor.  Override MetaNodeTracker and on node delete
bq.reset meta location (We used to do this over in MetaNodeTracker
bq.but to do that we had to have a CatalogTracker over in zk package
bq.which is silly -- bad package encapsulation).
bq.  
bq.(waitForRootServer) Renamed getRootServerConnection and change it
bq.from public to package private.
bq.(waitForRootServerConnectionDefault, getRootServerConnection) Removed.
bq.(getMetaServerConnection) Change from public to package private.
bq.Use MetaReader to read the meta location in root rather than a
bq.raw HRegionInterface so we get retrying.
bq.(remaining, timedout) Added utility methods.
bq.(waitForMetaServer) Changed from public to private.
bq.(resetMetaLocation) Made it synchronized on metaAvailable.
bq.Not all accesses were synchronized.
bq.  
bq.  M src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
bq.Refactor to use HTable instead of raw HRegionInterface so we get
bq.retrying.  For each operation we get an HTable, use it, then close it.
bq.(putToMetaTable, putsToMetaTable, etc) Utility methods.
bq.(updateRootWithMetaMigrationStatus, etc.) Moved out to own
bq.class since these classes are for a one-time migration only.
bq.  
bq.  A 
src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java
bq.New class that holds all Meta* methods updating meta table used
bq.doing the one-time migration done to meta on startup.  This class
bq.is marked deprecated because its going to be dropped in 0.94.
bq.  
b

[jira] [Commented] (HBASE-4520) Better handling of Bloom filter type discrepancy between HFile and CF config

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119398#comment-13119398
 ] 

jirapos...@reviews.apache.org commented on HBASE-4520:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2143/#review2266
---

Ship it!


+1

Had a sense of deja vu reviewing so maybe this is part of another patch 
already?  Maybe not.  Should go in regardless.

- Michael


On 2011-10-02 08:05:42, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2143/
bq.  ---
bq.  
bq.  (Updated 2011-10-02 08:05:42)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Jonathan Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Modify StoreFile to make it clear where Bloom filter type settings come 
from. We have two sources of truth: (1) HFile; and (2) CF configuration. (1) 
takes precedence in the reader, and (2) takes precedence in the writer.
bq.  
bq.  
bq.  This addresses bug HBASE-4520.
bq.  https://issues.apache.org/jira/browse/HBASE-4520
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
fa629c7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2143/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Running unit tests. I will post the results here -- please don't commit 
this yet.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> Better handling of Bloom filter type discrepancy between HFile and CF config
> 
>
> Key: HBASE-4520
> URL: https://issues.apache.org/jira/browse/HBASE-4520
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
> Fix For: 0.94.0
>
>
> Modify StoreFile to make it clear where Bloom filter type settings come from. 
> We have two sources of truth: (1) HFile; and (2) CF configuration. (1) takes 
> precedence in the reader, and (2) takes precedence in the writer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119397#comment-13119397
 ] 

stack commented on HBASE-4522:
--

@Mikhail We do Puppet at SU with per-cluster configs burned into the checked-in 
puppet scripts.

> Make hbase-site-custom.xml override the hbase-site.xml
> --
>
> Key: HBASE-4522
> URL: https://issues.apache.org/jira/browse/HBASE-4522
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Liyin Tang
>Priority: Minor
> Fix For: 0.94.0
>
>
> The motivation for diff is that we want to override some config change for 
> any specific cluster easily by just adding the config entries in the 
> hbase-site-custom.xml for that cluster. This change adds the 
> hbase-site-custom.xml configuration file into HBaseConfiguration.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4531) hbase-4454 failsafe broke mvn site; back it out or fix

[stack (Created) (JIRA)]

hbase-4454 failsafe broke mvn site; back it out or fix
--

 Key: HBASE-4531
 URL: https://issues.apache.org/jira/browse/HBASE-4531
 Project: HBase
  Issue Type: Bug
Reporter: stack
Priority: Blocker
 Fix For: 0.92.0


mvn site is broke in head of trunk.  If I back out the last pom change it works 
again:

{code}

r1177168 | stack | 2011-09-29 05:42:13 + (Thu, 29 Sep 2011) | 1 line

HBASE-4454 Add failsafe plugin to build and rename integration tests
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4530) ops_mgt.xml - expanding hbase backup section

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4530:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> ops_mgt.xml - expanding hbase backup section
> 
>
> Key: HBASE-4530
> URL: https://issues.apache.org/jira/browse/HBASE-4530
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4530.xml.patch
>
>
> * Adding 2 utilities to the utility list (Export, Import)
> * Correcting the RowCounter entry (it was missing the arguments)
> * Expanded the placeholder entry for backup into something more substantial 
> and categorized the approaches (full shutdown, live cluster).  I left a link 
> to Otis' blog entry in the section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4530) ops_mgt.xml - expanding hbase backup section

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4530:
-

Status: Patch Available  (was: Open)

> ops_mgt.xml - expanding hbase backup section
> 
>
> Key: HBASE-4530
> URL: https://issues.apache.org/jira/browse/HBASE-4530
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: ops_mgt_HBASE_4530.xml.patch
>
>
> * Adding 2 utilities to the utility list (Export, Import)
> * Correcting the RowCounter entry (it was missing the arguments)
> * Expanded the placeholder entry for backup into something more substantial 
> and categorized the approaches (full shutdown, live cluster).  I left a link 
> to Otis' blog entry in the section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira