[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121762#comment-13121762
 ] 

Lars Hofhansl commented on HBASE-4536:
--

That might be getting hard to understand. minVersions would have slightly 
different meaning depending on whether that extra flag is set. Without the flag 
minVersions is like "maxVersions for deleted rows", not sure who would need 
that.

Having just the flag for deleted rows would also make the code easier to follow 
as the column tracker would no longer need to distinguish between "normal" 
rows, delete markers, and deleted rows as it does in the current patch; but 
only between rows (deleted or not) and delete markers.


> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[Li Pi (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Pi updated HBASE-4482:
-

Attachment: hbase-4482v4.2.txt

final version.

> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt, hbase-4482v4.2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)

[Matt Corgan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121752#comment-13121752
 ] 

Matt Corgan commented on HBASE-4218:


Jacek - have you done anything with the KeyValue/scanner/searching interfaces?  
I'm curious to see your approach.  

Like you, I'm materializing a the iterator's current cell, but the materialized 
row/family/qualifier/timestamp/type/value all reside in separate arrays/fields. 
 The scanner can only materialize one cell at a time, which i think can work 
long term but doesn't play well with some of the current scanner interfaces.

The problem can be dodged by spawning a new array and copying everything into 
the KeyValue format, but we would see a massive speedup and could possibly 
eliminate all object instantiation (and furious garbage collection) if we could 
do comparisons on the intermediate arrays.  I've mocked up some cell interfaces 
and comparators but am wondering what you've already got in progress.

Regarding scanners - Supported operations on a block are next(), previous(), 
nextRow(), previousRow(), positionAt(KeyValue kv, boolean beforeIfMiss), and 
some others.  Main problem is that i can't peek() which is used in the current 
version of the KeyValue heap, though i've mocked an alternate approach without 
it.  I'm also starting to think that a traditional iterator's hasNext() method 
should not be supported so that true streaming can be done and so that blocks 
don't need to know about their neighbors.

> Delta Encoding of KeyValues  (aka prefix compression)
> -
>
> Key: HBASE-4218
> URL: https://issues.apache.org/jira/browse/HBASE-4218
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Reporter: Jacek Migdal
>  Labels: compression
>
> A compression for keys. Keys are sorted in HFile and they are usually very 
> similar. Because of that, it is possible to design better compression than 
> general purpose algorithms,
> It is an additional step designed to be used in memory. It aims to save 
> memory in cache as well as speeding seeks within HFileBlocks. It should 
> improve performance a lot, if key lengths are larger than value lengths. For 
> example, it makes a lot of sense to use it when value is a counter.
> Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) 
> shows that I could achieve decent level of compression:
>  key compression ratio: 92%
>  total compression ratio: 85%
>  LZO on the same data: 85%
>  LZO after delta encoding: 91%
> While having much better performance (20-80% faster decompression ratio than 
> LZO). Moreover, it should allow far more efficient seeking which should 
> improve performance a bit.
> It seems that a simple compression algorithms are good enough. Most of the 
> savings are due to prefix compression, int128 encoding, timestamp diffs and 
> bitfields to avoid duplication. That way, comparisons of compressed data can 
> be much faster than a byte comparator (thanks to prefix compression and 
> bitfields).
> In order to implement it in HBase two important changes in design will be 
> needed:
> -solidify interface to HFileBlock / HFileReader Scanner to provide seeking 
> and iterating; access to uncompressed buffer in HFileBlock will have bad 
> performance
> -extend comparators to support comparison assuming that N first bytes are 
> equal (or some fields are equal)
> Link to a discussion about something similar:
> http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4545) TestHLog doesn't clean up after itself

[Gary Helmling (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling resolved HBASE-4545.
--

   Resolution: Fixed
Fix Version/s: 0.94.0
 Assignee: Gary Helmling
 Hadoop Flags: Reviewed

Committed to trunk.

> TestHLog doesn't clean up after itself
> --
>
> Key: HBASE-4545
> URL: https://issues.apache.org/jira/browse/HBASE-4545
> Project: HBase
>  Issue Type: Test
>  Components: test
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.94.0
>
> Attachments: HBASE-4545.patch
>
>
> TestHLog has been hanging during shutdown of the mini cluster after all tests 
> are run.  Further investigation shows that there are many places where the 
> TestHLog tests are not cleaning up after themselves.
> Necessary changes are:
> * since all tests use HLog directly, a MiniHBaseCluster is not needed.  The 
> test should only launch a MiniDFSCluster
> * several tests do not close the created HLog at completion
> * the test class should shutdown the mini cluster in an @AfterClass method

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[dhruba borthakur (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121745#comment-13121745
 ] 

dhruba borthakur commented on HBASE-4528:
-

Hi Ted, if the HLog.sync() throws an exception, then it is not clear whether 
the transaction made it to the disk or not, but the changes to memstore are 
already made. So, it won't help to catch the exception from Hlog.sync() and 
continue; instead it is better to fail completely, do you agree?

Kannan: thanks for pointing out the problem related to memstore flushes. I had 
a solution for this deployed internally, will upload a new patch that will 
include the fix to the problem. Essentially, the flush will wait for all 
currently running transactions to quiesce before committing the flush.

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121743#comment-13121743
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/#review2386
---

Ship it!


- Ted


On 2011-10-06 05:46:49, Li Pi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2051/
bq.  ---
bq.  
bq.  (Updated 2011-10-06 05:46:49)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Bugfix, kills a race condition.
bq.  
bq.  Ignore r1, thats the wrong patch.
bq.  
bq.  
bq.  This addresses bug HBASE-4482.
bq.  https://issues.apache.org/jira/browse/HBASE-4482
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 
fe8b95a 
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
4072387 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 
0814f41 
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 
bq.  
bq.  Diff: https://reviews.apache.org/r/2051/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Looped tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Li
bq.  
bq.



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4545) TestHLog doesn't clean up after itself

[Gary Helmling (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling updated HBASE-4545:
-

Issue Type: Test  (was: Bug)

> TestHLog doesn't clean up after itself
> --
>
> Key: HBASE-4545
> URL: https://issues.apache.org/jira/browse/HBASE-4545
> Project: HBase
>  Issue Type: Test
>  Components: test
>Reporter: Gary Helmling
> Attachments: HBASE-4545.patch
>
>
> TestHLog has been hanging during shutdown of the mini cluster after all tests 
> are run.  Further investigation shows that there are many places where the 
> TestHLog tests are not cleaning up after themselves.
> Necessary changes are:
> * since all tests use HLog directly, a MiniHBaseCluster is not needed.  The 
> test should only launch a MiniDFSCluster
> * several tests do not close the created HLog at completion
> * the test class should shutdown the mini cluster in an @AfterClass method

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121742#comment-13121742
 ] 

Ted Yu commented on HBASE-4536:
---

Introducing new flag would allow maximum flexibility.
For column family where the flag isn't specified, we can handle deleted rows 
and the delete markers according to the presence of minVersions.

> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4402) Retaining locality after restart broken

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121741#comment-13121741
 ] 

Lars Hofhansl commented on HBASE-4402:
--

We just ran into this yesterday in our 0.92 based (small) test cluster.
Nice patch!

> Retaining locality after restart broken
> ---
>
> Key: HBASE-4402
> URL: https://issues.apache.org/jira/browse/HBASE-4402
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4402.txt, hbase-4402.txt
>
>
> In DefaultLoadBalancer, we implement the "retain assignment" function like so:
> {code}
>   if (sn != null && servers.contains(sn)) {
> assignments.get(sn).add(region.getKey());
> {code}
> but this will never work since after a cluster restart, all servers have a 
> new ServerName with a new startcode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121739#comment-13121739
 ] 

Ted Yu commented on HBASE-4528:
---

For Dhruba's comment @ 06/Oct/11 04:09, the following call would prevent 
inconsistency Gary mentioned from happening:
{code}
Runtime.getRuntime().halt(1);
{code}
But this is not as amenable as guarding the following in the finally block 
against possible IOE (from sync()):
{code}
if (w != null) rwcc.completeMemstoreInsert(w);
{code}

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121740#comment-13121740
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/
---

(Updated 2011-10-06 05:46:49.397731)


Review request for hbase.


Changes
---

fixed as per ted yu's request.


Summary
---

Bugfix, kills a race condition.

Ignore r1, thats the wrong patch.


This addresses bug HBASE-4482.
https://issues.apache.org/jira/browse/HBASE-4482


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java fe8b95a 
  
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 4072387 
  src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 0814f41 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 

Diff: https://reviews.apache.org/r/2051/diff


Testing
---

Looped tests.


Thanks,

Li



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121734#comment-13121734
 ] 

Lars Hofhansl commented on HBASE-4536:
--

Thinking about this a bit more. I think this would better be covered by a 
separate flag.

The reason is this: Users might set TTL for a store in order to recreate any 
state of the store within that TTL. This does not currently work deleted rows.
MinVersins is somewhat orthogonal to this.
If KEEP_DELETED_ROWS (or however we want to call this) is set on a CF, it would 
just mean that deleted rows and the delete markers are subject to the usual 
MaxVersions, TTL/MinVersions rules.

Do people think this is not a good idea generally, and this should be rather 
handled by the application - by simply not deleting anything? (there's fairly 
little discussion here)


> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121735#comment-13121735
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--



bq.  On 2011-10-06 05:30:17, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 
397
bq.  > 
bq.  >
bq.  > Should be (offHeapCacheSize <= 0 || !enableOffHeapCache)

Fixed.


bq.  On 2011-10-06 05:30:17, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java, line 
401
bq.  > 
bq.  >
bq.  > Whitespace.

Fixed.


bq.  On 2011-10-06 05:30:17, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 
225
bq.  > 
bq.  >
bq.  > Indent to the right two spaces.

Fixed.


- Li


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/#review2384
---


On 2011-10-06 04:25:43, Li Pi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2051/
bq.  ---
bq.  
bq.  (Updated 2011-10-06 04:25:43)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Bugfix, kills a race condition.
bq.  
bq.  Ignore r1, thats the wrong patch.
bq.  
bq.  
bq.  This addresses bug HBASE-4482.
bq.  https://issues.apache.org/jira/browse/HBASE-4482
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 
fe8b95a 
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
4072387 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 
0814f41 
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 
bq.  
bq.  Diff: https://reviews.apache.org/r/2051/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Looped tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Li
bq.  
bq.



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121731#comment-13121731
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/#review2384
---


After looping TestSlabCache 50 times and TestSingleSizeCache 30 times, I didn't 
see test failure.
There is one bug, see below.


src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java


Indent to the right two spaces.



src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java


Should be (offHeapCacheSize <= 0 || !enableOffHeapCache)



src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java


Whitespace.


- Ted


On 2011-10-06 04:25:43, Li Pi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2051/
bq.  ---
bq.  
bq.  (Updated 2011-10-06 04:25:43)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Bugfix, kills a race condition.
bq.  
bq.  Ignore r1, thats the wrong patch.
bq.  
bq.  
bq.  This addresses bug HBASE-4482.
bq.  https://issues.apache.org/jira/browse/HBASE-4482
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 
fe8b95a 
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
4072387 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 
0814f41 
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 
bq.  
bq.  Diff: https://reviews.apache.org/r/2051/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Looped tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Li
bq.  
bq.



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4545) TestHLog doesn't clean up after itself

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121724#comment-13121724
 ] 

stack commented on HBASE-4545:
--

+1

Good on G.

> TestHLog doesn't clean up after itself
> --
>
> Key: HBASE-4545
> URL: https://issues.apache.org/jira/browse/HBASE-4545
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Gary Helmling
> Attachments: HBASE-4545.patch
>
>
> TestHLog has been hanging during shutdown of the mini cluster after all tests 
> are run.  Further investigation shows that there are many places where the 
> TestHLog tests are not cleaning up after themselves.
> Necessary changes are:
> * since all tests use HLog directly, a MiniHBaseCluster is not needed.  The 
> test should only launch a MiniDFSCluster
> * several tests do not close the created HLog at completion
> * the test class should shutdown the mini cluster in an @AfterClass method

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121716#comment-13121716
 ] 

jirapos...@reviews.apache.org commented on HBASE-4528:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2141/#review2383
---


Along the lines of the concern Gary raises, because of insert into memstore 
happening before "sync" succeeding, I think another scenario can cause problems 
too:

1) say, memstore insert happens to CF1 & CF2.
2) then, memstore for CF1 gets flushed 
3) region server crashes before the sync is successful *and* before CF2 was 
flushed.

Now, when the region is opened in a new region server, the region will contain 
a partial transaction (only CF1 changes will survive)-- and cross CF atomicity 
will be lost.

- Kannan


On 2011-10-03 05:54:07, Dhruba Borthakur wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2141/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 05:54:07)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The changes the multiPut operation so that the sync to the wal occurs 
outside the rowlock.
bq.  
bq.  This enhancement is done only to HRegion.mut(Put[]) because this is the 
only method that gets invoked from an application. The HRegion.put(Put) is used 
only by unit tests and should possibly be deprecated.
bq.  
bq.  I have attached a unit test. I have not yet run all unit tests, but early 
feedback on this patch will be very helpful.
bq.  
bq.  
bq.  This addresses bug HBASE-4528.
bq.  https://issues.apache.org/jira/browse/HBASE-4528
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178298 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2141/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Not yet run the full suite of unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Dhruba
bq.  
bq.



> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121713#comment-13121713
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/
---

(Updated 2011-10-06 04:25:43.653003)


Review request for hbase.


Changes
---

Addressed JGray's reviews above. 

Testing reveals this one not to fail (yet).


Summary
---

Bugfix, kills a race condition.

Ignore r1, thats the wrong patch.


This addresses bug HBASE-4482.
https://issues.apache.org/jira/browse/HBASE-4482


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java fe8b95a 
  
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 4072387 
  src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 0814f41 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 

Diff: https://reviews.apache.org/r/2051/diff


Testing
---

Looped tests.


Thanks,

Li



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121712#comment-13121712
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--



bq.  On 2011-09-26 19:52:28, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 
221
bq.  > 
bq.  >
bq.  > This log would be expensive.

Got rid of that.


- Li


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/#review2070
---


On 2011-10-06 04:20:33, Li Pi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2051/
bq.  ---
bq.  
bq.  (Updated 2011-10-06 04:20:33)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Bugfix, kills a race condition.
bq.  
bq.  Ignore r1, thats the wrong patch.
bq.  
bq.  
bq.  This addresses bug HBASE-4482.
bq.  https://issues.apache.org/jira/browse/HBASE-4482
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 
fe8b95a 
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
3840279 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
4072387 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 
0814f41 
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 
bq.  
bq.  Diff: https://reviews.apache.org/r/2051/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Looped tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Li
bq.  
bq.



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121711#comment-13121711
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--



bq.  On 2011-09-26 20:01:05, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 
222
bq.  > 
bq.  >
bq.  > Can you comment here on the change?  And should this use LOG instead 
of System.out?

Removed this part entirely.


bq.  On 2011-09-26 20:01:05, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 
312
bq.  > 
bq.  >
bq.  > add a reference to the JIRA # in this comment... and break this to 
two lines.

Will do.


bq.  On 2011-09-26 20:01:05, Jonathan Gray wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java,
 lines 33-34
bq.  > 
bq.  >
bq.  > remove from javadoc

Will do.


bq.  On 2011-09-26 20:01:05, Jonathan Gray wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, 
line 1272
bq.  > 
bq.  >
bq.  > Do we need this?  At the least should be a DEBUG (seems like RS logs 
will be filled with this though, is that intended?)

Will remove.


- Li


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/#review2071
---


On 2011-10-06 04:20:33, Li Pi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2051/
bq.  ---
bq.  
bq.  (Updated 2011-10-06 04:20:33)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Bugfix, kills a race condition.
bq.  
bq.  Ignore r1, thats the wrong patch.
bq.  
bq.  
bq.  This addresses bug HBASE-4482.
bq.  https://issues.apache.org/jira/browse/HBASE-4482
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 
fe8b95a 
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
3840279 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
4072387 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 
0814f41 
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 
bq.  
bq.  Diff: https://reviews.apache.org/r/2051/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Looped tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Li
bq.  
bq.



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121710#comment-13121710
 ] 

jirapos...@reviews.apache.org commented on HBASE-4482:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2051/
---

(Updated 2011-10-06 04:20:33.696931)


Review request for hbase.


Changes
---

v4 of the diff. Got rid of the whole putIfAbsent/onEviction loops. Replaced 
with synchronize and thread.sleep/wait.


Summary
---

Bugfix, kills a race condition.

Ignore r1, thats the wrong patch.


This addresses bug HBASE-4482.
https://issues.apache.org/jira/browse/HBASE-4482


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
3798a06 
  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java fe8b95a 
  
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 91b1603 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3840279 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 4072387 
  src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 0814f41 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
e021780 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
8dd5159 

Diff: https://reviews.apache.org/r/2051/diff


Testing
---

Looped tests.


Thanks,

Li



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[dhruba borthakur (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121709#comment-13121709
 ] 

dhruba borthakur commented on HBASE-4528:
-

I agree that if the HLog.sync() throws an exception (but does not abort the 
regiomnserver), then we have a serious problem in our hands.

The existing trunk-code has the problem that if we are able to successfully 
update the WAL but insertion into the memstore resulted in an exception, then 
the client will be notified of the failed-update, but actually the transaction 
could come back to life if wal-recovery occurs.

One proposal is to change the last few lines of HLog.syncer(long txid) to throw 
an Runtime exception instead of requesting a log roll. 

{code}
} catch (IOException e) {
  LOG.fatal("Could not sync. Earlier was Requesting close of hlog " +
" but instead we should bail out completely", e);
  Runtime.exit(1);
}

{code}

Do you think that this code is palatable? Do folks usually see this code path 
being executed in their clusters?

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4416) OpenedRegionHandler running for a dead assignment will kill the master

[stack (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4416.
--

Resolution: Duplicate

Resolving as duplicate of HBASE-4540 (Ram is working on this issue over there).

> OpenedRegionHandler running for a dead assignment will kill the master
> --
>
> Key: HBASE-4416
> URL: https://issues.apache.org/jira/browse/HBASE-4416
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.90.5
>
>
> It goes like this:
>  - Master balances region R from server A to server B
>  - B reports OPENED of R
>  - Master queues a OpenedRegionHandler for R on B
>  - B dies (sad)
>  - Master processes the splits and reassigns R to C
>  - C reports OPENED of R
>  - Master queues a OpenedRegionHandler for R on C
>  - OpenedRegionHandler for R on B is finally processed, but leaves the region 
> in a weird state (log #1)
>  - OpenedRegionHandler for R on C is processed, fails when it tries to delete 
> the znode, kills the master (log #2)
> If the master didn't commit seppuku, it would have had a wrong view of the 
> state of that region because it wouldn't link it to the region server that 
> really opened it in the end.
> I'm not sure how I would go fixing this though...
> Log 1:
> {quote}
> 2011-09-15 01:57:47,430 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: The server is not in online 
> servers, ServerName=sv4r23s44,60020,1316076811761, 
> region=6e22b45f4288ea4d73f612ccf111aea6
> 2011-09-15 01:57:47,430 DEBUG 
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region 
> 6e22b45f4288ea4d73f612ccf111aea6 on sv4r23s44,60020,1316076811761
> {quote}
> Log 2:
> {quote}
> 2011-09-15 01:58:10,171 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x231d7b21aba0480 Deleting existing unassigned node for 
> 6e22b45f4288ea4d73f612ccf111aea6 that is in expected state RS_ZK_REGION_OPENED
> 2011-09-15 01:58:10,204 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: 
> master:6-0x231d7b21aba0480 Unable to get data of znode 
> /hbase/unassigned/6e22b45f4288ea4d73f612ccf111aea6 because node does not 
> exist (not necessarily an error)
> 2011-09-15 01:58:10,204 FATAL org.apache.hadoop.hbase.master.HMaster: Error 
> deleting OPENED node in ZK for transition ZK node 
> (6e22b45f4288ea4d73f612ccf111aea6)
> {quote} 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4545) TestHLog doesn't clean up after itself

[Gary Helmling (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling updated HBASE-4545:
-

Attachment: HBASE-4545.patch

The attached patch changes TestHLog to only use a MiniDFSCluster and fixes the 
unclosed HLog instances.

TestHLog now executes in about half the time of previously.  And it passes 
reliably for me in a batch of 10 test runs.

> TestHLog doesn't clean up after itself
> --
>
> Key: HBASE-4545
> URL: https://issues.apache.org/jira/browse/HBASE-4545
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Gary Helmling
> Attachments: HBASE-4545.patch
>
>
> TestHLog has been hanging during shutdown of the mini cluster after all tests 
> are run.  Further investigation shows that there are many places where the 
> TestHLog tests are not cleaning up after themselves.
> Necessary changes are:
> * since all tests use HLog directly, a MiniHBaseCluster is not needed.  The 
> test should only launch a MiniDFSCluster
> * several tests do not close the created HLog at completion
> * the test class should shutdown the mini cluster in an @AfterClass method

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121707#comment-13121707
 ] 

ramkrishna.s.vasudevan commented on HBASE-4540:
---

@JD
Yes the bug is same as hbase-4416. I did not know one already was there.
@Jon
I am working on writing test cases 
Will submit it once done

Thanks guys for your reviews
@Ted
will address your comment in the updated patch.

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4545) TestHLog doesn't clean up after itself

[Gary Helmling (Created) (JIRA)]

TestHLog doesn't clean up after itself
--

 Key: HBASE-4545
 URL: https://issues.apache.org/jira/browse/HBASE-4545
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Gary Helmling


TestHLog has been hanging during shutdown of the mini cluster after all tests 
are run.  Further investigation shows that there are many places where the 
TestHLog tests are not cleaning up after themselves.

Necessary changes are:
* since all tests use HLog directly, a MiniHBaseCluster is not needed.  The 
test should only launch a MiniDFSCluster
* several tests do not close the created HLog at completion
* the test class should shutdown the mini cluster in an @AfterClass method

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1744) Thrift server to match the new java api.

[Bob Copeland (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121698#comment-13121698
 ] 

Bob Copeland commented on HBASE-1744:
-

Also, seems checkAndPut() should allow value=null like checkAndDelete() -- I 
also made this change on my branch.

> Thrift server to match the new java api.
> 
>
> Key: HBASE-1744
> URL: https://issues.apache.org/jira/browse/HBASE-1744
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Tim Sell
>Assignee: Lars Francke
>Priority: Critical
> Fix For: 0.94.0
>
> Attachments: HBASE-1744.2.patch, HBASE-1744.3.patch, 
> HBASE-1744.preview.1.patch, thriftexperiment.patch
>
>
> This mutateRows, etc.. is a little confusing compared to the new cleaner java 
> client.
> Thinking of ways to make a thrift client that is just as elegant. something 
> like:
> void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
> with:
> struct TColumn {
>   1:Bytes family,
>   2:Bytes qualifier,
>   3:i64 timestamp
> }
> struct TPut {
>   1:Bytes row,
>   2:map values
> }
> This creates more verbose rpc  than if the columns in TPut were just 
> map>, but that is harder to fit timestamps into and 
> still be intuitive from say python.
> Presumably the goal of a thrift gateway is to be easy first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4386) NPE in TaskMonitor

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121687#comment-13121687
 ] 

Hudson commented on HBASE-4386:
---

Integrated in HBase-0.92 #47 (See 
[https://builds.apache.org/job/HBase-0.92/47/])
HBASE-4386  Fix a potential NPE in TaskMonitor

todd : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java


> NPE in TaskMonitor
> --
>
> Key: HBASE-4386
> URL: https://issues.apache.org/jira/browse/HBASE-4386
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: hbase-4386.txt
>
>
> Saw the following hitting /rs-status
> INTERNAL_SERVER_ERRORCaused 
> by:java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.purgeExpiredTasks(TaskMonitor.java:97)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.getTasks(TaskMonitor.java:127)
> at 
> org.apache.hbase.tmpl.common.TaskMonitorTmplImpl.renderNoFlush(TaskMonitorTmplImpl.java:50)
> at 
> org.apache.hbase.tmpl.common.TaskMonitorTmpl.renderNoFlush(TaskMonitorTmpl.java:170)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:70)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:176)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:167)
> at 
> org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121667#comment-13121667
 ] 

Ted Yu commented on HBASE-2856:
---

FYI patch v11 from HBASE-4344 didn't contain 4485-v4.txt for HBASE-4485.

> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 
> acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner whereby the first 
> peek() of a row is done at time T then the rest of the read is done at T+1 
> after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' 
> data becomes committed and flushed to disk.
> One possible solution is to introduce the memstoreTS (or similarly equivalent 
> value) to the HFile thus allowing us to preserve read consistency past 
> flushes.  Another solution involves fixing the scanners so that peek() is not 
> destructive (and thus might return different things at different times alas).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121660#comment-13121660
 ] 

jirapos...@reviews.apache.org commented on HBASE-4528:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2141/#review2380
---



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


We should check this.htableDescriptor.isDeferredLogFlush()


- Ted


On 2011-10-03 05:54:07, Dhruba Borthakur wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2141/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 05:54:07)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The changes the multiPut operation so that the sync to the wal occurs 
outside the rowlock.
bq.  
bq.  This enhancement is done only to HRegion.mut(Put[]) because this is the 
only method that gets invoked from an application. The HRegion.put(Put) is used 
only by unit tests and should possibly be deprecated.
bq.  
bq.  I have attached a unit test. I have not yet run all unit tests, but early 
feedback on this patch will be very helpful.
bq.  
bq.  
bq.  This addresses bug HBASE-4528.
bq.  https://issues.apache.org/jira/browse/HBASE-4528
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178298 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2141/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Not yet run the full suite of unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Dhruba
bq.  
bq.



> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121652#comment-13121652
 ] 

jirapos...@reviews.apache.org commented on HBASE-4528:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2141/#review2373
---


Two potential issues that I see:

#1 - If I'm reading it correctly, this change breaks current behavior for 
tables where DEFERRED_LOG_FLUSH=true, where HLog.sync() should not be triggered.

#2 - It also seems like moving the memstore updates prior to the WAL append and 
sync has the potential to leave around memstore entries that are not present in 
the WAL if:
1) the log.sync() call throws an IOException
2) the memstore entries are made visible via the call to 
rwcc.completeMemstoreInsert() in the finally block
3) the client sees the puts as failed due to the IOException
4) the subsequent roll of the HLog writer triggered by the IOException succeeds 
-- the region server does not abort in this case.  

In this case, wouldn't the client see the update as failed (due to the 
IOException), while the memstore entries persist?  So do the memstore changes 
need to be tracked and rolled back?  Or am I missing something?


/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


Doesn't this break current behavior in respect to tables with deferred log 
flush set to "true"?  Previously no sync was triggered for these tables.

Also, if the sync call throws an IOException that gets sent back to the 
client, how do the memstore entries get cleared?


- Gary


On 2011-10-03 05:54:07, Dhruba Borthakur wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2141/
bq.  ---
bq.  
bq.  (Updated 2011-10-03 05:54:07)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The changes the multiPut operation so that the sync to the wal occurs 
outside the rowlock.
bq.  
bq.  This enhancement is done only to HRegion.mut(Put[]) because this is the 
only method that gets invoked from an application. The HRegion.put(Put) is used 
only by unit tests and should possibly be deprecated.
bq.  
bq.  I have attached a unit test. I have not yet run all unit tests, but early 
feedback on this patch will be very helpful.
bq.  
bq.  
bq.  This addresses bug HBASE-4528.
bq.  https://issues.apache.org/jira/browse/HBASE-4528
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1178298 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2141/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Not yet run the full suite of unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Dhruba
bq.  
bq.



> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4386) NPE in TaskMonitor

[Todd Lipcon (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-4386:
---

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed for 92 and trunk

> NPE in TaskMonitor
> --
>
> Key: HBASE-4386
> URL: https://issues.apache.org/jira/browse/HBASE-4386
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: hbase-4386.txt
>
>
> Saw the following hitting /rs-status
> INTERNAL_SERVER_ERRORCaused 
> by:java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.purgeExpiredTasks(TaskMonitor.java:97)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.getTasks(TaskMonitor.java:127)
> at 
> org.apache.hbase.tmpl.common.TaskMonitorTmplImpl.renderNoFlush(TaskMonitorTmplImpl.java:50)
> at 
> org.apache.hbase.tmpl.common.TaskMonitorTmpl.renderNoFlush(TaskMonitorTmpl.java:170)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:70)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:176)
> at 
> org.apache.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:167)
> at 
> org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4388) Second start after migration from 90 to trunk crashes

[Todd Lipcon (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121614#comment-13121614
 ] 

Todd Lipcon commented on HBASE-4388:


A few small comments:
- we should have a constant like META_MIGRATION_CURRENT_VERSION which maps to 
the most recent - we can use this when we have completed migration
- it seems to make more sense to have the constants just be ints, and then 
convert the cell contents to an int where we do the comparison, rather than 
have the constants be byte arrays and use byte array comparison.

> Second start after migration from 90 to trunk crashes
> -
>
> Key: HBASE-4388
> URL: https://issues.apache.org/jira/browse/HBASE-4388
> Project: HBase
>  Issue Type: Bug
>  Components: master, migration
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4388-v2.txt, 4388-v3.txt, 4388.txt, meta.tgz
>
>
> I started a trunk cluster to upgrade from 90, inserted a ton of data, then 
> did a clean shutdown. When I started again, I got the following exception:
> 11/09/13 12:29:09 INFO master.HMaster: Meta has HRI with HTDs. Updating meta 
> now.
> 11/09/13 12:29:09 FATAL master.HMaster: Unhandled exception. Starting 
> shutdown.
> java.lang.NegativeArraySizeException: -102
> at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:147)
> at 
> org.apache.hadoop.hbase.HTableDescriptor.readFields(HTableDescriptor.java:606)
> at 
> org.apache.hadoop.hbase.migration.HRegionInfo090x.readFields(HRegionInfo090x.java:641)
> at 
> org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:133)
> at 
> org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:103)
> at 
> org.apache.hadoop.hbase.util.Writables.getHRegionInfoForMigration(Writables.java:228)
> at 
> org.apache.hadoop.hbase.catalog.MetaEditor.getHRegionInfoForMigration(MetaEditor.java:350)
> at 
> org.apache.hadoop.hbase.catalog.MetaEditor$1.visit(MetaEditor.java:273)
> at 
> org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:633)
> at 
> org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:255)
> at 
> org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:235)
> at 
> org.apache.hadoop.hbase.catalog.MetaEditor.updateMetaWithNewRegionInfo(MetaEditor.java:284)
> at 
> org.apache.hadoop.hbase.catalog.MetaEditor.migrateRootAndMeta(MetaEditor.java:298)
> at 
> org.apache.hadoop.hbase.master.HMaster.updateMetaWithNewHRI(HMaster.java:529)
> at 
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:472)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4402) Retaining locality after restart broken

[Todd Lipcon (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121610#comment-13121610
 ] 

Todd Lipcon commented on HBASE-4402:


Your fix looks good. I didn't think about the fact that there might be nulls 
due to regions that were added while the server was offline, or if a table was 
disabled when shutdown, but ZK was wiped before the cluster was brought back up.

> Retaining locality after restart broken
> ---
>
> Key: HBASE-4402
> URL: https://issues.apache.org/jira/browse/HBASE-4402
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4402.txt, hbase-4402.txt
>
>
> In DefaultLoadBalancer, we implement the "retain assignment" function like so:
> {code}
>   if (sn != null && servers.contains(sn)) {
> assignments.get(sn).add(region.getKey());
> {code}
> but this will never work since after a cluster restart, all servers have a 
> new ServerName with a new startcode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121609#comment-13121609
 ] 

jirapos...@reviews.apache.org commented on HBASE-2856:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2224/#review2376
---


Rest of the diff reviewed, and comments inline. (Joint review with Kannan).


/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


Rename to: hasMemstoreTS



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


if () {
}



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


Remove comment



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


Should check against the constant:
Bytes.toInt(keyValueFormatVersion) == KEY_VALUE_VER_WITH_MEMSTORE



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java


if () {
}

1. Discussed with Amitanand, he is planning to move this to the end of the 
KV (to play nice with delta encoding). 
2. Also planning to store this in varying-length format
3. Also, if (kv.memstoreTS < current read point across all scanners) then 
we can just write a 0. This would be the case for most of the KV's except the 
last few written. 



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java


Need a:
if (this.includeMemstoreTS) {
}


- Karthik


On 2011-10-05 19:18:51, Amitanand Aiyer wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2224/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 19:18:51)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and 
Karthik Ranganathan.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  address the 2856 issues by writing the memstoreTS to the disk.
bq.  
bq.  version v11 of the patch.
bq.  
bq.  uploading it here for easier review process.
bq.  
bq.  
bq.  This addresses bug HBASE-2856.
bq.  https://issues.apache.org/jira/browse/HBASE-2856
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
1175027 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1175027 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1174515 
bq.  
bq.  Diff: https://reviews.apache.org/r/2224/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  mvn test
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Amitanand
bq.  
bq.



> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 
> acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner whereby the first 
> peek() of a row is done at time T then the rest of the read is done at T+1 
> after a flush, thus the memstoreTS data is lost, and previou

[jira] [Commented] (HBASE-4241) Optimize flushing of the Store cache for max versions and (new) min versions

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121594#comment-13121594
 ] 

Lars Hofhansl commented on HBASE-4241:
--

During memstore flush we only want to read from the snapshot. There is no 
principal reason why using memstore scanner for this wouldn't work (need to 
make sure that kvsetIt never has anything to iterate).

The memstore scanner now also mucks with readpoints (although I think that is 
new). It seemed easier to just wrap a simple scanner wrapper instance around 
the snapshot.

Wanna try using memstore scanner? If it turns out to be simpler I am happy to 
change it.

> Optimize flushing of the Store cache for max versions and (new) min versions
> 
>
> Key: HBASE-4241
> URL: https://issues.apache.org/jira/browse/HBASE-4241
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4241-v2.txt, 4241-v8.txt, 4241.txt
>
>
> As discussed with with Jon, there is room for improvement in how the memstore 
> is flushed to disk.
> Currently only expired KVs are pruned before flushing, but we can also prune 
> versions if we find at least maxVersions versions in the memstore.
> The same holds for the new minversion feature: If we find at least minVersion 
> versions in the store we can remove all further versions that are expired.
> Generally we should use the same mechanism here that is used for Compaction. 
> I.e. StoreScanner. We only need to add a scanner to Memstore that can scan 
> along the current snapshot.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121588#comment-13121588
 ] 

jirapos...@reviews.apache.org commented on HBASE-2856:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2224/#review2374
---

Ship it!


LGTM (Caveat the feedback by the boys above).  Minors in the below to consider 
if making new patch.  Oh, lots of whitespace as the lads say that you might 
purge though no biggie.


/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java


If new version, fix this ... add parens or put it all on one line.. hard to 
read as is.



/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java


Add parens if you are making new patch here.



/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java


ditto



/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java


Parens



/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java


Is this a good name?  Its come back in from the storefile metadata.  There 
is no memstore in this context.  No biggie.  Just saying.


- Michael


On 2011-10-05 19:18:51, Amitanand Aiyer wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2224/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 19:18:51)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and 
Karthik Ranganathan.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  address the 2856 issues by writing the memstoreTS to the disk.
bq.  
bq.  version v11 of the patch.
bq.  
bq.  uploading it here for easier review process.
bq.  
bq.  
bq.  This addresses bug HBASE-2856.
bq.  https://issues.apache.org/jira/browse/HBASE-2856
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
1175027 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1175027 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1174515 
bq.  
bq.  Diff: https://reviews.apache.org/r/2224/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  mvn test
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Amitanand
bq.  
bq.



> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 
> acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner whereby the first 
> peek() of a row is done at time T then the rest of the read is done at T+1 
> after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' 
> data becomes committed and flushed to disk.
> One possible solution is to introduce the memstoreTS (or similarly equivalent 
> value) to the HFile thus allowing us to preserve read consistency past 
> flushes.  Another solution involves fixing the

[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121583#comment-13121583
 ] 

jirapos...@reviews.apache.org commented on HBASE-2856:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2224/#review2365
---


this is some good stuff.  nice work amit.


/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


whitespace added in this file



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


maybe add one line of javadoc to describe why we need this method (this is 
a method that gets used pretty widely in the code to support this change)



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


should we be pulling this stuff from KeyValue constants instead of Bytes 
directly?



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


debug?



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


maybe move this Bytes.SIZEOF_LONG into some other constant that reflects 
what this is for?  KeyValue.MEMSTORE_TS_SIZE or some such thing



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java


comment here



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java


accidental new line w/ whitespace



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


introducing whitespace here and throughout file



/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java


since this is a public constructor, add a comment that this initializes to 0



/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java


add javadoc to the public method and maybe turn it into a setter.  i 
actually have a chance i'd like to get in eventually that requires disabling 
rwcc for a specified read



/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java


this javadoc is stale now



/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java


just remove


- Jonathan


On 2011-10-05 19:18:51, Amitanand Aiyer wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2224/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 19:18:51)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and 
Karthik Ranganathan.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  address the 2856 issues by writing the memstoreTS to the disk.
bq.  
bq.  version v11 of the patch.
bq.  
bq.  uploading it here for easier review process.
bq.  
bq.  
bq.  This addresses bug HBASE-2856.
bq.  https://issues.apache.org/jira/browse/HBASE-2856
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
1175027 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1175027 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1174515 
bq.  
bq.  Diff: https://reviews.apache.org/r/2224/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  mvn test
bq.  
bq.  
bq. 

[jira] [Commented] (HBASE-4241) Optimize flushing of the Store cache for max versions and (new) min versions

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121568#comment-13121568
 ] 

Lars Hofhansl commented on HBASE-4241:
--

Hi Liyin, just seeing this now. Let me have a look.

> Optimize flushing of the Store cache for max versions and (new) min versions
> 
>
> Key: HBASE-4241
> URL: https://issues.apache.org/jira/browse/HBASE-4241
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4241-v2.txt, 4241-v8.txt, 4241.txt
>
>
> As discussed with with Jon, there is room for improvement in how the memstore 
> is flushed to disk.
> Currently only expired KVs are pruned before flushing, but we can also prune 
> versions if we find at least maxVersions versions in the memstore.
> The same holds for the new minversion feature: If we find at least minVersion 
> versions in the store we can remove all further versions that are expired.
> Generally we should use the same mechanism here that is used for Compaction. 
> I.e. StoreScanner. We only need to add a scanner to Memstore that can scan 
> along the current snapshot.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4501) [replication] Shutting down a stream leaves recovered sources running

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121564#comment-13121564
 ] 

Hudson commented on HBASE-4501:
---

Integrated in HBase-0.92 #46 (See 
[https://builds.apache.org/job/HBase-0.92/46/])
HBASE-4501  [replication] Shutting down a stream leaves recovered
   sources running

jdcryans : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java


> [replication] Shutting down a stream leaves recovered sources running
> -
>
> Key: HBASE-4501
> URL: https://issues.apache.org/jira/browse/HBASE-4501
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 0.90.5
>
> Attachments: HBASE-4501.patch
>
>
> When removing a peer it will call ReplicationSourceManager.removePeer which 
> calls closeRecoveredQueue which does this:
> {code}
> LOG.info("Done with the recovered queue " + src.getPeerClusterZnode());
> this.oldsources.remove(src);
> this.zkHelper.deleteSource(src.getPeerClusterZnode(), false);
> {code}
> This works in the case where the recovered source is done and is calling this 
> method, but when removing a peer it never calls terminate on thus it leaving 
> it running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4402) Retaining locality after restart broken

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121559#comment-13121559
 ] 

stack commented on HBASE-4402:
--

Looks like the testMergeTable failure is because of this patch.  Adding the 
below gets us past the testMergeTable failure (NPEs causing master shutdown and 
then test would hang looking for the absconded master).  This test is weird too 
anyways hand-creating regions in fs then bringing all online; needs to be 
refactored but meantime

{code}
68,69c140,143
< +  List localServers = serversByHostname.get(
< +  oldServerName.getHostname());
---
> +  List localServers = new ArrayList();
> +  if (oldServerName != null) {
> +localServers = serversByHostname.get(oldServerName.getHostname());
> +  }
76c150
< +oldHostsNoLongerPresent.add(oldServerName.getHostname());
---
> +if (oldServerName != null) 
> oldHostsNoLongerPresent.add(oldServerName.getHostname());
{code}

> Retaining locality after restart broken
> ---
>
> Key: HBASE-4402
> URL: https://issues.apache.org/jira/browse/HBASE-4402
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4402.txt, hbase-4402.txt
>
>
> In DefaultLoadBalancer, we implement the "retain assignment" function like so:
> {code}
>   if (sn != null && servers.contains(sn)) {
> assignments.get(sn).add(region.getKey());
> {code}
> but this will never work since after a cluster restart, all servers have a 
> new ServerName with a new startcode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4544) Rename RWCC to MVCC

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121546#comment-13121546
 ] 

stack commented on HBASE-4544:
--

+1

Thanks Amit.

> Rename RWCC to MVCC
> ---
>
> Key: HBASE-4544
> URL: https://issues.apache.org/jira/browse/HBASE-4544
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
> Fix For: 0.94.0
>
> Attachments: 4544-v1.txt
>
>
> ReadWriteConcurrencyControl should be called MultiVersionConcurrencyControl.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121529#comment-13121529
 ] 

jirapos...@reviews.apache.org commented on HBASE-2856:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2224/#review2369
---


Find inline some partial review comments (joint review with Kannan)

- Karthik


On 2011-10-05 19:18:51, Amitanand Aiyer wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2224/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 19:18:51)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and 
Karthik Ranganathan.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  address the 2856 issues by writing the memstoreTS to the disk.
bq.  
bq.  version v11 of the patch.
bq.  
bq.  uploading it here for easier review process.
bq.  
bq.  
bq.  This addresses bug HBASE-2856.
bq.  https://issues.apache.org/jira/browse/HBASE-2856
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
1175027 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1175027 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1174515 
bq.  
bq.  Diff: https://reviews.apache.org/r/2224/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  mvn test
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Amitanand
bq.  
bq.



> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 
> acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner whereby the first 
> peek() of a row is done at time T then the rest of the read is done at T+1 
> after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' 
> data becomes committed and flushed to disk.
> One possible solution is to introduce the memstoreTS (or similarly equivalent 
> value) to the HFile thus allowing us to preserve read consistency past 
> flushes.  Another solution involves fixing the scanners so that peek() is not 
> destructive (and thus might return different things at different times alas).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121530#comment-13121530
 ] 

jirapos...@reviews.apache.org commented on HBASE-2856:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2224/#review2366
---



/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java


Rename to shouldIncludeMemstoreTS



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


Whitespace



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


Whitespace



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


whitespace



/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java


Is this constructor used anywhere? We could remove both this and the 
default constructor as this.memstoreRead and this.memstoreWrite should be 0 by 
default.



/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java


Rename to useRWCC



/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java


if () {
  ...
}



/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java


Comment update - sequence id should be timestamp



/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java


Pass hint to StoreScanner about whether rwcc is to be used or not 
(non-compaction versus compaction)


- Karthik


On 2011-10-05 19:18:51, Amitanand Aiyer wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2224/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 19:18:51)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and 
Karthik Ranganathan.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  address the 2856 issues by writing the memstoreTS to the disk.
bq.  
bq.  version v11 of the patch.
bq.  
bq.  uploading it here for easier review process.
bq.  
bq.  
bq.  This addresses bug HBASE-2856.
bq.  https://issues.apache.org/jira/browse/HBASE-2856
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 
1175027 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1174515 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
1174515 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1175027 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1174515 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1174515 
bq.  
bq.  Diff: https://reviews.apache.org/r/2224/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  mvn test
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Amitanand
bq.  
bq.



> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 
> acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner

[jira] [Commented] (HBASE-1744) Thrift server to match the new java api.

[Bob Copeland (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121512#comment-13121512
 ] 

Bob Copeland commented on HBASE-1744:
-

Hi,

I put some fixes for the above issues Ted Yu noted and another fix in my git 
tree here (will send Tim S. a pull request in a sec):

http://github.com/bcopeland/hbase-thrift

The other fix was to allow timeRange to be null in scannerOpen's TScan.

(I work on a small python library that wraps the thrift API.  I just forward 
ported it to thrift2 today and these were the first issues to pop up.)

> Thrift server to match the new java api.
> 
>
> Key: HBASE-1744
> URL: https://issues.apache.org/jira/browse/HBASE-1744
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Tim Sell
>Assignee: Lars Francke
>Priority: Critical
> Fix For: 0.94.0
>
> Attachments: HBASE-1744.2.patch, HBASE-1744.3.patch, 
> HBASE-1744.preview.1.patch, thriftexperiment.patch
>
>
> This mutateRows, etc.. is a little confusing compared to the new cleaner java 
> client.
> Thinking of ways to make a thrift client that is just as elegant. something 
> like:
> void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
> with:
> struct TColumn {
>   1:Bytes family,
>   2:Bytes qualifier,
>   3:i64 timestamp
> }
> struct TPut {
>   1:Bytes row,
>   2:map values
> }
> This creates more verbose rpc  than if the columns in TPut were just 
> map>, but that is harder to fit timestamps into and 
> still be intuitive from say python.
> Presumably the goal of a thrift gateway is to be easy first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121511#comment-13121511
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/
---

(Updated 2011-10-05 21:45:30.186260)


Review request for hbase and Mingjie Lai.


Changes
---

add missing  in RSStatusTmpl.jamon.


Summary
---

Proposed fix for HBASE-4070. 


This addresses bug HBASE-4070.
https://issues.apache.org/jira/browse/HBASE-4070


Diffs (updated)
-

  src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 
  src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
  src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 
  src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3840279 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
eda5a9b 

Diff: https://reviews.apache.org/r/2029/diff


Testing
---

Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.


Thanks,

Eugene



> [Coprocessors] Improve region server metrics to report loaded coprocessors to 
> master
> 
>
> Key: HBASE-4070
> URL: https://issues.apache.org/jira/browse/HBASE-4070
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Mingjie Lai
>Assignee: Eugene Koontz
> Attachments: HBASE-4070.patch, HBASE-4070.patch, HBASE-4070.patch, 
> master-web-ui.jpg, rs-status-web-ui.jpg
>
>
> HBASE-3512 is about listing loaded cp classes at shell. To make it more 
> generic, we need a way to report this piece of information from region to 
> master (or just at region server level). So later on, we can display the 
> loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121508#comment-13121508
 ] 

Jonathan Gray commented on HBASE-4465:
--

Nice work Liyin and Mikhail!

> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
> Attachments: 
> HBASE-4465_Lazy-seek_optimization_for_St-20111005121052-b2ea8753.patch
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121507#comment-13121507
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/
---

(Updated 2011-10-05 21:40:25.842211)


Review request for hbase and Mingjie Lai.


Changes
---

Addressed Michael Stack's review (thanks St^Ack).

-in HServerLoad, clarify purpose of data member "coprocessors" versus 
"getCoprocessors()" method.
-remove unneeded String Comparator.
-moved tests to TestClassLoading.java (thanks to Mingjie Lai for the suggestion)


Summary
---

Proposed fix for HBASE-4070. 


This addresses bug HBASE-4070.
https://issues.apache.org/jira/browse/HBASE-4070


Diffs (updated)
-

  src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 
  src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
  src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 
  src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3840279 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
eda5a9b 

Diff: https://reviews.apache.org/r/2029/diff


Testing
---

Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.


Thanks,

Eugene



> [Coprocessors] Improve region server metrics to report loaded coprocessors to 
> master
> 
>
> Key: HBASE-4070
> URL: https://issues.apache.org/jira/browse/HBASE-4070
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Mingjie Lai
>Assignee: Eugene Koontz
> Attachments: HBASE-4070.patch, HBASE-4070.patch, HBASE-4070.patch, 
> master-web-ui.jpg, rs-status-web-ui.jpg
>
>
> HBASE-3512 is about listing loaded cp classes at shell. To make it more 
> generic, we need a way to report this piece of information from region to 
> master (or just at region server level). So later on, we can display the 
> loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[Mikhail Bautin (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121497#comment-13121497
 ] 

Mikhail Bautin commented on HBASE-4465:
---

Thanks, Jonathan! I just chatted with Nicolas and he said we should not worry 
about the 89 branch yet because he will be syncing our internal changes into 
the public 89 branch.

> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
> Attachments: 
> HBASE-4465_Lazy-seek_optimization_for_St-20111005121052-b2ea8753.patch
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[Mikhail Bautin (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin resolved HBASE-4465.
---

Resolution: Fixed

> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.94.0, 0.89.20100924
>
> Attachments: 
> HBASE-4465_Lazy-seek_optimization_for_St-20111005121052-b2ea8753.patch
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121482#comment-13121482
 ] 

Jonathan Gray commented on HBASE-4465:
--

Committed to trunk.  What's the status on the 89 branch?  Should we keep this 
open?

> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
> Attachments: 
> HBASE-4465_Lazy-seek_optimization_for_St-20111005121052-b2ea8753.patch
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121470#comment-13121470
 ] 

Ted Yu commented on HBASE-4540:
---

For ZKAssign.deleteNode(), the javadoc needs to be updated as it covers both 
opened and closed states:
{code}
   * Returns false if the node was not in the proper state but did exist.
   *
   * This method is used during table disables when a region finishes
   * successfully closing.  This is the Master acknowledging completion
   * of the specified regions transition to being closed.
{code}

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4544) Rename RWCC to MVCC

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121467#comment-13121467
 ] 

Jonathan Gray commented on HBASE-4544:
--

Nice!

Do you want to get this in before/after HBASE-2856?

> Rename RWCC to MVCC
> ---
>
> Key: HBASE-4544
> URL: https://issues.apache.org/jira/browse/HBASE-4544
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
> Fix For: 0.94.0
>
> Attachments: 4544-v1.txt
>
>
> ReadWriteConcurrencyControl should be called MultiVersionConcurrencyControl.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[Mikhail Bautin (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-4465:
--

Attachment: 
HBASE-4465_Lazy-seek_optimization_for_St-20111005121052-b2ea8753.patch

Attaching the latest version of the patch.

> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
> Attachments: 
> HBASE-4465_Lazy-seek_optimization_for_St-20111005121052-b2ea8753.patch
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121461#comment-13121461
 ] 

Jonathan Gray commented on HBASE-4465:
--

Please attach the final patch to JIRA.

> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[Mikhail Bautin (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121458#comment-13121458
 ] 

Mikhail Bautin commented on HBASE-4465:
---

All unit tests passed, ready to be committed.

> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4481) TestMergeTool failed in 0.92 build 20

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121445#comment-13121445
 ] 

Hudson commented on HBASE-4481:
---

Integrated in HBase-TRUNK #2297 (See 
[https://builds.apache.org/job/HBase-TRUNK/2297/])
HBASE-4481 TestMergeTool failed in 0.92 build 20

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java


> TestMergeTool failed in 0.92 build 20
> -
>
> Key: HBASE-4481
> URL: https://issues.apache.org/jira/browse/HBASE-4481
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Priority: Critical
> Attachments: 4481-dbg.txt, 4481.txt
>
>
> TestMergeTool failed due to the following exception:
> {code}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>   at java.lang.String.substring(String.java:1937)
>   at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81)
>   at org.apache.hadoop.hbase.ServerName.(ServerName.java:63)
>   at 
> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:62)
>   at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:583)
>   at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:108)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1588)
>   at org.apache.hadoop.hbase.util.Merge.run(Merge.java:94)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>   at 
> org.apache.hadoop.hbase.util.TestMergeTool.mergeAndVerify(TestMergeTool.java:186)
>   at 
> org.apache.hadoop.hbase.util.TestMergeTool.testMergeTool(TestMergeTool.java:264)
> {code}
> Log can be found at:
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/20/testReport/junit/org.apache.hadoop.hbase.util/TestMergeTool/testMergeTool/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4501) [replication] Shutting down a stream leaves recovered sources running

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121444#comment-13121444
 ] 

Hudson commented on HBASE-4501:
---

Integrated in HBase-TRUNK #2297 (See 
[https://builds.apache.org/job/HBase-TRUNK/2297/])
HBASE-4501  [replication] Shutting down a stream leaves recovered
   sources running

jdcryans : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java


> [replication] Shutting down a stream leaves recovered sources running
> -
>
> Key: HBASE-4501
> URL: https://issues.apache.org/jira/browse/HBASE-4501
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 0.90.5
>
> Attachments: HBASE-4501.patch
>
>
> When removing a peer it will call ReplicationSourceManager.removePeer which 
> calls closeRecoveredQueue which does this:
> {code}
> LOG.info("Done with the recovered queue " + src.getPeerClusterZnode());
> this.oldsources.remove(src);
> this.zkHelper.deleteSource(src.getPeerClusterZnode(), false);
> {code}
> This works in the case where the recovered source is done and is calling this 
> method, but when removing a peer it never calls terminate on thus it leaving 
> it running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4541) [book] book.xml - claritying schema design section on versions

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121446#comment-13121446
 ] 

Hudson commented on HBASE-4541:
---

Integrated in HBase-TRUNK #2297 (See 
[https://builds.apache.org/job/HBase-TRUNK/2297/])
HBASE-4541 clarifying versions in Schema Design

dmeil : 
Files : 
* /hbase/trunk/src/docbkx/book.xml


> [book] book.xml - claritying schema design section on versions
> --
>
> Key: HBASE-4541
> URL: https://issues.apache.org/jira/browse/HBASE-4541
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: book_HBASE_4541.xml.patch
>
>
> In the Schema Design chapter there was a section on versions and it wasn't 
> clearly differentiating between max and min versions.
> * Moving the existing section on versions to a subsection called "Maximum 
> Number of Versions" so it's at the same level as "Minimum Number of 
> Versions", and added the "max" qualifier in that sub-section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121441#comment-13121441
 ] 

jirapos...@reviews.apache.org commented on HBASE-4070:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2029/#review2352
---



src/main/java/org/apache/hadoop/hbase/HServerLoad.java


You are right; thanks. No stringComparator needed.



src/main/java/org/apache/hadoop/hbase/HServerLoad.java


I think there's a need for a data member here because HServerLoad needs an 
initial set of regionserver-level coprocessors supplied to it when it is 
constructed (this is the last parameter "final Set coprocessors" that 
we have added to the HServerLoad constructor).

Then, when getCoprocessors() is called, we take the union of those 
regionserver-level coprocessors with all of the currently-loaded region-level 
coprocessors. 

I am creating a local returnValue that holds this set union, in 
getCoprocessors(), to make this more clear hopefully, and adding documentation 
to that effect.



src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java


Removed stringComparator since it's not needed for TreeSet - thanks.



src/main/java/org/apache/hadoop/hbase/master/HMaster.java


Good point, thank you.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


Thanks, good idea.



src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java


Mingjie recommends moving to TestClassLoading.java - sounds good to me.


- Eugene


On 2011-10-04 01:21:46, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2029/
bq.  ---
bq.  
bq.  (Updated 2011-10-04 01:21:46)
bq.  
bq.  
bq.  Review request for hbase and Mingjie Lai.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Proposed fix for HBASE-4070. 
bq.  
bq.  
bq.  This addresses bug HBASE-4070.
bq.  https://issues.apache.org/jira/browse/HBASE-4070
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon 
abeb850 
bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
be6fceb 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd 
bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
dbae4fd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
3840279 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2029/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Two new tests : testRegionServerCoprocessorReported() and 
testMasterServerCoprocessorsReported() included in a new source file 
src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Eugene
bq.  
bq.



> [Coprocessors] Improve region server metrics to report loaded coprocessors to 
> master
> 
>
> Key: HBASE-4070
> URL: https://issues.apache.org/jira/browse/HBASE-4070
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Mingjie Lai
>Assignee: Eugene Koontz
> Attachments: HBASE-4070.patch, HBASE-4070.patch, HBASE-4070.patch, 
> master-web-ui.jpg, rs-status-web-ui.jpg
>
>
> HBASE-3512 is about listing loaded cp classes at shell. To make it more 
> generic, we need a way to report this piece of information from region to 
> master (or just at region server level). So later on, we can display the 
> loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4505) SubstringComparator - Javadoc refers to a class that doesn't exist

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4505:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> SubstringComparator - Javadoc refers to a class that doesn't exist
> --
>
> Key: HBASE-4505
> URL: https://issues.apache.org/jira/browse/HBASE-4505
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Trivial
> Attachments: SubstringComparator_HBASE_4505.java.patch
>
>
> For example... 
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SubstringComparator.html
> "This comparator is for use with ColumnValueFilter,"
> No such class exists.  It should be SingleColumnValueFilter.  The code 
> example is wrong too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4505) SubstringComparator - Javadoc refers to a class that doesn't exist

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4505:
-

Status: Patch Available  (was: Open)

> SubstringComparator - Javadoc refers to a class that doesn't exist
> --
>
> Key: HBASE-4505
> URL: https://issues.apache.org/jira/browse/HBASE-4505
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Trivial
> Attachments: SubstringComparator_HBASE_4505.java.patch
>
>
> For example... 
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SubstringComparator.html
> "This comparator is for use with ColumnValueFilter,"
> No such class exists.  It should be SingleColumnValueFilter.  The code 
> example is wrong too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4505) SubstringComparator - Javadoc refers to a class that doesn't exist

[Doug Meil (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4505:
-

Attachment: SubstringComparator_HBASE_4505.java.patch

> SubstringComparator - Javadoc refers to a class that doesn't exist
> --
>
> Key: HBASE-4505
> URL: https://issues.apache.org/jira/browse/HBASE-4505
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Trivial
> Attachments: SubstringComparator_HBASE_4505.java.patch
>
>
> For example... 
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SubstringComparator.html
> "This comparator is for use with ColumnValueFilter,"
> No such class exists.  It should be SingleColumnValueFilter.  The code 
> example is wrong too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4505) SubstringComparator - Javadoc refers to a class that doesn't exist

[Doug Meil (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121437#comment-13121437
 ] 

Doug Meil commented on HBASE-4505:
--

Note:  this patch is a Javadoc-only patch.

> SubstringComparator - Javadoc refers to a class that doesn't exist
> --
>
> Key: HBASE-4505
> URL: https://issues.apache.org/jira/browse/HBASE-4505
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Trivial
> Attachments: SubstringComparator_HBASE_4505.java.patch
>
>
> For example... 
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SubstringComparator.html
> "This comparator is for use with ColumnValueFilter,"
> No such class exists.  It should be SingleColumnValueFilter.  The code 
> example is wrong too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121429#comment-13121429
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/#review2363
---

Ship it!


- Ted


On 2011-10-05 19:11:26, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2180/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 19:11:26)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Previously, if we had several StoreFiles for a column family in a region, 
we would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).
bq.  
bq.  This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.
bq.  
bq.  
bq.  This addresses bug HBASE-4465.
bq.  https://issues.apache.org/jira/browse/HBASE-4465
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 
1be0280 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 
b8d33e8 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
abb5931 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
64a6e3e 
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 
9d2b2a7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2180/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All unit tests should be passing now. Will rebase and re-run again just in 
case.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> di

[jira] [Updated] (HBASE-4544) Rename RWCC to MVCC

[Amitanand Aiyer (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4544:
---

Attachment: 4544-v1.txt

> Rename RWCC to MVCC
> ---
>
> Key: HBASE-4544
> URL: https://issues.apache.org/jira/browse/HBASE-4544
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
> Fix For: 0.94.0
>
> Attachments: 4544-v1.txt
>
>
> ReadWriteConcurrencyControl should be called MultiVersionConcurrencyControl.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4544) Rename RWCC to MVCC

[Amitanand Aiyer (Created) (JIRA)]

Rename RWCC to MVCC
---

 Key: HBASE-4544
 URL: https://issues.apache.org/jira/browse/HBASE-4544
 Project: HBase
  Issue Type: Sub-task
Reporter: Amitanand Aiyer
 Attachments: 4544-v1.txt

ReadWriteConcurrencyControl should be called MultiVersionConcurrencyControl.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121407#comment-13121407
 ] 

Ted Yu commented on HBASE-4540:
---

For ZKAssign.deleteNode(), the following can be removed:
{code}
LOG.debug(zkw.prefix("Successfully deleted unassigned node for region " +
regionName + " in expected state " + expectedState));
{code}

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4501) [replication] Shutting down a stream leaves recovered sources running

[Jean-Daniel Cryans (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans updated HBASE-4501:
--

Fix Version/s: (was: 0.92.0)
   0.90.5

> [replication] Shutting down a stream leaves recovered sources running
> -
>
> Key: HBASE-4501
> URL: https://issues.apache.org/jira/browse/HBASE-4501
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 0.90.5
>
> Attachments: HBASE-4501.patch
>
>
> When removing a peer it will call ReplicationSourceManager.removePeer which 
> calls closeRecoveredQueue which does this:
> {code}
> LOG.info("Done with the recovered queue " + src.getPeerClusterZnode());
> this.oldsources.remove(src);
> this.zkHelper.deleteSource(src.getPeerClusterZnode(), false);
> {code}
> This works in the case where the recovered source is done and is calling this 
> method, but when removing a peer it never calls terminate on thus it leaving 
> it running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[Jean-Daniel Cryans (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121398#comment-13121398
 ] 

Jean-Daniel Cryans commented on HBASE-4540:
---

This reminds me of HBASE-4416.

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121384#comment-13121384
 ] 

Jonathan Gray commented on HBASE-4540:
--

Looks pretty good.  Once you get the unit tests passing, want to put it up on 
RB?

Also, it'd be really good if you could start thinking about how to mock these 
scenarios better in our unit tests.  You are finding lots of great bugs but 
without tests it will be hard to prevent regressions.

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121380#comment-13121380
 ] 

jirapos...@reviews.apache.org commented on HBASE-2856:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2224/
---

Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik 
Ranganathan.


Summary
---

address the 2856 issues by writing the memstoreTS to the disk.

version v11 of the patch.

uploading it here for easier review process.


This addresses bug HBASE-2856.
https://issues.apache.org/jira/browse/HBASE-2856


Diffs
-

  /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 
1174515 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1174515 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1174515 
  /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1175027 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1174515 
  
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1174515 
  /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
1174515 
  /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1174515 
  /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1174515 
  /src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1174515 
  /src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1175027 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1174515 
  /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 
1174515 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
1174515 

Diff: https://reviews.apache.org/r/2224/diff


Testing
---

mvn test


Thanks,

Amitanand



> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 
> acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner whereby the first 
> peek() of a row is done at time T then the rest of the read is done at T+1 
> after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' 
> data becomes committed and flushed to disk.
> One possible solution is to introduce the memstoreTS (or similarly equivalent 
> value) to the HFile thus allowing us to preserve read consistency past 
> flushes.  Another solution involves fixing the scanners so that peek() is not 
> destructive (and thus might return different things at different times alas).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121373#comment-13121373
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/
---

(Updated 2011-10-05 19:11:26.781722)


Review request for hbase.


Changes
---

Getting rid of duplicate import of AtomicLong in StoreFileScanner.


Summary
---

Previously, if we had several StoreFiles for a column family in a region, we 
would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).

This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.


This addresses bug HBASE-4465.
https://issues.apache.org/jira/browse/HBASE-4465


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
  
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
  src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 1be0280 
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java b8d33e8 
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
  src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
  
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java abb5931 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 64a6e3e 
  src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 9d2b2a7 

Diff: https://reviews.apache.org/r/2180/diff


Testing
---

All unit tests should be passing now. Will rebase and re-run again just in case.


Thanks,

Mikhail



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4543) major_compact '.META.' has no effect

[Amitanand Aiyer (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4543:
---

Attachment: 0001-fix-the-issue-of-.META.-getting-ignored.patch

was not able to upload the patch to review board. just uploading it here.

> major_compact '.META.' has no effect
> 
>
> Key: HBASE-4543
> URL: https://issues.apache.org/jira/browse/HBASE-4543
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621, 0.89.20100924
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
>Priority: Minor
> Attachments: 0001-fix-the-issue-of-.META.-getting-ignored.patch
>
>
> major_compact '.META.' has no effect, although major_compact 
> 'any_other_table' works fine from the shell.
> This issue seems to only affect 0.89. The apache-trunk seems to handle this 
> case properly.
> The issue is that getTableRegions() in HMaster.java only works if the 
> tableName given is a "normal" table.
> The methodology (using a MetaScanner to look through the .META. table for the 
> tableName) does not work if
> the tableName is .META.
> The fix modifies getTableRegions() to check if the tableName is .META.; and 
> if so, handle it accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121359#comment-13121359
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/#review2361
---


Thanks Mikhail.  I have reviewed this diff internally and it looks good to me.

- Liyin


On 2011-10-05 18:00:03, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2180/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 18:00:03)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Previously, if we had several StoreFiles for a column family in a region, 
we would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).
bq.  
bq.  This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.
bq.  
bq.  
bq.  This addresses bug HBASE-4465.
bq.  https://issues.apache.org/jira/browse/HBASE-4465
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 
1be0280 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 
b8d33e8 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
abb5931 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
64a6e3e 
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 
9d2b2a7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2180/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All unit tests should be passing now. Will rebase and re-run again just in 
case.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is

[jira] [Updated] (HBASE-4543) major_compact '.META.' has no effect

[Amitanand Aiyer (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4543:
---

  Description: 
major_compact '.META.' has no effect, although major_compact 'any_other_table' 
works fine from the shell.

This issue seems to only affect 0.89. The apache-trunk seems to handle this 
case properly.


The issue is that getTableRegions() in HMaster.java only works if the tableName 
given is a "normal" table.
The methodology (using a MetaScanner to look through the .META. table for the 
tableName) does not work if
the tableName is .META.

The fix modifies getTableRegions() to check if the tableName is .META.; and if 
so, handle it accordingly.

  was:
major_compact '.META.' has no effect, although major_compact 'any_other_table' 
works fine from the shell.

The issue is that getTableRegions() in HMaster.java only works if the tableName 
given is a "normal" table.
The methodology (using a MetaScanner to look through the .META. table for the 
tableName) does not work if
the tableName is .META.

The fix modifies getTableRegions() to check if the tableName is .META.; and if 
so, handle it accordingly.

Affects Version/s: 0.89.20100621
   0.89.20100924

> major_compact '.META.' has no effect
> 
>
> Key: HBASE-4543
> URL: https://issues.apache.org/jira/browse/HBASE-4543
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621, 0.89.20100924
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
>Priority: Minor
>
> major_compact '.META.' has no effect, although major_compact 
> 'any_other_table' works fine from the shell.
> This issue seems to only affect 0.89. The apache-trunk seems to handle this 
> case properly.
> The issue is that getTableRegions() in HMaster.java only works if the 
> tableName given is a "normal" table.
> The methodology (using a MetaScanner to look through the .META. table for the 
> tableName) does not work if
> the tableName is .META.
> The fix modifies getTableRegions() to check if the tableName is .META.; and 
> if so, handle it accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121352#comment-13121352
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/#review2360
---

Ship it!


This is some really awesome work, Mikhail and Liyin.  You guys have taken our 
codez to a new level with all the changes you guys are making.  And the fake KV 
idea is super elegant.  Nice job!  This is going to be a major performance win 
across so many applications.

Bring back the old get!

- Jonathan


On 2011-10-05 18:00:03, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2180/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 18:00:03)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Previously, if we had several StoreFiles for a column family in a region, 
we would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).
bq.  
bq.  This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.
bq.  
bq.  
bq.  This addresses bug HBASE-4465.
bq.  https://issues.apache.org/jira/browse/HBASE-4465
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 
1be0280 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 
b8d33e8 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
abb5931 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
64a6e3e 
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 
9d2b2a7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2180/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All unit tests should be passing now. Will rebase and re-run again just in 
case.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that f

[jira] [Created] (HBASE-4543) major_compact '.META.' has no effect

[Amitanand Aiyer (Created) (JIRA)]

major_compact '.META.' has no effect


 Key: HBASE-4543
 URL: https://issues.apache.org/jira/browse/HBASE-4543
 Project: HBase
  Issue Type: Bug
Reporter: Amitanand Aiyer
Priority: Minor


major_compact '.META.' has no effect, although major_compact 'any_other_table' 
works fine from the shell.

The issue is that getTableRegions() in HMaster.java only works if the tableName 
given is a "normal" table.
The methodology (using a MetaScanner to look through the .META. table for the 
tableName) does not work if
the tableName is .META.

The fix modifies getTableRegions() to check if the tableName is .META.; and if 
so, handle it accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4543) major_compact '.META.' has no effect

[Amitanand Aiyer (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer reassigned HBASE-4543:
--

Assignee: Amitanand Aiyer

> major_compact '.META.' has no effect
> 
>
> Key: HBASE-4543
> URL: https://issues.apache.org/jira/browse/HBASE-4543
> Project: HBase
>  Issue Type: Bug
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
>Priority: Minor
>
> major_compact '.META.' has no effect, although major_compact 
> 'any_other_table' works fine from the shell.
> The issue is that getTableRegions() in HMaster.java only works if the 
> tableName given is a "normal" table.
> The methodology (using a MetaScanner to look through the .META. table for the 
> tableName) does not work if
> the tableName is .META.
> The fix modifies getTableRegions() to check if the tableName is .META.; and 
> if so, handle it accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121335#comment-13121335
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/#review2358
---


That is a pretty cool idea.
I was looking at this w.r.t. to HBASE-4071 and HBASE-4241, and it looks good 
from that viewpoint.

- Lars


On 2011-10-05 18:00:03, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2180/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 18:00:03)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Previously, if we had several StoreFiles for a column family in a region, 
we would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).
bq.  
bq.  This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.
bq.  
bq.  
bq.  This addresses bug HBASE-4465.
bq.  https://issues.apache.org/jira/browse/HBASE-4465
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 
1be0280 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 
b8d33e8 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
abb5931 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
64a6e3e 
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 
9d2b2a7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2180/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All unit tests should be passing now. Will rebase and re-run again just in 
case.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dar

[jira] [Updated] (HBASE-4542) add filter info to slow query logging

[Kannan Muthukkaruppan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kannan Muthukkaruppan updated HBASE-4542:
-

Description: 
Slow query log doesn't report filters in effect.

For example:
{code}
(operationTooSlow): \
{"processingtimems":3468,"client":"10.138.43.206:40035","timeRange": 
[0,9223372036854775807],\
"starttimems":1317772005821,"responsesize":42411, \
"class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
"row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
"method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
{code}

the above would suggest that all columns of myTable:CF1 are being requested for 
the given row. But in reality there could be filters in effect (such as 
ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should 
enhance the slow query log to capture & report this information.



  was:
Slow query log doesn't report filter in effect.

For example:
{code}
(operationTooSlow): \
{"processingtimems":3468,"client":"10.138.43.206:40035","timeRange": 
[0,9223372036854775807],\
"starttimems":1317772005821,"responsesize":42411, \
"class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
"row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
"method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
{code}

the above would suggest that all columns of myTable:CF1 are being requested for 
the given row. But in reality there could be filters in effect (such as 
ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should 
enhance the slow query log to capture & report this information.




> add filter info to slow query logging
> -
>
> Key: HBASE-4542
> URL: https://issues.apache.org/jira/browse/HBASE-4542
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Assignee: Madhuwanti Vaidya
>
> Slow query log doesn't report filters in effect.
> For example:
> {code}
> (operationTooSlow): \
> {"processingtimems":3468,"client":"10.138.43.206:40035","timeRange": 
> [0,9223372036854775807],\
> "starttimems":1317772005821,"responsesize":42411, \
> "class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
> "row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
> "method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
> {code}
> the above would suggest that all columns of myTable:CF1 are being requested 
> for the given row. But in reality there could be filters in effect (such as 
> ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should 
> enhance the slow query log to capture & report this information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121320#comment-13121320
 ] 

jirapos...@reviews.apache.org commented on HBASE-4536:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2178/
---

(Updated 2011-10-05 18:17:40.058774)


Review request for hbase.


Summary (updated)
---

In order to use a replicated cluster as backup and to make true use of HBase's 
timestamps it has to be possible to get/scan rows that are hidden by a delete 
marker. If a marker was placed at time T, it should be possible to retrieve 
rows with get/scan timerange of [0-T).

This changes the following:
1. MinVersions now also means: Keep this number of version even when the row 
was deleted.
2. Allow gets/scans to retrieve rows hidden by a delete marker (only if 
minVersions is set for the CF).
3. Do not unconditionally collect all deleted rows during a compaction.

The change is pretty small, but the logic is intricate, so please review 
carefully.


This addresses bug HBASE-4536.
https://issues.apache.org/jira/browse/HBASE-4536


Diffs
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
 1178891 

Diff: https://reviews.apache.org/r/2178/diff


Testing (updated)
---

All test (except TestAdmin, which fail with or without my change) pass now.


Thanks,

Lars



> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4542) add filter info to slow query logging

[Kannan Muthukkaruppan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kannan Muthukkaruppan updated HBASE-4542:
-

Assignee: Madhuwanti Vaidya

> add filter info to slow query logging
> -
>
> Key: HBASE-4542
> URL: https://issues.apache.org/jira/browse/HBASE-4542
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Assignee: Madhuwanti Vaidya
>
> Slow query log doesn't report filter in effect.
> For example:
> {code}
> (operationTooSlow): \
> {"processingtimems":3468,"client":"10.138.43.206:40035","timeRange": 
> [0,9223372036854775807],\
> "starttimems":1317772005821,"responsesize":42411, \
> "class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
> "row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
> "method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
> {code}
> the above would suggest that all columns of myTable:CF1 are being requested 
> for the given row. But in reality there could be filters in effect (such as 
> ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should 
> enhance the slow query log to capture & report this information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121314#comment-13121314
 ] 

jirapos...@reviews.apache.org commented on HBASE-4536:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2178/
---

(Updated 2011-10-05 18:16:19.303065)


Review request for hbase.


Changes
---

All tests pass now (except TestAdmin, which fails with or without my changes).


Summary
---

In order to use a replicated cluster as backup and to make true use of HBase's 
timestamps it has to be possible to get/scan rows that are hidden by a delete 
marker. If a marker was placed at time T, it should be possible to retrieve 
rows with get/scan timerange of [0-T).

This changes the following:
1. MinVersions now also means: Keep this number of version even when the row 
was deleted.
2. Allow gets/scans to retrieve rows hidden by a delete marker.
3. Do not unconditionally collect all deleted rows during a compaction.

The change is pretty small, but the logic is intricate, so please review 
carefully.


This addresses bug HBASE-4536.
https://issues.apache.org/jira/browse/HBASE-4536


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java
 1178891 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
 1178891 

Diff: https://reviews.apache.org/r/2178/diff


Testing
---

TestMinVersions (with a new test for this) passed. Have not run full suite, 
yet. This is for initial feedback.


Thanks,

Lars



> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4481) TestMergeTool failed in 0.92 build 20

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121312#comment-13121312
 ] 

Hudson commented on HBASE-4481:
---

Integrated in HBase-0.92 #45 (See 
[https://builds.apache.org/job/HBase-0.92/45/])
HBASE-4481 TestMergeTool failed in 0.92 build 20

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java


> TestMergeTool failed in 0.92 build 20
> -
>
> Key: HBASE-4481
> URL: https://issues.apache.org/jira/browse/HBASE-4481
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Priority: Critical
> Attachments: 4481-dbg.txt, 4481.txt
>
>
> TestMergeTool failed due to the following exception:
> {code}
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>   at java.lang.String.substring(String.java:1937)
>   at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81)
>   at org.apache.hadoop.hbase.ServerName.(ServerName.java:63)
>   at 
> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:62)
>   at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:583)
>   at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:108)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1588)
>   at org.apache.hadoop.hbase.util.Merge.run(Merge.java:94)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>   at 
> org.apache.hadoop.hbase.util.TestMergeTool.mergeAndVerify(TestMergeTool.java:186)
>   at 
> org.apache.hadoop.hbase.util.TestMergeTool.testMergeTool(TestMergeTool.java:264)
> {code}
> Log can be found at:
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/20/testReport/junit/org.apache.hadoop.hbase.util/TestMergeTool/testMergeTool/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4542) add filter info to slow query logging

[Kannan Muthukkaruppan (Created) (JIRA)]

add filter info to slow query logging
-

 Key: HBASE-4542
 URL: https://issues.apache.org/jira/browse/HBASE-4542
 Project: HBase
  Issue Type: Improvement
Reporter: Kannan Muthukkaruppan


Slow query log doesn't report filter in effect.

For example:
{code}
(operationTooSlow): \
{"processingtimems":3468,"client":"10.138.43.206:40035","timeRange": 
[0,9223372036854775807],\
"starttimems":1317772005821,"responsesize":42411, \
"class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
"row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
"method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
{code}

the above would suggest that all columns of myTable:CF1 are being requested for 
the given row. But in reality there could be filters in effect (such as 
ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should 
enhance the slow query log to capture & report this information.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121299#comment-13121299
 ] 

ramkrishna.s.vasudevan commented on HBASE-4540:
---

The reason for getting the znode version for the following scenario
-> RS1 tries opening a region by transiting it to OPENED
->OpenedRegionHandler has still not processed.
-> RS1 goes down and the region is assigned to RS2.
-> RS2 has transited the node to OPENED
-> Now the OpenedRegionHandler will try to delete the znode and it will succeed 
thinking the region is in RS1.
-> To avoid the above scenario i have tried to use the znode version that comes 
along when we get the callback after transiting the node to OPENED state.

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121298#comment-13121298
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/#review2354
---


TestBlocksRead changes look good.

Nice work Mikhail/Liyin-- this is a *huge* optimization!


- Kannan


On 2011-10-05 18:00:03, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2180/
bq.  ---
bq.  
bq.  (Updated 2011-10-05 18:00:03)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Previously, if we had several StoreFiles for a column family in a region, 
we would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).
bq.  
bq.  This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.
bq.  
bq.  
bq.  This addresses bug HBASE-4465.
bq.  https://issues.apache.org/jira/browse/HBASE-4465
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 
1be0280 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 
b8d33e8 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
abb5931 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 
64a6e3e 
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 
9d2b2a7 
bq.  
bq.  Diff: https://reviews.apache.org/r/2180/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All unit tests should be passing now. Will rebase and re-run again just in 
case.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Mikhail
bq.  
bq.



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurem

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121292#comment-13121292
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/
---

(Updated 2011-10-05 18:00:03.234737)


Review request for hbase.


Changes
---

Updating testing done.


Summary
---

Previously, if we had several StoreFiles for a column family in a region, we 
would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).

This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.


This addresses bug HBASE-4465.
https://issues.apache.org/jira/browse/HBASE-4465


Diffs
-

  src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
  
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
  src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 1be0280 
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java b8d33e8 
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
  src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
  
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java abb5931 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 64a6e3e 
  src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 9d2b2a7 

Diff: https://reviews.apache.org/r/2180/diff


Testing (updated)
---

All unit tests should be passing now. Will rebase and re-run again just in case.


Thanks,

Mikhail



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121291#comment-13121291
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/
---

(Updated 2011-10-05 17:59:30.185594)


Review request for hbase.


Changes
---

Addressing Ted's comments and fixing TestBlocksRead because the number of 
blocks read has decreased in a few cases.


Summary
---

Previously, if we had several StoreFiles for a column family in a region, we 
would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).

This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.


This addresses bug HBASE-4465.
https://issues.apache.org/jira/browse/HBASE-4465


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
  
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
  src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 1be0280 
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java b8d33e8 
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
  src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
  
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java abb5931 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 64a6e3e 
  src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 9d2b2a7 

Diff: https://reviews.apache.org/r/2180/diff


Testing
---

Running unit tests -- please do not commit yet.


Thanks,

Mikhail



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[ramkrishna.s.vasudevan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-4540:
--

Attachment: HBASE-4540_1.patch

I have not yet completely run the test suite.
Once run will let you know the results.  

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[ramkrishna.s.vasudevan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-4540:
--

Status: Patch Available  (was: Open)

Addresses HBASE-4539 also.

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121287#comment-13121287
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/
---

(Updated 2011-10-05 17:56:40.110499)


Review request for hbase.


Changes
---

Replying to Ted's comments. Will post a new version of the diff shortly.


Summary
---

Previously, if we had several StoreFiles for a column family in a region, we 
would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).

This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.


This addresses bug HBASE-4465.
https://issues.apache.org/jira/browse/HBASE-4465


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
  
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
  src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 1be0280 
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java b8d33e8 
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
  src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 035f765 
  
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 
dad278a 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java abb5931 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 
31bfea7 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 64a6e3e 
  src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java 
8ad5aab 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 
b3beabb 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java 9d2b2a7 

Diff: https://reviews.apache.org/r/2180/diff


Testing
---

Running unit tests -- please do not commit yet.


Thanks,

Mikhail



> Lazy-seek optimization for StoreFile scanners
> -
>
> Key: HBASE-4465
> URL: https://issues.apache.org/jira/browse/HBASE-4465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>  Labels: optimization, seek
> Fix For: 0.89.20100924, 0.94.0
>
>
> Previously, if we had several StoreFiles for a column family in a region, we 
> would seek in each of them and only then merge the results, even though the 
> row/column we are looking for might only be in the most recent (and the 
> smallest) file. Now we prioritize our reads from those files so that we check 
> the most recent file first. This is done by doing a "lazy seek" which 
> pretends that the next value in the StoreFile is (seekRow, seekColumn, 
> lastTimestampInStoreFile), which is earlier in the KV order than anything 
> that might actually occur in the file. So if we don't find the result in 
> earlier files, that fake KV will bubble up to the top of the KV heap and a 
> real seek will be done. This is expected to significantly reduce the amount 
> of disk IO (as of 09/22/2011 we are doing dark launch testing and 
> measurement).
> This is joint work with Liyin Tang -- huge thanks to him for many helpful 
> discussions on this and the idea of putting fake KVs with the highest 
> timestamp of the StoreFile in the scanner priority queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121283#comment-13121283
 ] 

jirapos...@reviews.apache.org commented on HBASE-4465:
--



bq.  On 2011-10-04 23:56:22, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java, 
line 306
bq.  > 
bq.  >
bq.  > Should be "lazily-sought"

Somehow "sought" does not sound right for me -- "seek" is a very specific 
computer science term here. Replaced with "has done a seek operation" here and 
below.


bq.  On 2011-10-04 23:56:22, Ted Yu wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java, line 
246
bq.  > 
bq.  >
bq.  > Should realSeekDone be set before returning ?

realSeekDone is set to true by enforceSeek() in case we decide we need to do it 
as part of requestSeek()

  realSeekDone = false;  // <-- setting this by default in case lazy seek 
 // takes effect or enforceSeek() fails
  . . .
  if (seekTimestamp > maxTimestampInFile) {
// Create a fake key that is not greater than the real next key.
// (Lower timestamps correspond to higher KVs.)
// To understand this better, consider that we are asked to seek to
// a higher timestamp than the max timestamp in this file. We know that
// the next point when we have to consider this file again is when we
// pass the max timestamp of this file (with the same row/column).
cur = kv.createFirstOnRowColTS(maxTimestampInFile);
  } else {
// This will be the case e.g. when we need to seek to the next
// row/column, and we don't know exactly what they are, so we set the
// seek key's timestamp to OLDEST_TIMESTAMP to skip the rest of this
// row/column.
enforceSeek();  // <-- this sets realSeekDone
  }


bq.  On 2011-10-04 23:56:22, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java, 
line 371
bq.  > 
bq.  >
bq.  > Should be 'real-sought' and 'lazily-sought'

Done.


bq.  On 2011-10-04 23:56:22, Ted Yu wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java, 
line 101
bq.  > 
bq.  >
bq.  > Should be 'is sought'

Done.


- Mikhail


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2180/#review2332
---


On 2011-10-04 22:10:40, Mikhail Bautin wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2180/
bq.  ---
bq.  
bq.  (Updated 2011-10-04 22:10:40)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Previously, if we had several StoreFiles for a column family in a region, 
we would seek in each of them and only then merge the results, even though the 
row/column we are looking for might only be in the most recent (and the 
smallest) file. Now we prioritize our reads from those files so that we check 
the most recent file first. This is done by doing a "lazy seek" which pretends 
that the next value in the StoreFile is (seekRow, seekColumn, 
lastTimestampInStoreFile), which is earlier in the KV order than anything that 
might actually occur in the file. So if we don't find the result in earlier 
files, that fake KV will bubble up to the top of the KV heap and a real seek 
will be done. This is expected to significantly reduce the amount of disk IO 
(as of 09/22/2011 we are doing dark launch testing and measurement).
bq.  
bq.  This is joint work with Liyin Tang – huge thanks to him for many helpful 
discussions on this and the idea of putting fake KVs with the highest timestamp 
of the StoreFile in the scanner priority queue.
bq.  
bq.  
bq.  This addresses bug HBASE-4465.
bq.  https://issues.apache.org/jira/browse/HBASE-4465
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java aa34006 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java 
94ddce7 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 
1be0280 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 
b8d33e8 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java 
fbcd276 
bq.src/main/java/org/apache/hadoop/hbase/regionserv

[jira] [Resolved] (HBASE-4501) [replication] Shutting down a stream leaves recovered sources running

[Jean-Daniel Cryans (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans resolved HBASE-4501.
---

  Resolution: Fixed
Assignee: Jean-Daniel Cryans
Hadoop Flags: Reviewed

Thanks for the review guys, committed to 0.90/2 and trunk.

> [replication] Shutting down a stream leaves recovered sources running
> -
>
> Key: HBASE-4501
> URL: https://issues.apache.org/jira/browse/HBASE-4501
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 0.92.0
>
> Attachments: HBASE-4501.patch
>
>
> When removing a peer it will call ReplicationSourceManager.removePeer which 
> calls closeRecoveredQueue which does this:
> {code}
> LOG.info("Done with the recovered queue " + src.getPeerClusterZnode());
> this.oldsources.remove(src);
> this.zkHelper.deleteSource(src.getPeerClusterZnode(), false);
> {code}
> This works in the case where the recovered source is done and is calling this 
> method, but when removing a peer it never calls terminate on thus it leaving 
> it running.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4536) Allow CF to retain deleted rows

[Lars Hofhansl (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4536:
-

Hadoop Flags:   (was: Incompatible change)

Now version of patch does not have the incompatible behavior.

> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121166#comment-13121166
 ] 

Lars Hofhansl commented on HBASE-4536:
--

You are right.

I have a new version of the patch now that only does that when minimum versions 
is enabled for the CF. As minversions is not released, yet, this should be ok.


> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[Li Pi (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121155#comment-13121155
 ] 

Li Pi commented on HBASE-4482:
--

The interesting bit: http://pastebin.com/Xj6bdFMs

The same thread is calling both putIfAbsent and the remove loops. Assuming I am 
reading this stacktrace right.

The loops are supposed to block eachother. I'm confused for now.



> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache

[Li Pi (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121147#comment-13121147
 ] 

Li Pi commented on HBASE-4482:
--

After applying patch v2. Current status is, It is very difficult to reproduce 
the bug. Ted Yu has a stacktrace.

The interested bit I posted to http://pastebin.com/Xj6bdFMs.

The SlabCache maintains an invariant: that SlabCache and SingleSlabCache, will, 
in steady state, either both have an entry or neither of them have an entry.

In order to maintain this invariant, SlabCache does a few things.

For any given caching operation, SlabCache will write to its backingStore 
EXACTLY once. If an entry already exists.

This is why theres this loop here:
while (backingStore.putIfAbsent(blockName, scache) != null); (I should put 
a thread.yield() in here).

SlabCache cannot remove an entry from its backingStore unless a SingleSizeCache 
calls its onEviction. SingleSizeCache will only call onEviction once and only 
once for any given key. This is enforced with a synchronize block.

Once onEviction is called, SlabCache will remove one, and EXACTLY one entry. It 
will loop until it does this.

while ((backingStore.remove(key)) == null) {
  Thread.yield();
}

> Race Condition Concerning Eviction in SlabCache
> ---
>
> Key: HBASE-4482
> URL: https://issues.apache.org/jira/browse/HBASE-4482
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Li Pi
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: hbase-4482v1.txt, hbase-4482v2.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4536) Allow CF to retain deleted rows

[Jonathan Gray (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121139#comment-13121139
 ] 

Jonathan Gray commented on HBASE-4536:
--

This changes default behavior now?  I disagree that expected behavior is to 
ever uncover previously deleted data.  I'm okay with this as an option.


> Allow CF to retain deleted rows
> ---
>
> Key: HBASE-4536
> URL: https://issues.apache.org/jira/browse/HBASE-4536
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0, 0.94.0
>
>
> Parent allows for a cluster to retain rows for a TTL or keep a minimum number 
> of versions.
> However, if a client deletes a row all version older than the delete tomb 
> stone will be remove at the next major compaction (and even at memstore flush 
> - see HBASE-4241).
> There should be a way to retain those version to guard against software error.
> I see two options here:
> 1. Add a new flag HColumnDescriptor. Something like "RETAIN_DELETED".
> 2. Folds this into the parent change. I.e. keep minimum-number-of-versions of 
> versions even past the delete marker.
> #1 would allow for more flexibility. #2 comes somewhat naturally with parent 
> (from a user viewpoint)
> Comments? Any other options?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4541) [book] book.xml - claritying schema design section on versions

[Doug Meil (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121134#comment-13121134
 ] 

Doug Meil commented on HBASE-4541:
--

Thanks!

> [book] book.xml - claritying schema design section on versions
> --
>
> Key: HBASE-4541
> URL: https://issues.apache.org/jira/browse/HBASE-4541
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: book_HBASE_4541.xml.patch
>
>
> In the Schema Design chapter there was a section on versions and it wasn't 
> clearly differentiating between max and min versions.
> * Moving the existing section on versions to a subsection called "Maximum 
> Number of Versions" so it's at the same level as "Minimum Number of 
> Versions", and added the "max" qualifier in that sub-section.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira