[jira] [Updated] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[dhruba borthakur (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dhruba borthakur updated HBASE-4528:


Attachment: appendNoSyncPut4.txt

Addressed Gary's concern. The HLog.sync() does not throw an exception if it 
encounters a HDFS error. Instead it triggers a logroll as usual.

If the put code encounter an error while syncing to hdfs, then it rollbacks the 
change to the memstore and throws an exception to the client.

Kannan: I will rerun my performance benchmark and report results.

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
> appendNoSyncPut3.txt, appendNoSyncPut4.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123403#comment-13123403
 ] 

jirapos...@reviews.apache.org commented on HBASE-4528:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2141/
---

(Updated 2011-10-08 07:50:25.069515)


Review request for hbase.


Changes
---

The HLog.sync() does not throw an exception if it encounters a HDFS error. 
Instead it triggers a logroll as usual. If the put code encounter an error 
while syncing to hdfs, then it rollbacks the change to the memstore and throws 
an exception to the client.


Summary
---

The changes the multiPut operation so that the sync to the wal occurs outside 
the rowlock.

This enhancement is done only to HRegion.mut(Put[]) because this is the only 
method that gets invoked from an application. The HRegion.put(Put) is used only 
by unit tests and should possibly be deprecated.

I have attached a unit test. I have not yet run all unit tests, but early 
feedback on this patch will be very helpful.


This addresses bug HBASE-4528.
https://issues.apache.org/jira/browse/HBASE-4528


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1180314 
  /src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 1180314 
  
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1180314 
  /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1180314 
  /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java 1180314 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java 
PRE-CREATION 
  /src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 1180314 
  /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 
1180314 

Diff: https://reviews.apache.org/r/2141/diff


Testing
---

Not yet run the full suite of unit tests.


Thanks,

Dhruba



> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
> appendNoSyncPut3.txt, appendNoSyncPut4.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[dhruba borthakur (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123404#comment-13123404
 ] 

dhruba borthakur commented on HBASE-4528:
-

It appears to me that deleting kvs from the memstore (as part of the 
transaction rollback) should be ok. It is done without the rowlock because the 
kvs are not yet visible to scanners because the rwcc is not yet advanced. 

One doubtful case is when a thread A is trying to insert kv via a Put call.  
Thread A inserted into memstore but has failed to sync to wal. Now suppose 
another thread B has concurrently inserted exactly the same kv into the 
memstore successfully and committed the transaction by syncing to wal 
successfully. Now thread A tries to rollback its failed transaction and removes 
the kv from the memstore. Is this scenario possible? In that case, should the 
rollback code in Thread A delete the kv from the memstore only if its 
kv.memstoreTS matches its own?

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
> appendNoSyncPut3.txt, appendNoSyncPut4.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123500#comment-13123500
 ] 

jirapos...@reviews.apache.org commented on HBASE-4528:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2141/#review2463
---


Nice work.
See comments below which aim to solve the case Dhruba raised @ 08/Oct/11 08:01


/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


We should pass w.getWriteNumber() to this method and use it to verify the 
memstoreTS of kv's matches the writeNumber.
w wouldn't be null in this case since step 8 shouldn't have been executed 
yet.



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


This line should be moved into an else block for the if block above.



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


We should verify that kv.getMemstoreTS() matches w.getWriteNumber() before 
removing from store.



/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java


Should be 'remove a KeyValue'


- Ted


On 2011-10-08 07:50:25, Dhruba Borthakur wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2141/
bq.  ---
bq.  
bq.  (Updated 2011-10-08 07:50:25)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The changes the multiPut operation so that the sync to the wal occurs 
outside the rowlock.
bq.  
bq.  This enhancement is done only to HRegion.mut(Put[]) because this is the 
only method that gets invoked from an application. The HRegion.put(Put) is used 
only by unit tests and should possibly be deprecated.
bq.  
bq.  I have attached a unit test. I have not yet run all unit tests, but early 
feedback on this patch will be very helpful.
bq.  
bq.  
bq.  This addresses bug HBASE-4528.
bq.  https://issues.apache.org/jira/browse/HBASE-4528
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 
1180314 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java 
1180314 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java 
PRE-CREATION 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 
1180314 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 
1180314 
bq.  
bq.  Diff: https://reviews.apache.org/r/2141/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Not yet run the full suite of unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Dhruba
bq.  
bq.



> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
> appendNoSyncPut3.txt, appendNoSyncPut4.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123503#comment-13123503
 ] 

jirapos...@reviews.apache.org commented on HBASE-4528:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2141/#review2464
---


+1

- ramkrishna


On 2011-10-08 07:50:25, Dhruba Borthakur wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2141/
bq.  ---
bq.  
bq.  (Updated 2011-10-08 07:50:25)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The changes the multiPut operation so that the sync to the wal occurs 
outside the rowlock.
bq.  
bq.  This enhancement is done only to HRegion.mut(Put[]) because this is the 
only method that gets invoked from an application. The HRegion.put(Put) is used 
only by unit tests and should possibly be deprecated.
bq.  
bq.  I have attached a unit test. I have not yet run all unit tests, but early 
feedback on this patch will be very helpful.
bq.  
bq.  
bq.  This addresses bug HBASE-4528.
bq.  https://issues.apache.org/jira/browse/HBASE-4528
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 
1180314 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java 
1180314 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java 
PRE-CREATION 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 
1180314 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 
1180314 
bq.  
bq.  Diff: https://reviews.apache.org/r/2141/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Not yet run the full suite of unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Dhruba
bq.  
bq.



> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
> appendNoSyncPut3.txt, appendNoSyncPut4.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123505#comment-13123505
 ] 

Ted Yu commented on HBASE-4528:
---

Minor suggestion for the new MemStore.remove() method:
Since it is called only in case of error recovery, I feel a better name maybe 
rollback().

> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
> appendNoSyncPut3.txt, appendNoSyncPut4.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123510#comment-13123510
 ] 

jirapos...@reviews.apache.org commented on HBASE-4218:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2308/#review2466
---


I ran unit tests with Jacek's patch. 1199 unit tests passed. The only one that 
failed was ServerCustomProtocol, which also seems to fail sporadically without 
the patch. Without the patch, there are only 1028 tests, so the patch is 
apparently very well unit-tested.

- Mikhail


On 2011-10-08 00:51:01, Jacek Migdal wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2308/
bq.  ---
bq.  
bq.  (Updated 2011-10-08 00:51:01)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Delta encoding for key values.
bq.  
bq.  
bq.  This addresses bug HBASE-4218.
bq.  https://issues.apache.org/jira/browse/HBASE-4218
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/BitsetKeyDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/BufferedDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/CompressionState.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/CopyKeyDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncodedBlock.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderAlgorithms.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderToSmallBufferException.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DiffKeyDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/FastDiffDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/deltaencoder/PrefixKeyDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/EmptyBlockDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockDeltaEncoder.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
 1180113 
bq.
http://svn.apache.org/repos/asf/hbase/tr

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123513#comment-13123513
 ] 

jirapos...@reviews.apache.org commented on HBASE-4528:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2141/#review2467
---



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


If walSyncSuccessful is true, rwcc.completeMemstoreInsert(w) on line 1852 
would have been called.
If walSyncSuccessful is false, removeKeysFromMemstore() on line 1878 rolls 
back the changes to memstore.
Looks like this line is no longer needed.


- Ted


On 2011-10-08 07:50:25, Dhruba Borthakur wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2141/
bq.  ---
bq.  
bq.  (Updated 2011-10-08 07:50:25)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The changes the multiPut operation so that the sync to the wal occurs 
outside the rowlock.
bq.  
bq.  This enhancement is done only to HRegion.mut(Put[]) because this is the 
only method that gets invoked from an application. The HRegion.put(Put) is used 
only by unit tests and should possibly be deprecated.
bq.  
bq.  I have attached a unit test. I have not yet run all unit tests, but early 
feedback on this patch will be very helpful.
bq.  
bq.  
bq.  This addresses bug HBASE-4528.
bq.  https://issues.apache.org/jira/browse/HBASE-4528
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 
1180314 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java
 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1180314 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java 
1180314 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestParallelPut.java 
PRE-CREATION 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 
1180314 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 
1180314 
bq.  
bq.  Diff: https://reviews.apache.org/r/2141/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Not yet run the full suite of unit tests.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Dhruba
bq.  
bq.



> The put operation can release the rowlock before sync-ing the Hlog
> --
>
> Key: HBASE-4528
> URL: https://issues.apache.org/jira/browse/HBASE-4528
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
> appendNoSyncPut3.txt, appendNoSyncPut4.txt
>
>
> This allows for better throughput when there are hot rows. A single row 
> update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-1744) Thrift server to match the new java api.

[Tim Sell (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Sell updated HBASE-1744:


Status: Patch Available  (was: Open)

> Thrift server to match the new java api.
> 
>
> Key: HBASE-1744
> URL: https://issues.apache.org/jira/browse/HBASE-1744
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Tim Sell
>Assignee: Tim Sell
>Priority: Critical
> Fix For: 0.94.0
>
> Attachments: HBASE-1744.2.patch, HBASE-1744.3.patch, 
> HBASE-1744.4.patch, HBASE-1744.5.patch, HBASE-1744.6.patch, 
> HBASE-1744.7.patch, HBASE-1744.preview.1.patch, thriftexperiment.patch
>
>
> This mutateRows, etc.. is a little confusing compared to the new cleaner java 
> client.
> Thinking of ways to make a thrift client that is just as elegant. something 
> like:
> void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
> with:
> struct TColumn {
>   1:Bytes family,
>   2:Bytes qualifier,
>   3:i64 timestamp
> }
> struct TPut {
>   1:Bytes row,
>   2:map values
> }
> This creates more verbose rpc  than if the columns in TPut were just 
> map>, but that is harder to fit timestamps into and 
> still be intuitive from say python.
> Presumably the goal of a thrift gateway is to be easy first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-1744) Thrift server to match the new java api.

[Tim Sell (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Sell updated HBASE-1744:


Attachment: HBASE-1744.7.patch

new patch, added licenses and trivial python demo.

> Thrift server to match the new java api.
> 
>
> Key: HBASE-1744
> URL: https://issues.apache.org/jira/browse/HBASE-1744
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Tim Sell
>Assignee: Tim Sell
>Priority: Critical
> Fix For: 0.94.0
>
> Attachments: HBASE-1744.2.patch, HBASE-1744.3.patch, 
> HBASE-1744.4.patch, HBASE-1744.5.patch, HBASE-1744.6.patch, 
> HBASE-1744.7.patch, HBASE-1744.preview.1.patch, thriftexperiment.patch
>
>
> This mutateRows, etc.. is a little confusing compared to the new cleaner java 
> client.
> Thinking of ways to make a thrift client that is just as elegant. something 
> like:
> void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
> with:
> struct TColumn {
>   1:Bytes family,
>   2:Bytes qualifier,
>   3:i64 timestamp
> }
> struct TPut {
>   1:Bytes row,
>   2:map values
> }
> This creates more verbose rpc  than if the columns in TPut were just 
> map>, but that is harder to fit timestamps into and 
> still be intuitive from say python.
> Presumably the goal of a thrift gateway is to be easy first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1744) Thrift server to match the new java api.

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123531#comment-13123531
 ] 

Ted Yu commented on HBASE-1744:
---

+1 on patch v7.

> Thrift server to match the new java api.
> 
>
> Key: HBASE-1744
> URL: https://issues.apache.org/jira/browse/HBASE-1744
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Tim Sell
>Assignee: Tim Sell
>Priority: Critical
> Fix For: 0.94.0
>
> Attachments: HBASE-1744.2.patch, HBASE-1744.3.patch, 
> HBASE-1744.4.patch, HBASE-1744.5.patch, HBASE-1744.6.patch, 
> HBASE-1744.7.patch, HBASE-1744.preview.1.patch, thriftexperiment.patch
>
>
> This mutateRows, etc.. is a little confusing compared to the new cleaner java 
> client.
> Thinking of ways to make a thrift client that is just as elegant. something 
> like:
> void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
> with:
> struct TColumn {
>   1:Bytes family,
>   2:Bytes qualifier,
>   3:i64 timestamp
> }
> struct TPut {
>   1:Bytes row,
>   2:map values
> }
> This creates more verbose rpc  than if the columns in TPut were just 
> map>, but that is harder to fit timestamps into and 
> still be intuitive from say python.
> Presumably the goal of a thrift gateway is to be easy first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4557) Do something better than UnknownScannerException

[Jean-Daniel Cryans (Created) (JIRA)]

Do something better than UnknownScannerException


 Key: HBASE-4557
 URL: https://issues.apache.org/jira/browse/HBASE-4557
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
 Fix For: 0.94.0


UnknownScannerException is a plague, there's no reason we should not at least 
try to create a new scanner. If that fails again, maybe try automatically 
setting a lower scanner caching (if possible and with proper loggin) and retry 
again.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4102) atomicAppend: A put that appends to the latest version of a cell; i.e. reads current value then adds the bytes offered by the client to the tail and writes out a new entr

[Lars Hofhansl (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4102:
-

Assignee: Lars Hofhansl

This seems like a simple useful addition.
Should this be in 0.92 or 0.94?

> atomicAppend: A put that appends to the latest version of a cell; i.e. reads 
> current value then adds the bytes offered by the client to the tail and 
> writes out a new entry
> ---
>
> Key: HBASE-4102
> URL: https://issues.apache.org/jira/browse/HBASE-4102
> Project: HBase
>  Issue Type: New Feature
>Reporter: stack
>Assignee: Lars Hofhansl
>
> Its come up a few times that clients want to add to an existing cell rather 
> than make a new cell each time.  At our place, the frontend keeps a list of 
> urls a user has visited -- their md5s -- and updates it as user progresses.  
> Rather than read, modify client-side, then write new value back to hbase, it 
> would be sweet if could do it all in one operation in hbase server.  TSDB 
> aims to be space efficient.  Rather than pay the cost of the KV wrapper per 
> metric, it would rather have a KV for an interval an in this KV have a value 
> that is all the metrics for the period.
> It could be done as a coprocessor but this feels more like a fundamental 
> feature.
> Benoît suggests that atomicAppend take a flag to indicate whether or not the 
> client wants to see the resulting cell; often a client won't want to see the 
> result and in this case, why pay the price formulating and delivering a 
> response that client just drops.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123567#comment-13123567
 ] 

stack commented on HBASE-4553:
--

So, I see this in this failure up on jenkins in 
https://builds.apache.org/job/HBase-0.92/53/artifact/trunk/target/surefire-reports/org.apache.hadoop.hbase.avro.TestAvroServer-output.txt

I see that we updated .tableinfo and a check on the files modtime at about the 
same time fails with filenotfound.

Todd, should we spin here till file shows up?

> The update of .tableinfo is not atomic; we remove then rename
> -
>
> Key: HBASE-4553
> URL: https://issues.apache.org/jira/browse/HBASE-4553
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>
> This comes of HBASE-4547.  The rename in 0.20 hdfs fails if file exists 
> already.  In 0.20+ its better but still 'some' issues if existing reader when 
> file is renamed.  This issue is about fixing this (though we depend on fix 
> first being in hdfs).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123569#comment-13123569
 ] 

jirapos...@reviews.apache.org commented on HBASE-4540:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2251/#review2469
---

Ship it!


I'm good on commit.

Have some suggestions for future handler tests below.  I'm ok if we commit w/o 
addressing them here.

Nice fix Ram


http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java


We don't have this method already in our ZK* classes?



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java


Do you have to spin up the cluster twice?   Could you do it once only in 
@BeforeClass and then shut it down in @AfterClass?  So its run once only?



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java


Good test.

Would it be possible to test the handler without spinning up the cluster?  
See TestOpenRegionHandler over under regionserver.handler in tests -- they 
don't spin up a cluster, just zk.  Test can run faster if no dfs+hbase.  Not 
important.  For the future.


- Michael


On 2011-10-08 05:13:32, ramkrishna vasudevan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2251/
bq.  ---
bq.  
bq.  (Updated 2011-10-08 05:13:32)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jonathan Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Fix for handling HBASE-4539 and HBASE-4540.
bq.  Ran all the testcases.  Added one new testcase to verify 
OpenedRegionHandler scenarios.
bq.  Also addresses Ted's comments.
bq.  
bq.  
bq.  This addresses bug HBASE-4540.
bq.  https://issues.apache.org/jira/browse/HBASE-4540
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
 1179945 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java
 1179945 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java
 1179945 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
 1179945 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2251/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Yes
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  ramkrishna
bq.  
bq.



> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> tr

[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing

[Jesse Yates (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123570#comment-13123570
 ] 

Jesse Yates commented on HBASE-4480:


nitpick: I would prefer the options be labeled in the help as something like 
'-f FILE' rather than '-f=FILE' - it feels more clear.

If you could put together an actual patch and update the docs that we can 
actually push into the code base, that would be great.

> Testing script to simplfy local testing
> ---
>
> Key: HBASE-4480
> URL: https://issues.apache.org/jira/browse/HBASE-4480
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jesse Yates
>Priority: Minor
>  Labels: test
> Attachments: runtest.sh, runtest2.sh
>
>
> As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
> http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
> script that would handle more of the finer points of running/checking our 
> test suite.
> This script should:
> (1) Allow people to determine which tests are hanging/taking a long time to 
> run
> (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
> running the whole suite that caused the failure
> (3) Allow people to specify to run just unit tests or also integration tests 
> (essentially wrapping calls to 'maven test' and 'maven verify').
> This script should just be a convenience script - running tests directly from 
> maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4102) atomicAppend: A put that appends to the latest version of a cell; i.e. reads current value then adds the bytes offered by the client to the tail and writes out a new en

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123572#comment-13123572
 ] 

Lars Hofhansl commented on HBASE-4102:
--

I have this working now.

But now I realized two things:
1. I modeled it after the old ICV. I assume we want something like the new 
Increment API.
2. Is this something that even want to build into HBase? Or should a user 
implement this with a coprocessor endpoint? (It would be possible to do with a 
coprocessor, albeit not quite as efficient as an endpoint would have no access 
to the Stores.


> atomicAppend: A put that appends to the latest version of a cell; i.e. reads 
> current value then adds the bytes offered by the client to the tail and 
> writes out a new entry
> ---
>
> Key: HBASE-4102
> URL: https://issues.apache.org/jira/browse/HBASE-4102
> Project: HBase
>  Issue Type: New Feature
>Reporter: stack
>Assignee: Lars Hofhansl
>
> Its come up a few times that clients want to add to an existing cell rather 
> than make a new cell each time.  At our place, the frontend keeps a list of 
> urls a user has visited -- their md5s -- and updates it as user progresses.  
> Rather than read, modify client-side, then write new value back to hbase, it 
> would be sweet if could do it all in one operation in hbase server.  TSDB 
> aims to be space efficient.  Rather than pay the cost of the KV wrapper per 
> metric, it would rather have a KV for an interval an in this KV have a value 
> that is all the metrics for the period.
> It could be done as a coprocessor but this feels more like a fundamental 
> feature.
> Benoît suggests that atomicAppend take a flag to indicate whether or not the 
> client wants to see the resulting cell; often a client won't want to see the 
> result and in this case, why pay the price formulating and delivering a 
> response that client just drops.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (HBASE-4102) atomicAppend: A put that appends to the latest version of a cell; i.e. reads current value then adds the bytes offered by the client to the tail and writes out a new

[Lars Hofhansl (Work started) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-4102 started by Lars Hofhansl.

> atomicAppend: A put that appends to the latest version of a cell; i.e. reads 
> current value then adds the bytes offered by the client to the tail and 
> writes out a new entry
> ---
>
> Key: HBASE-4102
> URL: https://issues.apache.org/jira/browse/HBASE-4102
> Project: HBase
>  Issue Type: New Feature
>Reporter: stack
>Assignee: Lars Hofhansl
>
> Its come up a few times that clients want to add to an existing cell rather 
> than make a new cell each time.  At our place, the frontend keeps a list of 
> urls a user has visited -- their md5s -- and updates it as user progresses.  
> Rather than read, modify client-side, then write new value back to hbase, it 
> would be sweet if could do it all in one operation in hbase server.  TSDB 
> aims to be space efficient.  Rather than pay the cost of the KV wrapper per 
> metric, it would rather have a KV for an interval an in this KV have a value 
> that is all the metrics for the period.
> It could be done as a coprocessor but this feels more like a fundamental 
> feature.
> Benoît suggests that atomicAppend take a flag to indicate whether or not the 
> client wants to see the resulting cell; often a client won't want to see the 
> result and in this case, why pay the price formulating and delivering a 
> response that client just drops.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing

[Jesse Yates (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123573#comment-13123573
 ] 

Jesse Yates commented on HBASE-4480:


Oh, and also adding an option for running clean before the tests would be nice. 
Its odd when I go to run a single test (with the top 20 slowest) and it prints 
out more than 1 result.

> Testing script to simplfy local testing
> ---
>
> Key: HBASE-4480
> URL: https://issues.apache.org/jira/browse/HBASE-4480
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jesse Yates
>Priority: Minor
>  Labels: test
> Attachments: runtest.sh, runtest2.sh
>
>
> As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
> http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
> script that would handle more of the finer points of running/checking our 
> test suite.
> This script should:
> (1) Allow people to determine which tests are hanging/taking a long time to 
> run
> (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
> running the whole suite that caused the failure
> (3) Allow people to specify to run just unit tests or also integration tests 
> (essentially wrapping calls to 'maven test' and 'maven verify').
> This script should just be a convenience script - running tests directly from 
> maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123576#comment-13123576
 ] 

Ted Yu commented on HBASE-4218:
---

For BlockDeltaEncoder.afterBlockCache(), I am not sure if the following matches 
the logic:
{code}
  // Postcondition: if (isCompaction is set and onDisk is not NONR) or
  //inMemory is not set -> don;t encode.
{code}

> Delta Encoding of KeyValues  (aka prefix compression)
> -
>
> Key: HBASE-4218
> URL: https://issues.apache.org/jira/browse/HBASE-4218
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 0.94.0
>Reporter: Jacek Migdal
>  Labels: compression
> Attachments: open-source.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually very 
> similar. Because of that, it is possible to design better compression than 
> general purpose algorithms,
> It is an additional step designed to be used in memory. It aims to save 
> memory in cache as well as speeding seeks within HFileBlocks. It should 
> improve performance a lot, if key lengths are larger than value lengths. For 
> example, it makes a lot of sense to use it when value is a counter.
> Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) 
> shows that I could achieve decent level of compression:
>  key compression ratio: 92%
>  total compression ratio: 85%
>  LZO on the same data: 85%
>  LZO after delta encoding: 91%
> While having much better performance (20-80% faster decompression ratio than 
> LZO). Moreover, it should allow far more efficient seeking which should 
> improve performance a bit.
> It seems that a simple compression algorithms are good enough. Most of the 
> savings are due to prefix compression, int128 encoding, timestamp diffs and 
> bitfields to avoid duplication. That way, comparisons of compressed data can 
> be much faster than a byte comparator (thanks to prefix compression and 
> bitfields).
> In order to implement it in HBase two important changes in design will be 
> needed:
> -solidify interface to HFileBlock / HFileReader Scanner to provide seeking 
> and iterating; access to uncompressed buffer in HFileBlock will have bad 
> performance
> -extend comparators to support comparison assuming that N first bytes are 
> equal (or some fields are equal)
> Link to a discussion about something similar:
> http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4489) Better key splitting in RegionSplitter

[Dave Revell (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123577#comment-13123577
 ] 

Dave Revell commented on HBASE-4489:


@Nicolas, responding to your points one-by-one:

1 and 2. This is a perfectly valid key format (choosing readability over space 
efficiency). Hashed keys are good. There is no disagreement here, but as a 
default it is inefficient for people who don't need to print their HBase keys, 
or are willing to parse their keys before printing. Claims of equal 
compressibility need evidence (raw bytes vs. ASCII)
3. Why should Java integers or longs have anything to do with an MD5 hash, 
which is a sequence of 16 bytes? Do we expect clients to truncate their MD5 
hashes and make sure the high-order bit is a 0 (as required to be in the range 
0-7F)? This is a bizarre default and puts a strange burden on clients, whose 
MD5 generator is giving them an arbitrary 128-bit array.
4. What unevenness are you referring to? If you're referring to the unevenness 
that results from using arbitrary keys in a partitioning scheme designed for 
ASCII, it can be quite bad. The first ASCII character, '0', has the ordinal 48. 
So the first region would cover the range (empty)..48), which is 48/256 = 18% 
of the key space, regardless of how many regions there are.

Maybe we should chat on freenode or Monday? I think it would be fast and easy 
to figure out where we disagree if we were chatting in realtime. Also, thanks 
for the input.

> Better key splitting in RegionSplitter
> --
>
> Key: HBASE-4489
> URL: https://issues.apache.org/jira/browse/HBASE-4489
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Dave Revell
>Assignee: Dave Revell
> Attachments: HBASE-4489-branch0.90-v1.patch, HBASE-4489-trunk-v1.patch
>
>
> The RegionSplitter utility allows users to create a pre-split table from the 
> command line or do a rolling split on an existing table. It supports 
> pluggable split algorithms that implement the SplitAlgorithm interface. The 
> only/default SplitAlgorithm is one that assumes keys fall in the range from 
> ASCII string "" to ASCII string "7FFF". This is not a sane 
> default, and seems useless to most users. Users are likely to be surprised by 
> the fact that all the region splits occur in in the byte range of ASCII 
> characters.
> A better default split algorithm would be one that evenly divides the space 
> of all bytes, which is what this patch does. Making a table with five regions 
> would split at \x33\x33..., \x66\x66, \x99\x99..., \xCC\xCC..., and 
> \xFF\xFF.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123597#comment-13123597
 ] 

Ted Yu commented on HBASE-4218:
---

EmptyBlockDeltaEncoder, CompressionState, BlockDeltaEncoder need license.

> Delta Encoding of KeyValues  (aka prefix compression)
> -
>
> Key: HBASE-4218
> URL: https://issues.apache.org/jira/browse/HBASE-4218
> Project: HBase
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 0.94.0
>Reporter: Jacek Migdal
>  Labels: compression
> Attachments: open-source.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually very 
> similar. Because of that, it is possible to design better compression than 
> general purpose algorithms,
> It is an additional step designed to be used in memory. It aims to save 
> memory in cache as well as speeding seeks within HFileBlocks. It should 
> improve performance a lot, if key lengths are larger than value lengths. For 
> example, it makes a lot of sense to use it when value is a counter.
> Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) 
> shows that I could achieve decent level of compression:
>  key compression ratio: 92%
>  total compression ratio: 85%
>  LZO on the same data: 85%
>  LZO after delta encoding: 91%
> While having much better performance (20-80% faster decompression ratio than 
> LZO). Moreover, it should allow far more efficient seeking which should 
> improve performance a bit.
> It seems that a simple compression algorithms are good enough. Most of the 
> savings are due to prefix compression, int128 encoding, timestamp diffs and 
> bitfields to avoid duplication. That way, comparisons of compressed data can 
> be much faster than a byte comparator (thanks to prefix compression and 
> bitfields).
> In order to implement it in HBase two important changes in design will be 
> needed:
> -solidify interface to HFileBlock / HFileReader Scanner to provide seeking 
> and iterating; access to uncompressed buffer in HFileBlock will have bad 
> performance
> -extend comparators to support comparison assuming that N first bytes are 
> equal (or some fields are equal)
> Link to a discussion about something similar:
> http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename

[Todd Lipcon (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123603#comment-13123603
 ] 

Todd Lipcon commented on HBASE-4553:


atomic operations on filesystems are tricky... to do this "correctly" in the 
face of crashes, we need to have some process either do a rollback or 
roll-forward to recover from failures. Something like:

writer:
- create tableinfo.tmp
- delete tableinfo
- rename tableinfo.tmp to tableinfo

reader:
- try to read tableinfo
- on IOE (block missing, etc), that means that the file was deleted underneath. 
So spin until the file open succeeds.


if the writer crashes between the delete and rename, we need someone else to 
come in and "finish" the operation.

IMO we need some general purpose way of allowing the master to keep an "intent 
log" in ZK for this kind of thing - and then if the master fails over, it can 
complete the operation.

> The update of .tableinfo is not atomic; we remove then rename
> -
>
> Key: HBASE-4553
> URL: https://issues.apache.org/jira/browse/HBASE-4553
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>
> This comes of HBASE-4547.  The rename in 0.20 hdfs fails if file exists 
> already.  In 0.20+ its better but still 'some' issues if existing reader when 
> file is renamed.  This issue is about fixing this (though we depend on fix 
> first being in hdfs).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123616#comment-13123616
 ] 

stack commented on HBASE-4553:
--

@Todd Thanks boss.

> The update of .tableinfo is not atomic; we remove then rename
> -
>
> Key: HBASE-4553
> URL: https://issues.apache.org/jira/browse/HBASE-4553
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>
> This comes of HBASE-4547.  The rename in 0.20 hdfs fails if file exists 
> already.  In 0.20+ its better but still 'some' issues if existing reader when 
> file is renamed.  This issue is about fixing this (though we depend on fix 
> first being in hdfs).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4102) atomicAppend: A put that appends to the latest version of a cell; i.e. reads current value then adds the bytes offered by the client to the tail and writes out a new en

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123618#comment-13123618
 ] 

stack commented on HBASE-4102:
--

@Lars On whether it should be 0.92 or 0.94, I'm thinking 0.94 (because people 
are watching me -- J-D and Todd will kill me if I commit a new feature to 
0.92).  That said, I think at least SU will patch their 0.92 with this patch to 
get this feature; we need it.

I think we want it like new increment API.

On doing it as a CP, the argument is that this is a fundamental rather than 
something to do as a CP.

> atomicAppend: A put that appends to the latest version of a cell; i.e. reads 
> current value then adds the bytes offered by the client to the tail and 
> writes out a new entry
> ---
>
> Key: HBASE-4102
> URL: https://issues.apache.org/jira/browse/HBASE-4102
> Project: HBase
>  Issue Type: New Feature
>Reporter: stack
>Assignee: Lars Hofhansl
>
> Its come up a few times that clients want to add to an existing cell rather 
> than make a new cell each time.  At our place, the frontend keeps a list of 
> urls a user has visited -- their md5s -- and updates it as user progresses.  
> Rather than read, modify client-side, then write new value back to hbase, it 
> would be sweet if could do it all in one operation in hbase server.  TSDB 
> aims to be space efficient.  Rather than pay the cost of the KV wrapper per 
> metric, it would rather have a KV for an interval an in this KV have a value 
> that is all the metrics for the period.
> It could be done as a coprocessor but this feels more like a fundamental 
> feature.
> Benoît suggests that atomicAppend take a flag to indicate whether or not the 
> client wants to see the resulting cell; often a client won't want to see the 
> result and in this case, why pay the price formulating and delivering a 
> response that client just drops.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HBASE-4430) Disable TestSlabCache and TestSingleSizedCache temporarily to see if these are cause of build box failure though all tests pass

[stack (Reopened) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-4430:
--


Reopening because TestSlabCache failed in most recent build -- hung.  Going to 
disable this test again.

> Disable TestSlabCache and TestSingleSizedCache temporarily to see if these 
> are cause of build box failure though all tests pass
> ---
>
> Key: HBASE-4430
> URL: https://issues.apache.org/jira/browse/HBASE-4430
> Project: HBase
>  Issue Type: Task
>  Components: test
>Reporter: stack
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: TestSlabCache.trace
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4430) Disable TestSlabCache and TestSingleSizedCache temporarily to see if these are cause of build box failure though all tests pass

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4430:
-

Priority: Major  (was: Blocker)

Marking 'major' rather than blocker since this is experimental feature -- 
shouldn't hold up 0.92.

> Disable TestSlabCache and TestSingleSizedCache temporarily to see if these 
> are cause of build box failure though all tests pass
> ---
>
> Key: HBASE-4430
> URL: https://issues.apache.org/jira/browse/HBASE-4430
> Project: HBase
>  Issue Type: Task
>  Components: test
>Reporter: stack
>Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: TestSlabCache.trace
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4430) Disable TestSlabCache and TestSingleSizedCache temporarily to see if these are cause of build box failure though all tests pass

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123621#comment-13123621
 ] 

stack commented on HBASE-4430:
--

I disabled all the tests in TestSlabCache for now.  I looked at build output 
and there is nothing in the .txt and no -output.txt unfortunately making it 
harder debug.

> Disable TestSlabCache and TestSingleSizedCache temporarily to see if these 
> are cause of build box failure though all tests pass
> ---
>
> Key: HBASE-4430
> URL: https://issues.apache.org/jira/browse/HBASE-4430
> Project: HBase
>  Issue Type: Task
>  Components: test
>Reporter: stack
>Assignee: Li Pi
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: TestSlabCache.trace
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4430) Disable TestSlabCache and TestSingleSizedCache temporarily to see if these are cause of build box failure though all tests pass

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123623#comment-13123623
 ] 

Hudson commented on HBASE-4430:
---

Integrated in HBase-TRUNK #2309 (See 
[https://builds.apache.org/job/HBase-TRUNK/2309/])
HBASE-4430 Disable TestSlabCache and TestSingleSizedCache temporarily to 
see if these are cause of build box failure though all tests pass -- DISABLED 
TESTSLABCACHE AGAIN... its hanging

stack : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java


> Disable TestSlabCache and TestSingleSizedCache temporarily to see if these 
> are cause of build box failure though all tests pass
> ---
>
> Key: HBASE-4430
> URL: https://issues.apache.org/jira/browse/HBASE-4430
> Project: HBase
>  Issue Type: Task
>  Components: test
>Reporter: stack
>Assignee: Li Pi
> Fix For: 0.92.0
>
> Attachments: TestSlabCache.trace
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4480) Testing script to simplfy local testing

[Scott Kuehn (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Kuehn updated HBASE-4480:
---

Attachment: HBASE-4480.patch

> Testing script to simplfy local testing
> ---
>
> Key: HBASE-4480
> URL: https://issues.apache.org/jira/browse/HBASE-4480
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jesse Yates
>Priority: Minor
>  Labels: test
> Attachments: HBASE-4480.patch, runtest.sh, runtest2.sh
>
>
> As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
> http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
> script that would handle more of the finer points of running/checking our 
> test suite.
> This script should:
> (1) Allow people to determine which tests are hanging/taking a long time to 
> run
> (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
> running the whole suite that caused the failure
> (3) Allow people to specify to run just unit tests or also integration tests 
> (essentially wrapping calls to 'maven test' and 'maven verify').
> This script should just be a convenience script - running tests directly from 
> maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing

[Scott Kuehn (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123626#comment-13123626
 ] 

Scott Kuehn commented on HBASE-4480:


@Jesse - done.  Three fixes/improvements to the latest script:
- the -s option only reports duration of tests that were run (ignores stale 
surefire files)
- added -c option for cleaning prior to test running. this will just force a 
'mvn clean'
- more robust path management

> Testing script to simplfy local testing
> ---
>
> Key: HBASE-4480
> URL: https://issues.apache.org/jira/browse/HBASE-4480
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jesse Yates
>Priority: Minor
>  Labels: test
> Attachments: HBASE-4480.patch, runtest.sh, runtest2.sh
>
>
> As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
> http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
> script that would handle more of the finer points of running/checking our 
> test suite.
> This script should:
> (1) Allow people to determine which tests are hanging/taking a long time to 
> run
> (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
> running the whole suite that caused the failure
> (3) Allow people to specify to run just unit tests or also integration tests 
> (essentially wrapping calls to 'maven test' and 'maven verify').
> This script should just be a convenience script - running tests directly from 
> maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing

[Jesse Yates (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123629#comment-13123629
 ] 

Jesse Yates commented on HBASE-4480:


@Scott good stuff.

nitpick: if I'm doing a clean, it should run 'mvn clean test' rather than 
running 'mvn clean' and then 'mvn test'

> Testing script to simplfy local testing
> ---
>
> Key: HBASE-4480
> URL: https://issues.apache.org/jira/browse/HBASE-4480
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jesse Yates
>Priority: Minor
>  Labels: test
> Attachments: HBASE-4480.patch, runtest.sh, runtest2.sh
>
>
> As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
> http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
> script that would handle more of the finer points of running/checking our 
> test suite.
> This script should:
> (1) Allow people to determine which tests are hanging/taking a long time to 
> run
> (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
> running the whole suite that caused the failure
> (3) Allow people to specify to run just unit tests or also integration tests 
> (essentially wrapping calls to 'maven test' and 'maven verify').
> This script should just be a convenience script - running tests directly from 
> maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4480) Testing script to simplfy local testing

[Scott Kuehn (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Kuehn updated HBASE-4480:
---

Attachment: HBASE-4480_v2.patch

> Testing script to simplfy local testing
> ---
>
> Key: HBASE-4480
> URL: https://issues.apache.org/jira/browse/HBASE-4480
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jesse Yates
>Priority: Minor
>  Labels: test
> Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, runtest.sh, 
> runtest2.sh
>
>
> As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
> http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
> script that would handle more of the finer points of running/checking our 
> test suite.
> This script should:
> (1) Allow people to determine which tests are hanging/taking a long time to 
> run
> (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
> running the whole suite that caused the failure
> (3) Allow people to specify to run just unit tests or also integration tests 
> (essentially wrapping calls to 'maven test' and 'maven verify').
> This script should just be a convenience script - running tests directly from 
> maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4480) Testing script to simplfy local testing

[Scott Kuehn (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123632#comment-13123632
 ] 

Scott Kuehn commented on HBASE-4480:


@Jesse - uploaded a v2 of the patch with this change

> Testing script to simplfy local testing
> ---
>
> Key: HBASE-4480
> URL: https://issues.apache.org/jira/browse/HBASE-4480
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jesse Yates
>Priority: Minor
>  Labels: test
> Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, runtest.sh, 
> runtest2.sh
>
>
> As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
> http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
> script that would handle more of the finer points of running/checking our 
> test suite.
> This script should:
> (1) Allow people to determine which tests are hanging/taking a long time to 
> run
> (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
> running the whole suite that caused the failure
> (3) Allow people to specify to run just unit tests or also integration tests 
> (essentially wrapping calls to 'maven test' and 'maven verify').
> This script should just be a convenience script - running tests directly from 
> maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123633#comment-13123633
 ] 

jirapos...@reviews.apache.org commented on HBASE-4540:
--



bq.  On 2011-10-08 21:55:31, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java,
 line 873
bq.  > 
bq.  >
bq.  > We don't have this method already in our ZK* classes?

@Stack
ZKAssign() did have getDataAndWatch() that accepts stat object.  Only ZKUtil 
had but it returned data in bytes which had to be again converted to 
RegionTransitionData. 
Hence added an utility api in ZKAssign itself and thought it may be useful in 
future also.


bq.  On 2011-10-08 21:55:31, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java,
 line 102
bq.  > 
bq.  >
bq.  > Good test.
bq.  > 
bq.  > Would it be possible to test the handler without spinning up the 
cluster?  See TestOpenRegionHandler over under regionserver.handler in tests -- 
they don't spin up a cluster, just zk.  Test can run faster if no dfs+hbase.  
Not important.  For the future.

@Stack
I can do like that atleast for one of the testcases in TestOpenedRegionHandler. 
 But i have to use the MockServer and MockRegionServices.
I will raise one minor improvement task to do that. Currently MockServer and 
MockRegionServices are under regionserver.handler package but the new testcase 
is in master package.  So better we can move it to a test.utility package and 
then use it across. So i will currently go with this commit and then track the 
new improvement JIRA to closure.


- ramkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2251/#review2469
---


On 2011-10-08 05:13:32, ramkrishna vasudevan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2251/
bq.  ---
bq.  
bq.  (Updated 2011-10-08 05:13:32)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jonathan Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Fix for handling HBASE-4539 and HBASE-4540.
bq.  Ran all the testcases.  Added one new testcase to verify 
OpenedRegionHandler scenarios.
bq.  Also addresses Ted's comments.
bq.  
bq.  
bq.  This addresses bug HBASE-4540.
bq.  https://issues.apache.org/jira/browse/HBASE-4540
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
 1179945 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java
 1179945 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java
 1179945 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
 1179945 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2251/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Yes
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  ramkrishna
bq.  
bq.



> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,

[jira] [Created] (HBASE-4558) Refactor TestOpenedRegionHandler and TestOpenRegionHandler.

[ramkrishna.s.vasudevan (Created) (JIRA)]

Refactor TestOpenedRegionHandler and TestOpenRegionHandler.
---

 Key: HBASE-4558
 URL: https://issues.apache.org/jira/browse/HBASE-4558
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor


This is an improvement task taken up to refactor TestOpenedRegionandler and 
TestOpenRegionHandler so that MockServer and MockRegionServerServices can be 
accessed from a common utility package.
If we do this then one of the testcases in TestOpenedRegionHandler need not 
start up a cluster and also moving it into a common package will help in 
mocking the server for future testcases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4540) OpenedRegionHandler is not enforcing atomicity of the operation it is performing

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123637#comment-13123637
 ] 

ramkrishna.s.vasudevan commented on HBASE-4540:
---

Integrated to 0.92 and trunk.
Thanks for the reviews Ted, Gray, J-D and Stack.
Created HBASE-4558 to refactor Stack's comment on using testcases without 
starting cluster.

> OpenedRegionHandler is not enforcing atomicity of the operation it is 
> performing
> 
>
> Key: HBASE-4540
> URL: https://issues.apache.org/jira/browse/HBASE-4540
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4540_1.patch
>
>
> -> OpenedRegionHandler has not yet deleted the znode of the region R1 opened 
> by RS1.
> -> RS1 goes down.
> -> Servershutdownhandler assigns the region R1 to RS2.
> -> The znode of R1 is moved to OFFLINE state by master or OPENING state by 
> RS2 if RS2 has started opening the region.
> -> Now the first OpenedRegionHandler tries to delete the znode thinking its 
> in OPENED state but fails.
> -> Though it fails it removes the node from RIT and adds RS1 as the owner of 
> R1 in master's memory.
> -> Now when RS2 completes opening the region the master is not able to open 
> the region as already the reigon has been deleted from RIT.
> {code}
> Master
> ==
> 2011-10-05 20:49:45,301 INFO 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
> processing of shutdown of linux146,60020,1317827727647
> 2011-10-05 20:49:54,177 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
> running balancer because 1 region(s) in transition: 
> {3e69d628a8bd8e9b7c5e7a2a6e03aad9=t1,,1317827883842.3e69d628a8bd8e9b7c5e7a2a6e03aad9.
>  state=PENDING_OPEN, ts=1317827985272, server=linux76,60020,1317827746847}
> 2011-10-05 20:49:57,720 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=M_ZK_REGION_OFFLINE, server=linux76,6,1317827742012, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:14,501 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Deleting existing unassigned node for 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 that is in expected state RS_ZK_REGION_OPENED
> 2011-10-05 20:50:14,505 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: 
> master:6-0x132d3dc13090023 Attempting to delete unassigned node 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 in RS_ZK_REGION_OPENED state but node is in 
> RS_ZK_REGION_OPENING state
> After the region is opened in RS2
> =
> 2011-10-05 20:50:48,066 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9, which is more than 15 seconds late
> 2011-10-05 20:50:48,290 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> 2011-10-05 20:50:53,743 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Handling 
> transition=RS_ZK_REGION_OPENING, server=linux76,60020,1317827746847, 
> region=3e69d628a8bd8e9b7c5e7a2a6e03aad9
> 2011-10-05 20:50:54,182 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
> Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
> 2011-10-05 20:50:54,397 WARN 
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENING for region 
> 3e69d628a8bd8e9b7c5e7a2a6e03aad9 from server linux76,60020,1317827746847 but 
> region was in  the state null and not in expected PENDING_OPEN or OPENING 
> states
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira