[jira] [Commented] (HBASE-5747) Forward port "hbase-5708 [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test"
[ https://issues.apache.org/jira/browse/HBASE-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253856#comment-13253856 ] Mikhail Bautin commented on HBASE-5747: --- @Stack: thanks a lot for taking care of this forward-port! (I know we should normally port our 89-fb changes to trunk ourselves.) > Forward port "hbase-5708 [89-fb] Make MiniMapRedCluster directory a > subdirectory of target/test" > > > Key: HBASE-5747 > URL: https://issues.apache.org/jira/browse/HBASE-5747 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5474.txt, 5474v2.txt, 5474v3 (1).txt, 5474v3.txt, > 5708v4.txt, 5708v4.txt > > > Forward port as much as we can of Mikhail's hard-won test cleanups over on > 0.89 branch Will improve our being able to run unit tests in //. He also > found a few bugs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248932#comment-13248932 ] Mikhail Bautin commented on HBASE-5708: --- @Stack: yes, it would be great if you could forward-port the applicable parts of the 89-fb patch to trunk. I would be happy to answer any questions. > [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test > -- > > Key: HBASE-5708 > URL: https://issues.apache.org/jira/browse/HBASE-5708 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D2601.1.patch, D2601.2.patch, D2601.3.patch, > D2601.4.patch > > > Some map-reduce-based tests are failing when executed concurrently in 89-fb > because mini-map-reduce cluster uses /tmp/hadoop- for temporary > data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5355) Compressed RPC's for HBase
[ https://issues.apache.org/jira/browse/HBASE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248892#comment-13248892 ] Mikhail Bautin commented on HBASE-5355: --- We have committed this feature (internally) to 89-fb. The implementation is available at https://reviews.facebook.net/D1671. I tried to apply a patch to trunk, but it turned out to be very complex due to protocol refactoring done in trunk. Since trunk is moving to protocol buffers, this feature might be implemented very differently there compared to our RPC compression implementation 89-fb. Given these complications, we currently do not have enough resources to do the proper port to trunk, so I will commit the 89-fb patch into the 0.89-fb branch and leave the JIRA open for the trunk fix. > Compressed RPC's for HBase > -- > > Key: HBASE-5355 > URL: https://issues.apache.org/jira/browse/HBASE-5355 > Project: HBase > Issue Type: Improvement > Components: ipc >Affects Versions: 0.89.20100924 >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > Some application need ability to do large batched writes and reads from a > remote MR cluster. These eventually get bottlenecked on the network. These > results are also pretty compressible sometimes. > The aim here is to add the ability to do compressed calls to the server on > both the send and receive paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5708) [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test
[ https://issues.apache.org/jira/browse/HBASE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245815#comment-13245815 ] Mikhail Bautin commented on HBASE-5708: --- It is fixed on the trunk with a lot of refactoring done around test directory set up. Right now I am not looking at backporting all of that, but at fixing this particular issue. > [89-fb] Make MiniMapRedCluster directory a subdirectory of target/test > -- > > Key: HBASE-5708 > URL: https://issues.apache.org/jira/browse/HBASE-5708 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > > Some map-reduce-based tests are failing when executed concurrently in 89-fb > because mini-map-reduce cluster uses /tmp/hadoop- for temporary > data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239920#comment-13239920 ] Mikhail Bautin commented on HBASE-4218: --- No, DATA_BLOCK_ENCODING=NONE disables encoding, regardless of the value of ENCODE_ON_DISK. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt, > D1659.1.patch, D1659.2.patch, D1659.3.patch, D447.1.patch, D447.10.patch, > D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, > D447.16.patch, D447.17.patch, D447.18.patch, D447.19.patch, D447.2.patch, > D447.20.patch, D447.21.patch, D447.22.patch, D447.23.patch, D447.24.patch, > D447.25.patch, D447.26.patch, D447.3.patch, D447.4.patch, D447.5.patch, > D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding-2012-01-17_11_09_09.patch, > Delta-encoding-2012-01-25_00_45_29.patch, > Delta-encoding-2012-01-25_16_32_14.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta-encoding.patch-2012-01-05_15_16_43.patch, > Delta-encoding.patch-2012-01-05_16_31_44.patch, > Delta-encoding.patch-2012-01-05_16_31_44_copy.patch, > Delta-encoding.patch-2012-01-05_18_50_47.patch, > Delta-encoding.patch-2012-01-07_14_12_48.patch, > Delta-encoding.patch-2012-01-13_12_20_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners
[ https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239203#comment-13239203 ] Mikhail Bautin commented on HBASE-4465: --- @Stack: yes, this feature is on by default, because it has the same or better performance as before in all cases. > Lazy-seek optimization for StoreFile scanners > - > > Key: HBASE-4465 > URL: https://issues.apache.org/jira/browse/HBASE-4465 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Labels: optimization, seek > Fix For: 0.89.20100924, 0.94.0 > > Attachments: > HBASE-4465_Lazy-seek_optimization_for_St-20111005121052-b2ea8753.patch > > > Previously, if we had several StoreFiles for a column family in a region, we > would seek in each of them and only then merge the results, even though the > row/column we are looking for might only be in the most recent (and the > smallest) file. Now we prioritize our reads from those files so that we check > the most recent file first. This is done by doing a "lazy seek" which > pretends that the next value in the StoreFile is (seekRow, seekColumn, > lastTimestampInStoreFile), which is earlier in the KV order than anything > that might actually occur in the file. So if we don't find the result in > earlier files, that fake KV will bubble up to the top of the KV heap and a > real seek will be done. This is expected to significantly reduce the amount > of disk IO (as of 09/22/2011 we are doing dark launch testing and > measurement). > This is joint work with Liyin Tang -- huge thanks to him for many helpful > discussions on this and the idea of putting fake KVs with the highest > timestamp of the StoreFile in the scanner priority queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4607) Split log worker should terminate properly when waiting for znode
[ https://issues.apache.org/jira/browse/HBASE-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235926#comment-13235926 ] Mikhail Bautin commented on HBASE-4607: --- Tried to apply the patch, but it shows as already applied. Upon further analysis, very similar changes to SplitLogWorker have been integrated as part of HBASE-5542: svn diff -r 1303648:1303920 https://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java > Split log worker should terminate properly when waiting for znode > - > > Key: HBASE-4607 > URL: https://issues.apache.org/jira/browse/HBASE-4607 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Minor > Fix For: 0.94.0 > > Attachments: > HBASE-4607_SplitLogWorker_should_correct-20111017231456-47a82ef3.patch > > > This is an attempt to fix the fact that SplitLogWorker threads are not being > terminated properly in some unit tests. This probably does not happen in > production because the master always creates the log-splitting ZK node, but > it does happen in 89-fb. Thanks to Prakash Khemani for help on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5612) Data types for HBase values
[ https://issues.apache.org/jira/browse/HBASE-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234898#comment-13234898 ] Mikhail Bautin commented on HBASE-5612: --- @Nicolas: it is not totally clear to me what you mean by a struct schema. I was discussing data block encoding and its benefits when applied to our applications with Liyin, and the simple example of variable-length integer encoding for counters came up. We can take it in many different directions, and I just want to open a discussion to gauge interest. > Data types for HBase values > --- > > Key: HBASE-5612 > URL: https://issues.apache.org/jira/browse/HBASE-5612 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > > In many real-life applications all values in a certain column family are of a > certain data type, e.g. 64-bit integer. We could specify that in the column > descriptor and enable data type-specific compression such as variable-length > integer encoding. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5566) [89-fb] Region server can get stuck in getMaster on master failover
[ https://issues.apache.org/jira/browse/HBASE-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232762#comment-13232762 ] Mikhail Bautin commented on HBASE-5566: --- Code reviewed at: https://reviews.facebook.net/D2283 > [89-fb] Region server can get stuck in getMaster on master failover > --- > > Key: HBASE-5566 > URL: https://issues.apache.org/jira/browse/HBASE-5566 > Project: HBase > Issue Type: Bug >Affects Versions: 0.89-fb >Reporter: Prakash Khemani >Assignee: Mikhail Bautin > > This is specific to the 89-fb master. We have a retry loop in > HRegionServer.getMaster where we do not read the location of the master from > ZK, so a region server can get stuck there on master failover. We need to add > a unit test to reliably catch this, and fix the bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231692#comment-13231692 ] Mikhail Bautin commented on HBASE-5521: --- +1 I have sucessfully re-run tests that failed on Jenkins locally: Running org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.904 sec Running org.apache.hadoop.hbase.io.hfile.TestReseekTo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.846 sec Running org.apache.hadoop.hbase.io.hfile.TestHFile Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.11 sec Running org.apache.hadoop.hbase.coprocessor.TestProcessRowEndpoint Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.273 sec Results : Tests run: 10, Failures: 0, Errors: 0, Skipped: 0 @Committers: this patch has been out for a while, and Yongqiang is addressing final minor comments. Would anyone else +1 this? > Move compression/decompression to an encoder specific encoding context > -- > > Key: HBASE-5521 > URL: https://issues.apache.org/jira/browse/HBASE-5521 > Project: HBase > Issue Type: Improvement >Reporter: He Yongqiang >Assignee: He Yongqiang > Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, > HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, > HBASE-5521.D2097.5.patch, HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch > > > As part of working on HBASE-5313, we want to add a new columnar > encoder/decoder. It makes sense to move compression to be part of > encoder/decoder: > 1) a scanner for a columnar encoded block can do lazy decompression to a > specific part of a key value object > 2) avoid an extra bytes copy from encoder to hblock-writer. > If there is no encoder specified for a writer, the HBlock.Writer will use a > default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5575) Configure Arcanist lint engine for HBase
[ https://issues.apache.org/jira/browse/HBASE-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231604#comment-13231604 ] Mikhail Bautin commented on HBASE-5575: --- Reviewed at https://reviews.facebook.net/D2289. > Configure Arcanist lint engine for HBase > > > Key: HBASE-5575 > URL: https://issues.apache.org/jira/browse/HBASE-5575 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Attachments: Enabling-lint-2012-03-16_13_40_37.patch > > > We need to enable Arcanist lint engine in HBase, so that a commit could be > checked by running "arc lint". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5581) Creating a table with invalid syntax does not give an error message when it fails
[
https://issues.apache.org/jira/browse/HBASE-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231600#comment-13231600
]
Mikhail Bautin commented on HBASE-5581:
---
Binu: thanks for the patch!
Stack: thanks for committing!
> Creating a table with invalid syntax does not give an error message when it
> fails
> -
>
> Key: HBASE-5581
> URL: https://issues.apache.org/jira/browse/HBASE-5581
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Binu John
>Priority: Minor
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5581trunk.patch, D2343.1.patch
>
>
> Creating a table with invalid syntax does not give an error message when it
> fails. In this case, it doesn't actually create the CF requested, but doesn't
> give any indication to the user that it failed.
> create 'test', {NAME => 'test', VERSIONS => 1, BLOCKCACHE => true, NUMREGIONS
> => 20, SPLITALGO => "HexStringSplit", COMPRESSION => 'LZO', BLOOMFILTER =>
> 'ROW'}
> 0 row(s) in 3.0930 seconds
> hbase(main):002:0> describe 'test'
> DESCRIPTION
> ENABLED
> {NAME => 'test', FAMILIES => []}
> true
> 1 row(s) in 0.1430 seconds
>
> Putting {NUMREGIONS => 20, SPLITALGO => "HexStringSplit"} into a separate
> stanza works fine, so the feature is fine.
> create 'test', {NAME => 'test', VERSIONS => 1, BLOCKCACHE => true,
> COMPRESSION => 'LZO', BLOOMFILTER => 'ROW'}, {NUMREGIONS => 20, SPLITALGO =>
> "HexStringSplit"}
> 0 row(s) in 2.7860 seconds
> hbase(main):002:0> describe 'test'
> DESCRIPTION
> ENABLED
> {NAME => 'test', FAMILIES => [{NAME => 'test', DATA_BLOCK_ENCODING =>
> 'NONE', true
> BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'LZO',
> VERSIONS
> => '1', TTL => '2147483647', BLOCKSIZE => '65536', BLOOMFILTER_ERRORRATE =>
> '
> 0.01', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE =>
> 'true'}]}
>
> We should throw an error if we can't create the CF so it's clear to the user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5581) Creating a table with invalid syntax does not give an error message when it fails
[
https://issues.apache.org/jira/browse/HBASE-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231495#comment-13231495
]
Mikhail Bautin commented on HBASE-5581:
---
Is this OK to commit to trunk? Could someone +1? Thanks!
> Creating a table with invalid syntax does not give an error message when it
> fails
> -
>
> Key: HBASE-5581
> URL: https://issues.apache.org/jira/browse/HBASE-5581
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Binu John
>Priority: Minor
> Attachments: D2343.1.patch
>
>
> Creating a table with invalid syntax does not give an error message when it
> fails. In this case, it doesn't actually create the CF requested, but doesn't
> give any indication to the user that it failed.
> create 'test', {NAME => 'test', VERSIONS => 1, BLOCKCACHE => true, NUMREGIONS
> => 20, SPLITALGO => "HexStringSplit", COMPRESSION => 'LZO', BLOOMFILTER =>
> 'ROW'}
> 0 row(s) in 3.0930 seconds
> hbase(main):002:0> describe 'test'
> DESCRIPTION
> ENABLED
> {NAME => 'test', FAMILIES => []}
> true
> 1 row(s) in 0.1430 seconds
>
> Putting {NUMREGIONS => 20, SPLITALGO => "HexStringSplit"} into a separate
> stanza works fine, so the feature is fine.
> create 'test', {NAME => 'test', VERSIONS => 1, BLOCKCACHE => true,
> COMPRESSION => 'LZO', BLOOMFILTER => 'ROW'}, {NUMREGIONS => 20, SPLITALGO =>
> "HexStringSplit"}
> 0 row(s) in 2.7860 seconds
> hbase(main):002:0> describe 'test'
> DESCRIPTION
> ENABLED
> {NAME => 'test', FAMILIES => [{NAME => 'test', DATA_BLOCK_ENCODING =>
> 'NONE', true
> BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'LZO',
> VERSIONS
> => '1', TTL => '2147483647', BLOCKSIZE => '65536', BLOOMFILTER_ERRORRATE =>
> '
> 0.01', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE =>
> 'true'}]}
>
> We should throw an error if we can't create the CF so it's clear to the user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227737#comment-13227737 ] Mikhail Bautin commented on HBASE-5074: --- @Lars: what I committed was based on D1521.14.patch, but it will not be exactly the same patch, because I used "arc patch" to apply the patch from Differential, fixed some minor indentation problem, and committed using the git-svn bridge. I also re-ran all the unit tests before the commit. Sorry for a delay in replying. > support checksums in HBase block cache > -- > > Key: HBASE-5074 > URL: https://issues.apache.org/jira/browse/HBASE-5074 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.94.0, 0.96.0 > > Attachments: 5074-0.94.txt, D1521.1.patch, D1521.1.patch, > D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, > D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, > D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, > D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, > D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, > D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, > D1521.9.patch, D1521.9.patch > > > The current implementation of HDFS stores the data in one block file and the > metadata(checksum) in another block file. This means that every read into the > HBase block cache actually consumes two disk iops, one to the datafile and > one to the checksum file. This is a major problem for scaling HBase, because > HBase is usually bottlenecked on the number of random disk iops that the > storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well
[ https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226626#comment-13226626 ] Mikhail Bautin commented on HBASE-5292: --- Re-run failed tests locally: Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 244.11 sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.249 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 100.749 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.473 sec Results : Tests run: 19, Failures: 0, Errors: 0, Skipped: 0 Someone please confirm if it is OK to commit this. > getsize per-CF metric incorrectly counts compaction related reads as well > -- > > Key: HBASE-5292 > URL: https://issues.apache.org/jira/browse/HBASE-5292 > Project: HBase > Issue Type: Bug >Affects Versions: 0.89.20100924 >Reporter: Kannan Muthukkaruppan > Attachments: > 0001-jira-HBASE-5292-Prevent-counting-getSize-on-compacti.patch, > D1527.1.patch, D1527.2.patch, D1527.3.patch, D1527.4.patch, D1617.1.patch, > jira-HBASE-5292-Prevent-counting-getSize-on-compacti-2012-03-09_13_26_52.patch > > > The per-CF "getsize" metric's intent was to track bytes returned (to HBase > clients) per-CF. [Note: We already have metrics to track # of HFileBlock's > read for compaction vs. non-compaction cases -- e.g., compactionblockreadcnt > vs. fsblockreadcnt.] > Currently, the "getsize" metric gets updated for both client initiated > Get/Scan operations as well for compaction related reads. The metric is > updated in StoreScanner.java:next() when the Scan query matcher returns an > INCLUDE* code via a: > HRegion.incrNumericMetric(this.metricNameGetsize, copyKv.getLength()); > We should not do the above in case of compactions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well
[ https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226412#comment-13226412 ] Mikhail Bautin commented on HBASE-5292: --- @Stack: yes, I will commit this. Sorry it's been a while. > getsize per-CF metric incorrectly counts compaction related reads as well > -- > > Key: HBASE-5292 > URL: https://issues.apache.org/jira/browse/HBASE-5292 > Project: HBase > Issue Type: Bug >Affects Versions: 0.89.20100924 >Reporter: Kannan Muthukkaruppan > Attachments: > 0001-jira-HBASE-5292-Prevent-counting-getSize-on-compacti.patch, > D1527.1.patch, D1527.2.patch, D1527.3.patch, D1527.4.patch, D1617.1.patch > > > The per-CF "getsize" metric's intent was to track bytes returned (to HBase > clients) per-CF. [Note: We already have metrics to track # of HFileBlock's > read for compaction vs. non-compaction cases -- e.g., compactionblockreadcnt > vs. fsblockreadcnt.] > Currently, the "getsize" metric gets updated for both client initiated > Get/Scan operations as well for compaction related reads. The metric is > updated in StoreScanner.java:next() when the Scan query matcher returns an > INCLUDE* code via a: > HRegion.incrNumericMetric(this.metricNameGetsize, copyKv.getLength()); > We should not do the above in case of compactions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225703#comment-13225703 ] Mikhail Bautin commented on HBASE-5074: --- @Lars: yes. I have also re-run unit tests one more time. > support checksums in HBase block cache > -- > > Key: HBASE-5074 > URL: https://issues.apache.org/jira/browse/HBASE-5074 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.94.0 > > Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, > D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, > D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, > D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, > D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, > D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, > D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, > D1521.9.patch > > > The current implementation of HDFS stores the data in one block file and the > metadata(checksum) in another block file. This means that every read into the > HBase block cache actually consumes two disk iops, one to the datafile and > one to the checksum file. This is a major problem for scaling HBase, because > HBase is usually bottlenecked on the number of random disk iops that the > storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4542) add filter info to slow query logging
[
https://issues.apache.org/jira/browse/HBASE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223755#comment-13223755
]
Mikhail Bautin commented on HBASE-4542:
---
All unit tests passed locally. Waiting for Hadoop QA before committing.
> add filter info to slow query logging
> -
>
> Key: HBASE-4542
> URL: https://issues.apache.org/jira/browse/HBASE-4542
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.89.20100924
>Reporter: Kannan Muthukkaruppan
>Assignee: Madhuwanti Vaidya
> Attachments:
> 0001-jira-HBASE-4542-Add-filter-info-to-slow-query-loggin.patch,
> Add-filter-info-to-slow-query-logging-2012-03-06_14_28_13.patch,
> D1263.2.patch, D1539.1.patch
>
>
> Slow query log doesn't report filters in effect.
> For example:
> {code}
> (operationTooSlow): \
> {"processingtimems":3468,"client":"10.138.43.206:40035","timeRange":
> [0,9223372036854775807],\
> "starttimems":1317772005821,"responsesize":42411, \
> "class":"HRegionServer","table":"myTable","families":{"CF1":"ALL"]},\
> "row":"6c3b8efa132f0219b7621ed1e5c8c70b","queuetimems":0,\
> "method":"get","totalColumns":1,"maxVersions":1,"storeLimit":-1}
> {code}
> the above would suggest that all columns of myTable:CF1 are being requested
> for the given row. But in reality there could be filters in effect (such as
> ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should
> enhance the slow query log to capture & report this information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223567#comment-13223567 ] Mikhail Bautin commented on HBASE-5074: --- @Dhruba: could you please rerun the failed tests locally, as well as check the test reports? org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv > support checksums in HBase block cache > -- > > Key: HBASE-5074 > URL: https://issues.apache.org/jira/browse/HBASE-5074 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.94.0 > > Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, > D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, > D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, > D1521.13.patch, D1521.13.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, > D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, > D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, > D1521.8.patch, D1521.9.patch, D1521.9.patch > > > The current implementation of HDFS stores the data in one block file and the > metadata(checksum) in another block file. This means that every read into the > HBase block cache actually consumes two disk iops, one to the datafile and > one to the checksum file. This is a major problem for scaling HBase, because > HBase is usually bottlenecked on the number of random disk iops that the > storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223499#comment-13223499
]
Mikhail Bautin commented on HBASE-5010:
---
Actually here is the reason for those confusing updates. Ted seems to have
specified HBASE-5010 instead of HBASE-5510 in the commit message.
commit 5d773d9fa176cb056b993fdff8a2853f75315ec8
Author: tedyu
Date: Mon Mar 5 10:41:03 2012
HBASE-5010 Pass region info in
LoadBalancer.randomAssignment(List servers) (Anoop Sam
git-svn-id: http://svn.apache.org/repos/asf/hbase/trunk@1297155
13f79535-47bb-0310-9956-ffa450edef68
> Filter HFiles based on TTL
> --
>
> Key: HBASE-5010
> URL: https://issues.apache.org/jira/browse/HBASE-5010
> Project: HBase
> Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
> D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch, D909.6.patch
>
>
> In ScanWildcardColumnTracker we have
> {code:java}
>
> this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
> ...
> private boolean isExpired(long timestamp) {
> return timestamp < oldestStamp;
> }
> {code}
> but this time range filtering does not participate in HFile selection. In one
> real case this caused next() calls to time out because all KVs in a table got
> expired, but next() had to iterate over the whole table to find that out. We
> should be able to filter out those HFiles right away. I think a reasonable
> approach is to add a "default timerange filter" to every scan for a CF with a
> finite TTL and utilize existing filtering in
> StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223490#comment-13223490
]
Mikhail Bautin commented on HBASE-5010:
---
@Ram: I don't see any mentions of HBASE-5510 in this JIRA, except for your
comment. What updates are you referring to?
> Filter HFiles based on TTL
> --
>
> Key: HBASE-5010
> URL: https://issues.apache.org/jira/browse/HBASE-5010
> Project: HBase
> Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Fix For: 0.94.0
>
> Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
> D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch, D909.6.patch
>
>
> In ScanWildcardColumnTracker we have
> {code:java}
>
> this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
> ...
> private boolean isExpired(long timestamp) {
> return timestamp < oldestStamp;
> }
> {code}
> but this time range filtering does not participate in HFile selection. In one
> real case this caused next() calls to time out because all KVs in a table got
> expired, but next() had to iterate over the whole table to find that out. We
> should be able to filter out those HFiles right away. I think a reasonable
> approach is to add a "default timerange filter" to every scan for a CF with a
> finite TTL and utilize existing filtering in
> StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4932) Block cache can be mistakenly instantiated by tools
[ https://issues.apache.org/jira/browse/HBASE-4932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215864#comment-13215864 ] Mikhail Bautin commented on HBASE-4932: --- @Prakash: I've noticed that this is not there in 89-fb. Do we need this patch there as well? > Block cache can be mistakenly instantiated by tools > --- > > Key: HBASE-4932 > URL: https://issues.apache.org/jira/browse/HBASE-4932 > Project: HBase > Issue Type: Bug >Reporter: Prakash Khemani >Assignee: Prakash Khemani > Fix For: 0.94.0 > > Attachments: HBASE-4932.patch > > > Map Reduce tasks that create a writer to write HFiles inadvertently end up > creating block cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215789#comment-13215789 ] Mikhail Bautin commented on HBASE-5416: --- @Max: if you scan the 'flag' column family first, find the rows that you are interested in, and query only those rows from the 'snap' column family, you will avoid the slowness from scanning every row in 'snap'. With proper batching, the two-pass approach should work fine if you don't need atomicity. The problem with such deep changes to the scanner framework is that it would require comprehensive new unit tests. The included unit test only writes three rows and does not really check the new feature (or the old functionality) on a large scale. Take a look at TestMultiColumnScanner and TestSeekOptimizations. We will need something at least as comprehensive as those tests for this improvement, probably even a multithreaded test case to ensure we don't break atomicity. If we do not do that testing now, we will still have to do it before the next stable release, but it would be unfair to pass the hidden costs of testing to those who don't need this particular optimization right now but will soon need a stable system for another production release. > Improve performance of scans with some kind of filters. > --- > > Key: HBASE-5416 > URL: https://issues.apache.org/jira/browse/HBASE-5416 > Project: HBase > Issue Type: Improvement > Components: filters, performance, regionserver >Affects Versions: 0.90.4 >Reporter: Max Lapan >Assignee: Max Lapan > Attachments: 5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, > Filtered_scans_v2.patch, Filtered_scans_v3.patch, Filtered_scans_v4.patch > > > When the scan is performed, whole row is loaded into result list, after that > filter (if exists) is applied to detect that row is needed. > But when scan is performed on several CFs and filter checks only data from > the subset of these CFs, data from CFs, not checked by a filter is not needed > on a filter stage. Only when we decided to include current row. And in such > case we can significantly reduce amount of IO performed by a scan, by loading > only values, actually checked by a filter. > For example, we have two CFs: flags and snap. Flags is quite small (bunch of > megabytes) and is used to filter large entries from snap. Snap is very large > (10s of GB) and it is quite costly to scan it. If we needed only rows with > some flag specified, we use SingleColumnValueFilter to limit result to only > small subset of region. But current implementation is loading both CFs to > perform scan, when only small subset is needed. > Attached patch adds one routine to Filter interface to allow filter to > specify which CF is needed to it's operation. In HRegion, we separate all > scanners into two groups: needed for filter and the rest (joined). When new > row is considered, only needed data is loaded, filter applied, and only if > filter accepts the row, rest of data is loaded. At our data, this speeds up > such kind of scans 30-50 times. Also, this gives us the way to better > normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5442) Use builder pattern in StoreFile and HFile
[
https://issues.apache.org/jira/browse/HBASE-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215278#comment-13215278
]
Mikhail Bautin commented on HBASE-5442:
---
Re-ran failed unit tests locally:
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 212.073 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.808 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 98.041 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.51 sec
Running org.apache.hadoop.hbase.replication.TestReplicationPeer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.776 sec
> Use builder pattern in StoreFile and HFile
> --
>
> Key: HBASE-5442
> URL: https://issues.apache.org/jira/browse/HBASE-5442
> Project: HBase
> Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: D1893.1.patch, D1893.2.patch,
> HFile-StoreFile-builder-2012-02-22_22_49_00.patch
>
>
> We have five ways to create an HFile writer, two ways to create a StoreFile
> writer, and the sets of parameters keep changing, creating a lot of
> confusion, especially when porting patches across branches. The same thing is
> happening to HColumnDescriptor. I think we should move to a "builder pattern"
> solution, e.g.
> {code:java}
> HFileWriter w = HFile.getWriterBuilder(conf, )
> .setParameter1(value1)
> .setParameter2(value2)
> ...
> .build();
> {code}
> Each parameter setter being on its own line will make merges/cherry-pick work
> properly, we will not have to even mention default parameters again, and we
> can eliminate a dozen impossible-to-remember constructors.
> This particular JIRA addresses StoreFile and HFile refactoring. For
> HColumnDescriptor refactoring see HBASE-5357.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5357) Use builder pattern in HColumnDescriptor
[
https://issues.apache.org/jira/browse/HBASE-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215166#comment-13215166
]
Mikhail Bautin commented on HBASE-5357:
---
Re-ran failed unit tests locally:
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 211.925 sec
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.352 sec
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 105.09 sec
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.055 sec
Results :
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0
> Use builder pattern in HColumnDescriptor
>
>
> Key: HBASE-5357
> URL: https://issues.apache.org/jira/browse/HBASE-5357
> Project: HBase
> Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: D1851.1.patch, D1851.2.patch, D1851.3.patch,
> D1851.4.patch,
> Use-builder-pattern-for-HColumnDescriptor-2012-02-21_19_13_35.patch,
> Use-builder-pattern-for-HColumnDescriptor-2012-02-23_12_42_49.patch,
> Use-builder-pattern-for-HColumnDescriptor-20120223113155-e387d251.patch
>
>
> We have five ways to create an HFile writer, two ways to create a StoreFile
> writer, and the sets of parameters keep changing, creating a lot of
> confusion, especially when porting patches across branches. The same thing is
> happening to HColumnDescriptor. I think we should move to a "builder pattern"
> solution, e.g.
> {code:java}
> HFileWriter w = HFile.getWriterBuilder(conf, )
> .setParameter1(value1)
> .setParameter2(value2)
> ...
> .build();
> {code}
> Each parameter setter being on its own line will make merges/cherry-pick work
> properly, we will not have to even mention default parameters again, and we
> can eliminate a dozen impossible-to-remember constructors.
> This particular JIRA addresses the HColumnDescriptor refactoring. For
> StoreFile/HFile refactoring see HBASE-5442.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5357) Use builder pattern in HColumnDescriptor
[
https://issues.apache.org/jira/browse/HBASE-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215030#comment-13215030
]
Mikhail Bautin commented on HBASE-5357:
---
@Ted: I thought that was what I did. Attaching again.
> Use builder pattern in HColumnDescriptor
>
>
> Key: HBASE-5357
> URL: https://issues.apache.org/jira/browse/HBASE-5357
> Project: HBase
> Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: D1851.1.patch, D1851.2.patch, D1851.3.patch,
> D1851.4.patch,
> Use-builder-pattern-for-HColumnDescriptor-2012-02-21_19_13_35.patch,
> Use-builder-pattern-for-HColumnDescriptor-2012-02-23_12_42_25.patch,
> Use-builder-pattern-for-HColumnDescriptor-20120223113155-e387d251.patch
>
>
> We have five ways to create an HFile writer, two ways to create a StoreFile
> writer, and the sets of parameters keep changing, creating a lot of
> confusion, especially when porting patches across branches. The same thing is
> happening to HColumnDescriptor. I think we should move to a "builder pattern"
> solution, e.g.
> {code:java}
> HFileWriter w = HFile.getWriterBuilder(conf, )
> .setParameter1(value1)
> .setParameter2(value2)
> ...
> .build();
> {code}
> Each parameter setter being on its own line will make merges/cherry-pick work
> properly, we will not have to even mention default parameters again, and we
> can eliminate a dozen impossible-to-remember constructors.
> This particular JIRA addresses the HColumnDescriptor refactoring. For
> StoreFile/HFile refactoring see HBASE-5442.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5456) Introduce PowerMock into our unit tests to reduce unnecessary method exposure
[ https://issues.apache.org/jira/browse/HBASE-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214289#comment-13214289 ] Mikhail Bautin commented on HBASE-5456: --- On PowerMock: yes, I agree that in some cases it is a cleaner solution than creating test-only methods. @Ted: I see 1406 tests in my full test suite run. Do you have some reasons to believe that the total number of small/medium/large tests should be 885? > Introduce PowerMock into our unit tests to reduce unnecessary method exposure > - > > Key: HBASE-5456 > URL: https://issues.apache.org/jira/browse/HBASE-5456 > Project: HBase > Issue Type: Task >Reporter: Zhihong Yu > > We should introduce PowerMock into our unit tests so that we don't have to > expose methods intended to be used by unit tests. > Here was Benoit's reply to a user of asynchbase about testability: > OpenTSDB has unit tests that are mocking out HBaseClient just fine > [1]. You can mock out pretty much anything on the JVM: final, > private, JDK stuff, etc. All you need is the right tools. I've been > very happy with PowerMock. It supports Mockito and EasyMock. > I've never been keen on mutilating public interfaces for the sake of > testing. With tools like PowerMock, we can keep the public APIs tidy > while mocking and overriding anything, even in the most private guts > of the classes. > [1] > https://github.com/stumbleupon/opentsdb/blob/master/src/uid/TestUniqueId.java#L66 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5456) Introduce PowerMock into our unit tests to reduce unnecessary method exposure
[ https://issues.apache.org/jira/browse/HBASE-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214128#comment-13214128 ] Mikhail Bautin commented on HBASE-5456: --- I think most of our access restriction problems occur when the test and the class being accessed are in different packages. An alternative way to restrict test-only method access while avoiding using reflection is to define the test-only method as protected and create a subclass in the test. > Introduce PowerMock into our unit tests to reduce unnecessary method exposure > - > > Key: HBASE-5456 > URL: https://issues.apache.org/jira/browse/HBASE-5456 > Project: HBase > Issue Type: Task >Reporter: Zhihong Yu > > We should introduce PowerMock into our unit tests so that we don't have to > expose methods intended to be used by unit tests. > Here was Benoit's reply to a user of asynchbase about testability: > OpenTSDB has unit tests that are mocking out HBaseClient just fine > [1]. You can mock out pretty much anything on the JVM: final, > private, JDK stuff, etc. All you need is the right tools. I've been > very happy with PowerMock. It supports Mockito and EasyMock. > I've never been keen on mutilating public interfaces for the sake of > testing. With tools like PowerMock, we can keep the public APIs tidy > while mocking and overriding anything, even in the most private guts > of the classes. > [1] > https://github.com/stumbleupon/opentsdb/blob/master/src/uid/TestUniqueId.java#L66 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5456) Introduce PowerMock into our unit tests to reduce unnecessary method exposure
[ https://issues.apache.org/jira/browse/HBASE-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213990#comment-13213990 ] Mikhail Bautin commented on HBASE-5456: --- I think this only makes sense for HBase if people start running all unit tests for every patch (not just "small" and "medium" tests). These advanced reflection features convert some frequent types of errors from compile-time to test-time. Also, a lot of IDE search and refactoring features will be broken. > Introduce PowerMock into our unit tests to reduce unnecessary method exposure > - > > Key: HBASE-5456 > URL: https://issues.apache.org/jira/browse/HBASE-5456 > Project: HBase > Issue Type: Task >Reporter: Zhihong Yu > > We should introduce PowerMock into our unit tests so that we don't have to > expose methods intended to be used by unit tests. > Here was Benoit's reply to a user of asynchbase about testability: > OpenTSDB has unit tests that are mocking out HBaseClient just fine > [1]. You can mock out pretty much anything on the JVM: final, > private, JDK stuff, etc. All you need is the right tools. I've been > very happy with PowerMock. It supports Mockito and EasyMock. > I've never been keen on mutilating public interfaces for the sake of > testing. With tools like PowerMock, we can keep the public APIs tidy > while mocking and overriding anything, even in the most private guts > of the classes. > [1] > https://github.com/stumbleupon/opentsdb/blob/master/src/uid/TestUniqueId.java#L66 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207111#comment-13207111 ] Mikhail Bautin commented on HBASE-5387: --- @Ted: sure. I was rebasing and running a final local test run. > Reuse compression streams in HFileBlock.Writer > -- > > Key: HBASE-5387 > URL: https://issues.apache.org/jira/browse/HBASE-5387 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Critical > Fix For: 0.94.0 > > Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, > D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, > Fix-deflater-leak-2012-02-11_17_13_10.patch, > Fix-deflater-leak-2012-02-12_00_37_27.patch > > > We need to to reuse compression streams in HFileBlock.Writer instead of > allocating them every time. The motivation is that when using Java's built-in > implementation of Gzip, we allocate a new GZIPOutputStream object and an > associated native data structure every time we create a compression stream. > The native data structure is only deallocated in the finalizer. This is one > suspected cause of recent TestHFileBlock failures on Hadoop QA: > https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206758#comment-13206758 ] Mikhail Bautin commented on HBASE-5387: --- Those reported test failures seem unrelated. Is this OK to commit? > Reuse compression streams in HFileBlock.Writer > -- > > Key: HBASE-5387 > URL: https://issues.apache.org/jira/browse/HBASE-5387 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Critical > Fix For: 0.94.0 > > Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, > D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, > Fix-deflater-leak-2012-02-11_17_13_10.patch, > Fix-deflater-leak-2012-02-12_00_37_27.patch > > > We need to to reuse compression streams in HFileBlock.Writer instead of > allocating them every time. The motivation is that when using Java's built-in > implementation of Gzip, we allocate a new GZIPOutputStream object and an > associated native data structure every time we create a compression stream. > The native data structure is only deallocated in the finalizer. This is one > suspected cause of recent TestHFileBlock failures on Hadoop QA: > https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206361#comment-13206361 ] Mikhail Bautin commented on HBASE-5387: --- @Ted: thanks for attaching the patch. I will look into how to fix Phabricator to attach Jenkins-compatible patches. > Reuse compression streams in HFileBlock.Writer > -- > > Key: HBASE-5387 > URL: https://issues.apache.org/jira/browse/HBASE-5387 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Critical > Fix For: 0.94.0 > > Attachments: 5387.txt, D1719.1.patch, D1719.2.patch, D1719.3.patch, > D1719.4.patch, D1719.5.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch, > Fix-deflater-leak-2012-02-11_17_13_10.patch > > > We need to to reuse compression streams in HFileBlock.Writer instead of > allocating them every time. The motivation is that when using Java's built-in > implementation of Gzip, we allocate a new GZIPOutputStream object and an > associated native data structure every time we create a compression stream. > The native data structure is only deallocated in the finalizer. This is one > suspected cause of recent TestHFileBlock failures on Hadoop QA: > https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[
https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206325#comment-13206325
]
Mikhail Bautin commented on HBASE-5387:
---
@Lars: where are we creating a new GzipCodec frequently? We only instantiate
ReusableStreamGzipCodec once in the following block:
{code:title=Compression.java}
GZ("gz") {
private transient GzipCodec codec;
@Override
DefaultCodec getCodec(Configuration conf) {
if (codec == null) {
codec = new ReusableStreamGzipCodec();
codec.setConf(new Configuration(conf));
}
return codec;
}
},
{code}
What we are creating less frequently than before is the compressing output
stream (a subclass of GZIPOutputStream with the associated native data
structure): once per HFile writer with the patch vs. for every HFile block
previously.
> Reuse compression streams in HFileBlock.Writer
> --
>
> Key: HBASE-5387
> URL: https://issues.apache.org/jira/browse/HBASE-5387
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Critical
> Fix For: 0.94.0
>
> Attachments: D1719.1.patch, D1719.2.patch,
> Fix-deflater-leak-2012-02-10_18_48_45.patch,
> Fix-deflater-leak-2012-02-11_17_13_10.patch
>
>
> We need to to reuse compression streams in HFileBlock.Writer instead of
> allocating them every time. The motivation is that when using Java's built-in
> implementation of Gzip, we allocate a new GZIPOutputStream object and an
> associated native data structure every time we create a compression stream.
> The native data structure is only deallocated in the finalizer. This is one
> suspected cause of recent TestHFileBlock failures on Hadoop QA:
> https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206300#comment-13206300 ] Mikhail Bautin commented on HBASE-5387: --- The BAOS in question only exists during the lifetime of the writer, and will be deallocated when flush or compaction is complete. > Reuse compression streams in HFileBlock.Writer > -- > > Key: HBASE-5387 > URL: https://issues.apache.org/jira/browse/HBASE-5387 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Critical > Fix For: 0.94.0 > > Attachments: D1719.1.patch, > Fix-deflater-leak-2012-02-10_18_48_45.patch > > > We need to to reuse compression streams in HFileBlock.Writer instead of > allocating them every time. The motivation is that when using Java's built-in > implementation of Gzip, we allocate a new GZIPOutputStream object and an > associated native data structure every time we create a compression stream. > The native data structure is only deallocated in the finalizer. This is one > suspected cause of recent TestHFileBlock failures on Hadoop QA: > https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
[ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206012#comment-13206012 ] Mikhail Bautin commented on HBASE-5387: --- @Ted: I think this addresses the root cause of TestHFileBlock and TestForceCacheImportantBlocks failures. As you suspected, Hadoop QA was pointing to a real bug in HBase. However, I think we have had this issue for a while (even in HFile v1), and it just got exposed as I increased the volume of IO happening within a single unit test. I will add a ulimit setting to our internal test runs so that we catch memory leaks like this in the future. > Reuse compression streams in HFileBlock.Writer > -- > > Key: HBASE-5387 > URL: https://issues.apache.org/jira/browse/HBASE-5387 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Attachments: Fix-deflater-leak-2012-02-10_18_48_45.patch > > > We need to to reuse compression streams in HFileBlock.Writer instead of > allocating them every time. The motivation is that when using Java's built-in > implementation of Gzip, we allocate a new GZIPOutputStream object and an > associated native data structure every time we create a compression stream. > The native data structure is only deallocated in the finalizer. This is one > suspected cause of recent TestHFileBlock failures on Hadoop QA: > https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5382) Test that we always cache index and bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205727#comment-13205727 ] Mikhail Bautin commented on HBASE-5382: --- @Ted: I ran unit tests and the patch passed all of them (not just small and medium that Hadoop QA runs). I got a +1 on this from JD, and this code has been previously reviewed and approved as part of HBASE-4683. Sorry if this is a misunderstanding, but I thought we had plans of increasing the memory limit of HBase QA? > Test that we always cache index and bloom blocks > > > Key: HBASE-5382 > URL: https://issues.apache.org/jira/browse/HBASE-5382 > Project: HBase > Issue Type: Test >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Attachments: TestForceCacheImportantBlocks-2012-02-10_11_07_15.patch > > > This is a unit test that should have been part of HBASE-4683 but was not > committed. The original test was reviewed as part of > https://reviews.facebook.net/D807. Submitting unit test as a separate JIRA > and patch, and extending the scope of the test to also handle the case when > block cache is enabled for the column family. The new review is at > https://reviews.facebook.net/D1695. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4683) Always cache index and bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205728#comment-13205728 ] Mikhail Bautin commented on HBASE-4683: --- The new JIRA with the unit test is HBASE-5382. > Always cache index and bloom blocks > --- > > Key: HBASE-4683 > URL: https://issues.apache.org/jira/browse/HBASE-4683 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Mikhail Bautin >Priority: Minor > Fix For: 0.94.0, 0.92.0 > > Attachments: 0001-Cache-important-block-types.patch, 4683-v2.txt, > 4683.txt, D1695.1.patch, D807.1.patch, D807.2.patch, D807.3.patch, > HBASE-4683-0.92-v2.patch, HBASE-4683-v3.patch > > > This would add a new boolean config option: hfile.block.cache.datablocks > Default would be true. > Setting this to false allows HBase in a mode where only index blocks are > cached, which is useful for analytical scenarios where a useful working set > of the data cannot be expected to fit into the (aggregate) cache. > This is the equivalent of setting cacheBlocks to false on all scans > (including scans on behalf of gets). > I would like to get a general feeling about what folks think about this. > The change itself would be simple. > Update (Mikhail): we probably don't need a new conf option. Instead, we will > make index blocks cached by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4683) Always cache index and bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205200#comment-13205200 ] Mikhail Bautin commented on HBASE-4683: --- @jdcryans: it looks like TestForceCacheImportantBlocks.java was not included into your trunk commit of this JIRA. I will verify that the test still works and post an addendum patch. > Always cache index and bloom blocks > --- > > Key: HBASE-4683 > URL: https://issues.apache.org/jira/browse/HBASE-4683 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Mikhail Bautin >Priority: Minor > Fix For: 0.94.0, 0.92.0 > > Attachments: 0001-Cache-important-block-types.patch, 4683-v2.txt, > 4683.txt, D807.1.patch, D807.2.patch, D807.3.patch, HBASE-4683-0.92-v2.patch, > HBASE-4683-v3.patch > > > This would add a new boolean config option: hfile.block.cache.datablocks > Default would be true. > Setting this to false allows HBase in a mode where only index blocks are > cached, which is useful for analytical scenarios where a useful working set > of the data cannot be expected to fit into the (aggregate) cache. > This is the equivalent of setting cacheBlocks to false on all scans > (including scans on behalf of gets). > I would like to get a general feeling about what folks think about this. > The change itself would be simple. > Update (Mikhail): we probably don't need a new conf option. Instead, we will > make index blocks cached by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase maintenance gracefully
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198591#comment-13198591 ] Mikhail Bautin commented on HBASE-5320: --- Lars: collaboration would be very welcome :) I do not yet have a design in mind for this one, nor have I started actively working on this. I just noticed this missing piece in our deployment process. The main difficulty here probably lies in the fact that the master and the regionservers are all unreachable during maintenance. Therefore, there needs to be some other server to connect to in order to find out the cluster state. This server could even be shared across multiple clusters, thus unifying a piece of operational infrastructure that any company running multiple HBase clusters has to implement. Alternatively, this could be ZK-based, but the ZK instance has to be different from the one started and stopped with the HBase cluster. Please share your design ideas. > Create client API to handle HBase maintenance gracefully > > > Key: HBASE-5320 > URL: https://issues.apache.org/jira/browse/HBASE-5320 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Minor > > When we do HBase cluster maintenance, we typically have to manually stop or > disable the client temporarily. It would be nice to have a way for the client > to find out that HBase in undergoing maintenance through an appropriate API > and gracefully handle it on its own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5012) Unify data/index/bloom and meta block read methods in HFileReaderV{1,2}
[
https://issues.apache.org/jira/browse/HBASE-5012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196354#comment-13196354
]
Mikhail Bautin commented on HBASE-5012:
---
Another issue that Kannan mentioned as part of a code review for the 89-fb port
of HBASE-4422 (https://reviews.facebook.net/D1341). This seems like an
appropriate place to address it.
In the readBlockBuffer method in HFileReaderV1 we have
{code:title=HFileReaderV1.java|borderStyle=solid}
HFileBlock cachedBlock =
(HFileBlock) cacheConf.getBlockCache().getBlock(cacheKey,
cacheConf.shouldCacheDataOnRead());
{code}
but later in the method the condition used for actually caching the block is
different:
{code:title=HFileReaderV1.java|borderStyle=solid}
// Cache the block
if (cacheBlock && cacheConf.shouldCacheBlockOnRead(
hfileBlock.getBlockType().getCategory())) {
cacheConf.getBlockCache().cacheBlock(cacheKey, hfileBlock,
cacheConf.isInMemory());
}
{code}
We need to make these "cache block" flags two consistent across these two code
spots as we unify getMetaBlock and readBlock[Buffer], and double-check this
across HFileReaderV{1,2}.
> Unify data/index/bloom and meta block read methods in HFileReaderV{1,2}
> ---
>
> Key: HBASE-5012
> URL: https://issues.apache.org/jira/browse/HBASE-5012
> Project: HBase
> Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Minor
>
> Reduce code duplication between getMetaBlock and readBlock in HFileReaderV1
> and HFileReaderV2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193148#comment-13193148 ] Mikhail Bautin commented on HBASE-4218: --- Re-running unit tests that failed on Jenkins: Running org.apache.hadoop.hbase.client.TestFromClientSide Tests run: 52, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 181.919 sec Running org.apache.hadoop.hbase.client.TestAdmin Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 195.194 sec Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 223.405 sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.48 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.561 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.289 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.362 sec Results : Tests run: 122, Failures: 0, Errors: 0, Skipped: 3 > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt, > D447.1.patch, D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, > D447.14.patch, D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch, > D447.19.patch, D447.2.patch, D447.20.patch, D447.21.patch, D447.22.patch, > D447.23.patch, D447.24.patch, D447.25.patch, D447.3.patch, D447.4.patch, > D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding-2012-01-17_11_09_09.patch, > Delta-encoding-2012-01-25_00_45_29.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta-encoding.patch-2012-01-05_15_16_43.patch, > Delta-encoding.patch-2012-01-05_16_31_44.patch, > Delta-encoding.patch-2012-01-05_16_31_44_copy.patch, > Delta-encoding.patch-2012-01-05_18_50_47.patch, > Delta-encoding.patch-2012-01-07_14_12_48.patch, > Delta-encoding.patch-2012-01-13_12_20_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192943#comment-13192943 ] Mikhail Bautin commented on HBASE-4218: --- All unit tests passed (either in parallel on map-reduce, or when I re-ran the failed ones locally). > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt, > D447.1.patch, D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, > D447.14.patch, D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch, > D447.19.patch, D447.2.patch, D447.20.patch, D447.21.patch, D447.22.patch, > D447.23.patch, D447.24.patch, D447.25.patch, D447.3.patch, D447.4.patch, > D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding-2012-01-17_11_09_09.patch, > Delta-encoding-2012-01-25_00_45_29.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta-encoding.patch-2012-01-05_15_16_43.patch, > Delta-encoding.patch-2012-01-05_16_31_44.patch, > Delta-encoding.patch-2012-01-05_16_31_44_copy.patch, > Delta-encoding.patch-2012-01-05_18_50_47.patch, > Delta-encoding.patch-2012-01-07_14_12_48.patch, > Delta-encoding.patch-2012-01-13_12_20_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5268) Add delete column prefix delete marker
[ https://issues.apache.org/jira/browse/HBASE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192746#comment-13192746 ] Mikhail Bautin commented on HBASE-5268: --- @Lars, Todd: Yes, the column prefix delete will break lazy seek optimization. The lazy seek optimization is only enabled for multi-column queries, though, so the bug would not be seen when only reading one column or the whole row. More generally, our various types of deletes get in the way of "point queries", e.g. reading a particular column or a particular timestamp range, because we have to check for deletes first. As Todd pointed out, with column prefix deletes we would have to search the entire row for multi-column gets. We could trigger these optimizations using another "Column Prefix Delete" Bloom filter, but that might be an overkill. It might be easier to detect and delete just the required columns at the write path. > Add delete column prefix delete marker > -- > > Key: HBASE-5268 > URL: https://issues.apache.org/jira/browse/HBASE-5268 > Project: HBase > Issue Type: Improvement > Components: client, regionserver >Reporter: Lars Hofhansl > Attachments: 5268-v2.txt, 5268-v3.txt, 5268-v4.txt, 5268.txt > > > This is another part missing in the "wide row challenge". > Currently entire families of a row can be deleted or individual columns or > versions. > There is no facility to mark multiple columns for deletion by column prefix. > Turns out that be achieve with very little code (it's possible that I missed > some of the new delete bloom filter code, so please review this thoroughly). > I'll attach a patch soon, just working on some tests now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for monitoring and auto-tuning performance
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192504#comment-13192504 ] Mikhail Bautin commented on HBASE-5262: --- @Todd: I totally agree. We should also disable event logging for the event log table itself. > Structured event log for HBase for monitoring and auto-tuning performance > - > > Key: HBASE-5262 > URL: https://issues.apache.org/jira/browse/HBASE-5262 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin > > Creating this JIRA to open a discussion about a structured (machine-readable) > log that will record events such as compaction start/end times, compaction > input/output files, their sizes, the same for flushes, etc. This can be > stored e.g. in a new system table in HBase itself. The data from this log can > then be analyzed and used to optimize compactions at run time, or otherwise > auto-tune HBase configuration to reduce the number of knobs the user has to > configure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for monitoring and auto-tuning performance
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192491#comment-13192491 ] Mikhail Bautin commented on HBASE-5262: --- Some of the information needed for compaction decision making: * Compaction start time * For each compaction input file: * Size * Number of key/value pairs * Average key/value size * other metadata * Compaction end time * Compaction status (success, failure) * The same metadata as above for the compaction output file I am not yet sure what information we would like to collect for caching decision making—that needs more thinking. We could also collect "region history", e.g. region open / close events: * Region name * Event type * Server where the region was opened or closed * Reason That would allow to detect problematic regions that move from machine to machine automatically. I agree that it would make sense to isolate information collection logic from decision making logic, so that external adaptive cluster tuning tools and/or external sources of information could be plugged in. > Structured event log for HBase for monitoring and auto-tuning performance > - > > Key: HBASE-5262 > URL: https://issues.apache.org/jira/browse/HBASE-5262 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin > > Creating this JIRA to open a discussion about a structured (machine-readable) > log that will record events such as compaction start/end times, compaction > input/output files, their sizes, the same for flushes, etc. This can be > stored e.g. in a new system table in HBase itself. The data from this log can > then be analyzed and used to optimize compactions at run time, or otherwise > auto-tune HBase configuration to reduce the number of knobs the user has to > configure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192479#comment-13192479 ] Mikhail Bautin commented on HBASE-5230: --- Re-running failed unit tests locally. Running org.apache.hadoop.hbase.master.TestSplitLogManager Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.748 sec Running org.apache.hadoop.hbase.replication.TestReplication Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 128.254 sec Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 215.328 sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.367 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.431 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.762 sec Results : Tests run: 39, Failures: 0, Errors: 0, Skipped: > Unit test to ensure compactions don't cache data on write > - > > Key: HBASE-5230 > URL: https://issues.apache.org/jira/browse/HBASE-5230 > Project: HBase > Issue Type: Test >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Minor > Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, > D1353.4.patch, > Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, > Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch, > Don-t-cache-data-blocks-on-compaction-2012-01-23_15_27_23.patch > > > Create a unit test for HBASE-3976 (making sure we don't cache data blocks on > write during compactions even if cache-on-write is enabled generally > enabled). This is because we have very different implementations of > HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) > and with CacheConfig (presumably it's there but not sure if it even works, > since the patch in HBASE-3976 may not have been committed). We need to create > a unit test to verify that we don't cache data blocks on write during > compactions, and resolve HBASE-3976 so that this new unit test does not fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for monitoring and auto-tuning performance
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192441#comment-13192441 ] Mikhail Bautin commented on HBASE-5262: --- I think we could implement something small-scale in HBase itself, so that it can eventually be used for making optimization decisions for compactions and caching. OpenTSDB is probably better for large-scale time series collection. Also, we don't want to introduce a circular dependency between HBase and OpenTSDB. > Structured event log for HBase for monitoring and auto-tuning performance > - > > Key: HBASE-5262 > URL: https://issues.apache.org/jira/browse/HBASE-5262 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin > > Creating this JIRA to open a discussion about a structured (machine-readable) > log that will record events such as compaction start/end times, compaction > input/output files, their sizes, the same for flushes, etc. This can be > stored e.g. in a new system table in HBase itself. The data from this log can > then be analyzed and used to optimize compactions at run time, or otherwise > auto-tune HBase configuration to reduce the number of knobs the user has to > configure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for monitoring and auto-tuning performance
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192179#comment-13192179 ] Mikhail Bautin commented on HBASE-5262: --- I was thinking of a lightly-loaded system table different from .META.. This way it is easier to control this feature, and we can even make it optional based on the configuration. Storing event log data in HBase makes sense to me because we don't expect a huge number of records compared to the user's data itself, and we probably don't want to implement another logging framework with HDFS file management, cleaning up old files, etc. The multiple writers to one file question is also there. If we store this data in HBase, then yes, we can use TTL to set event log retention to e.g. one month or so, or make it configurable. > Structured event log for HBase for monitoring and auto-tuning performance > - > > Key: HBASE-5262 > URL: https://issues.apache.org/jira/browse/HBASE-5262 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin > > Creating this JIRA to open a discussion about a structured (machine-readable) > log that will record events such as compaction start/end times, compaction > input/output files, their sizes, the same for flushes, etc. This can be > stored e.g. in a new system table in HBase itself. The data from this log can > then be analyzed and used to optimize compactions at run time, or otherwise > auto-tune HBase configuration to reduce the number of knobs the user has to > configure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5262) Structured event log for HBase for monitoring and auto-tuning performance
[ https://issues.apache.org/jira/browse/HBASE-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191689#comment-13191689 ] Mikhail Bautin commented on HBASE-5262: --- JSON could be the encoding for the "value" part of each log entry. However, if we decide to store this type of information in HBase itself, we will need to think through the schema from the point of view of at least couple of different use cases, e.g. analyzing compaction performance, auto-tuning the compaction algorithm, maybe auto-tuning some block cache settings, etc. > Structured event log for HBase for monitoring and auto-tuning performance > - > > Key: HBASE-5262 > URL: https://issues.apache.org/jira/browse/HBASE-5262 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin > > Creating this JIRA to open a discussion about a structured (machine-readable) > log that will record events such as compaction start/end times, compaction > input/output files, their sizes, the same for flushes, etc. This can be > stored e.g. in a new system table in HBase itself. The data from this log can > then be analyzed and used to optimize compactions at run time, or otherwise > auto-tune HBase configuration to reduce the number of knobs the user has to > configure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191372#comment-13191372 ] Mikhail Bautin commented on HBASE-5230: --- The above failed tests passed locally: Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 216.187 sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.841 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.529 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.111 sec Running org.apache.hadoop.hbase.coprocessor.TestClassLoading Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.787 sec Results : Tests run: 24, Failures: 0, Errors: 0, Skipped: 0 > Unit test to ensure compactions don't cache data on write > - > > Key: HBASE-5230 > URL: https://issues.apache.org/jira/browse/HBASE-5230 > Project: HBase > Issue Type: Test >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Minor > Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, > Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, > Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch > > > Create a unit test for HBASE-3976 (making sure we don't cache data blocks on > write during compactions even if cache-on-write is enabled generally > enabled). This is because we have very different implementations of > HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) > and with CacheConfig (presumably it's there but not sure if it even works, > since the patch in HBASE-3976 may not have been committed). We need to create > a unit test to verify that we don't cache data blocks on write during > compactions, and resolve HBASE-3976 so that this new unit test does not fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5230) Unit test to ensure compactions don't cache data on write
[ https://issues.apache.org/jira/browse/HBASE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191320#comment-13191320 ] Mikhail Bautin commented on HBASE-5230: --- @Ted: the unit test failure at https://builds.apache.org/job/PreCommit-HBASE-Build/824//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testRowMutationMultiThreads/ seems unrelated. Is this patch OK to be committed? (We can wait for another run of unit tests if necessary, I've just re-uploaded the patch.) > Unit test to ensure compactions don't cache data on write > - > > Key: HBASE-5230 > URL: https://issues.apache.org/jira/browse/HBASE-5230 > Project: HBase > Issue Type: Test >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Minor > Attachments: D1353.1.patch, D1353.2.patch, D1353.3.patch, > Don-t-cache-data-blocks-on-compaction-2012-01-21_00_53_54.patch, > Don-t-cache-data-blocks-on-compaction-2012-01-23_10_23_45.patch > > > Create a unit test for HBASE-3976 (making sure we don't cache data blocks on > write during compactions even if cache-on-write is enabled generally > enabled). This is because we have very different implementations of > HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) > and with CacheConfig (presumably it's there but not sure if it even works, > since the patch in HBASE-3976 may not have been committed). We need to create > a unit test to verify that we don't cache data blocks on write during > compactions, and resolve HBASE-3976 so that this new unit test does not fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3796) Per-Store Entries in Compaction Queue
[ https://issues.apache.org/jira/browse/HBASE-3796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191303#comment-13191303 ] Mikhail Bautin commented on HBASE-3796: --- Sorry, it seems like I re-opened the wrong patch instead of HBASE-3976. Restoring the "Fixed" status. > Per-Store Entries in Compaction Queue > - > > Key: HBASE-3796 > URL: https://issues.apache.org/jira/browse/HBASE-3796 > Project: HBase > Issue Type: Bug >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Fix For: 0.92.1 > > Attachments: HBASE-3796-fixed.patch, HBASE-3796.patch > > > Although compaction is decided on a per-store basis, right now the > CompactSplitThread only deals at the Region level for queueing. Store-level > compaction queue entries will give us more visibility into compaction > workload + allow us to stop summarizing priorities. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188812#comment-13188812
]
Mikhail Bautin commented on HBASE-5010:
---
@Stack: there a small item pending here. The 89-fb patch contains more testing
for the compaction case, which has to be ported to trunk.
@Lars: I will work with Prakash to ensure this does not break HBASE-4721. I
believe HBASE-4721 is only used for a replication approach that is still in
development.
I think we'll keep this JIRA open until the two above items are resolved,
unless anyone objects.
> Filter HFiles based on TTL
> --
>
> Key: HBASE-5010
> URL: https://issues.apache.org/jira/browse/HBASE-5010
> Project: HBase
> Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Zhihong Yu
> Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
> D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch, D909.6.patch
>
>
> In ScanWildcardColumnTracker we have
> {code:java}
>
> this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
> ...
> private boolean isExpired(long timestamp) {
> return timestamp < oldestStamp;
> }
> {code}
> but this time range filtering does not participate in HFile selection. In one
> real case this caused next() calls to time out because all KVs in a table got
> expired, but next() had to iterate over the whole table to find that out. We
> should be able to filter out those HFiles right away. I think a reasonable
> approach is to add a "default timerange filter" to every scan for a CF with a
> finite TTL and utilize existing filtering in
> StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
[
https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188784#comment-13188784
]
Mikhail Bautin commented on HBASE-3976:
---
The last commit I can see for this JIRA is the following:
{quote}
Author: stack
Date: Tue Jun 14 16:47:37 2011
HBASE-3976 Disable Block Cache On Compactions -- REVERT
{quote}
This makes me think this path was actually reverted in trunk. Besides, when
Jonathan's HBASE-4422 (CacheConfig) went in, it completely changed the way this
feature would be implemented. This is why it is not possible to verify whether
the patch has been applied or not anymore from looking at the current version
of Store.java in trunk. I can't say off the top of my head whether we cache
blocks on write during compactions in HBase trunk these days with the new logic
implemented by HBASE-4422 in effect. However, I am planning to write a new unit
test for both trunk and 89-fb to verify that, and address this JIRA if
necessary in a consistent way across the two branches.
> Disable Block Cache On Compactions
> --
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Karthick Sankarachary
>Assignee: Nicolas Spiegelberg
>Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch,
> HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is
> beneficial? Currently, if block cache is enabled on a certain family, then
> every time it's compacted, we load all of its blocks into the (LRU) cache, at
> the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which
> rightly points out that, "we should not bog down the LRU with unneccessary
> blocks" during compaction. Even though that issue has been marked as "fixed",
> it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions
> period (as illustrated in the attached patch)? Or, can we be selectively
> aggressive about what blocks do get cached during compaction (e.g., only
> cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188175#comment-13188175 ] Mikhail Bautin commented on HBASE-4218: --- @Ted: we are in the process of doing a final review internally. It will probably be a couple more days -- we will post an update. Thanks! --Mikhail > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt, > D447.1.patch, D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, > D447.14.patch, D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch, > D447.19.patch, D447.2.patch, D447.20.patch, D447.21.patch, D447.22.patch, > D447.23.patch, D447.24.patch, D447.3.patch, D447.4.patch, D447.5.patch, > D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding-2012-01-17_11_09_09.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta-encoding.patch-2012-01-05_15_16_43.patch, > Delta-encoding.patch-2012-01-05_16_31_44.patch, > Delta-encoding.patch-2012-01-05_16_31_44_copy.patch, > Delta-encoding.patch-2012-01-05_18_50_47.patch, > Delta-encoding.patch-2012-01-07_14_12_48.patch, > Delta-encoding.patch-2012-01-13_12_20_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in MonitoredRPCHandlerImpl
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182135#comment-13182135 ] Mikhail Bautin commented on HBASE-5141: --- Actually, the committed patch contains more stuff than the patch attached to the JIRA: svn diff http://svn.apache.org/repos/asf/hbase/trunk -r1228740 Was the security version of the patch committed into trunk or something? > Memory leak in MonitoredRPCHandlerImpl > -- > > Key: HBASE-5141 > URL: https://issues.apache.org/jira/browse/HBASE-5141 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.92.0, 0.94.0 > > Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot > 2012-01-06 at 3.03.09 PM.png > > > I got a pretty reliable way of OOME'ing my region servers. Using a big > payload (64MB in my case), a default heap and default number of handlers, > it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB > reference and once a compaction kicks in it kills everything. > The issue is that even after the RPC call is done, the packet still lives in > MonitoredRPCHandlerImpl. > Will attach a screen shot of jprofiler's analysis in a moment. > This is a blocker for 0.92.0, anyone using a high number of handlers and > bigish values will kill themselves. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in MonitoredRPCHandlerImpl
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182136#comment-13182136 ] Mikhail Bautin commented on HBASE-5141: --- Correction: use this svn command: svn diff http://svn.apache.org/repos/asf/hbase/trunk -r1228739 > Memory leak in MonitoredRPCHandlerImpl > -- > > Key: HBASE-5141 > URL: https://issues.apache.org/jira/browse/HBASE-5141 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.92.0, 0.94.0 > > Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot > 2012-01-06 at 3.03.09 PM.png > > > I got a pretty reliable way of OOME'ing my region servers. Using a big > payload (64MB in my case), a default heap and default number of handlers, > it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB > reference and once a compaction kicks in it kills everything. > The issue is that even after the RPC call is done, the packet still lives in > MonitoredRPCHandlerImpl. > Will attach a screen shot of jprofiler's analysis in a moment. > This is a blocker for 0.92.0, anyone using a high number of handlers and > bigish values will kill themselves. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in MonitoredRPCHandlerImpl
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182132#comment-13182132 ] Mikhail Bautin commented on HBASE-5141: --- I get the error shown at http://pastebin.com/AdAp0M35 when trying to build the following commit: Author: stack Date: Sat Jan 7 14:16:11 2012 HBASE-5141 Memory leak in MonitoredRPCHandlerImpl git-svn-id: http://svn.apache.org/repos/asf/hbase/trunk@1228740 > Memory leak in MonitoredRPCHandlerImpl > -- > > Key: HBASE-5141 > URL: https://issues.apache.org/jira/browse/HBASE-5141 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.92.0, 0.94.0 > > Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot > 2012-01-06 at 3.03.09 PM.png > > > I got a pretty reliable way of OOME'ing my region servers. Using a big > payload (64MB in my case), a default heap and default number of handlers, > it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB > reference and once a compaction kicks in it kills everything. > The issue is that even after the RPC call is done, the packet still lives in > MonitoredRPCHandlerImpl. > Will attach a screen shot of jprofiler's analysis in a moment. > This is a blocker for 0.92.0, anyone using a high number of handlers and > bigish values will kill themselves. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182134#comment-13182134 ] Mikhail Bautin commented on HBASE-4218: --- @Ted: I was running a load test with LZO compression and PREFIX encoding and everything was fine, but then I switched to encoding in cache only and compactions started failing. I need to look into this. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, 4218.txt, D447.1.patch, > D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, > D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch, D447.19.patch, > D447.2.patch, D447.20.patch, D447.21.patch, D447.3.patch, D447.4.patch, > D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta-encoding.patch-2012-01-05_15_16_43.patch, > Delta-encoding.patch-2012-01-05_16_31_44.patch, > Delta-encoding.patch-2012-01-05_16_31_44_copy.patch, > Delta-encoding.patch-2012-01-05_18_50_47.patch, > Delta-encoding.patch-2012-01-07_14_12_48.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5141) Memory leak in MonitoredRPCHandlerImpl
[ https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182127#comment-13182127 ] Mikhail Bautin commented on HBASE-5141: --- FYI: The build is broken in trunk because of this patch. > Memory leak in MonitoredRPCHandlerImpl > -- > > Key: HBASE-5141 > URL: https://issues.apache.org/jira/browse/HBASE-5141 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.92.0, 0.94.0 > > Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot > 2012-01-06 at 3.03.09 PM.png > > > I got a pretty reliable way of OOME'ing my region servers. Using a big > payload (64MB in my case), a default heap and default number of handlers, > it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB > reference and once a compaction kicks in it kills everything. > The issue is that even after the RPC call is done, the packet still lives in > MonitoredRPCHandlerImpl. > Will attach a screen shot of jprofiler's analysis in a moment. > This is a blocker for 0.92.0, anyone using a high number of handlers and > bigish values will kill themselves. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180987#comment-13180987 ] Mikhail Bautin commented on HBASE-4218: --- The failed tests above pass locally: Running org.apache.hadoop.hbase.replication.TestReplication Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.447 sec Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 216.844 sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.119 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.373 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.574 sec Results : Tests run: 27, Failures: 0, Errors: 0, Skipped: 0 The patch also works good (so far) in a LoadTestTool 5-node cluster test with LZO compression and PREFIX encoding. I have a couple more minor changes to the patch, so please don't commit yet. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, 4218.txt, D447.1.patch, > D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, > D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch, D447.2.patch, > D447.3.patch, D447.4.patch, D447.5.patch, D447.6.patch, D447.7.patch, > D447.8.patch, D447.9.patch, Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta-encoding.patch-2012-01-05_15_16_43.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179959#comment-13179959 ] Mikhail Bautin commented on HBASE-4218: --- Actually, I think it is OK to ignore cached encoded blocks on compaction. We can get encoded blocks in cache and have a compaction write an unencoded file in two cases: * Encoding is turned on in cache only. In that case we don't want to use encoded blocks during compaction at all, because the in-cache-only mode implies that we don't trust our encoding algorithms 100% and want to guard against possible persistent data corruption. * Encoding was turned on (either in cache only or everywhere) and it was turned off entirely. Since this is not a very frequent case, I think we could probably optimize this after the patch is stabilized. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, 4218.txt, D447.1.patch, > D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, > D447.15.patch, D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, > D447.4.patch, D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, > D447.9.patch, Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179960#comment-13179960 ] Mikhail Bautin commented on HBASE-4218: --- Re-reading my previous post, I want to make an addition: we still use cached encoded blocks when compacting a fully-encoded column family. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, 4218.txt, D447.1.patch, > D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, > D447.15.patch, D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, > D447.4.patch, D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, > D447.9.patch, Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179945#comment-13179945 ] Mikhail Bautin commented on HBASE-4218: --- I think that with an 8K line patch we probably should not try to put more complexity into the first version of delta encoding. We can always make things more complicated later. I like the two-parameter setup: DATA_BLOCK_ENCODING sets the encoding type (on-disk and in-cache by default) and ENCODE_ON_DISK (true by default) allows to use in-cache-only encoding (when explicitly setting ENCODE_ON_DISK=false) and get the benefit of encoding in cache even before we are 100% sure that our encoding algorithms and encoded scanners are stable. If everyone agrees with that, I will finish the patch by (1) adding a unit test for switching data block encoding column family settings; (2) including encoding type in the cache key; and (3) simplifying the HFileDataBlockEncoder interface, since we assume that the "in-memory format" (used by scanners) is always the same as the in-cache format and don't need methods such as afterReadFromDiskAndPuttingInCache anymore. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, 4218.txt, D447.1.patch, > D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, > D447.15.patch, D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, > D447.4.patch, D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, > D447.9.patch, Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179349#comment-13179349 ] Mikhail Bautin commented on HBASE-4218: --- A brief status update. I am in the process of implementing support for column family data block encoding configuration changes. Those changes are coming in the next version of the patch that I will post tomorrow. After discussing this with Kannan, our solution is: * Assign an in-cache data block encoding to every HFile reader. This in-cache encoding is determined as follows: ** If the HFile is not encoded on disk, the in-cache encoding is set to the column family's DATA_BLOCK_ENCODING. ** If the HFile is encoded on disk, the in-cache encoding is set to the HFile encoding to avoid the wasted effort of re-encoding blocks for cache. * When a non-encoded block is loaded from disk, it is encoded using the in-cache encoding and put in cache. * When an encoded block is loaded from disk, its encoding is left as is. * To reduce the complexity of data block encoding switching, we can include the in-cache encoding type in the block cache key. For example, if ENCODED_IN_CACHE_ONLY is turned on without encoding on disk, and then the encoding is turned off altogether, the cache will be populated with non-encoded blocks (since they will have completely different keys) and encoded blocks will age out from the cache. While this is suboptimal, the implementation is very simple and the common case (when the CF encoding options do not change) is not complicated with unnecessary corner cases. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, 4218.txt, D447.1.patch, > D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, > D447.15.patch, D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, > D447.4.patch, D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, > D447.9.patch, Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179161#comment-13179161 ] Mikhail Bautin commented on HBASE-4218: --- Here is another update after discussing this with Jerry. Actually, the real value of in-cache-only encoding for us is that if we can get a benefit of data block encoding in production faster without risking data corruption, so we still want to support that option. This benefit should come from being able to put more stuff in cache, and (based on Jacek's experiments, I haven't confirmed this myself) from faster encoded scanners. We really need to make sure that we don't go through encoding/decoding on compactions when in-cache-only encoding is enabled, though. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, D447.1.patch, D447.10.patch, > D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, > D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, D447.4.patch, > D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179153#comment-13179153 ] Mikhail Bautin commented on HBASE-4218: --- @Matt: what do you call "the settings UI"? I thought HColumnDescriptor was part of the user-visible API, and if we allowed more flexible options there, we would have to fully support them everywhere. On the performance issue: HBase is IO-bound for most production workloads, so if we can fit more data into cache, we should get a performance win. Jacek reported that encoded scanners were faster in his experiments, and if they are not, we should optimize them or disable prefix compression for that particular workload. In a CPU-bound situation, one reason encoded scanners could be slower is that the data does not compress well, so delta encoding introduces an unnecessary CPU overhead and does not really save any space in cache. For that type of workload, using prefix compression probably is not the right thing to do. Could you please share some more details about the workload in your test? Is it CPU-bound or IO-bound? Is it similar to your envisioned use case for data block encoding? Are you planning to use the PREFIX algorithm or your trie implementation? Does the trie algorithm have the same encoded scanner performance problem? @Lars, Matt: "We have all the framework in place" and "features or already working code" are relative concepts. The framework still needs to be tweaked to (1) support all real use cases people have in mind; and (2) allow to solidify the existing implementation and test it really well. Jacek's original patch did not handle switching data block encoding settings in the column family, and I am in the process of modifying the patch to support that. The more flexibility we allow for column family encoding configuration, the more cases we have to test, and the more exotic edge cases we get. A couple more notes on supporting switching data block encoding column family settings. Kannan and I discussed this, and we came up with a plan for allowing a seamless migration to a new data block encoding. Blocks read from existing HFiles will still be brought into cache using their original encoding, and we will allow storing a mixture of different data block encodings in the cache. The new encoding configuration will only be applied on flushes and compactions. This is similar to the seamless HFile format upgrade that we have already done successfully. Another possible way to simplify things even further could be to get rid of the ENCODE_IN_CACHE_ONLY option completely. We introduced it for testing, but it seems to be causing more trouble than it is worth, and actually slows down patch stabilization and testing. Such "test-mode" encoding would require extra care to avoid using encoding during compactions, because that could actually corrupt on-disk data. I think a better way would be to add more unit tests for various edge cases and transitions for simplified configuration options, and do more synthetic load testing with those. For dark launch cluster it is always possible to take a backup and roll back if a data corruption happens. I still need to discuss that option with Kannan and the rest of our team, but please let me know what you think. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, D447.1.patch, D447.10.patch, > D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, > D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, D447.4.patch, > D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key len
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179036#comment-13179036 ] Mikhail Bautin commented on HBASE-4218: --- @Matt, Ted: the problem with the previous settings was that they were too flexible, and allowed for different encodings in cache and in memory. We definitely don't want to re-encode a block using a different encoding algorithm after loading it from disk. After a discussion with Kannan we decided that the whole benefit of delta encoding is in encoded scanners and allowing to put more data into cache. If we want to use a compression algorithm on disk but not in cache, it is possible to implement that using the existing compression framework. Furthermore, Jacek found in his experiments that encoded scanners were actually faster than scanners on decoded blocks. Please let me know what use case you have in mind that would require storing decoded blocks in cache and would not allow for efficient scanning over encoded blocks. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, 4218-v16.txt, D447.1.patch, D447.10.patch, > D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, > D447.16.patch, D447.17.patch, D447.2.patch, D447.3.patch, D447.4.patch, > D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176976#comment-13176976 ] Mikhail Bautin commented on HBASE-4218: --- Just a quick note from an offline conversation with Kannan: we need to support modifying data block encoding column family settings. In the most recent version of the patch (https://reviews.facebook.net/D447?vs=&id=3237&whitespace=ignore-all) there are the following user-facing column family settings: * DATA_BLOCK_ENCODING - specifies data block encoding type or NONE * ENCODE_IN_CACHE_ONLY - boolean (false by default). If true, data blocks are only encoded in cache but not on disk We removed the "encoded scanner" flag, and we use encoded scanners by default any time we use data block encoding. Given the above column family settings, we need to unit-test at least the following transitions: # Switching from no data block encoding to a data block encoding everywhere, and vice versa # Switching from no data block encoding to a data block encoding in cache only, and vice versa # Flipping the "in cache only" flag but keeping the data block encoding type the same # Switching from one data block encoding everywhere to another one # Switching from one data block encoding in cache only to another one # Switching to a different data block encoding and flipping the "in cache only" flag. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, D447.1.patch, D447.10.patch, D447.11.patch, > D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, D447.2.patch, > D447.3.patch, D447.4.patch, D447.5.patch, D447.6.patch, D447.7.patch, > D447.8.patch, D447.9.patch, Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5031) [89-fb] Remove hard-coded non-existent host name from TestScanner
[ https://issues.apache.org/jira/browse/HBASE-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176787#comment-13176787 ] Mikhail Bautin commented on HBASE-5031: --- @Nicolas: TestScanner was only failing for this reason in 89-fb. Looking at the trunk version, I see that it does not use HServerAddress (which is deprecated in trunk) and does not perform a hostname lookup on "bar.foo.com". So yes, this is only relevant to 89-fb. > [89-fb] Remove hard-coded non-existent host name from TestScanner > -- > > Key: HBASE-5031 > URL: https://issues.apache.org/jira/browse/HBASE-5031 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > Attachments: D867.1.patch > > > TestScanner is failing on 0.89-fb because it has a hard-coded fake host name > that it is trying to look up. Replacing this with 127.0.0.1: > instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176024#comment-13176024
]
Mikhail Bautin commented on HBASE-5010:
---
@Ted: thanks for taking care of the patch! A couple of questions:
(1) Do you know why these spurious NumberFormatExceptions occur? Has this
happened before?
(2) Do you think we need this optimization in 0.92?
@Lars: thanks for the review!
> Filter HFiles based on TTL
> --
>
> Key: HBASE-5010
> URL: https://issues.apache.org/jira/browse/HBASE-5010
> Project: HBase
> Issue Type: Bug
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
> Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
> D909.2.patch
>
>
> In ScanWildcardColumnTracker we have
> {code:java}
>
> this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
> ...
> private boolean isExpired(long timestamp) {
> return timestamp < oldestStamp;
> }
> {code}
> but this time range filtering does not participate in HFile selection. In one
> real case this caused next() calls to time out because all KVs in a table got
> expired, but next() had to iterate over the whole table to find that out. We
> should be able to filter out those HFiles right away. I think a reasonable
> approach is to add a "default timerange filter" to every scan for a CF with a
> finite TTL and utilize existing filtering in
> StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175791#comment-13175791 ] Mikhail Bautin commented on HBASE-4218: --- @Ted: could you please revert the patch for now? It is not ready yet (sorry for not indicating this clearly, I will let you know when it's good to go). Even though it passes all unit tests, on Thursday I uncovered bugs in data block encoding handling during cluster testing. A simple load test with delta encoding turned on fails as soon as the first store file is written out. I am not sure if Jacek did this kind of testing during his internship, or if this is a new problem that I introduced while iterating on the patch. Furthermore, there is a design problem related to changing the encoding algorithm for an existing CF: if an encoded block has different encoding than what's configured by the CF, an assertion is thrown. These issues should not be that difficult to fix, though, and I still think the patch is very close to being finished. > Data Block Encoding of KeyValues (aka delta encoding / prefix compression) > --- > > Key: HBASE-4218 > URL: https://issues.apache.org/jira/browse/HBASE-4218 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jacek Migdal >Assignee: Mikhail Bautin > Labels: compression > Fix For: 0.94.0 > > Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, > 0001-Delta-encoding.patch, D447.1.patch, D447.10.patch, D447.11.patch, > D447.12.patch, D447.13.patch, D447.2.patch, D447.3.patch, D447.4.patch, > D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, > Data-block-encoding-2011-12-23.patch, > Delta-encoding.patch-2011-12-22_11_52_07.patch, > Delta_encoding_with_memstore_TS.patch, open-source.diff > > > A compression for keys. Keys are sorted in HFile and they are usually very > similar. Because of that, it is possible to design better compression than > general purpose algorithms, > It is an additional step designed to be used in memory. It aims to save > memory in cache as well as speeding seeks within HFileBlocks. It should > improve performance a lot, if key lengths are larger than value lengths. For > example, it makes a lot of sense to use it when value is a counter. > Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) > shows that I could achieve decent level of compression: > key compression ratio: 92% > total compression ratio: 85% > LZO on the same data: 85% > LZO after delta encoding: 91% > While having much better performance (20-80% faster decompression ratio than > LZO). Moreover, it should allow far more efficient seeking which should > improve performance a bit. > It seems that a simple compression algorithms are good enough. Most of the > savings are due to prefix compression, int128 encoding, timestamp diffs and > bitfields to avoid duplication. That way, comparisons of compressed data can > be much faster than a byte comparator (thanks to prefix compression and > bitfields). > In order to implement it in HBase two important changes in design will be > needed: > -solidify interface to HFileBlock / HFileReader Scanner to provide seeking > and iterating; access to uncompressed buffer in HFileBlock will have bad > performance > -extend comparators to support comparison assuming that N first bytes are > equal (or some fields are equal) > Link to a discussion about something similar: > http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)
[
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169842#comment-13169842
]
Mikhail Bautin commented on HBASE-4218:
---
bq. Maybe we could call it KeyValueEncoding, DataBlockEncoding, HCellEncoding,
BlockEncoding...
Matt: do you have a specific re-naming of delta encoders in mind? Jacek's
original delta encoding algorithm names are
{Bitset,Prefix,Diff,FastDiff}KeyDeltaEncoder. How do these correspond to the
alternative encoder names you are suggesting?
> Delta Encoding of KeyValues (aka prefix compression)
> -
>
> Key: HBASE-4218
> URL: https://issues.apache.org/jira/browse/HBASE-4218
> Project: HBase
> Issue Type: Improvement
> Components: io
>Affects Versions: 0.94.0
>Reporter: Jacek Migdal
>Assignee: Mikhail Bautin
> Labels: compression
> Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch,
> D447.1.patch, D447.2.patch, D447.3.patch, D447.4.patch, D447.5.patch,
> D447.6.patch, D447.7.patch, D447.8.patch,
> Delta_encoding_with_memstore_TS.patch, open-source.diff
>
>
> A compression for keys. Keys are sorted in HFile and they are usually very
> similar. Because of that, it is possible to design better compression than
> general purpose algorithms,
> It is an additional step designed to be used in memory. It aims to save
> memory in cache as well as speeding seeks within HFileBlocks. It should
> improve performance a lot, if key lengths are larger than value lengths. For
> example, it makes a lot of sense to use it when value is a counter.
> Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes)
> shows that I could achieve decent level of compression:
> key compression ratio: 92%
> total compression ratio: 85%
> LZO on the same data: 85%
> LZO after delta encoding: 91%
> While having much better performance (20-80% faster decompression ratio than
> LZO). Moreover, it should allow far more efficient seeking which should
> improve performance a bit.
> It seems that a simple compression algorithms are good enough. Most of the
> savings are due to prefix compression, int128 encoding, timestamp diffs and
> bitfields to avoid duplication. That way, comparisons of compressed data can
> be much faster than a byte comparator (thanks to prefix compression and
> bitfields).
> In order to implement it in HBase two important changes in design will be
> needed:
> -solidify interface to HFileBlock / HFileReader Scanner to provide seeking
> and iterating; access to uncompressed buffer in HFileBlock will have bad
> performance
> -extend comparators to support comparison assuming that N first bytes are
> equal (or some fields are equal)
> Link to a discussion about something similar:
> http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windows&subj=Re+prefix+compression
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4683) Always cache index and bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169720#comment-13169720 ] Mikhail Bautin commented on HBASE-4683: --- +1 on J-D's patch for 0.92 > Always cache index and bloom blocks > --- > > Key: HBASE-4683 > URL: https://issues.apache.org/jira/browse/HBASE-4683 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Mikhail Bautin >Priority: Minor > Fix For: 0.92.0, 0.94.0 > > Attachments: 4683-v2.txt, 4683.txt, D807.1.patch, D807.2.patch, > HBASE-4683-0.92-v2.patch, HBASE-4683-v3.patch > > > This would add a new boolean config option: hfile.block.cache.datablocks > Default would be true. > Setting this to false allows HBase in a mode where only index blocks are > cached, which is useful for analytical scenarios where a useful working set > of the data cannot be expected to fit into the (aggregate) cache. > This is the equivalent of setting cacheBlocks to false on all scans > (including scans on behalf of gets). > I would like to get a general feeling about what folks think about this. > The change itself would be simple. > Update (Mikhail): we probably don't need a new conf option. Instead, we will > make index blocks cached by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4965) Monitor the open file descriptors and the threads counters during the unit tests
[ https://issues.apache.org/jira/browse/HBASE-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167879#comment-13167879 ] Mikhail Bautin commented on HBASE-4965: --- Everything compiles again, thanks! It seems that only part of the files in this patch were committed the first time around. > Monitor the open file descriptors and the threads counters during the unit > tests > > > Key: HBASE-4965 > URL: https://issues.apache.org/jira/browse/HBASE-4965 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.0 > Environment: all >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Fix For: 0.94.0 > > Attachments: 4965-v2.txt, 4965-v2.txt, 4965-v3.txt, 4965_all.patch, > 4965_all.patch, 4965_all.patch, ResourceChecker.java, > ResourceCheckerJUnitRule.java > > > We're seeing a lot of issues with hadoop-qa related to threads or file > descriptors. > Monitoring these counters would ease the analysis. > Note as well that > - if we want to execute the tests in the same jvm (because the test is small > or because we want to share the cluster) we can't afford to leak too many > resources > - if the tests leak, it's more difficult to detect a leak in the software > itself. > I attach piece of code that I used. It requires two lines in a unit test > class to: > - before every test, count the threads and the open file descriptor > - after every test, compare with the previous value. > I ran it on some tests; we have for example: > - client.TestMultiParallel#testBatchWithManyColsInOneRowGetAndPut: 232 > threads (was 231), 390 file descriptors (was 390). => TestMultiParallel uses > 232 threads! > - client.TestMultipleTimestamps#testWithColumnDeletes: 152 threads (was 151), > 283 file descriptors (was 282). > - client.TestAdmin#testCheckHBaseAvailableClosesConnection: 477 threads (was > 294), 815 file descriptors (was 461) > - client.TestMetaMigrationRemovingHTD#testMetaMigration: 149 threads (was > 148), 310 file descriptors (was 307). > It's not always leaks, we can expect some pooling effects. But still... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4965) Monitor the open file descriptors and the threads counters during the unit tests
[ https://issues.apache.org/jira/browse/HBASE-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167858#comment-13167858 ] Mikhail Bautin commented on HBASE-4965: --- I am getting a ton of compile errors caused by this change: http://pastebin.com/Lw7HNt75. I am using hadoop-0.20.205.0 (the default one in pom.xml). Can we get this fixed (change the default Hadoop version, etc. etc.)? > Monitor the open file descriptors and the threads counters during the unit > tests > > > Key: HBASE-4965 > URL: https://issues.apache.org/jira/browse/HBASE-4965 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.0 > Environment: all >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Fix For: 0.94.0 > > Attachments: 4965-v2.txt, 4965-v2.txt, 4965-v3.txt, 4965_all.patch, > 4965_all.patch, 4965_all.patch, ResourceChecker.java, > ResourceCheckerJUnitRule.java > > > We're seeing a lot of issues with hadoop-qa related to threads or file > descriptors. > Monitoring these counters would ease the analysis. > Note as well that > - if we want to execute the tests in the same jvm (because the test is small > or because we want to share the cluster) we can't afford to leak too many > resources > - if the tests leak, it's more difficult to detect a leak in the software > itself. > I attach piece of code that I used. It requires two lines in a unit test > class to: > - before every test, count the threads and the open file descriptor > - after every test, compare with the previous value. > I ran it on some tests; we have for example: > - client.TestMultiParallel#testBatchWithManyColsInOneRowGetAndPut: 232 > threads (was 231), 390 file descriptors (was 390). => TestMultiParallel uses > 232 threads! > - client.TestMultipleTimestamps#testWithColumnDeletes: 152 threads (was 151), > 283 file descriptors (was 282). > - client.TestAdmin#testCheckHBaseAvailableClosesConnection: 477 threads (was > 294), 815 file descriptors (was 461) > - client.TestMetaMigrationRemovingHTD#testMetaMigration: 149 threads (was > 148), 310 file descriptors (was 307). > It's not always leaks, we can expect some pooling effects. But still... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4683) Always cache index blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167824#comment-13167824 ] Mikhail Bautin commented on HBASE-4683: --- Actually shouldCacheDataOnRead is weird. It is true by default as long as block cache is enabled, unless it is explicitly disabled in cacheConf. So I don't really know if that particular line ever becomes a problem when block cache is enabled. > Always cache index blocks > - > > Key: HBASE-4683 > URL: https://issues.apache.org/jira/browse/HBASE-4683 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Mikhail Bautin >Priority: Minor > Fix For: 0.94.0 > > Attachments: 4683-v2.txt, 4683.txt > > > This would add a new boolean config option: hfile.block.cache.datablocks > Default would be true. > Setting this to false allows HBase in a mode where only index blocks are > cached, which is useful for analytical scenarios where a useful working set > of the data cannot be expected to fit into the (aggregate) cache. > This is the equivalent of setting cacheBlocks to false on all scans > (including scans on behalf of gets). > I would like to get a general feeling about what folks think about this. > The change itself would be simple. > Update (Mikhail): we probably don't need a new conf option. Instead, we will > make index blocks cached by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4683) Create config option to only cache index blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167660#comment-13167660 ] Mikhail Bautin commented on HBASE-4683: --- Lars: we won't be able to cache index blocks if the block cache is disabled completely. However, if it is enabled, we should always cache index blocks. Please let me know if your understanding of this issue is different. > Create config option to only cache index blocks > --- > > Key: HBASE-4683 > URL: https://issues.apache.org/jira/browse/HBASE-4683 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Fix For: 0.94.0 > > Attachments: 4683-v2.txt, 4683.txt > > > This would add a new boolean config option: hfile.block.cache.datablocks > Default would be true. > Setting this to false allows HBase in a mode where only index blocks are > cached, which is useful for analytical scenarios where a useful working set > of the data cannot be expected to fit into the (aggregate) cache. > This is the equivalent of setting cacheBlocks to false on all scans > (including scans on behalf of gets). > I would like to get a general feeling about what folks think about this. > The change itself would be simple. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4683) Create config option to only cache index blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166601#comment-13166601 ] Mikhail Bautin commented on HBASE-4683: --- This is from HFileBlockIndex.java, line 202 (trunk): boolean shouldCache = cacheBlocks || (lookupLevel < searchTreeLevel) On the other hand, in HFileReaderV2 we have: cacheBlock &= cacheConf.shouldCacheDataOnRead(). So currently, when we disable block caching, we don't even cache data blocks, and there is a valid issue to be fixed in this JIRA. > Create config option to only cache index blocks > --- > > Key: HBASE-4683 > URL: https://issues.apache.org/jira/browse/HBASE-4683 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Priority: Minor > Fix For: 0.94.0 > > Attachments: 4683-v2.txt, 4683.txt > > > This would add a new boolean config option: hfile.block.cache.datablocks > Default would be true. > Setting this to false allows HBase in a mode where only index blocks are > cached, which is useful for analytical scenarios where a useful working set > of the data cannot be expected to fit into the (aggregate) cache. > This is the equivalent of setting cacheBlocks to false on all scans > (including scans on behalf of gets). > I would like to get a general feeling about what folks think about this. > The change itself would be simple. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4835) ConcurrentModificationException out of ZKConfig.makeZKProps
[
https://issues.apache.org/jira/browse/HBASE-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154366#comment-13154366
]
Mikhail Bautin commented on HBASE-4835:
---
@Andrew: thanks for the fix!
The simple approach with copying the configuration sounds good -- I presume we
don't create too many unique ZooKeeperWatchers in a single JVM.
Alternatively, we could somehow get an immutable snapshot of the
configuration's key set and iterate that instead of the configuration itself.
> ConcurrentModificationException out of ZKConfig.makeZKProps
> ---
>
> Key: HBASE-4835
> URL: https://issues.apache.org/jira/browse/HBASE-4835
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Attachments: HBASE-4835.patch
>
>
> Mikhail reported this from a five-node, three-RS cluster test:
> {code}
> 2011-11-21 01:30:15,188 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> ,60020,1321867814890: Initialization of RS failed. Hence
> aborting RS.
> java.util.ConcurrentModificationException
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1031)
> at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1042)
> at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:75)
> at
> org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:144)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:124)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1262)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:568)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.(HConnectionManager.java:559)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:183)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.(CatalogTracker.java:177)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:575)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:534)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:642)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154088#comment-13154088 ] Mikhail Bautin commented on HBASE-2418: --- I just saw this regionserver crash in my five-node, three-RS cluster test. Since this is a ZK-related patch that went in recently, I am attaching the stack trace here just in case. 2011-11-21 01:30:15,188 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ,60020,1321867814890: Initialization of RS failed. Hence aborting RS. java.util.ConcurrentModificationException at java.util.Hashtable$Enumerator.next(Hashtable.java:1031) at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1042) at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:75) at org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:144) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:124) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1262) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:568) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.(HConnectionManager.java:559) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:183) at org.apache.hadoop.hbase.catalog.CatalogTracker.(CatalogTracker.java:177) at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:575) at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:534) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:642) at java.lang.Thread.run(Thread.java:619) > add support for ZooKeeper authentication > > > Key: HBASE-2418 > URL: https://issues.apache.org/jira/browse/HBASE-2418 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Patrick Hunt >Assignee: Eugene Koontz >Priority: Critical > Labels: security, zookeeper > Fix For: 0.92.0, 0.94.0 > > Attachments: 2418.addendum, HBASE-2418-6.patch, HBASE-2418-6.patch > > > Some users may run a ZooKeeper cluster in "multi tenant mode" meaning that > more than one client service would > like to share a single ZooKeeper service instance (cluster). In this case the > client services typically want to protect > their data (ZK znodes) from access by other services (tenants) on the > cluster. Say you are running HBase and Solr > and Neo4j, or multiple HBase instances, etc... having > authentication/authorization on the znodes is important for both > security and helping to ensure that services don't interact negatively (touch > each other's data). > Today HBase does not have support for authentication or authorization. This > should be added to the HBase clients > that are accessing the ZK cluster. In general it means calling addAuthInfo > once after a session is established: > http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, > byte[]) > with a user specific credential, often times this is a shared secret or > certificate. You may be able to statically configure this > in some cases (config string or file to read from), however in my case in > particular you may need to access it programmatically, > which adds complexity as the end user may need to load code into HBase for > accessing the credential. > Secondly you need to specify a non "world" ACL when interacting with znodes > (create primarily): > http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html > http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html > Feel free to ping the ZooKeeper team if you have questions. It might also be > good to discuss with some > potential end users - in particular regarding how the end user can specify > the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4824) TestZKLeaderManager is flaky
[ https://issues.apache.org/jira/browse/HBASE-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153368#comment-13153368 ] Mikhail Bautin commented on HBASE-4824: --- Probably the same issue. Closing. Thanks! > TestZKLeaderManager is flaky > > > Key: HBASE-4824 > URL: https://issues.apache.org/jira/browse/HBASE-4824 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Priority: Minor > > TestZKLeaderManager is flaky. It failed in a full test suite run for me, then > passed when I reran it locally, but then failed when I ran it in a loop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153319#comment-13153319 ] Mikhail Bautin commented on HBASE-4821: --- Thanks everyone for your comments. I will read up on BigTop and Puppet. I am also fine with using a JVM-based language for load tests themselves, as long as there is a way to do something like "kill -9", which we can't really do in our unit tests. We could also try to reuse/modify the MiniHBaseCluster framework to talk to a real HBase cluster and script various distributed test scenarios in pure Java. However, I want to emphasize one thing. Once configured, this HBase integration test tool should be extremely easy to use, as simple as: hbase_integration_test.sh . We might have to write some amount of nontrivial "glue" script code (e.g. in bash) to make that happen. > A fully automated comprehensive distributed integration test for HBase > -- > > Key: HBASE-4821 > URL: https://issues.apache.org/jira/browse/HBASE-4821 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Critical > > To properly verify that a particular version of HBase is good for production > deployment we need a better way to do real cluster testing after incremental > changes. Running unit tests is good, but we also need to deploy HBase to a > cluster, run integration tests, load tests, Thrift server tests, kill some > region servers, kill the master, and produce a report. All of this needs to > happen in 20-30 minutes with minimal manual intervention. I think this way we > can combine agile development with high stability of the codebase. I am > envisioning a high-level framework written in a scripting language (e.g. > Python) that would abstract external operations such as "deploy to test > cluster", "kill a particular server", "run load test A", "run load test B" > (we already have a few kinds of load tests implemented in Java, and we could > write a Thrift load test in Python). This tool should also produce > intermediate output, allowing to catch problems early and restart the test. > No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion
[ https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150754#comment-13150754 ] Mikhail Bautin commented on HBASE-4768: --- @Ted: thanks for clarifying and for taking care of this! Sorry for breaking the unit test. Now I remember that this issue also came up when porting HFile v2. +1 on the addendum > Per-(table, columnFamily) metrics with configurable table name inclusion > > > Key: HBASE-4768 > URL: https://issues.apache.org/jira/browse/HBASE-4768 > Project: HBase > Issue Type: New Feature >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Fix For: 0.94.0 > > Attachments: 4768.addendum, D363.1.patch, D363.2.patch, D363.3.patch, > D363.4.patch, D363.5.patch > > > As we kept adding more granular block read and block cache usage statistics, > a combinatorial explosion of various cases to monitor started to happen, > especially when we wanted both per-table/column family/block type statistics > and aggregate statistics on various subsets of these dimensions. Here, we > un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a > centralized class that knows how to update all kinds of per-table/CF/block > type counters. > Table name and column family configuration have been pushed to a base class, > SchemaConfigured. This is convenient as many of existing classes that have > these properties (HFile readers/writers, HFile blocks, etc.) did not have a > base class. Whether to collect per-(table, columnFamily) or per-columnFamily > only metrics can be configured with the hbase.metrics.showTableName > configuration key. We don't expect this configuration to change at runtime, > so we cache the setting statically and log a warning when an attempt is made > to flip it once already set. This way we don't have to pass configuration to > a lot more places, e.g. everywhere an HFile reader is instantiated. > Thanks to Liyin for his initial version of per-table metrics patch and a lot > of valuable feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4768) Per-(table, columnFamily) metrics with configurable table name inclusion
[ https://issues.apache.org/jira/browse/HBASE-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150686#comment-13150686 ] Mikhail Bautin commented on HBASE-4768: --- @Ted: actually, TestHFileBlock passes when I run it locally, and I don't see how BYTE_BUFFER_HEAP_SIZE could have changed with my changes. Does the unit test pass or fail when you run it locally? public static final int BYTE_BUFFER_HEAP_SIZE = (int) ClassSize.estimateBase( ByteBuffer.wrap(new byte[0], 0, 0).getClass(), false); Do you know of anything in the Hudson testing environment that could have affected this? > Per-(table, columnFamily) metrics with configurable table name inclusion > > > Key: HBASE-4768 > URL: https://issues.apache.org/jira/browse/HBASE-4768 > Project: HBase > Issue Type: New Feature >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Fix For: 0.94.0 > > Attachments: 4768.addendum, D363.1.patch, D363.2.patch, D363.3.patch, > D363.4.patch, D363.5.patch > > > As we kept adding more granular block read and block cache usage statistics, > a combinatorial explosion of various cases to monitor started to happen, > especially when we wanted both per-table/column family/block type statistics > and aggregate statistics on various subsets of these dimensions. Here, we > un-clutters HFile readers, LruBlockCache, StoreFile, etc. by creating a > centralized class that knows how to update all kinds of per-table/CF/block > type counters. > Table name and column family configuration have been pushed to a base class, > SchemaConfigured. This is convenient as many of existing classes that have > these properties (HFile readers/writers, HFile blocks, etc.) did not have a > base class. Whether to collect per-(table, columnFamily) or per-columnFamily > only metrics can be configured with the hbase.metrics.showTableName > configuration key. We don't expect this configuration to change at runtime, > so we cache the setting statically and log a warning when an attempt is made > to flip it once already set. This way we don't have to pass configuration to > a lot more places, e.g. everywhere an HFile reader is instantiated. > Thanks to Liyin for his initial version of per-table metrics patch and a lot > of valuable feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4704) A JRuby script for identifying active master
[ https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149811#comment-13149811 ] Mikhail Bautin commented on HBASE-4704: --- @Stack/Ted: could you please commit this patch? Thanks! --Mikhail > A JRuby script for identifying active master > > > Key: HBASE-4704 > URL: https://issues.apache.org/jira/browse/HBASE-4704 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Trivial > Fix For: 0.94.0 > > Attachments: D117.1.patch, HBASE-4704.D117.2.patch, > HBASE-4704.D117.3.patch > > > This simple script reads the HBase master ZK node and outputs the hostname of > the active master. This is needed so that operational scripts can decide > where the primary master is running. I am also including a one-line > hbase-jruby script so we can make our jruby scripts proper UNIX executables > by including an "#!/usr/bin/env hbase-jruby" at the top. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4746) Use a random ZK client port in unit tests so we can run them in parallel
[ https://issues.apache.org/jira/browse/HBASE-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145979#comment-13145979 ] Mikhail Bautin commented on HBASE-4746: --- > Why did the statics have to be removed from TestMasterReplication but not > from TestReplication and TestMultislaveReplication (and I do apologize for > the cut-and-pasting here)? @Lars: I am not sure why TestReplication and TestMultislveReplication worked fine without changing them to set up and tear down a mini cluster for every test. These tests are complex enough and sequential test methods were most likely stepping on each other's state in TestMasterReplication. I think it only makes sense to isolate test methods within a unit test. > Use a random ZK client port in unit tests so we can run them in parallel > > > Key: HBASE-4746 > URL: https://issues.apache.org/jira/browse/HBASE-4746 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Fix For: 0.94.0 > > Attachments: 4746-trunk-v2.txt, D255.1.patch, D255.2.patch, > D279-trunk-v5.txt, D279-trunk-v7.txt, D279.1.patch, D279.2.patch, > D279.3.patch, D279.4.patch, D279.5.patch, D279.6.patch, D279.7.patch, D279.92 > > > The hard-coded ZK client port has long been a problem for running HBase test > suite in parallel. The mini ZK cluster should run on a random free port, and > that port should be passed to all parts of the unit tests that need to talk > to the mini cluster. In fact, randomizing the port exposes a lot of places in > the code where a new configuration is instantiated, and as a result the > client tries to talk to the default ZK client port and times out. > The initial fix is for 0.89-fb, where it already allows to run unit tests in > parallel in 10 minutes. A fix for the trunk will follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4746) Use a random ZK client port in unit tests so we can run them in parallel
[ https://issues.apache.org/jira/browse/HBASE-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145689#comment-13145689 ] Mikhail Bautin commented on HBASE-4746: --- Never mind my last comment :) got an update as I was writing it. Thanks a lot Ted and Stack! > Use a random ZK client port in unit tests so we can run them in parallel > > > Key: HBASE-4746 > URL: https://issues.apache.org/jira/browse/HBASE-4746 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Fix For: 0.94.0 > > Attachments: 4746-trunk-v2.txt, D255.1.patch, D255.2.patch, > D279-trunk-v5.txt, D279-trunk-v7.txt, D279.1.patch, D279.2.patch, > D279.3.patch, D279.4.patch, D279.5.patch, D279.6.patch, D279.7.patch > > > The hard-coded ZK client port has long been a problem for running HBase test > suite in parallel. The mini ZK cluster should run on a random free port, and > that port should be passed to all parts of the unit tests that need to talk > to the mini cluster. In fact, randomizing the port exposes a lot of places in > the code where a new configuration is instantiated, and as a result the > client tries to talk to the default ZK client port and times out. > The initial fix is for 0.89-fb, where it already allows to run unit tests in > parallel in 10 minutes. A fix for the trunk will follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4746) Use a random ZK client port in unit tests so we can run them in parallel
[ https://issues.apache.org/jira/browse/HBASE-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145686#comment-13145686 ] Mikhail Bautin commented on HBASE-4746: --- @Ted: thanks for fixing the part of the patch that did not apply (I have no idea why). I think D279-trunk-v7.txt is ready to be committed. > Use a random ZK client port in unit tests so we can run them in parallel > > > Key: HBASE-4746 > URL: https://issues.apache.org/jira/browse/HBASE-4746 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Fix For: 0.94.0 > > Attachments: 4746-trunk-v2.txt, D255.1.patch, D255.2.patch, > D279-trunk-v5.txt, D279-trunk-v7.txt, D279.1.patch, D279.2.patch, > D279.3.patch, D279.4.patch, D279.5.patch, D279.6.patch, D279.7.patch > > > The hard-coded ZK client port has long been a problem for running HBase test > suite in parallel. The mini ZK cluster should run on a random free port, and > that port should be passed to all parts of the unit tests that need to talk > to the mini cluster. In fact, randomizing the port exposes a lot of places in > the code where a new configuration is instantiated, and as a result the > client tries to talk to the default ZK client port and times out. > The initial fix is for 0.89-fb, where it already allows to run unit tests in > parallel in 10 minutes. A fix for the trunk will follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4746) Use a random ZK client port in unit tests so we can run them in parallel
[ https://issues.apache.org/jira/browse/HBASE-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145098#comment-13145098 ] Mikhail Bautin commented on HBASE-4746: --- @Ted: how did you generate 4746-trunk-v2.txt? When diffing D279.2.patch against 4746-trunk-v2.txt, I get a lot of differences. Can we use the D279.2.patch? The default ThriftHBaseServiceHandler constructor is not needed anymore in ThriftServer, because I instantiated the configuration in ThriftServer on constructor invocation. But if you think I should add the default constructor back (e.g. if it part of client API), please let me know and I will upload a D279.3.patch. Thank you! --Mikhail > Use a random ZK client port in unit tests so we can run them in parallel > > > Key: HBASE-4746 > URL: https://issues.apache.org/jira/browse/HBASE-4746 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Attachments: 4746-trunk-v2.txt, D255.1.patch, D279.1.patch, > D279.2.patch > > > The hard-coded ZK client port has long been a problem for running HBase test > suite in parallel. The mini ZK cluster should run on a random free port, and > that port should be passed to all parts of the unit tests that need to talk > to the mini cluster. In fact, randomizing the port exposes a lot of places in > the code where a new configuration is instantiated, and as a result the > client tries to talk to the default ZK client port and times out. > The initial fix is for 0.89-fb, where it already allows to run unit tests in > parallel in 10 minutes. A fix for the trunk will follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144884#comment-13144884 ] Mikhail Bautin commented on HBASE-4737: --- I tried to run trunk unit tests using my map-reduce test runner (which unfortunately only works in Facebook environment at the moment). To my surprise, it worked extremely well without HBASE-4746, confirming N's observation. The job completed in under 10 minutes, just like the one for 89-fb, and ran 239 test classes with 1096 tests. The only failure was TestHRegionInfo.testGetSetOfHTD (an NPE; will investigate). I started working on HBASE-4767 patch for trunk, but most of its value will be in refactoring and cleanup, since it looks like there are very few bugs left in the trunk related to not passing configuration from MiniHBaseCluster to clients in unit tests. > Split the tests in small/medium/large; allow small tests to be ran in > parallel within a single JVM > -- > > Key: HBASE-4737 > URL: https://issues.apache.org/jira/browse/HBASE-4737 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Attachments: 2003_4737_pom.dummy.patch, 2003_4737_pom.patch, > 2003_4737_pom.patch, 2003_4737_pom.patch, 2003_4737_pom.patch, > 2003_4737_pom.v2.patch, 2003_4737_pom.v2.patch, hbasetests.sh > > > 1) Split the tests in 3 categories > - small: no cluster, less than 15s, can be run in parallel with other tests > in a JVM > - medium: <45s, no flaky, useful to detect bugs immediatly > - large: remaining > > 2) Allow to run a subset: developpers should need to run only small and > medium before submitting a patch > - will need a surefire patch, see > http://jira.codehaus.org/browse/SUREFIRE-329 > Small is the default. All other tests will have to be marked Medium or Large > with a JUnit category. > Proposed split: > Small:122 classes, 479 methods, ~3 minutes when no //) > Medium: 78 classes, 373 methods, ~23 minutes > Large: 34 classes, 221 methods, ~60 minutes > I will have to extract the methods that are today in large or medium but > could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it > will be done in a second step (and another JIRA). > MEDIUM LIST (name; number of methods, time) > org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 > org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 > org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 > org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 > org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 > 0.762 > org.apache.hadoop.hbase.client.TestHCM3 21.961 > org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 > org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 > org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 > org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 > org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 > org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 > org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 > org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 > org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort > 1 13.874 > org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove > 1 16.923 > org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 > org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 > org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 > 33.353 > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort > 1 16.596 > org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove > 1 18.183 > org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 > org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 > org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 > org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 > org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 > org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 > org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 > org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 > org.apache.hadoop.hbase.master.
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics aggregation
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137649#comment-13137649 ] Mikhail Bautin commented on HBASE-4686: --- Sorry for confusing file names. The most recent patch is: https://issues.apache.org/jira/secure/attachment/12501169/HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch > [89-fb] Fix per-store metrics aggregation > -- > > Key: HBASE-4686 > URL: https://issues.apache.org/jira/browse/HBASE-4686 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Attachments: > HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, > > HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch > > > In r1182034 per-Store metrics were broken, because the aggregation of > StoreFile metrics over all stores in a region was replaced by overriding them > every time. We saw these metrics drop by a factor of numRegions on a > production cluster -- thanks to Kannan for noticing this! We need to fix the > metrics and add a unit test to ensure regressions like this don't happen in > the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4686) [89-fb] Fix per-store metrics aggregation
[ https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137643#comment-13137643 ] Mikhail Bautin commented on HBASE-4686: --- I put another version of the patch on Differential: https://reviews.facebook.net/D87 > [89-fb] Fix per-store metrics aggregation > -- > > Key: HBASE-4686 > URL: https://issues.apache.org/jira/browse/HBASE-4686 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Attachments: > HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch, > > HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch > > > In r1182034 per-Store metrics were broken, because the aggregation of > StoreFile metrics over all stores in a region was replaced by overriding them > every time. We saw these metrics drop by a factor of numRegions on a > production cluster -- thanks to Kannan for noticing this! We need to fix the > metrics and add a unit test to ensure regressions like this don't happen in > the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4191) hbase load balancer needs locality awareness
[ https://issues.apache.org/jira/browse/HBASE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134803#comment-13134803 ] Mikhail Bautin commented on HBASE-4191: --- @Ted: could you please elaborate on how you express the region assignment problem as a Max Flow problem? If we define the "cost" of assigning a region to a server based on locality, and define a constraint of "load balancedness" to be such that each regionserver is assigned no more than approximately ceil(numRegions / numServers) + C regions for some small value of C, then I can see how the problem becomes a min-cost max flow (http://en.wikipedia.org/wiki/Minimum_cost_flow_problem). However, I don't see how we could reduce the assignment problem to the max-flow problem directly (http://en.wikipedia.org/wiki/Maximum_flow_problem). > hbase load balancer needs locality awareness > > > Key: HBASE-4191 > URL: https://issues.apache.org/jira/browse/HBASE-4191 > Project: HBase > Issue Type: New Feature >Reporter: Ted Yu >Assignee: Liyin Tang > > Previously, HBASE-4114 implements the metrics for HFile HDFS block locality, > which provides the HFile level locality information. > But in order to work with load balancer and region assignment, we need the > region level locality information. > Let's define the region locality information first, which is almost the same > as HFile locality index. > HRegion locality index (HRegion A, RegionServer B) = > (Total number of HDFS blocks that can be retrieved locally by the > RegionServer B for the HRegion A) / ( Total number of the HDFS blocks for the > Region A) > So the HRegion locality index tells us that how much locality we can get if > the HMaster assign the HRegion A to the RegionServer B. > So there will be 2 steps involved to assign regions based on the locality. > 1) During the cluster start up time, the master will scan the hdfs to > calculate the "HRegion locality index" for each pair of HRegion and Region > Server. It is pretty expensive to scan the dfs. So we only needs to do this > once during the start up time. > 2) During the cluster run time, each region server will update the "HRegion > locality index" as metrics periodically as HBASE-4114 did. The Region Server > can expose them to the Master through ZK, meta table, or just RPC messages. > Based on the "HRegion locality index", the assignment manager in the master > would have a global knowledge about the region locality distribution. Imaging > the "HRegion locality index" as the capacity between the region server set > and region set, the assignment manager could the run the MAXIMUM FLOW solver > to reach the global optimization. > Also the master should share this global view to secondary master in case the > master fail over happens. > In addition, the HBASE-4491 (Locality Checker) is the tool, which is based on > the same metrics, to proactively to scan dfs to calculate the global locality > information in the cluster. It will help us to verify data locality > information during the run time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4607) Split log worker should terminate properly when waiting for znode
[ https://issues.apache.org/jira/browse/HBASE-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129822#comment-13129822 ] Mikhail Bautin commented on HBASE-4607: --- Unit tests seem to succeed, except for usual spurious failures. 2011-10-17_23_26_13 commit: Checking exitWorker in the loop | tests: 1039, fail: 0, err: 0, skip: 16, time: 5872.1 2011-10-18_01_45_52 commit: Checking exitWorker in the loop | tests: 1039, fail: 1, err: 1, skip: 16, time: 5797.7, failed: CoprocessorEndpoint, MasterObserver 2011-10-18_04_05_43 commit: Checking exitWorker in the loop | tests: 1039, fail: 0, err: 0, skip: 16, time: 5681.3 2011-10-18_06_23_25 commit: Checking exitWorker in the loop | tests: 1039, fail: 1, err: 0, skip: 16, time: 5958.7, failed: ServerCustomProtocol > Split log worker should terminate properly when waiting for znode > - > > Key: HBASE-4607 > URL: https://issues.apache.org/jira/browse/HBASE-4607 > Project: HBase > Issue Type: Bug >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Minor > Attachments: > HBASE-4607_SplitLogWorker_should_correct-20111017231456-47a82ef3.patch > > > This is an attempt to fix the fact that SplitLogWorker threads are not being > terminated properly in some unit tests. This probably does not happen in > production because the master always creates the log-splitting ZK node, but > it does happen in 89-fb. Thanks to Prakash Khemani for help on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners
[ https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121497#comment-13121497 ] Mikhail Bautin commented on HBASE-4465: --- Thanks, Jonathan! I just chatted with Nicolas and he said we should not worry about the 89 branch yet because he will be syncing our internal changes into the public 89 branch. > Lazy-seek optimization for StoreFile scanners > - > > Key: HBASE-4465 > URL: https://issues.apache.org/jira/browse/HBASE-4465 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Labels: optimization, seek > Fix For: 0.89.20100924, 0.94.0 > > Attachments: > HBASE-4465_Lazy-seek_optimization_for_St-20111005121052-b2ea8753.patch > > > Previously, if we had several StoreFiles for a column family in a region, we > would seek in each of them and only then merge the results, even though the > row/column we are looking for might only be in the most recent (and the > smallest) file. Now we prioritize our reads from those files so that we check > the most recent file first. This is done by doing a "lazy seek" which > pretends that the next value in the StoreFile is (seekRow, seekColumn, > lastTimestampInStoreFile), which is earlier in the KV order than anything > that might actually occur in the file. So if we don't find the result in > earlier files, that fake KV will bubble up to the top of the KV heap and a > real seek will be done. This is expected to significantly reduce the amount > of disk IO (as of 09/22/2011 we are doing dark launch testing and > measurement). > This is joint work with Liyin Tang -- huge thanks to him for many helpful > discussions on this and the idea of putting fake KVs with the highest > timestamp of the StoreFile in the scanner priority queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4465) Lazy-seek optimization for StoreFile scanners
[ https://issues.apache.org/jira/browse/HBASE-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121458#comment-13121458 ] Mikhail Bautin commented on HBASE-4465: --- All unit tests passed, ready to be committed. > Lazy-seek optimization for StoreFile scanners > - > > Key: HBASE-4465 > URL: https://issues.apache.org/jira/browse/HBASE-4465 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > Labels: optimization, seek > Fix For: 0.89.20100924, 0.94.0 > > > Previously, if we had several StoreFiles for a column family in a region, we > would seek in each of them and only then merge the results, even though the > row/column we are looking for might only be in the most recent (and the > smallest) file. Now we prioritize our reads from those files so that we check > the most recent file first. This is done by doing a "lazy seek" which > pretends that the next value in the StoreFile is (seekRow, seekColumn, > lastTimestampInStoreFile), which is earlier in the KV order than anything > that might actually occur in the file. So if we don't find the result in > earlier files, that fake KV will bubble up to the top of the KV heap and a > real seek will be done. This is expected to significantly reduce the amount > of disk IO (as of 09/22/2011 we are doing dark launch testing and > measurement). > This is joint work with Liyin Tang -- huge thanks to him for many helpful > discussions on this and the idea of putting fake KVs with the highest > timestamp of the StoreFile in the scanner priority queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120522#comment-13120522 ] Mikhail Bautin commented on HBASE-4522: --- OK, discarding this JIRA. Implemented as an internal patch instead. > Make hbase-site-custom.xml override the hbase-site.xml > -- > > Key: HBASE-4522 > URL: https://issues.apache.org/jira/browse/HBASE-4522 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Liyin Tang >Priority: Minor > Fix For: 0.94.0 > > > The motivation for diff is that we want to override some config change for > any specific cluster easily by just adding the config entries in the > hbase-site-custom.xml for that cluster. This change adds the > hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general
[ https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120392#comment-13120392 ] Mikhail Bautin commented on HBASE-4534: --- This is ready to be committed (all unit tests pass). The diff is in reviewboard (https://reviews.apache.org/r/2155/diff/raw/). > A new unit test for lazy seek and StoreScanner in general > - > > Key: HBASE-4534 > URL: https://issues.apache.org/jira/browse/HBASE-4534 > Project: HBase > Issue Type: Test >Affects Versions: 0.94.0 >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > > A randomized unit test for Gets/Scans (all-row, single-row, multi-row, > all-column, single-column, and multi-column). Also all combinations of Bloom > filters and compression (NONE vs GZIP) are tested. The unit test flushes > multiple StoreFiles with disjoint timestamp ranges and runs various types of > queries against them. Currently we are not testing overlapping timestamp > ranges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general
[ https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119853#comment-13119853 ] Mikhail Bautin commented on HBASE-4534: --- Regarding Lars's comment on AtomicLong performance: https://issues.apache.org/jira/browse/LUCENE-3408 is a similar issue in Lucene. We need to dig more to find out what their solution is, i.e. exactly how Lucene's Counter is better than AtomicLong. > A new unit test for lazy seek and StoreScanner in general > - > > Key: HBASE-4534 > URL: https://issues.apache.org/jira/browse/HBASE-4534 > Project: HBase > Issue Type: Test >Affects Versions: 0.94.0 >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > > A randomized unit test for Gets/Scans (all-row, single-row, multi-row, > all-column, single-column, and multi-column). Also all combinations of Bloom > filters and compression (NONE vs GZIP) are tested. The unit test flushes > multiple StoreFiles with disjoint timestamp ranges and runs various types of > queries against them. Currently we are not testing overlapping timestamp > ranges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
