[
https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229453#comment-13229453
]
[email protected] commented on HBASE-4608:
------------------------------------------------------
bq. On 2012-03-14 17:42:21, Lars Hofhansl wrote:
bq. >
src/main/java/org/apache/hadoop/hbase/regionserver/wal/LRUDictionary.java, line
32
bq. > <https://reviews.apache.org/r/4328/diff/2/?file=92107#file92107line32>
bq. >
bq. > I think I had that question to Li Pi... How much memory do we expect
this dictionary to take worst case?
bq. > I guess since there is one WAL per region server and it is rolled
periodically it is not a problem at all.
bq.
bq. Li Pi wrote:
bq. 65536 * 5 ( Regionname, Row key, CF, Column qual, table) * 100 bytes
(these are some big names) = 32768000 bytes. Or 32 megabytes.
bq.
bq. If you want to get silly, even at 1kb entries (wtf are you naming
things?), it maxes out at 320 megabytes.
Actually halve those amounts, 2^15, not 2^16.
- Li
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4328/#review5951
-----------------------------------------------------------
On 2012-03-14 07:34:58, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4328/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-14 07:34:58)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. See issue
bq.
bq.
bq. This addresses bug hbase-4608.
bq. https://issues.apache.org/jira/browse/hbase-4608
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/HConstants.java 045c6f3
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/wal/Compressor.java
PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/wal/Dictionary.java
PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java b5049b1
bq. src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
311ea1b
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/wal/KeyValueCompression.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/wal/LRUDictionary.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
ff63a5f
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java
01ebb5c
bq. src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
d8f317c
bq. src/main/java/org/apache/hadoop/hbase/util/Bytes.java de8e40b
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestCompressor.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestKeyValueCompression.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLRUDictionary.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
a11899c
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplayCompressed.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4328/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> HLog Compression
> ----------------
>
> Key: HBASE-4608
> URL: https://issues.apache.org/jira/browse/HBASE-4608
> Project: HBase
> Issue Type: New Feature
> Reporter: Li Pi
> Assignee: stack
> Fix For: 0.94.0
>
> Attachments: 4608-v19.txt, 4608-v20.txt, 4608-v22.txt, 4608v1.txt,
> 4608v13.txt, 4608v13.txt, 4608v14.txt, 4608v15.txt, 4608v16.txt, 4608v17.txt,
> 4608v18.txt, 4608v23.txt, 4608v24.txt, 4608v25.txt, 4608v27.txt, 4608v5.txt,
> 4608v6.txt, 4608v7.txt, 4608v8fixed.txt, hbase-4608-v28-delta.txt,
> hbase-4608-v28.txt, hbase-4608-v28.txt
>
>
> The current bottleneck to HBase write speed is replicating the WAL appends
> across different datanodes. We can speed up this process by compressing the
> HLog. Current plan involves using a dictionary to compress table name, region
> id, cf name, and possibly other bits of repeated data. Also, HLog format may
> be changed in other ways to produce a smaller HLog.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
