[ https://issues.apache.org/jira/browse/IMPALA-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790095#comment-16790095 ]
ASF subversion and git services commented on IMPALA-6658: --------------------------------------------------------- Commit 729fcf34764928bebfe7cf31153b57f7246404af in impala's branch refs/heads/master from Andrew Sherman [ https://gitbox.apache.org/repos/asf?p=impala.git;h=729fcf3 ] IMPALA-8279: Revert IMPALA-6658 to avoid ETL performance regression. IMPALA-6658 changed RleEncoder to have the ability to use run lengths other than 8. It seemed that a slightly more complex RleEncoder could save a small amount of disk space by using the longer run lengths, in particular for bit width of 1. We now see a performance regression on a simple ETL query. Overall it seems that the costs of IMPALA-6658 exceed the benefits. This change removes IMPALA-6658. The strategy for this was that the change to rle-encoding.h, which contains the code, was undone using 'git revert'. I removed the test changes in rle-test.cc that rely on different encoding lengths. This allows us to keep some useful new tests that were written as part of IMPALA-6658 TESTING: Ran all end-to-end tests. Change-Id: If6bcbaf564fbbe6dc83ba3afc100b4e5ccc7af40 Reviewed-on: http://gerrit.cloudera.org:8080/12680 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > Parquet RLE encoding can waste space with small repeated runs > ------------------------------------------------------------- > > Key: IMPALA-6658 > URL: https://issues.apache.org/jira/browse/IMPALA-6658 > Project: IMPALA > Issue Type: Improvement > Components: Backend > Reporter: Csaba Ringhofer > Assignee: Andrew Sherman > Priority: Minor > Labels: parquet, ramp-up > Fix For: Impala 3.1.0 > > > Currently RleEncoder creates repeated runs from 8 repeated values, which can > be less space efficient than bit-packed if bit width is 1 or 2. In the worst > case, the whole data page can be ~2X larger if bit width is 1, and ~1.25X > larger if bit is 2 compared to bit-packing. > A comment in rle_encoding.h writes different numbers, but it probably does > not calculate with the overhead of splitting long runs into smaller ones > (every run adds +1 byte for its length): > [https://github.com/apache/impala/blob/8079cd9d2a87051f81a41910b74fab15e35f36ea/be/src/util/rle-encoding.h#L62] > Note that if the data page is compressed, this size difference probably > disappears, but the larger uncompressed buffer size can still affect > performance. > Parquet RLE encoding is described here: > [https://github.com/apache/parquet-format/blob/master/Encodings.md#run-length-encoding-bit-packing-hybrid-rle-3] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org