[
https://issues.apache.org/jira/browse/PARQUET-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035310#comment-17035310
]
ASF GitHub Bot commented on PARQUET-1622:
-----------------------------------------
gszadovszky commented on pull request #705: PARQUET-1622: Add implementation
for BYTE_STREAM_SPLIT
URL: https://github.com/apache/parquet-mr/pull/705
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add BYTE_STREAM_SPLIT encoding
> ------------------------------
>
> Key: PARQUET-1622
> URL: https://issues.apache.org/jira/browse/PARQUET-1622
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-cpp, parquet-format, parquet-mr, parquet-thrift
> Reporter: Martin Radev
> Assignee: Martin Radev
> Priority: Minor
> Labels: features, pull-request-available
> Fix For: format-2.8.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Apache Parquet does not have any encodings suitable for FP data and the
> available text compressors (zstd, gzip, etc) do not handle FP data very well.
> It is possible to apply a simple data transformation named "stream
> splitting". Such could be "byte stream splitting" which creates K streams of
> length N where K is the number of bytes in the data type (4 for floats, 8 for
> doubles) and N is the number of elements in the sequence.
> The transformed data compresses significantly better on average than the
> original data and for some cases there is a performance improvement in
> compression and decompression speed.
> You can read a more detailed report here:
> https://drive.google.com/file/d/1wfLQyO2G5nofYFkS7pVbUW0-oJkQqBvv/view
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
