Uri Laserson created HADOOP-9591:
------------------------------------
Summary: Hadoop SnappyCodec (incorrectly?) uses block compression
Key: HADOOP-9591
URL: https://issues.apache.org/jira/browse/HADOOP-9591
Project: Hadoop Common
Issue Type: Bug
Components: io
Reporter: Uri Laserson
org.apache.hadoop.io.compress.SnappyCodec ends up calling BlockCompressorStream
and BlockDecompressorStream. However, this really takes in the SequenceFile
abstraction unnecessarily, as it wraps chunks of Snappy-compressed data with
extra metainformation. It's also not documented that this Snappy coded uses
block compression.
This has turned out to be a problem with Parquet-MR, since it uses this
SnappyCodec to compress its pages, but rather than just Snappy-compressed data,
it's Snappy-compressed wrapped in extra data. So tools that just expect
regular Snappy compression break (e.g., Impala).
There should be another codec added that uses CompressorStream instead. Here
is a rough outline, though I'm not sure how to correctly test for whether the
Snappy libs are present wrt backwards compatibility.
https://github.com/laserson/parquet-mr/blob/raw_snappy/parquet-hadoop/src/main/java/parquet/hadoop/util/RawSnappyCodec.java
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
