Uri Laserson created HADOOP-9591: ------------------------------------ Summary: Hadoop SnappyCodec (incorrectly?) uses block compression Key: HADOOP-9591 URL: https://issues.apache.org/jira/browse/HADOOP-9591 Project: Hadoop Common Issue Type: Bug Components: io Reporter: Uri Laserson
org.apache.hadoop.io.compress.SnappyCodec ends up calling BlockCompressorStream and BlockDecompressorStream. However, this really takes in the SequenceFile abstraction unnecessarily, as it wraps chunks of Snappy-compressed data with extra metainformation. It's also not documented that this Snappy coded uses block compression. This has turned out to be a problem with Parquet-MR, since it uses this SnappyCodec to compress its pages, but rather than just Snappy-compressed data, it's Snappy-compressed wrapped in extra data. So tools that just expect regular Snappy compression break (e.g., Impala). There should be another codec added that uses CompressorStream instead. Here is a rough outline, though I'm not sure how to correctly test for whether the Snappy libs are present wrt backwards compatibility. https://github.com/laserson/parquet-mr/blob/raw_snappy/parquet-hadoop/src/main/java/parquet/hadoop/util/RawSnappyCodec.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira