[ https://issues.apache.org/jira/browse/AVRO-135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Philip Zeyliger updated AVRO-135: --------------------------------- Attachment: AVRO-135.patch.txt Attaching a patch that attempts gzip compression. The patch itself introduces an enum (CompressionCodec), and introduces switch statements in DataFileWriter and DataFileStream. Compression involves writing the original data into a buffer, then compressing that in one go into another buffer, and finally writing the length of that second buffer followed by its contents into the output stream. For decompression, there's a filter that restricts the inputstream to a certain length. I've modified TestDataFile to be parameterized across all possible values of codec, and added a test that's more simple, that I used to hammer out the early bugs. I've added a setCompressionCodec() API to DataFileWriter. In theory, for inflate/deflate and for gzip, we don't actually need to store the length. (See http://www.gzip.org/format.txt .) It's irritating to do so with the GzipInputStream API, but we could fix it so that the bytes that weren't used in decompression are returned back to the stream. (The Inflater API has a getRemaining() API that helps do just that, and, in fact, it's used by GzipInputStream in readTrailer().) Do folks have an opinion on whether we should store the length or not? If we don't store it, it's possible we could avoid a memory copy or two by writing straight into the output buffer, but it'll complicate the read path a little bit. Do folks have an opinion on gzip vs deflate? Gzip costs 10 bytes header, 4 bytes CRC-32, and 4 bytes uncompressed size. > add compression to data files > ----------------------------- > > Key: AVRO-135 > URL: https://issues.apache.org/jira/browse/AVRO-135 > Project: Avro > Issue Type: New Feature > Components: java, spec > Reporter: Doug Cutting > Assignee: Philip Zeyliger > Priority: Blocker > Fix For: 1.3.0 > > Attachments: AVRO-135.patch.txt > > > We should add support for at least one compression codec to data files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.