[
https://issues.apache.org/jira/browse/PIG-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Reed updated PIG-42:
-----------------------------
Attachment: gzip.patch
The attached patch implements the method of splitting GZipped files as outlined
in the issue description. It uses the same hooks as BZip. We need to review to
make sure it terminates properly.
If the gzipped file is not setup for splits, we fall back to not splitting the
file.
An unsplittable gzipped dataset can be converted to a splittable one with the
following Pig Latin:
a = load 'orig.gz';
store a into 'splittable.gz';
> Pig should be able to split Gzip files like it can split Bzip files
> -------------------------------------------------------------------
>
> Key: PIG-42
> URL: https://issues.apache.org/jira/browse/PIG-42
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Benjamin Reed
> Attachments: gzip.patch
>
>
> It would be nice to be able to split gzip files like we can split bzip files.
> Unfortunately, we don't have a sync point for the split in the gzip format.
> Gzip file format supports the notion of concatenate gzipped files. When
> gzipped files are concatenated together they are treated as a single file. So
> to make a gzipped file splittable we can used an empty compressed file with
> some salt in the headers as a sync signature. Then we can make the gzip file
> splittable by using this sync signature between compressed segments of the
> file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
