[
https://issues.apache.org/jira/browse/PIG-3017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487186#comment-13487186
]
Jonathan Coveney commented on PIG-3017:
---------------------------------------
Well, I don't know the absolute size because I had a script where the JobConf
was failing out at about 6.5MB...I'm not sure if it fails as soon as it crosses
the thresh-hold, or if it fails after serializing everything. That said, after
this patch, the same JobConf was 600KB, so about 10x (note that I also changed
it to use Base64 encoding). Also, as far as serialization time, it's still in
the realm of ~5MB, so compression time is negligible. I did not do extensive
testing around the specifics, though.
> Pig's object serialization should use compression
> -------------------------------------------------
>
> Key: PIG-3017
> URL: https://issues.apache.org/jira/browse/PIG-3017
> Project: Pig
> Issue Type: Bug
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Fix For: 0.12
>
> Attachments: PIG-3017-0.patch
>
>
> We have run into cases of very large JobConf objects, and part of this is the
> fact that serialized objects are quite large. There is no reason not to use
> compression here, and ratios should be quite high.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
