[
https://issues.apache.org/jira/browse/PIG-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776769#comment-13776769
]
Rohini Palaniswamy commented on PIG-3480:
-----------------------------------------
We do have complex types like bag of maps and bag of bags and one or two levels
of nesting. But I assume you have way more nesting than we do. Does that matter
though as what is written to TFile is just bytes for both key and value?
We use lzo. It would be good to try gz and see if the problem is with lzo for
you.
2013-09-24 21:10:21,289 INFO [main]
com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2013-09-24 21:10:21,291 INFO [main] com.hadoop.compression.lzo.LzoCodec:
Successfully loaded & initialized native-lzo library
2013-09-24 21:10:21,293 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor [.lzo_deflate]
I don't think hadoop version should matter as we had hadoop 1.x till mid 2012.
> TFile-based tmpfile compression crashes in some cases
> -----------------------------------------------------
>
> Key: PIG-3480
> URL: https://issues.apache.org/jira/browse/PIG-3480
> Project: Pig
> Issue Type: Bug
> Reporter: Dmitriy V. Ryaboy
> Fix For: 0.12
>
> Attachments: PIG-3480.patch
>
>
> When pig tmpfile compression is on, some jobs fail inside core hadoop
> internals.
> Suspect TFile is the problem, because an experiment in replacing TFile with
> SequenceFile succeeded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
