have you hadded that before SET? register file:/ibm-compression.jar
regards Sébastien Cavecchi 6scaling.com - E: [email protected] T: 06.52.32.51.66 Skype: scavecchi - 2014-11-19 5:10 GMT+01:00 lulynn_2008 <[email protected]>: > Yes, seems pig.tmpfilecompression.codec does not support CMX now, > following is my testing: > SET pig.tmpfilecompression true; > SET pig.tmpfilecompression.codec cmx; > Error output: > Caused by: java.io.IOException: Invalid temporary file compression codec > [cmx]. Expected compression codecs for org.apache.pig.impl.io.TFileStorage > are GZ,GZIP,LZO. > > Is there anybody has any suggestion on this? Thanks > > > > > > > > > > > At 2014-11-18 18:47:18, "Krishna Kalyan" <[email protected]> wrote: > >Oh your question is regarding compression codec for temporary files?. > >I am not sure about that. > > > > > > > >On Tue, Nov 18, 2014 at 4:13 PM, Krishna Kalyan <[email protected] > > > >wrote: > > > >> Hi lulynn, > >> If you are using biginsights follow the steps to enable compression. > >> > >> set output.compression.enabled true; > >> set output.compression.codec org.apache.hadoop.io.compress.GzipCodec; > >> data = LOAD > >> '/bigdata/sample_data/NOAA_Weather_csv/2011/999999-53019-2011.csv.gz' > >> using PigStorage(',') as (projectname:chararray); > >> STORE data INTO '/comCodecGzip'; > >> > >> Cheers, > >> Krishna > >> > >> On Tue, Nov 18, 2014 at 2:04 PM, lulynn_2008 <[email protected]> > wrote: > >> > >>> BTW, cmx is "com.ibm.biginsights.compress.CmxCodec", the related jar is > >>> ibm-compression.jar. > >>> > >>> > >>> > >>> At 2014-11-18 15:49:53, "lulynn_2008" <[email protected]> wrote: > >>> > >>> Hi All, > >>> I am trying to use CMX as temp file compression > >>> > >>> codec, i.e > >>> SET pig.tmpfilecompression true; > >>> SET pig.tmpfilecompression.codec cmx; > >>> > >>> but following errors happened: > >>> Caused by: java.io.IOException: Invalid temporary file compression > codec > >>> []. Expected compression codecs are gz and lzo > >>> > >>> from pig cookbook I found following line > >>> > >>> "pig.tmpfilecompression.codec - Specifies which compression codec to > use. > >>> Currently, Pig accepts "gz" and "lzo" as possible values. However, > because > >>> LZO is under GPL license (and disabled by default) you will need to > >>> configure your cluster to use the LZO codec to take advantage of this > >>> feature. For details, see > >>> http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ. " > >>> > >>> > >>> Is there any workaround? or there are roadmaps for adding cmx as a > >>> > >>> supported codec ? I was using pig 0.12.0. > >>> > >>> Thanks > >>> > >>> > >>> > >>> > >> >
