Harish,Thank you very much for your valuable/assertive suggestion :-)I was able
to identify the problem and fix it.Else where in the code, we were setting a
different mapred-site.xml in the configuration.I still do not know why it is
using the DefaultCodec for compression (instead of the one I set -
SnappyCodec), but I am hopeful I will get there. Thanks again. Regards, - kiru
From: Harsh J <ha...@cloudera.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Sent: Tuesday, October 27, 2015 8:34 AM
Subject: Re: lzo error while running mr job
The stack trace is pretty certain you do, as it clearly tries to load a class
not belonging within Apache Hadoop. Try looking at the XML files the
application uses? Perhaps you've missed all the spots.
If I had to guess, given the JobSubmitter entry in the trace, it'd be in the
submitting host's /etc/hadoop/conf/* files, or in the dir pointed by
$HADOOP_CONF_DIR (if thats specifically set). Alternatively, it'd be in the
code.
If you have control over the code, you can also make it dump the XML before
submit via: job.getConfiguration().writeXml(System.out);. The XML dump will
carry the source of all properties along with their value.
On Tue, Oct 27, 2015 at 8:52 PM Kiru Pakkirisamy <kirupakkiris...@yahoo.com>
wrote:
| Harish,We don't have lzo in the io.compression.codecs list.That is what is
puzzling me.Regards, Kiru
| From:"Harsh J" <ha...@cloudera.com>
Date:Mon, Oct 26, 2015 at 11:39 PM
Subject:Re: lzo error while running mr job
|
|
|
| Every codec in the io.compression.codecs list of classes will be
initialised, regardless of actual further use. Since the Lzo*Codec classes
require the native library to initialise, the failure is therefore expected.
On Tue, Oct 27, 2015 at 11:42 AM Kiru Pakkirisamy <kirupakkiris...@yahoo.com>
wrote:
I am seeing a weird error after we moved to the new hadoop mapreduce java
packages in 2.4We are not using lzo (as in io.compression.codecs) but we still
get this error. Does it mean we have to have lzo installed even though we are
not using ? Thanks.
Regards,- kiru
2015-10-27 00:18:57,994 ERROR com.hadoop.compression.lzo.GPLNativeCodeLoader |
Could not load native gpl libraryjava.lang.UnsatisfiedLinkError: no
gplcompression in java.library.path at
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886) ~[?:1.7.0_85] at
java.lang.Runtime.loadLibrary0(Runtime.java:849) ~[?:1.7.0_85] at
java.lang.System.loadLibrary(System.java:1088) ~[?:1.7.0_85] at
com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31)
[flow-trunk.242-470787.jar:?] at
com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:60)
[flow-trunk.242-470787.jar:?] at java.lang.Class.forName0(Native Method)
[?:1.7.0_85] at java.lang.Class.forName(Class.java:278) [?:1.7.0_85] at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1834)
[flow-trunk.242-470787.jar:?] at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1799)
[flow-trunk.242-470787.jar:?] at
org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
[flow-trunk.242-470787.jar:?] at
org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
[flow-trunk.242-470787.jar:?] at
org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.isSplitable(CombineFileInputFormat.java:159)
[flow-trunk.242-470787.jar:?] at
org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:283)
[flow-trunk.242-470787.jar:?] at
org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:243)
[flow-trunk.242-470787.jar:?] at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
[flow-trunk.242-470787.jar:?]
Regards, - kiru
|
|
