On 11/17/14, 4:54 AM, Tsuyoshi OZAWA wrote:
I'm facing with same problem in Apache version. Can you run jobs with
snappy compression with 0.5.2 release?
There is a thread going on in the Apache mailing list about stopping the
shipping of the .so files as part of the binary distribution.
Are you by any chance not building hadoop without -Pnative?
mvn clean package -Pdist -Dtar -Dmaven.javadoc.skip=true -DskipTests
-fail-at-end -Pnative
is how I built it for my perf cluster.
Cheers,
Gopal
On Mon, Nov 17, 2014 at 9:26 PM, Hitesh Shah <hit...@apache.org> wrote:
Hi Subroto,
It could be an installer/distro issue. You may need to redirect your question
to the
Hortonworks forums to get an answer on this.
But yes, you are right - the env needs to be setup correctly to load the
required native
libs. Out of curiosity, are you using Ambari to setup your cluster?
thanks
— Hitesh
On Nov 17, 2014, at 11:07 AM, Subroto Sanyal <sanyalsubr...@gmail.com> wrote:
Hi,
With HDP-2.1.2 we are encountering the problem of:
2014-09-11 10:14:42,881 ERROR [TezChild]
org.apache.tez.runtime.task.TezTaskRunner:
Exception of type Error. Exiting now
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1198)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091)
at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274)
To workaround the problem we have to set the properties explicitly:
tez.am.launch.env
tez.task.launch.env
Is this a problem with HDP installation?
--
Cheers,
Subroto Sanyal
--
- Tsuyoshi
