Tez by default sets
"LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native/" in the
task env variables. So if $HADOOP_COMMON_HOME is correctly set on the
NodeManager machine then LD_LIBRARY_PATH should be correctly loaded without
you needing to do anything special.
Here is how to debug this.
Set the following in your yarn-site.xml for all node-manager (or 1 of them
in case you know that a task will run on that machine). This sets the
debug-delay to 60000 seconds but you could use a value that works for you.
This makes the NodeManager retain container specific launch files for the
specified time interval after the container exits.
<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>60000</value>
</property>
Now run your job and some tasks/containers will fail. Go to the node manager
for a failed task/container and locate directory specified by config
"yarn.nodemanager.local-dirs" in yarn-site.xml. Under that directory there
will be an nmPrivate directory.
Open the launch_container.sh file for that container. On my local cluster it
is the following
/tmp/hadoop-/nm-local-dir/nmPrivate/application_1452203679463_0001/container
_1452203679463_0001_01_000002/launch_container.sh
Inside that file your environment variable specifications. In my case
LD_LIBRARY_PATH is set as follows in the above file
export LD_LIBRARY_PATH="$PWD:$PWD:/deploy/deploy-trunk/hadoop/lib/native/"
You can check the value being set in your launch_container.sh for the
library path and try to debug from there about what went wrong. If
everything is setup correctly then you should find it set to the location of
the snappy native libs.
Bikas
From: Mich Talebzadeh [mailto:[email protected]]
Sent: Thursday, January 7, 2016 1:02 AM
To: [email protected]
Subject: RE: Hive on TEZ fails on buildSupportsSnappy()Z
Yes I have now Jeff.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native
hduser@rhes564::/home/hduser/hadoop-2.6.0/lib/native
<mailto:hduser@rhes564::/home/hduser/hadoop-2.6.0/lib/native> > ls -l
libsnappy.so
lrwxrwxrwx 1 root root 23 Jan 6 17:00 libsnappy.so ->
/usr/lib64/libsnappy.so
But it still fails as below
0: jdbc:hive2://rhes564:10010/default> set hive.execution.engine=tez;
No rows affected (0.003 seconds)
0: jdbc:hive2://rhes564:10010/default> select count(1) from sales;
INFO : Tez session hasn't been created yet. Opening session
INFO :
INFO : Status: Running (Executing on YARN cluster with App id
application_1452119781900_0002)
INFO : Map 1: -/- Reducer 2: 0/1
INFO : Map 1: 0/1 Reducer 2: 0/1
INFO : Map 1: 0(+1)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-1)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-1)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-2)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-2)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-3)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-3)/1 Reducer 2: 0/1
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1,
vertexId=vertex_1452119781900_0002_1_00, diagnostics=[Task failed,
taskId=task_1452119781900_0002_1_00_000000, diagnostics=[TaskAttempt 0
failed, info=[Error: Failure while running task:java.lang.RuntimeException:
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
Dr Mich Talebzadeh
LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUr
V8Pw
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.
pdf
Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN:
978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
one out shortly
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.
From: Jianfeng (Jeff) Zhang [mailto:[email protected]]
Sent: 07 January 2016 08:31
To: [email protected] <mailto:[email protected]>
Subject: Re: Hive on TEZ fails on buildSupportsSnappy()Z
Do you have snappy installed and put it on LD_LIBRARY_PATH ?
Best Regard,
Jeff Zhang
From: Mich Talebzadeh <[email protected] <mailto:[email protected]> >
Reply-To: "[email protected] <mailto:[email protected]> "
<[email protected] <mailto:[email protected]> >
Date: Tuesday, January 5, 2016 at 8:30 AM
To: "[email protected] <mailto:[email protected]> " <[email protected]
<mailto:[email protected]> >
Subject: Hive on TEZ fails on buildSupportsSnappy()Z
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
