yes. JAVA_LIBRARY_PATH seems to be the approach that works (rather
than just putting it into tasktracker_opts etc.)
Thanks.
On Wed, Mar 28, 2012 at 9:57 PM, Harsh J wrote:
> George,
>
> This ought to work. Did you restart all your TTs to have it set into effect?
>
> Also, the right way to do thi
George,
This ought to work. Did you restart all your TTs to have it set into effect?
Also, the right way to do this across Hadoop (in 1.0/cdh3/whatever) is
to add into your hadoop-env.sh:
JAVA_LIBRARY_PATH=/path/to/your/libs:$JAVA_LIBRARY_PATH
This way you do not stand to lose Hadoop's native l
Dmitriy
I've tested it on hadoop 1.0.0 and 1.0.1. (I don't know which version
cdh3u3 is based off of)
In hadoop-env.sh if I set
HADOOP_TASKTRACKER_OPTS="-Djava.library.path=/usr/blah" the TaskTracker
sees that option. Then it gets passed along to all M/R child tasks on
that node. Can you
Hm. doesn't seem to work for me (with cdh3u3)
I defined
export HADOOP_TASKTRACKER_OPTS="-Djava.library.path=/usr/"
and it doesn't seem to work (as opposed to when i set is with
property mapred.child.java.opts on the data node).
Still puzzling.
On Tue, Mar 27, 2012 at 7:17 PM, George Datsko
Dmitriy,
You can set for map or reduce tasks.
Please refer this link:
http://hadoop.apache.org/common/docs/r1.0.1/mapred_tutorial.html#Task+Execution+%26+Environment
mapred.map.child.java.opts
-Xmx512M -Djava.library.path=/home/mycompany/lib -verbose:gc
-Xloggc:/tmp/@taskid@.gc
Thank you, George. I assume you are referring to setenv.sh on the data
nodes to set library paths for task tracker, right?
On Mar 27, 2012 7:19 PM, "George Datskos"
wrote:
> Dmitriy,
>
> I just double-checked, and the caveat I stated earlier is incorrect. So,
> "-Djava.library.path" set in the
Dmitriy,
I just double-checked, and the caveat I stated earlier is incorrect.
So, "-Djava.library.path" set in the client's {mapred.child.java.opts}
should just append to to the "-Djava.library.path" that each TaskTracker
has when creating the library path for each child (M/R) task. So that
Dmitriy,
To deal with different servers having various shared libraries in
different locations, you can simply make sure the _TaskTracker_'s
-Djava.library.path is set correctly on each server. That library path
should be passed along to each child (M/R) task. (in *addition* to the
{mapred.
Hello,
I have a couple of questions regarding mapreduce configurations.
We install various platforms on data nodes that require mixed set of
native libraries.
Part of the problem is that in general case, this software platforms
may be installed into different locations in the backend. (we try to
