Hi,
I am new to hadoop and its not very familiar with internal working. I had
some questions about HADOOP_CLASSPATH.
We are currently suppose to use a Hadoop cluster with 4 machines and its
HADOOP_CLASSPATH in hadoop-env.sh is as below.
export
HADOOP_CLASSPATH="/home/user/app/www/WEB-INF/classes:$HADOOP_CLASSPATH"
Now my,
/home/user/app/www/WEB-INF/classes has a class called Application.class
>From a remote machine I submit a map-reduce job to this cluster, with a jar
called MyJar.jar. [This has a Application.class too, but with some
modifications]
When the TaskTracker spawns a child Java process for the Mapper the
classpath I see is as below in that order,
Lets say my hadoop is installed at: /home/user/hadoop/
/home/user/hadoop/jar1,
/home/user/hadoop/jar2,
.
.
.
/home/user/hadoop/jarN,
/home/user/hadoop/lib/jar1,
/home/user/hadoop/lib/jar2,
/home/user/hadoop/lib/jarN,
1./home/user/app/www/WEB-INF/classes,
2/${mapred.local.dir}/taskTracker/{user}/jobcache/{jobid}/jars/Myjar.jar
[note:- basically this has the modified class that I need to use for my
Map-Reduce job]
Well its clear from this classpath that i will end up using the
Application.class from the classes folder. with gives me incorrect results.
Now my Question is, how do I make sure i reverse the order of 1 & 2.
Some pointer that I found was,
1) if MyJar.jar is not changing much then I can put in a shared location and
modify my hadoop-env.sh to
export
HADOOP_CLASSPATH="/some/share/location/lib:/home/user/app/www/WEB-INF/classes:$HADOOP_CLASSPATH"
2) get rid of /home/user/app/www/WEB-INF/classes, from my hadoop-env.sh
3) is there any property taht suggest to add before classpath ?
Any help is greatly appreciated.
To Summarize,
If I have HADOOP_CLASSPTH in hadoop-env.sh already set, then how do I add
application jar before this classpath.
Again. I saw the DistributedCache.java [hadoop src] and the code looks like.
public static void addFileToClassPath(Path file, Configuration conf)
throws IOException {
String classpath = conf.get("mapred.job.classpath.files");
conf.set("mapred.job.classpath.files", classpath == null ? file
.toString() : classpath + System.getProperty("path.separator")
+ file.toString());
.............
}
basically new files are added to the end of existing classpath.
Thanks,
Dharin.
--
View this message in context:
http://old.nabble.com/Hadoopp_ClassPath-issue.-tp33666009p33666009.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
