Dharin, I believe the properties you are looking for are the following: HADOOP_USER_CLASSPATH_FIRST: When defined, this will let the user suggested classpath to the beginning of global classpath. So, you would have to do something like 'export HADOOP_USER_CLASSPATH_FIRST=true'. If you are on 2.0 (or 0.23), please refer bin/hadoop-config.sh for more information. If you are on 1.0 (or 0.20), refer to hadoop script.
Now, if you want to run an M/R job by passing your own jar and you want that jar to be used first, you want to set the config parameter 'mapreduce.job.user.classpath.first' and then the user provided jar will be put in before $HADOOP_CLASSPATH. Hope this makes sense. Also, these will work on 1.0 (or 0.23) above. Refer: https://issues.apache.org/jira/browse/MAPREDUCE-3696 (for 2.0, 0.23) https://issues.apache.org/jira/browse/MAPREDUCE-1938 (1.0, 0.20) Thanks, John George -----Original Message----- From: dmaniar <dharin.man...@gmail.com> Reply-To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org> Date: Tue, 10 Apr 2012 21:09:10 -0700 To: "core-u...@hadoop.apache.org" <core-u...@hadoop.apache.org> Subject: Hadoopp_ClassPath issue. > >Hi, > >I am new to hadoop and its not very familiar with internal working. I had >some questions about HADOOP_CLASSPATH. > >We are currently suppose to use a Hadoop cluster with 4 machines and its >HADOOP_CLASSPATH in hadoop-env.sh is as below. >export >HADOOP_CLASSPATH="/home/user/app/www/WEB-INF/classes:$HADOOP_CLASSPATH" > >Now my, >/home/user/app/www/WEB-INF/classes has a class called Application.class > >From a remote machine I submit a map-reduce job to this cluster, with a >jar >called MyJar.jar. [This has a Application.class too, but with some >modifications] > >When the TaskTracker spawns a child Java process for the Mapper the >classpath I see is as below in that order, > >Lets say my hadoop is installed at: /home/user/hadoop/ >/home/user/hadoop/jar1, >/home/user/hadoop/jar2, >. >. >. >/home/user/hadoop/jarN, >/home/user/hadoop/lib/jar1, >/home/user/hadoop/lib/jar2, >/home/user/hadoop/lib/jarN, >1./home/user/app/www/WEB-INF/classes, >2/${mapred.local.dir}/taskTracker/{user}/jobcache/{jobid}/jars/Myjar.jar >[note:- basically this has the modified class that I need to use for my >Map-Reduce job] > >Well its clear from this classpath that i will end up using the >Application.class from the classes folder. with gives me incorrect >results. > >Now my Question is, how do I make sure i reverse the order of 1 & 2. > >Some pointer that I found was, >1) if MyJar.jar is not changing much then I can put in a shared location >and >modify my hadoop-env.sh to >export >HADOOP_CLASSPATH="/some/share/location/lib:/home/user/app/www/WEB-INF/clas >ses:$HADOOP_CLASSPATH" > >2) get rid of /home/user/app/www/WEB-INF/classes, from my hadoop-env.sh > >3) is there any property taht suggest to add before classpath ? > >Any help is greatly appreciated. > >To Summarize, >If I have HADOOP_CLASSPTH in hadoop-env.sh already set, then how do I add >application jar before this classpath. > >Again. I saw the DistributedCache.java [hadoop src] and the code looks >like. > >public static void addFileToClassPath(Path file, Configuration conf) > throws IOException { > String classpath = conf.get("mapred.job.classpath.files"); > conf.set("mapred.job.classpath.files", classpath == null ? file > .toString() : classpath + System.getProperty("path.separator") > + file.toString()); > ............. >} > >basically new files are added to the end of existing classpath. > > >Thanks, >Dharin. > > > > >-- >View this message in context: >http://old.nabble.com/Hadoopp_ClassPath-issue.-tp33666009p33666009.html >Sent from the Hadoop core-user mailing list archive at Nabble.com. >
