Re: Re: Re: Re: Doubt in Hadoop
You need to send a jar to the cluster so it can run your code there. Hadoop doesn't magically know which jar is the one containing your main class, or that of your mapper/reducer -- so you need to tell it via that call so it knows which jar file to upload. - Aaron On Sun, Nov 29, 2009 at 7:54 AM, wrote: > Hi, > Actually, I just made the change suggested by Aaron and my code worked. > But I > still would like to know why does the setJarbyClass() method have to be > called > when the Main class and the Map and Reduce classes are in the same package > ? > > Thank You > > Abhishek Agrawal > > SUNY- Buffalo > (716-435-7122) > > On Sun 11/29/09 10:39 AM , aa...@buffalo.edu sent: > > Hi, > > I dont set job.setJarByClass(Map.class). But my main class, the map class > > andthe reduce class are all in the same package. Does this make any > difference > > at ordo I still have to call > > > > Thank You > > > > Abhishek Agrawal > > > > SUNY- Buffalo > > (716-435-7122) > > > > On Fri 11/27/09 1:42 PM , Aaron Kimball aa...@clou > > dera.com sent:> When you set up the Job object, do you > > call> job.setJarByClass(Map.class)? That will tell > > Hadoop which jar file to> ship with the job and to use for classloading > in > > your code.> > > > - Aaron > > > On Thu, Nov 26, 2009 at 11:56 PM, > > wrote:> Hi, > > >  I am running the job from command > > line. The job runs fine in the> local mode > > > but something happens when I try to run the job > > in the distributed> mode. > > > Abhishek Agrawal > > > SUNY- Buffalo > > > (716-435-7122) > > > On Fri 11/27/09  2:31 AM , Jeff > > Zhang sent:> > Do you run the map reduce job in command > > line or IDE? in map> reduce > > > > mode, you should put the jar containing the > > map and reduce class> in > > > > your classpath > > > > Jeff Zhang > > > > On Fri, Nov 27, 2009 at 2:19 PM, > >  wrote:> > Hello Everybody, > > > >    > >      I have > > a doubt in Haddop and was wondering> if > > > > anybody has faced a > > > > similar problem. I have a package called > > test. Inside that I have> > class called > > > > A.java, Map.java, Reduce.java. In A.java I > > have the main method> > where I am trying > > > > to initialize the jobConf object. I have > > written> > jobConf.setMapperClass(Map.class) and > > similarly for the reduce> class > > > > as well. The > > > > code works correctly when I run the code > > locally via> > > > jobConf.set("mapred.job.tracker","local") but I get an > > exception> > when I try to > > > > run this code on my cluster. The stack > > trace of the exception is> as > > > > under. I > > > > cannot understand the problem. Any help > > would be appreciated.> > java.lang.RuntimeException: > > java.lang.RuntimeException:> > java.lang.ClassNotFoundException: > > test.Map> >    > >  at> > > > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752)> > >    > >  at> > > > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690)> >  >   > >  at> > > > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)> >   >  > >  at> > > > > > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)> > >    > >  at> > > > > > > > > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) > > > >    > >  at> > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)> >    > >  at > > org.apache.hadoop.mapred.Child.main(Child.java:158)> > Caused by: > > java.lang.RuntimeException:> > > > java.lang.ClassNotFoundException:> > Markowitz.covarMatrixMap > > > >    > >  at> > > > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720)> > >    > >  at> > > > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744)> > >    > >  ... 6 more> > Caused by: > > java.lang.ClassNotFoundException: test.Map> >    > >  at> > > java.net.URLClassLoader$1.run(URLClassLoader.java:200)> >    > >  at > > java.security.AccessController.doPrivileged(Native> > Method) > > > >    > >  at> > > > java.net.URLClassLoader.findClass(URLClassLoader.java:188)> >    > >  at> > > java.lang.ClassLoader.loadClass(ClassLoader.java:306)> >    > >  at> > > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)> >    > >  at> > > java.lang.ClassLoader.loadClass(ClassLoader.java:251)> >    > >  at> > > > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)> >    > >  at java.lang.Class.forName0(Native Method)> >    > >  at java.lang.Class.forName(Class.java:247)> >    > >  at> > > > > > > > > > > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) > > > >    > >  at> > > > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718)> > >    > >  ... 7 more> > Thank You > > > > Abhishek Agrawal > > > > SUNY- Buffalo > > > > (716-435-7122) > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Re: Re: Re: Doubt in Hadoop
Hi, Actually, I just made the change suggested by Aaron and my code worked. But I still would like to know why does the setJarbyClass() method have to be called when the Main class and the Map and Reduce classes are in the same package ? Thank You Abhishek Agrawal SUNY- Buffalo (716-435-7122) On Sun 11/29/09 10:39 AM , aa...@buffalo.edu sent: > Hi, > I dont set job.setJarByClass(Map.class). But my main class, the map class > andthe reduce class are all in the same package. Does this make any difference > at ordo I still have to call > > Thank You > > Abhishek Agrawal > > SUNY- Buffalo > (716-435-7122) > > On Fri 11/27/09 1:42 PM , Aaron Kimball aa...@clou > dera.com sent:> When you set up the Job object, do you > call> job.setJarByClass(Map.class)? That will tell > Hadoop which jar file to> ship with the job and to use for classloading in > your code.> > > - Aaron > > On Thu, Nov 26, 2009 at 11:56 PM, > wrote:> Hi, > >  I am running the job from command > line. The job runs fine in the> local mode > > but something happens when I try to run the job > in the distributed> mode. > > Abhishek Agrawal > > SUNY- Buffalo > > (716-435-7122) > > On Fri 11/27/09  2:31 AM , Jeff > Zhang sent:> > Do you run the map reduce job in command > line or IDE? in map> reduce > > > mode, you should put the jar containing the > map and reduce class> in > > > your classpath > > > Jeff Zhang > > > On Fri, Nov 27, 2009 at 2:19 PM, >  wrote:> > Hello Everybody, > > >    >      I have > a doubt in Haddop and was wondering> if > > > anybody has faced a > > > similar problem. I have a package called > test. Inside that I have> > class called > > > A.java, Map.java, Reduce.java. In A.java I > have the main method> > where I am trying > > > to initialize the jobConf object. I have > written> > jobConf.setMapperClass(Map.class) and > similarly for the reduce> class > > > as well. The > > > code works correctly when I run the code > locally via> > > jobConf.set("mapred.job.tracker","local") but I get an > exception> > when I try to > > > run this code on my cluster. The stack > trace of the exception is> as > > > under. I > > > cannot understand the problem. Any help > would be appreciated.> > java.lang.RuntimeException: > java.lang.RuntimeException:> > java.lang.ClassNotFoundException: > test.Map> >    >  at> > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752)> >  >   >  at> > > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690)> >    >  at> > > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)> >    >  at> > > > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)> >  >   >  at> > > > > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) > > >    >  at> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)> >    >  at > org.apache.hadoop.mapred.Child.main(Child.java:158)> > Caused by: > java.lang.RuntimeException:> > > java.lang.ClassNotFoundException:> > Markowitz.covarMatrixMap > > >    >  at> > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720)> >  >   >  at> > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744)> >  >   >  ... 6 more> > Caused by: > java.lang.ClassNotFoundException: test.Map> >    >  at> > java.net.URLClassLoader$1.run(URLClassLoader.java:200)> >    >  at > java.security.AccessController.doPrivileged(Native> > Method) > > >    >  at> > > java.net.URLClassLoader.findClass(URLClassLoader.java:188)> >    >  at> > java.lang.ClassLoader.loadClass(ClassLoader.java:306)> >    >  at> > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)> >    >  at> > java.lang.ClassLoader.loadClass(ClassLoader.java:251)> >    >  at> > > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)> >    >  at java.lang.Class.forName0(Native Method)> >    >  at java.lang.Class.forName(Class.java:247)> >    >  at> > > > > > > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) > > >    >  at> > > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718)> >  >   >  ... 7 more> > Thank You > > > Abhishek Agrawal > > > SUNY- Buffalo > > > (716-435-7122) > > > > > > > > > > > > > > >
Re: Re: Re: Doubt in Hadoop
Hi, I dont set job.setJarByClass(Map.class). But my main class, the map class and the reduce class are all in the same package. Does this make any difference at or do I still have to call Thank You Abhishek Agrawal SUNY- Buffalo (716-435-7122) On Fri 11/27/09 1:42 PM , Aaron Kimball aa...@cloudera.com sent: > When you set up the Job object, do you call > job.setJarByClass(Map.class)? That will tell Hadoop which jar file to > ship with the job and to use for classloading in your code. > > - Aaron > On Thu, Nov 26, 2009 at 11:56 PM, wrote: > Hi, > I am running the job from command line. The job runs fine in the > local mode > but something happens when I try to run the job in the distributed > mode. > Abhishek Agrawal > SUNY- Buffalo > (716-435-7122) > On Fri 11/27/09 2:31 AM , Jeff Zhang sent: > > Do you run the map reduce job in command line or IDE? in map > reduce > > mode, you should put the jar containing the map and reduce class > in > > your classpath > > Jeff Zhang > > On Fri, Nov 27, 2009 at 2:19 PM, wrote: > > Hello Everybody, > > I have a doubt in Haddop and was wondering > if > > anybody has faced a > > similar problem. I have a package called test. Inside that I have > > class called > > A.java, Map.java, Reduce.java. In A.java I have the main method > > where I am trying > > to initialize the jobConf object. I have written > > jobConf.setMapperClass(Map.class) and similarly for the reduce > class > > as well. The > > code works correctly when I run the code locally via > > jobConf.set("mapred.job.tracker","local") but I get an exception > > when I try to > > run this code on my cluster. The stack trace of the exception is > as > > under. I > > cannot understand the problem. Any help would be appreciated. > > java.lang.RuntimeException: java.lang.RuntimeException: > > java.lang.ClassNotFoundException: test.Map > > at > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752) > > at > > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690) > > at > > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) > > at > > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > > at > > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) > > at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) > > at org.apache.hadoop.mapred.Child.main(Child.java:158) > > Caused by: java.lang.RuntimeException: > > java.lang.ClassNotFoundException: > > Markowitz.covarMatrixMap > > at > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720) > > at > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744) > > ... 6 more > > Caused by: java.lang.ClassNotFoundException: test.Map > > at > java.net.URLClassLoader$1.run(URLClassLoader.java:200) > > at java.security.AccessController.doPrivileged(Native > > Method) > > at > > java.net.URLClassLoader.findClass(URLClassLoader.java:188) > > at > java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) > > at > java.lang.ClassLoader.loadClass(ClassLoader.java:251) > > at > > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:247) > > at > > > > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) > > at > > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718) > > ... 7 more > > Thank You > > Abhishek Agrawal > > SUNY- Buffalo > > (716-435-7122) > > > > > >
Re: Re: Doubt in Hadoop
When you set up the Job object, do you call job.setJarByClass(Map.class)? That will tell Hadoop which jar file to ship with the job and to use for classloading in your code. - Aaron On Thu, Nov 26, 2009 at 11:56 PM, wrote: > Hi, > I am running the job from command line. The job runs fine in the local > mode > but something happens when I try to run the job in the distributed mode. > > > Abhishek Agrawal > > SUNY- Buffalo > (716-435-7122) > > On Fri 11/27/09 2:31 AM , Jeff Zhang zjf...@gmail.com sent: > > Do you run the map reduce job in command line or IDE? in map reduce > > mode, you should put the jar containing the map and reduce class in > > your classpath > > Jeff Zhang > > On Fri, Nov 27, 2009 at 2:19 PM, wrote: > > Hello Everybody, > >I have a doubt in Haddop and was wondering if > > anybody has faced a > > similar problem. I have a package called test. Inside that I have > > class called > > A.java, Map.java, Reduce.java. In A.java I have the main method > > where I am trying > > to initialize the jobConf object. I have written > > jobConf.setMapperClass(Map.class) and similarly for the reduce class > > as well. The > > code works correctly when I run the code locally via > > jobConf.set("mapred.job.tracker","local") but I get an exception > > when I try to > > run this code on my cluster. The stack trace of the exception is as > > under. I > > cannot understand the problem. Any help would be appreciated. > > java.lang.RuntimeException: java.lang.RuntimeException: > > java.lang.ClassNotFoundException: test.Map > >at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752) > >at > > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690) > >at > > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) > >at > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > >at > > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) > >at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) > >at org.apache.hadoop.mapred.Child.main(Child.java:158) > > Caused by: java.lang.RuntimeException: > > java.lang.ClassNotFoundException: > > Markowitz.covarMatrixMap > >at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720) > >at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744) > >... 6 more > > Caused by: java.lang.ClassNotFoundException: test.Map > >at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > >at java.security.AccessController.doPrivileged(Native > > Method) > >at > > java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > >at > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) > >at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > >at > > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > >at java.lang.Class.forName0(Native Method) > >at java.lang.Class.forName(Class.java:247) > >at > > > > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) > >at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718) > >... 7 more > > Thank You > > Abhishek Agrawal > > SUNY- Buffalo > > (716-435-7122) > > > > > >
Re: Re: Doubt in Hadoop
Hi, I am running the job from command line. The job runs fine in the local mode but something happens when I try to run the job in the distributed mode. Abhishek Agrawal SUNY- Buffalo (716-435-7122) On Fri 11/27/09 2:31 AM , Jeff Zhang zjf...@gmail.com sent: > Do you run the map reduce job in command line or IDE? in map reduce > mode, you should put the jar containing the map and reduce class in > your classpath > Jeff Zhang > On Fri, Nov 27, 2009 at 2:19 PM, wrote: > Hello Everybody, > I have a doubt in Haddop and was wondering if > anybody has faced a > similar problem. I have a package called test. Inside that I have > class called > A.java, Map.java, Reduce.java. In A.java I have the main method > where I am trying > to initialize the jobConf object. I have written > jobConf.setMapperClass(Map.class) and similarly for the reduce class > as well. The > code works correctly when I run the code locally via > jobConf.set("mapred.job.tracker","local") but I get an exception > when I try to > run this code on my cluster. The stack trace of the exception is as > under. I > cannot understand the problem. Any help would be appreciated. > java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: test.Map > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752) > at > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690) > at > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > at > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) > at org.apache.hadoop.mapred.Child.main(Child.java:158) > Caused by: java.lang.RuntimeException: > java.lang.ClassNotFoundException: > Markowitz.covarMatrixMap > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744) > ... 6 more > Caused by: java.lang.ClassNotFoundException: test.Map > at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > at java.security.AccessController.doPrivileged(Native > Method) > at > java.net.URLClassLoader.findClass(URLClassLoader.java:188) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718) > ... 7 more > Thank You > Abhishek Agrawal > SUNY- Buffalo > (716-435-7122) > >
Re: Doubt in Hadoop
Do you run the map reduce job in command line or IDE? in map reduce mode, you should put the jar containing the map and reduce class in your classpath Jeff Zhang On Fri, Nov 27, 2009 at 2:19 PM, wrote: > Hello Everybody, >I have a doubt in Haddop and was wondering if anybody has > faced a > similar problem. I have a package called test. Inside that I have class > called > A.java, Map.java, Reduce.java. In A.java I have the main method where I am > trying > to initialize the jobConf object. I have written > jobConf.setMapperClass(Map.class) and similarly for the reduce class as > well. The > code works correctly when I run the code locally via > jobConf.set("mapred.job.tracker","local") but I get an exception when I try > to > run this code on my cluster. The stack trace of the exception is as under. > I > cannot understand the problem. Any help would be appreciated. > > java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: test.Map >at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752) >at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690) >at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) >at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) >at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) >at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) >at org.apache.hadoop.mapred.Child.main(Child.java:158) > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > Markowitz.covarMatrixMap >at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720) >at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744) >... 6 more > Caused by: java.lang.ClassNotFoundException: test.Map >at java.net.URLClassLoader$1.run(URLClassLoader.java:200) >at java.security.AccessController.doPrivileged(Native Method) >at java.net.URLClassLoader.findClass(URLClassLoader.java:188) >at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) >at java.lang.ClassLoader.loadClass(ClassLoader.java:251) >at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) >at java.lang.Class.forName0(Native Method) >at java.lang.Class.forName(Class.java:247) >at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) >at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718) >... 7 more > > Thank You > > > Abhishek Agrawal > > SUNY- Buffalo > (716-435-7122) > > > >