Re: Re: Re: Re: Doubt in Hadoop
Hi, Actually, I just made the change suggested by Aaron and my code worked. But I still would like to know why does the setJarbyClass() method have to be called when the Main class and the Map and Reduce classes are in the same package ? Thank You Abhishek Agrawal SUNY- Buffalo (716-435-7122) On Sun 11/29/09 10:39 AM , aa...@buffalo.edu sent: Hi, I dont set job.setJarByClass(Map.class). But my main class, the map class andthe reduce class are all in the same package. Does this make any difference at ordo I still have to call Thank You Abhishek Agrawal SUNY- Buffalo (716-435-7122) On Fri 11/27/09 1:42 PM , Aaron Kimball aa...@clou dera.com sent: When you set up the Job object, do you call job.setJarByClass(Map.class)? That will tell Hadoop which jar file to ship with the job and to use for classloading in your code. - Aaron On Thu, Nov 26, 2009 at 11:56 PM, wrote: Hi,  I am running the job from command line. The job runs fine in the local mode but something happens when I try to run the job in the distributed mode. Abhishek Agrawal SUNY- Buffalo (716-435-7122) On Fri 11/27/09  2:31 AM , Jeff Zhang sent: Do you run the map reduce job in command line or IDE? in map reduce mode, you should put the jar containing the map and reduce class in your classpath Jeff Zhang On Fri, Nov 27, 2009 at 2:19 PM,  wrote: Hello Everybody,         I have a doubt in Haddop and was wondering if anybody has faced a similar problem. I have a package called test. Inside that I have class called A.java, Map.java, Reduce.java. In A.java I have the main method where I am trying to initialize the jobConf object. I have written jobConf.setMapperClass(Map.class) and similarly for the reduce class as well. The code works correctly when I run the code locally via jobConf.set(mapred.job.tracker,local) but I get an exception when I try to run this code on my cluster. The stack trace of the exception is as under. I cannot understand the problem. Any help would be appreciated. java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.Map     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752)     at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690)     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)     at org.apache.hadoop.mapred.Child.main(Child.java:158) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Markowitz.covarMatrixMap     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720)     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744)     ... 6 more Caused by: java.lang.ClassNotFoundException: test.Map     at java.net.URLClassLoader$1.run(URLClassLoader.java:200)     at java.security.AccessController.doPrivileged(Native Method)     at java.net.URLClassLoader.findClass(URLClassLoader.java:188)     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)     at java.lang.ClassLoader.loadClass(ClassLoader.java:251)     at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)     at java.lang.Class.forName0(Native Method)     at java.lang.Class.forName(Class.java:247)     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)     at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718)     ... 7 more Thank You Abhishek Agrawal SUNY- Buffalo (716-435-7122)
Re: Re: Doubt in Hadoop
When you set up the Job object, do you call job.setJarByClass(Map.class)? That will tell Hadoop which jar file to ship with the job and to use for classloading in your code. - Aaron On Thu, Nov 26, 2009 at 11:56 PM, aa...@buffalo.edu wrote: Hi, I am running the job from command line. The job runs fine in the local mode but something happens when I try to run the job in the distributed mode. Abhishek Agrawal SUNY- Buffalo (716-435-7122) On Fri 11/27/09 2:31 AM , Jeff Zhang zjf...@gmail.com sent: Do you run the map reduce job in command line or IDE? in map reduce mode, you should put the jar containing the map and reduce class in your classpath Jeff Zhang On Fri, Nov 27, 2009 at 2:19 PM, wrote: Hello Everybody, I have a doubt in Haddop and was wondering if anybody has faced a similar problem. I have a package called test. Inside that I have class called A.java, Map.java, Reduce.java. In A.java I have the main method where I am trying to initialize the jobConf object. I have written jobConf.setMapperClass(Map.class) and similarly for the reduce class as well. The code works correctly when I run the code locally via jobConf.set(mapred.job.tracker,local) but I get an exception when I try to run this code on my cluster. The stack trace of the exception is as under. I cannot understand the problem. Any help would be appreciated. java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.Map at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752) at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) at org.apache.hadoop.mapred.Child.main(Child.java:158) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Markowitz.covarMatrixMap at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744) ... 6 more Caused by: java.lang.ClassNotFoundException: test.Map at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718) ... 7 more Thank You Abhishek Agrawal SUNY- Buffalo (716-435-7122)
Re: Doubt in Hadoop
Do you run the map reduce job in command line or IDE? in map reduce mode, you should put the jar containing the map and reduce class in your classpath Jeff Zhang On Fri, Nov 27, 2009 at 2:19 PM, aa...@buffalo.edu wrote: Hello Everybody, I have a doubt in Haddop and was wondering if anybody has faced a similar problem. I have a package called test. Inside that I have class called A.java, Map.java, Reduce.java. In A.java I have the main method where I am trying to initialize the jobConf object. I have written jobConf.setMapperClass(Map.class) and similarly for the reduce class as well. The code works correctly when I run the code locally via jobConf.set(mapred.job.tracker,local) but I get an exception when I try to run this code on my cluster. The stack trace of the exception is as under. I cannot understand the problem. Any help would be appreciated. java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.Map at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752) at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) at org.apache.hadoop.mapred.Child.main(Child.java:158) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Markowitz.covarMatrixMap at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744) ... 6 more Caused by: java.lang.ClassNotFoundException: test.Map at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718) ... 7 more Thank You Abhishek Agrawal SUNY- Buffalo (716-435-7122)
Re: Re: Doubt in Hadoop
Hi, I am running the job from command line. The job runs fine in the local mode but something happens when I try to run the job in the distributed mode. Abhishek Agrawal SUNY- Buffalo (716-435-7122) On Fri 11/27/09 2:31 AM , Jeff Zhang zjf...@gmail.com sent: Do you run the map reduce job in command line or IDE? in map reduce mode, you should put the jar containing the map and reduce class in your classpath Jeff Zhang On Fri, Nov 27, 2009 at 2:19 PM, wrote: Hello Everybody, I have a doubt in Haddop and was wondering if anybody has faced a similar problem. I have a package called test. Inside that I have class called A.java, Map.java, Reduce.java. In A.java I have the main method where I am trying to initialize the jobConf object. I have written jobConf.setMapperClass(Map.class) and similarly for the reduce class as well. The code works correctly when I run the code locally via jobConf.set(mapred.job.tracker,local) but I get an exception when I try to run this code on my cluster. The stack trace of the exception is as under. I cannot understand the problem. Any help would be appreciated. java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: test.Map at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752) at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338) at org.apache.hadoop.mapred.Child.main(Child.java:158) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Markowitz.covarMatrixMap at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744) ... 6 more Caused by: java.lang.ClassNotFoundException: test.Map at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:718) ... 7 more Thank You Abhishek Agrawal SUNY- Buffalo (716-435-7122)
