Thanks for the example. I did look at the logs and also at the admin page
and all I see is the exception that I posted initially.
I am not sure why adding an extra jar to the classpath in DistributedCache
causes that exception. I tried to look at Configuration code in hadoop.util
package but it doesn't tell much. It looks like it's throwing on this line
"configureMethod.invoke(theObject, conf);" in below code.
*private* *static* *void* setJobConf(Object theObject, Configuration conf) {
//If JobConf and JobConfigurable are in classpath, AND
//theObject is of type JobConfigurable AND
//conf is of type JobConf then
//invoke configure on theObject
*try* {
Class<?> jobConfClass =
conf.getClassByName("org.apache.hadoop.mapred.JobConf");
Class<?> jobConfigurableClass =
conf.getClassByName("org.apache.hadoop.mapred.JobConfigurable");
*if* (jobConfClass.isAssignableFrom(conf.getClass()) &&
jobConfigurableClass.isAssignableFrom(theObject.getClass())) {
Method configureMethod =
jobConfigurableClass.getMethod("configure", jobConfClass);
configureMethod.invoke(theObject, conf);
}
} *catch* (ClassNotFoundException e) {
//JobConf/JobConfigurable not in classpath. no need to configure
} *catch* (Exception e) {
*throw* *new* RuntimeException("Error in configuring object", e);
}
}
On Tue, Feb 28, 2012 at 9:25 PM, Harsh J <ha...@cloudera.com> wrote:
> Mohit,
>
> If you visit the failed task attempt on the JT Web UI, you can see the
> complete, informative stack trace on it. It would point the exact line
> the trouble came up in and what the real error during the
> configure-phase of task initialization was.
>
> A simple attempts page goes like the following (replace job ID and
> task ID of course):
>
>
> http://host:50030/taskdetails.jsp?jobid=job_201202041249_3964&tipid=task_201202041249_3964_m_000000
>
> Once there, find and open the "All" logs link to see stdout, stderr,
> and syslog of the specific failed task attempt. You'll have more info
> sifting through this to debug your issue.
>
> This is also explained in Tom's book under the title "Debugging a Job"
> (p154, Hadoop: The Definitive Guide, 2nd ed.).
>
> On Wed, Feb 29, 2012 at 1:40 AM, Mohit Anchlia <mohitanch...@gmail.com>
> wrote:
> > It looks like adding this line causes invocation exception. I looked in
> > hdfs and I see that file in that path
> >
> > DistributedCache.*addFileToClassPath*(*new* Path("/jars/common.jar"),
> conf);
> >
> > I have similar code for another jar
> > "DistributedCache.*addFileToClassPath*(*new* Path("/jars/analytics.jar"),
> > conf);" but this works just fine.
> >
> >
> > On Tue, Feb 28, 2012 at 11:44 AM, Mohit Anchlia <mohitanch...@gmail.com
> >wrote:
> >
> >> I commented reducer and combiner both and still I see the same
> exception.
> >> Could it be because I have 2 jars being added?
> >>
> >> On Mon, Feb 27, 2012 at 8:23 PM, Subir S <subir.sasiku...@gmail.com
> >wrote:
> >>
> >>> On Tue, Feb 28, 2012 at 4:30 AM, Mohit Anchlia <mohitanch...@gmail.com
> >>> >wrote:
> >>>
> >>> > For some reason I am getting invocation exception and I don't see any
> >>> more
> >>> > details other than this exception:
> >>> >
> >>> > My job is configured as:
> >>> >
> >>> >
> >>> > JobConf conf = *new* JobConf(FormMLProcessor.*class*);
> >>> >
> >>> > conf.addResource("hdfs-site.xml");
> >>> >
> >>> > conf.addResource("core-site.xml");
> >>> >
> >>> > conf.addResource("mapred-site.xml");
> >>> >
> >>> > conf.set("mapred.reduce.tasks", "0");
> >>> >
> >>> > conf.setJobName("mlprocessor");
> >>> >
> >>> > DistributedCache.*addFileToClassPath*(*new*
> Path("/jars/analytics.jar"),
> >>> > conf);
> >>> >
> >>> > DistributedCache.*addFileToClassPath*(*new* Path("/jars/common.jar"),
> >>> > conf);
> >>> >
> >>> > conf.setOutputKeyClass(Text.*class*);
> >>> >
> >>> > conf.setOutputValueClass(Text.*class*);
> >>> >
> >>> > conf.setMapperClass(Map.*class*);
> >>> >
> >>> > conf.setCombinerClass(Reduce.*class*);
> >>> >
> >>> > conf.setReducerClass(IdentityReducer.*class*);
> >>> >
> >>>
> >>> Why would you set the Reducer when the number of reducers is set to
> zero.
> >>> Not sure if this is the real cause.
> >>>
> >>>
> >>> >
> >>> > conf.setInputFormat(SequenceFileAsTextInputFormat.*class*);
> >>> >
> >>> > conf.setOutputFormat(TextOutputFormat.*class*);
> >>> >
> >>> > FileInputFormat.*setInputPaths*(conf, *new* Path(args[0]));
> >>> >
> >>> > FileOutputFormat.*setOutputPath*(conf, *new* Path(args[1]));
> >>> >
> >>> > JobClient.*runJob*(conf);
> >>> >
> >>> > ---------
> >>> > *
> >>> >
> >>> > java.lang.RuntimeException*: Error in configuring object
> >>> >
> >>> > at org.apache.hadoop.util.ReflectionUtils.setJobConf(*
> >>> > ReflectionUtils.java:93*)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.util.ReflectionUtils.setConf(*ReflectionUtils.java:64*)
> >>> >
> >>> > at org.apache.hadoop.util.ReflectionUtils.newInstance(*
> >>> > ReflectionUtils.java:117*)
> >>> >
> >>> > at org.apache.hadoop.mapred.MapTask.runOldMapper(*MapTask.java:387*)
> >>> >
> >>> > at org.apache.hadoop.mapred.MapTask.run(*MapTask.java:325*)
> >>> >
> >>> > at org.apache.hadoop.mapred.Child$4.run(*Child.java:270*)
> >>> >
> >>> > at java.security.AccessController.doPrivileged(*Native Method*)
> >>> >
> >>> > at javax.security.auth.Subject.doAs(*Subject.java:396*)
> >>> >
> >>> > at org.apache.hadoop.security.UserGroupInformation.doAs(*
> >>> > UserGroupInformation.java:1157*)
> >>> >
> >>> > at org.apache.hadoop.mapred.Child.main(*Child.java:264*)
> >>> >
> >>> > Caused by: *java.lang.reflect.InvocationTargetException
> >>> > *
> >>> >
> >>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(*Native Method*)
> >>> >
> >>> > at sun.reflect.NativeMethodAccessorImpl.invoke(*
> >>> > NativeMethodAccessorImpl.java:39*)
> >>> >
> >>> > at
> >>> >
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
> >>> >
> >>>
> >>
> >>
>
>
>
> --
> Harsh J
>
