Adding the directory to my classpath worked!
Thank you both for the prompt assistance.
On Fri, Jul 11, 2008 at 12:33 PM, Matt Kent <[EMAIL PROTECTED]> wrote:
> The best way to configure all that stuff is in hadoop-site.xml, which
> lives in the hadoop conf directory. Make sure that directory is on the
> classpath of your application.
>
> On Fri, 2008-07-11 at 11:55 -0400, Larry Compton wrote:
> > Thanks. Is this the correct syntax?
> >
> > conf.set("mapred.job.tracker", "localhost:54311");
> >
> > It does appear to be communicating with the JobTracker now, but I get the
> > following stack trace. Is there anything else that needs to be done to
> > configure the job?
> >
> > Exception in thread "main" org.apache.hadoop.ipc.RemoteException:
> > java.io.IOException:
> > /home/larry/pkg/hadoop/hdfs/mapred/system/job_200807111114_0001/job.xml:
> No
> > such file or directory
> > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
> > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149)
> > at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155)
> > at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136)
> > at
> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175)
> > at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at $Proxy0.submitJob(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:585)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy0.submitJob(Unknown Source)
> > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:758)
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
> > at jobclient.MaterializeMain.main(MaterializeMain.java:44)
> >
> >
> > On Fri, Jul 11, 2008 at 11:41 AM, Shengkai Zhu <[EMAIL PROTECTED]>
> wrote:
> >
> > > You should provide JobTracker address and port through configuration.
> > >
> > >
> > > On 7/11/08, Larry Compton <[EMAIL PROTECTED]> wrote:
> > > >
> > > > I'm coming up to speed on the Hadoop APIs. I need to be able to
> invoke a
> > > > job
> > > > from within a Java application (as opposed to running from the
> > > command-line
> > > > "hadoop" executable). The JobConf and JobClient appear to support
> this
> > > and
> > > > I've written a test program to configure and run a job. However, the
> job
> > > > doesn't appear to be submitted to the JobTracker. Here's a code
> excerpt
> > > > from
> > > > my client...
> > > >
> > > > String rdfInputPath = args[0];
> > > > String outputPath = args[1];
> > > > String uriInputPath = args[2];
> > > > String jarPath = args[3];
> > > >
> > > > JobConf conf = new JobConf(MaterializeMap.class);
> > > > conf.setJobName("materialize");
> > > >
> > > > conf.setOutputKeyClass(Text.class);
> > > > conf.setOutputValueClass(Text.class);
> > > >
> > > > conf.setMapperClass(MaterializeMapper.class);
> > > > conf.setCombinerClass(MaterializeReducer.class);
> > > > conf.setReducerClass(MaterializeReducer.class);
> > > > conf.setJar(jarPath);
> > > >
> > > > DistributedCache.addCacheFile(new Path(uriInputPath).toUri(),
> > > conf);
> > > >
> > > > FileInputFormat.setInputPaths(conf, new Path(rdfInputPath));
> > > > FileOutputFormat.setOutputPath(conf, new Path(outputPath));
> > > >
> > > > conf.setInputFormat(TextInputFormat.class);
> > > > conf.setOutputFormat(TextOutputFormat.class);
> > > >
> > > > JobClient.runJob(conf);
> > > >
> > > > It seems like I should be providing a URL to the JobTracker
> somewhere,
> > > but
> > > > I
> > > > can't figure out where to provide the information.
> > > >
> > > > --
> > > > Larry Compton
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > 朱盛凯
> > >
> > > Jash Zhu
> > >
> > > 复旦大学软件学院
> > >
> > > Software School, Fudan University
> > >
> >
> >
> >
>
>
--
Larry Compton
