wait and sleep are not what you are looking for. you can use 'nohup' to run a job in the background and have its output piped to a file.
On Tue, Jun 10, 2008 at 5:48 PM, Meng Mao <[EMAIL PROTECTED]> wrote: > I'm interested in the same thing -- is there a recommended way to batch > Hadoop jobs together? > > On Tue, Jun 10, 2008 at 5:45 PM, Richard Zhang <[EMAIL PROTECTED]> > wrote: > >> Hello folks: >> I am running several hadoop applications on hdfs. To save the efforts in >> issuing the set of commands every time, I am trying to use bash script to >> run the several applications sequentially. To let the job finishes before >> it >> is proceeding to the next job, I am using wait in the script like below. >> >> sh bin/start-all.sh >> wait >> echo cluster start >> (bin/hadoop jar hadoop-0.17.0-examples.jar randomwriter -D >> test.randomwrite.bytes_per_map=107374182 rand) >> wait >> bin/hadoop jar hadoop-0.17.0-examples.jar randomtextwriter -D >> test.randomtextwrite.total_bytes=107374182 rand-text >> bin/stop-all.sh >> echo finished hdfs randomwriter experiment >> >> >> However, it always give the error like below. Does anyone have better idea >> on how to run the multiple sequential jobs with bash script? >> >> HadoopScript.sh: line 39: wait: pid 10 is not a child of this shell >> >> org.apache.hadoop.ipc.RemoteException: >> org.apache.hadoop.mapred.JobTracker$IllegalStateException: Job tracker >> still >> initializing >> at >> org.apache.hadoop.mapred.JobTracker.ensureRunning(JobTracker.java:1722) >> at >> org.apache.hadoop.mapred.JobTracker.getNewJobId(JobTracker.java:1730) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:557) >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) >> at $Proxy1.getNewJobId(Unknown Source) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >> at >> >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >> at $Proxy1.getNewJobId(Unknown Source) >> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:696) >> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973) >> at >> org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:276) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at >> org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:287) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> >> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) >> at >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) >> at >> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:53) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:155) >> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220) >> > > > > -- > hustlin, hustlin, everyday I'm hustlin >
