Re: Running mapred job from remote machine to a pseudo-distributed hadoop
Arv Mistry wrote: I'll try again, can anyone tell me should it be possible to run hadoop in a pseudo-distributed mode (i.e. everything on one machine) and then submit a mapred job using the ToolRunner from another machine on that hadoop configuration? Cheers Arv Yes. It is possible to do. You can start hadoop cluster on single node. Documentation available at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed Once the cluster is up, you can submit jobs from any client, but the client configuration should be aware of Namenode and JobTracker nodes. You can use the generic options *-fs* and *-jt* on commandline for the same. Thanks Amareshwari -Original Message- From: Arv Mistry [mailto:[EMAIL PROTECTED] Sent: Thursday, July 31, 2008 2:32 PM To: core-user@hadoop.apache.org Subject: Running mapred job from remote machine to a pseudo-distributed hadoop I have hadoop setup in a pseudo-distributed mode i.e. everything on one machine, And I'm trying to submit a hadoop mapred job from another machine to that hadoop setup. At the point that I run the mapred job I get the following error. Any ideas as to what I'm doing wrong? Is this possible in a pseudo-distributed mode? Cheers Arv INFO | jvm 1| 2008/07/31 14:01:00 | 2008-07-31 14:01:00,547 ERROR [HadoopJobTool] java.io.IOException: /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such file or directory INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:175) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1| 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) INFO | jvm 1| 2008/07/31 14:01:00 | INFO | jvm 1| 2008/07/31 14:01:00 | org.apache.hadoop.ipc.RemoteException: java.io.IOException: /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such file or directory INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:175) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1| 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) INFO | jvm 1| 2008/07/31 14:01:00 | INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Client.call(Client.java:557) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) INFO | jvm 1| 2008/07/31 14:01:00 | at $Proxy5.submitJob(Unknown Source) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAcce
Re: Running mapred job from remote machine to a pseudo-distributed hadoop
On Fri, Aug 1, 2008 at 7:13 AM, Arv Mistry <[EMAIL PROTECTED]> wrote: > > I'll try again, can anyone tell me should it be possible to run hadoop > in a pseudo-distributed mode (i.e. everything on one machine) That's not quite what pseudo-distributed mode is. You can run regular hadoop jobs on a cluster that consists of one machine, just change the hostname in your hadoop-site.xml file to the real hostname of your machine. If you've got "localhost" in the conf, Hadoop is going to use LocalJobRunner, and that may be related to your issue. I may be wrong on this - I haven't spent much time looking at that code. Take a look at ./src/java/org/apache/hadoop/mapred/JobClient.java for what gets kicked off (for 0.17.1 at least). -- James Moore | [EMAIL PROTECTED] Ruby and Ruby on Rails consulting blog.restphone.com
RE: Running mapred job from remote machine to a pseudo-distributed hadoop
I'll try again, can anyone tell me should it be possible to run hadoop in a pseudo-distributed mode (i.e. everything on one machine) and then submit a mapred job using the ToolRunner from another machine on that hadoop configuration? Cheers Arv -Original Message- From: Arv Mistry [mailto:[EMAIL PROTECTED] Sent: Thursday, July 31, 2008 2:32 PM To: core-user@hadoop.apache.org Subject: Running mapred job from remote machine to a pseudo-distributed hadoop I have hadoop setup in a pseudo-distributed mode i.e. everything on one machine, And I'm trying to submit a hadoop mapred job from another machine to that hadoop setup. At the point that I run the mapred job I get the following error. Any ideas as to what I'm doing wrong? Is this possible in a pseudo-distributed mode? Cheers Arv INFO | jvm 1| 2008/07/31 14:01:00 | 2008-07-31 14:01:00,547 ERROR [HadoopJobTool] java.io.IOException: /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such file or directory INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:175) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1| 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) INFO | jvm 1| 2008/07/31 14:01:00 | INFO | jvm 1| 2008/07/31 14:01:00 | org.apache.hadoop.ipc.RemoteException: java.io.IOException: /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such file or directory INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:175) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1| 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) INFO | jvm 1| 2008/07/31 14:01:00 | INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Client.call(Client.java:557) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) INFO | jvm 1| 2008/07/31 14:01:00 | at $Proxy5.submitJob(Unknown Source) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1| 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo cationHandler.java:82) INFO | jvm 1| 2008
Running mapred job from remote machine to a pseudo-distributed hadoop
I have hadoop setup in a pseudo-distributed mode i.e. everything on one machine, And I'm trying to submit a hadoop mapred job from another machine to that hadoop setup. At the point that I run the mapred job I get the following error. Any ideas as to what I'm doing wrong? Is this possible in a pseudo-distributed mode? Cheers Arv INFO | jvm 1| 2008/07/31 14:01:00 | 2008-07-31 14:01:00,547 ERROR [HadoopJobTool] java.io.IOException: /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such file or directory INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:175) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1| 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) INFO | jvm 1| 2008/07/31 14:01:00 | INFO | jvm 1| 2008/07/31 14:01:00 | org.apache.hadoop.ipc.RemoteException: java.io.IOException: /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such file or directory INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:175) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1| 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) INFO | jvm 1| 2008/07/31 14:01:00 | INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.Client.call(Client.java:557) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) INFO | jvm 1| 2008/07/31 14:01:00 | at $Proxy5.submitJob(Unknown Source) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) INFO | jvm 1| 2008/07/31 14:01:00 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) INFO | jvm 1| 2008/07/31 14:01:00 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo cationHandler.java:82) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation Handler.java:59) INFO | jvm 1| 2008/07/31 14:01:00 | at $Proxy5.submitJob(Unknown Source) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:758) INFO | jvm 1| 2008/07/31 14:01:00 | at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973) INFO | jvm 1| 2008/07/31 14:01:00 | at com.rialto.p
