Apparently, the fix to my original error is because hadoop is setup for a single local machine out of the box and i had to change these directories <property> <name>mapred.local.dir</name> <value>/hadoop/mapred/local</value> </property> <property> <name>mapred.system.dir</name> <value>/hadoop/mapred/system</value> </property> <property> <name>mapred.temp.dir</name> <value>/hadoop/mapred/temp</value> </property>
to be hdfs instead of "hadoop.tmp.dir" So now distcp works as a non-hadoop user and mapred works as a non-hadoop user from the name node, however, from a workstation i get this now blue:hadoop-0.17.1 mdidomenico$ bin/hadoop distcp "file:///Users/mdidomenico/hadoop/1gTestfile" "1gTestfile-1" 08/09/09 13:44:19 INFO util.CopyFiles: srcPaths=[file:/Users/mdidomenico/hadoop/1gTestfile] 08/09/09 13:44:19 INFO util.CopyFiles: destPath=1gTestfile-1 08/09/09 13:44:20 INFO util.CopyFiles: srcCount=1 08/09/09 13:44:22 INFO mapred.JobClient: Running job: job_200809091332_0004 08/09/09 13:44:23 INFO mapred.JobClient: map 0% reduce 0% 08/09/09 13:44:31 INFO mapred.JobClient: Task Id : task_200809091332_0004_m_000000_0, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.util.CopyFiles$CopyFilesMapper.close(CopyFiles.java:527) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124) 08/09/09 13:44:50 INFO mapred.JobClient: Task Id : task_200809091332_0004_m_000000_1, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.util.CopyFiles$CopyFilesMapper.close(CopyFiles.java:527) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124) 08/09/09 13:45:07 INFO mapred.JobClient: Task Id : task_200809091332_0004_m_000000_2, Status : FAILED java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 at org.apache.hadoop.util.CopyFiles$CopyFilesMapper.close(CopyFiles.java:527) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124) 08/09/09 13:45:26 INFO mapred.JobClient: map 100% reduce 100% With failures, global counters are inaccurate; consider running with -i Copy failed: java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062) at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:604) at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) On Tue, Sep 9, 2008 at 1:14 PM, Michael Di Domenico <[EMAIL PROTECTED]>wrote: > manually creating the "system" directory gets me past the first error, but > now i get this. i'm not necessarily sure its a step forward though, because > the map task never shows up in the jobtracker > [EMAIL PROTECTED] hadoop-0.17.1]$ bin/hadoop distcp > "file:///home/mdidomenico/1gTestfile" "1gTestfile" > 08/09/09 13:12:06 INFO util.CopyFiles: > srcPaths=[file:/home/mdidomenico/1gTestfile] > 08/09/09 13:12:06 INFO util.CopyFiles: destPath=1gTestfile > 08/09/09 13:12:07 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:07 INFO dfs.DFSClient: Abandoning block > blk_5758513071638050362 > 08/09/09 13:12:13 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:13 INFO dfs.DFSClient: Abandoning block > blk_1691495306775808049 > 08/09/09 13:12:17 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:17 INFO dfs.DFSClient: Abandoning block > blk_1027634596973755899 > 08/09/09 13:12:19 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:19 INFO dfs.DFSClient: Abandoning block > blk_4535302510016050282 > 08/09/09 13:12:23 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:23 INFO dfs.DFSClient: Abandoning block > blk_7022658012001626339 > 08/09/09 13:12:25 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:25 INFO dfs.DFSClient: Abandoning block > blk_-4509681241839967328 > 08/09/09 13:12:29 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:29 INFO dfs.DFSClient: Abandoning block > blk_8318033979013580420 > 08/09/09 13:12:31 WARN dfs.DFSClient: DataStreamer Exception: > java.io.IOException: Unable to create new block. > 08/09/09 13:12:31 WARN dfs.DFSClient: Error Recovery for block > blk_-4509681241839967328 bad datanode[0] > 08/09/09 13:12:35 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:35 INFO dfs.DFSClient: Abandoning block > blk_2848354798649979411 > 08/09/09 13:12:41 WARN dfs.DFSClient: DataStreamer Exception: > java.io.IOException: Unable to create new block. > 08/09/09 13:12:41 WARN dfs.DFSClient: Error Recovery for block > blk_2848354798649979411 bad datanode[0] > Exception in thread "Thread-0" java.util.ConcurrentModificationException > at java.util.TreeMap$PrivateEntryIterator.nextEntry(Unknown Source) > at java.util.TreeMap$KeyIterator.next(Unknown Source) > at org.apache.hadoop.dfs.DFSClient.close(DFSClient.java:217) > at > org.apache.hadoop.dfs.DistributedFileSystem.close(DistributedFileSystem.java:214) > at > org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1324) > at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:224) > at > org.apache.hadoop.fs.FileSystem$ClientFinalizer.run(FileSystem.java:209) > 08/09/09 13:12:41 INFO dfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Could not read from stream > 08/09/09 13:12:41 INFO dfs.DFSClient: Abandoning block > blk_9189111926428577428 > > On Tue, Sep 9, 2008 at 1:03 PM, Michael Di Domenico < > [EMAIL PROTECTED]> wrote: > >> a little more digging and it appears i cannot run distcp as someone other >> then hadoop on the namenode >> /tmp/hadoop-hadoop/mapred/system/job_200809091231_0005/job.xml >> >> looking at this directory from the error file the "system" directory does >> not exist on the namenode, i only have a "local" directory >> >> >> On Tue, Sep 9, 2008 at 12:41 PM, Michael Di Domenico < >> [EMAIL PROTECTED]> wrote: >> >>> i'm not sure that's the issue, i basically tarred up the hadoop directory >>> from the cluster and copied it over to the non-data node >>> but i do agree i've likely got a setting wrong, since i can run distcp >>> from the namenode and it works fine. the question is which one >>> >>> On Mon, Sep 8, 2008 at 7:04 PM, Aaron Kimball <[EMAIL PROTECTED]>wrote: >>> >>>> It is likely that you mapred.system.dir and/or fs.default.name settings >>>> are >>>> incorrect on the non-datanode machine that you are launching the task >>>> from. >>>> These two settings (in your conf/hadoop-site.xml file) must match the >>>> settings on the cluster itself. >>>> >>>> - Aaron >>>> >>>> On Sun, Sep 7, 2008 at 8:58 PM, Michael Di Domenico >>>> <[EMAIL PROTECTED]>wrote: >>>> >>>> > I'm attempting to load data into hadoop (version 0.17.1), from a >>>> > non-datanode machine in the cluster. I can run jobs and copyFromLocal >>>> > works >>>> > fine, but when i try to use distcp i get the below. I'm don't >>>> understand >>>> > what the error, can anyone help? >>>> > Thanks >>>> > >>>> > blue:hadoop-0.17.1 mdidomenico$ time bin/hadoop distcp -overwrite >>>> > file:///Users/mdidomenico/hadoop/1gTestfile >>>> /user/mdidomenico/1gTestfile >>>> > 08/09/07 23:56:06 INFO util.CopyFiles: >>>> > srcPaths=[file:/Users/mdidomenico/hadoop/1gTestfile] >>>> > 08/09/07 23:56:06 INFO util.CopyFiles: >>>> > destPath=/user/mdidomenico/1gTestfile1 >>>> > 08/09/07 23:56:07 INFO util.CopyFiles: srcCount=1 >>>> > With failures, global counters are inaccurate; consider running with >>>> -i >>>> > Copy failed: org.apache.hadoop.ipc.RemoteException: >>>> java.io.IOException: >>>> > /tmp/hadoop-hadoop/mapred/system/job_200809072254_0005/job.xml: No >>>> such >>>> > file >>>> > or directory >>>> > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215) >>>> > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149) >>>> > at >>>> > org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155) >>>> > at >>>> > org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136) >>>> > at >>>> > org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175) >>>> > at >>>> > org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755) >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) >>>> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>>> Source) >>>> > at java.lang.reflect.Method.invoke(Unknown Source) >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>>> > >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557) >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) >>>> > at $Proxy1.submitJob(Unknown Source) >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> > at >>>> > >>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>> > at >>>> > >>>> > >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>> > at java.lang.reflect.Method.invoke(Method.java:585) >>>> > at >>>> > >>>> > >>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >>>> > at >>>> > >>>> > >>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >>>> > at $Proxy1.submitJob(Unknown Source) >>>> > at >>>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:758) >>>> > at >>>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973) >>>> > at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:604) >>>> > at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) >>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>>> > at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) >>>> > >>>> >>> >>> >> >