[
https://issues.apache.org/jira/browse/HADOOP-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507130
]
Devaraj Das commented on HADOOP-1502:
-------------------------------------
A comment here is that the File class's mkdirs( ) is vulnerable to the problem
(as I commented
https://issues.apache.org/jira/browse/HADOOP-1513#action_12507128), but
File.mkdir( ) is not. File class doesn't cache updates (as is true with HDFS).
File.mkdir( ), File.exists( ) calls are passed on to the OS. So for example, if
a thread calls File.mkdir(), and, if another thread makes a call to
File.exists() while the mkdir( ) is in progress, that exists( ) call will
return false. So if a call to File.exists() returns true, we can be sure that
the native filesystem contains that path.
> IOException at task startup "No valid local directories in property:
> dfs.client.buffer.dir"
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-1502
> URL: https://issues.apache.org/jira/browse/HADOOP-1502
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.12.3
> Reporter: Koji Noguchi
> Priority: Minor
>
> When tasks started up on a brand new mapred cluster, I saw couples of them
> fail immediately by
> java.io.IOException: No valid local directories in property:
> dfs.client.buffer.dir
> at org.apache.hadoop.conf.Configuration.getFile(Configuration.java:401)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(DFSClient.java:1037)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.(DFSClient.java:1004)
> at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:276)
> at
> org.apache.hadoop.dfs.DistributedFileSystem$RawDistributedFileSystem.create(DistributedFileSystem.java:143)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.(ChecksumFileSystem.java:367)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:442)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:346)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:253)
> at
> org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.copy(CopyFiles.java:215)
> at
> org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.map(CopyFiles.java:410)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
> (Run on version 0.12.3 + many patches, so the line number could be a little
> off)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
