[
https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686514#comment-13686514
]
Konstantin Shvachko commented on HDFS-4849:
-------------------------------------------
Sorry, should correct myself. The code I posted previously works correctly
because FSDataOutputStream keeps track of the offset in the stream. I'll remove
the comment to avoid confusion in reading. It will still be available in the
history.
I am still responding to
> As regards creating blocks simultaneously, it is prevented precisely because
> multiple creates are not allowed today and hence the application does not
> even get to allocating blocks.
The snippet below shows that multiple creates are allowed in current code, and
the things break because there is a competition for block allocations not file
creates.
{code}
static volatile int nrDone;
public static void main(String[] args) throws Exception {
Configuration conf = new HdfsConfiguration();
conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024);
MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(3).build();
final FileSystem fs = cluster.getFileSystem();
final Path path = new Path("testFile.txt");
nrDone = 0;
for(int i = 0; i < 10; i++) {
new Thread() {
@Override
public void run() {
try {
final FSDataOutputStream out = fs.create(path);
for(int bi = 0; bi < 12; bi++)
out.write(new byte[1024], 0, 1024);
out.close();
} catch (IOException e) {
LOG.error("Write failed: ", e);
}
nrDone++;
}
}.start();
}
while(nrDone < 10) ;
LOG.info("Done. " + fs.getFileStatus(path));
for(BlockLocation bl : fs.getFileBlockLocations(path, 0, Long.MAX_VALUE))
LOG.info(bl);
}
{code}
> Idempotent create and append operations.
> ----------------------------------------
>
> Key: HDFS-4849
> URL: https://issues.apache.org/jira/browse/HDFS-4849
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.4-alpha
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Attachments: idempotentCreate.patch, idempotentCreate.patch,
> idempotentCreate.patch
>
>
> create, append and delete operations can be made idempotent. This will reduce
> chances for a job or other app failures when NN fails over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
