[
https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686471#comment-13686471
]
Konstantin Shvachko commented on HDFS-4849:
-------------------------------------------
> As regards creating blocks simultaneously, it is prevented precisely because
> multiple creates are not allowed today and hence the application does not
> even get to allocating blocks.
My example is different. You create file once. THEN spawn 10 threads to add
blocks in the same file.
To demonstrate my point I wrote a snippet below. Just past© it into your
favourite test and run. You will be amazed what you will get, I was. Also try
to vary the {{bufLen}}. Some interesting things happen when you get to the
second block.
The system can be abused and this has nothing to do with the retries.
{code}
static volatile int nrDone;
public static void main(String[] args) throws Exception {
Configuration conf = new HdfsConfiguration();
MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
numDataNodes(3).build();
FileSystem fs = cluster.getFileSystem();
Path path = new Path("testFile.txt");
final FSDataOutputStream out = fs.create(path);
nrDone = 0;
for(int i = 0; i < 10; i++) {
new Thread() {
@Override
public void run() {
int bufLen = 128*1024*1024 + 7;
byte[] toWrite = new byte[bufLen];
try {
out.write(toWrite, 0, toWrite.length);
} catch (IOException e) {
LOG.error("Failed to write a string: ", e);
}
nrDone++;
}
}.start();
}
while(nrDone < 10) ;
out.close();
LOG.info("Done. " + fs.getFileStatus(path));
for(BlockLocation bl : fs.getFileBlockLocations(path, 0, Long.MAX_VALUE))
LOG.info(bl);
}
{code}
> Idempotent create and append operations.
> ----------------------------------------
>
> Key: HDFS-4849
> URL: https://issues.apache.org/jira/browse/HDFS-4849
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.4-alpha
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Attachments: idempotentCreate.patch, idempotentCreate.patch,
> idempotentCreate.patch
>
>
> create, append and delete operations can be made idempotent. This will reduce
> chances for a job or other app failures when NN fails over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
