[ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090068#comment-13090068 ]
Konstantin Shvachko commented on HDFS-1108: ------------------------------------------- bq. I and several others am working on approach #2 Thanks Todd for clarifying this. Could you please also share the design document for your approach? I would like to learn the details and understand why you choose an approach which at this point does not seem to me optimal for the project. bq. option 1 causes dataloss regardless of your opinions on HA This is not a data loss, Todd. This is a tradeoff between performance and the persistence of data. Having flush and sync one can control when to choose performance in favor of guaranteed persistence and when vice versa. Regardless of my opinion on HA Is anybody asking to remove this flexibility? bq. How do you differentiate between logSync() to that stream Vs stream to the disk? I do not differentiate between streams. As I said addBlock() transaction should be treated the same way as setTimes(), that is it is logged (and batched) but not synced. There is no consistency issue here. Transactions will eventually be committed to the journal by another sync-able transaction or by a file close(). bq. The approach 2 has different requirements that many are interested in. I have repeated this many times. Suresh, yes you do repeat it. But you never answered MY question, which HA approach are you implementing. As you see you have to make choices even with issues that seemed to be common part for all approaches. I like Milind's idea about an implementation "without shared storage assumption". Sticking to the point. - Without HA consideration this patch removes flexibility to choose between performance and guaranteed persistence of data - There should be a good reason for that - HA solution with shared storage seems to be the reason - The community has not seen the design, hasn't discussed it, and is not aware of why this one is better than other three (or was it four) published in different jiras. > Log newly allocated blocks > -------------------------- > > Key: HDFS-1108 > URL: https://issues.apache.org/jira/browse/HDFS-1108 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node > Reporter: dhruba borthakur > Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, hdfs-1108.txt > > > The current HDFS design says that newly allocated blocks for a file are not > persisted in the NN transaction log when the block is allocated. Instead, a > hflush() or a close() on the file persists the blocks into the transaction > log. It would be nice if we can immediately persist newly allocated blocks > (as soon as they are allocated) for specific files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira