[
https://issues.apache.org/jira/browse/HDFS-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bogdan Raducanu updated HDFS-7268:
--
Description:
It seems to me that currently it's not possible to start appending to a file
with CreateFlag.SYNC_BLOCK behavior (i.e. hsync after each block). So, I think,
the only way to guarantee durability when appending is if the user calls hsync
at the end of each block.
FileSystem.append doesn't accept a CreateFlag argument.
FileSystem.create, on the other hand, ignores CreateFlag.APPEND afaics. I think
the plan in HDFS-744 was to use this method if durability is needed.
It seems it might work through FileContext but in the end DFSOutputStream never
sets shouldSyncBlock when appending anyway.
Or am I missing something?
was:
It seems to me that currently it's not possible to start appending to a file
with CreateFlag.SYNC_BLOCK behavior (i.e. hsync after each block). This means
proper fsync is not possible when appending (unless doing hsync after every
write, I guess).
FileSystem.append doesn't accept a CreateFlag argument (probably shouldn't
anyway).
FileSystem.create ignores CreateFlag.APPEND
It seems it might work through FileContext but in the end DFSOutputStream can't
be passed a CreateFlag for appending anyway.
Am I missing something?
I will think of a way to fix it. An immediate thought is to add a
setSyncBlock() method to DFSOutputStream and remove CreateFlag.SYNC_BLOCK.
> Can't append with SYNC_BLOCK
>
>
> Key: HDFS-7268
> URL: https://issues.apache.org/jira/browse/HDFS-7268
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.4.1
>Reporter: Bogdan Raducanu
>Assignee: Bogdan Raducanu
>
> It seems to me that currently it's not possible to start appending to a file
> with CreateFlag.SYNC_BLOCK behavior (i.e. hsync after each block). So, I
> think, the only way to guarantee durability when appending is if the user
> calls hsync at the end of each block.
> FileSystem.append doesn't accept a CreateFlag argument.
> FileSystem.create, on the other hand, ignores CreateFlag.APPEND afaics. I
> think the plan in HDFS-744 was to use this method if durability is needed.
> It seems it might work through FileContext but in the end DFSOutputStream
> never sets shouldSyncBlock when appending anyway.
> Or am I missing something?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)