[
https://issues.apache.org/jira/browse/HDFS-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bogdan Raducanu updated HDFS-7268:
--
Description:
It seems to me that currently it's not possible to start appending to a file
with CreateFlag.SYNC_BLOCK behavior (i.e. hsync after each block). So, I think,
the only way to guarantee durability when appending is if the user calls hsync
at the end of each block.
FileSystem.append doesn't accept a CreateFlag argument.
FileSystem.create, on the other hand, ignores CreateFlag.APPEND afaics. I think
the plan in HDFS-744 was to use this method if durability is needed.
It seems it might work through FileContext but in the end DFSOutputStream never
sets shouldSyncBlock when appending anyway.
Or am I missing something?
was:
It seems to me that currently it's not possible to start appending to a file
with CreateFlag.SYNC_BLOCK behavior (i.e. hsync after each block). This means
proper fsync is not possible when appending (unless doing hsync after every
write, I guess).
FileSystem.append doesn't accept a CreateFlag argument (probably shouldn't
anyway).
FileSystem.create ignores CreateFlag.APPEND
It seems it might work through FileContext but in the end DFSOutputStream can't
be passed a CreateFlag for appending anyway.
Am I missing something?
I will think of a way to fix it. An immediate thought is to add a
setSyncBlock() method to DFSOutputStream and remove CreateFlag.SYNC_BLOCK.
Can't append with SYNC_BLOCK
Key: HDFS-7268
URL: https://issues.apache.org/jira/browse/HDFS-7268
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.4.1
Reporter: Bogdan Raducanu
Assignee: Bogdan Raducanu
It seems to me that currently it's not possible to start appending to a file
with CreateFlag.SYNC_BLOCK behavior (i.e. hsync after each block). So, I
think, the only way to guarantee durability when appending is if the user
calls hsync at the end of each block.
FileSystem.append doesn't accept a CreateFlag argument.
FileSystem.create, on the other hand, ignores CreateFlag.APPEND afaics. I
think the plan in HDFS-744 was to use this method if durability is needed.
It seems it might work through FileContext but in the end DFSOutputStream
never sets shouldSyncBlock when appending anyway.
Or am I missing something?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
