[ https://issues.apache.org/jira/browse/HDFS-7941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316948#comment-15316948 ]
Johannes Herr commented on HDFS-7941: ------------------------------------- The file may not appear to grow because the new length is not sent to the Namenode on hsync or because Block-compression is used and the data is still in the compress buffer. See http://stackoverflow.com/a/37664114/136247 for more details. > hsync() not working > ------------------- > > Key: HDFS-7941 > URL: https://issues.apache.org/jira/browse/HDFS-7941 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client > Affects Versions: 2.6.0 > Environment: HDP 2.2 running on Redhat > Reporter: Sverre Bakke > > When using SequenceFile.Writer and appending+syncing to file repeatedly, the > sync does not appear to work other than: > - once after writing headers > - when closing. > Imagine the following test case: > http://pastebin.com/Y9xysCRX > This code would append a new record every second and then immediately sync > it. One would also imagine that the file would grow for every append, > however, this does not happen. > After watching the behavior I have noticed that it only syncs the headers at > the very beginning (providing a file of 164 bytes) and then never again until > its closed. This despite it is asked to hsync() after every append. > Looking into the debug logs, this also claims the same behavior (executed the > provided code example and grepped for "sync"): > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > 2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms > This was the only time the code ran fsync throughout the entire execution. > This has been tested (with similar result) for the following deployments: > - sequencefile with no compression > - sequencefile with record compression > - sequencefile with block compression > - textfile with no compression -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org