[
https://issues.apache.org/jira/browse/HDFS-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835452#action_12835452
]
Zheng Shao commented on HDFS-987:
---------------------------------
Sure.
We uses http://github.com/facebook/scribe to write log data to HDFS. Scribe
rotates the log file based on both file size and time.
We are currently working on a "ContinuousCopier" that will continuously copy
data from one directory to another (in a different hadoop cluster).
The ContinuousCopier needs to know whether scribe has finished writing one file
and rotated to the next.
Without this information, it will be much harder/inefficient for the
ContinuousCopier to switch to the next file. Basically, we need to list the
directory, test whether the next file exists, if so, then we need to open the
current file once more to make sure we've copied it completely, and then we can
switch to the next file.
> HDFS/HFTP InputStream to expose the information about whether a file is under
> construction or not
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-987
> URL: https://issues.apache.org/jira/browse/HDFS-987
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Guanghao Shen
>
> This will allow a reader to know that whether the file is complete or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
