Zheng Shao created HDFS-14229: --------------------------------- Summary: Nonblocking HDFS create|write Key: HDFS-14229 URL: https://issues.apache.org/jira/browse/HDFS-14229 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client Reporter: Zheng Shao
Right now, the create call on HDFS is blocking. The write call can also be blocking if the write buffer reached its limit. However, for most applications, the only requirement is that when "close" on a file is called, the file is persisted and visible in HDFS. There is no need to make "create" visible right after the "create" call returns. A particular use case of this is to use HDFS as a place to store shuffle data (in Spark, Map-Reduce, or other loose-coupled applications). This Jira proposes that we add a new "async-hdfs://" protocol that maps to a new AsyncDistributedFileSystem class, whose create call is nonblocking but still returns a FSOutputStream that is never blocked on write (even when the file has not been physically created on HDFS yet). The close call on the FSOutputStream will block until the creation and all previous writes are completed and the file is closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org