[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532772#comment-17532772
]
Yuanbo Liu commented on HDFS-16569:
-----------------------------------
[~ayushtkn] Thanks for your comments.
> You mean to say, since the client has already reported the block location, no
> need to wait for the datanode to report it again, right?
yes
` {{{}dfs.namenode.file.close.num-committed-allowed{}}}` is not helpful when
most files' size is less than block size.
We've tried to seperated ibr thread from heartbeating thread, and applied in
our product env. It helps to make writing more stable especially for streaming
writing(flink, spark)
This Jira is trying to make further improvement. I think security is most
concerned reason that we cannot rely on client to report block locations. But
in a relatively closed env, it's worth doing it.
> Consider attaching block location info from client when closing a completed
> file
> --------------------------------------------------------------------------------
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Yuanbo Liu
> Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
