[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675889#comment-17675889
]
Doris Gu commented on HDFS-16569:
-
> ` {{{}dfs.namenode.file.close.num-committed-allowed{}}}` is not helpful when
> most files' size is less than block size.
@[~yuanbo] I think ` {{{}dfs.namenode.file.close.num-committed-allowed{}}}`
is more helpful when most files' size is less than block size. Can you explain
why?
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534276#comment-17534276
]
Xiaoqiao He commented on HDFS-16569:
{quote}Regarding Tectonic, seperating deamon threads from NN and isolating DN
from NN are most attracting features. Maybe we can design and implement those
features on hadoop-4 ?{quote}
Hmmm, It is very similar to Ozone, and will be a big architecture refactor.
Maybe we could start with only one nit feature (of course it need to discuss
and reach agreement first.), such as report location from client only as above
mentioned.
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533186#comment-17533186
]
Yuanbo Liu commented on HDFS-16569:
---
Regarding Tectonic, seperating deamon threads from NN and isolating DN from NN
are most attracting features. Maybe we can design and implement those features
on hadoop-4 ?
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532790#comment-17532790
]
Xiaoqiao He commented on HDFS-16569:
Thanks [~yuanbo] for your detailed explanation.
It is an interesting proposal. IIRC, there are some others DFS has implemented
as your proposal, such as
Tectonic(https://www.usenix.org/conference/fast21/presentation/pan).
I believe it will improve performance for write flow. I am sure if there are
some historical reasons or some security issues consideration when following
above proposal.
{quote}You mean to say, since the client has already reported the block
location, no need to wait for the datanode to report it again, right? So, the
answer is No, that is per design, the namenode doesn't blindly trust a client,
and there are other reasons as well. So, it requires at least one datanode to
confirm that it has received the block with same amount of data as the client
claims.{quote}
[~ayushtkn] Thanks Ayush to give one point that NameNode could not trust client
hundred percent. +1 for me, this is basic of the original HDFS design. And
another one, it could introduce security issue, such as one fake client report
random information to NameNode could cause the following read failed and mount
of missing blocks at NameNode side seen. I am concern if we could tradeoff
initial assumptions and try to find the best solution. Anyway, it is valuable
to have more discussion.
Thanks [~yuanbo] and [~ayushtkn].
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532772#comment-17532772
]
Yuanbo Liu commented on HDFS-16569:
---
[~ayushtkn] Thanks for your comments.
> You mean to say, since the client has already reported the block location, no
> need to wait for the datanode to report it again, right?
yes
` {{{}dfs.namenode.file.close.num-committed-allowed{}}}` is not helpful when
most files' size is less than block size.
We've tried to seperated ibr thread from heartbeating thread, and applied in
our product env. It helps to make writing more stable especially for streaming
writing(flink, spark)
This Jira is trying to make further improvement. I think security is most
concerned reason that we cannot rely on client to report block locations. But
in a relatively closed env, it's worth doing it.
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532765#comment-17532765
]
Yuanbo Liu commented on HDFS-16569:
---
[~hexiaoqiao] Thanks for your comment.
The process is here:
1. client get block location from nn
2. setup pipeline and write block data
3. when the block is finished, then retry multiple times to make sure DN has
sent RECEIVED_BLOCK to NN by IBR
4. when batch ibr is enabled (the interval is usually 300ms), client will not
mark block as finshed before 300ms(even longer if DN is very busy)
5. repeat step3,4 until last block is finshed, then client try to close file
multiple times(default is 5 times with around 15 seconds).
6. If DN is busy, or ibr batch is not triggerred when interval is not reached,
We can see log 'is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in
file' in namenode
As we know, client already gets the location info from nn, and gets acks from
dn, it seems not necessary to rely on ibr to mark block as completed. So we can
send block locations to NN in step3
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532761#comment-17532761
]
Ayush Saxena commented on HDFS-16569:
-
The client has location, that is why the block went into the committed state,
once the client reports then only the block moves to committed state, once the
ibr is received then it marks the block complete.
{quote}Since client already has the last block locations, it's not necessary to
rely on the ibr from DN when closing file.
{quote}
You mean to say, since the client has already reported the block location, no
need to wait for the datanode to report it again, right? So, the answer is No,
that is per design, the namenode doesn't blindly trust a client, and there are
other reasons as well. So, it requires at least one datanode to confirm that it
has received the block with same amount of data as the client claims.
If you are seeing this log a lot, then most probably your cluster is slow, that
IBR's are reaching late. Try this config
{{dfs.namenode.file.close.num-committed-allowed}}, this should allow you to
close the file with committed blocks.
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532747#comment-17532747
]
Xiaoqiao He commented on HDFS-16569:
[~yuanbo] Thanks for your report. Do you mean that last block size is 0 and no
anymore ibr but namenode wait util client timeout?
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
