[
https://issues.apache.org/jira/browse/HDFS-13356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-13356:
---------------------------------------
Description:
It seems we can run into a problem while a rolling upgrade with this.
The Balancer is upgraded after NameNodes, so once NN is upgraded it will expect
{{minBlockSize}} parameter in {{getBlocks()}}. The Balancer cannot send it yet,
so NN will use the default, which you set to 0. So NN will start unexpectedly
sending small blocks to the Balancer. So we should
# either change the default in protobuf to 10 MB
# or treat {{minBlockSize == 0}} in {{NameNodeRpcServer}} as a signal to use
the configuration variable
{{DFSConfigKeys.DFS_BALANCER_GETBLOCKS_MIN_BLOCK_SIZE_KEY}}.
If you agree, we should create a follow up jira. I wanted to backport this down
the chain of branches, but this upgrade scenario is stopping me.
[~shv] commented this in HDFS-13222 jira.
https://issues.apache.org/jira/browse/HDFS-13222?focusedCommentId=16414855&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16414855
was:
It seems we can run into a problem while a rolling upgrade with this.
The Balancer is upgraded after NameNodes, so once NN is upgraded it will expect
{{minBlockSize}}parameter in {{getBlocks()}}. The Balancer cannot send it yet,
so NN will use the default, which you set to 0. So NN will start unexpectedly
sending small blocks to the Balancer. So we should
# either change the default in protobuf to 10 MB
# or treat {{minBlockSize == 0}} in {{NameNodeRpcServer}} as a signal to use
the configuration variable
{{DFSConfigKeys.DFS_BALANCER_GETBLOCKS_MIN_BLOCK_SIZE_KEY}}.
If you agree, we should create a follow up jira. I wanted to backport this down
the chain of branches, but this upgrade scenario is stopping me.
[~shv]] commented this in HDFS-13222 jira.
https://issues.apache.org/jira/browse/HDFS-13222?focusedCommentId=16414855&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16414855
> Balancer:Set default value of minBlockSize to 10mb
> ---------------------------------------------------
>
> Key: HDFS-13356
> URL: https://issues.apache.org/jira/browse/HDFS-13356
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Affects Versions: 2.7.5
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Major
> Labels: balancer, upgrades
> Attachments: HDFS-13356.00.patch, HDFS-13356.01.patch,
> HDFS-13356.02.patch
>
>
> It seems we can run into a problem while a rolling upgrade with this.
> The Balancer is upgraded after NameNodes, so once NN is upgraded it will
> expect {{minBlockSize}} parameter in {{getBlocks()}}. The Balancer cannot
> send it yet, so NN will use the default, which you set to 0. So NN will start
> unexpectedly sending small blocks to the Balancer. So we should
> # either change the default in protobuf to 10 MB
> # or treat {{minBlockSize == 0}} in {{NameNodeRpcServer}} as a signal to use
> the configuration variable
> {{DFSConfigKeys.DFS_BALANCER_GETBLOCKS_MIN_BLOCK_SIZE_KEY}}.
> If you agree, we should create a follow up jira. I wanted to backport this
> down the chain of branches, but this upgrade scenario is stopping me.
> [~shv] commented this in HDFS-13222 jira.
> https://issues.apache.org/jira/browse/HDFS-13222?focusedCommentId=16414855&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16414855
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
