[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
--------------------------------
Description:
This is a follow-up of [HDFS-9379].
Though for actual benchmarking the defaults are rarely used, it would be good
to change the default for {{numThreads}} as a >=10 value and may be
{{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
[HDFS-9379] is tested in the future.
was:
Currently, the {{NNThroughputBenchmark}} test {{BlockReportStats}} relies on
sorted {{datanodes}} array in the lexicographical order of datanode's
{{xferAddr}}.
* There is an assertion of datanode's {{xferAddr}} lexicographical order when
filling the {{datanodes}}, see [the
code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java#L1152].
* When searching the datanode by {{DatanodeInfo}}, it uses binary search
against the {{datanodes}} array, see [the
code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java#L1187]
In {{DatanodeID}}, the {{xferAddr}} is defined as {{host:port}}. In
{{NNThroughputBenchmark}}, the port is simply _the index of the tiny datanode_
plus one.
The problem here is that, when there are more than 9 tiny datanodes
({{numThreads}}), the lexicographical order of datanode's {{xferAddr}} will be
invalid as the string value of datanode index is not in lexicographical order
any more. For example,
{code}
...
192.168.54.40:8
192.168.54.40:9
192.168.54.40:10
192.168.54.40:11
...
{code}
{{192.168.54.40:9}} is greater than {{192.168.54.40:10}}. The assertion will
fail and the binary search won't work.
The simple fix is to calculate the datanode index by port directly, instead of
using binary search.
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
> ----------------------------------------------------------------------------
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Reporter: Mingliang Liu
> Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
