[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-10677
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch,
> HDFS-9436.002.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-9436:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I just committed this. Thank you [~liuml07].
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch,
> HDFS-9436.002.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Attachment: HDFS-9436.002.patch
Thank you very much [~shv] for your review. Sending a couple of block reports
makes perfect sense to me. The v2 patch is to address.
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch,
> HDFS-9436.002.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-9436:
--
Target Version/s: 2.8.0
Fix Version/s: (was: 2.8.0)
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Attachment: HDFS-9436.001.patch
Thanks for your review and comment.
No, the change was not intentional. Actually I should have been more careful
when I made the change. I was thinking the overall overhead to run the tests is
very small even if all ops have a larger default scale.
I don't think they all need this change. The v1 patch is to change the
{{BlockReportStats}} only.
While debugging the code, I've no idea how {{numOpsRequired}} is explained in
{{BlockReportStats}} (perhaps a bad design). When {{addBlock()}} adds blocks in
datanode, the {{TinyDatanode}} is not really updating the {{blocks}} list
(perhaps a bug). Moreover, the {{blockReportList}} is not built using the
{{blocks}} either, so the block report will be empty (I comment in
[HDFS-7435]). All the above makes the {{BlockReportStats}} dubious to me,
though this patch seems independent.
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch, HDFS-9436.001.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Status: Patch Available (was: Open)
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Attachment: HDFS-9436.000.patch
The v0 patch makes number of datanodes in {{BlockReportStat}} 10 by default.
This is only to test more cases, especially brought by [HDFS-9379].
The default value should be rarely used. Meanwhile, the {{nameNodeProto}} and
{{dataNodeProto}} are static members of the benchmark, thus running multiple
operation tests with different arguments (e.g. number of datanodes) may get
unexpected result. Because of this, the default number of datanodes in
{{ReplicationReportStat}} is also changed as 10.
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9436.000.patch
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Priority: Minor (was: Major)
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 2.8.0
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Hadoop Flags: (was: Reviewed)
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9436) Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
[
https://issues.apache.org/jira/browse/HDFS-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-9436:
Description:
This is a follow-up of [HDFS-9379].
Though for actual benchmarking the defaults are rarely used, it would be good
to change the default for {{numThreads}} as a >=10 value and may be
{{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
[HDFS-9379] is tested in the future.
was:
Currently, the {{NNThroughputBenchmark}} test {{BlockReportStats}} relies on
sorted {{datanodes}} array in the lexicographical order of datanode's
{{xferAddr}}.
* There is an assertion of datanode's {{xferAddr}} lexicographical order when
filling the {{datanodes}}, see [the
code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java#L1152].
* When searching the datanode by {{DatanodeInfo}}, it uses binary search
against the {{datanodes}} array, see [the
code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java#L1187]
In {{DatanodeID}}, the {{xferAddr}} is defined as {{host:port}}. In
{{NNThroughputBenchmark}}, the port is simply _the index of the tiny datanode_
plus one.
The problem here is that, when there are more than 9 tiny datanodes
({{numThreads}}), the lexicographical order of datanode's {{xferAddr}} will be
invalid as the string value of datanode index is not in lexicographical order
any more. For example,
{code}
...
192.168.54.40:8
192.168.54.40:9
192.168.54.40:10
192.168.54.40:11
...
{code}
{{192.168.54.40:9}} is greater than {{192.168.54.40:10}}. The assertion will
fail and the binary search won't work.
The simple fix is to calculate the datanode index by port directly, instead of
using binary search.
> Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default
>
>
> Key: HDFS-9436
> URL: https://issues.apache.org/jira/browse/HDFS-9436
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>
> This is a follow-up of [HDFS-9379].
> Though for actual benchmarking the defaults are rarely used, it would be good
> to change the default for {{numThreads}} as a >=10 value and may be
> {{numOpsRequired}} in {{BlockReportStats}} just to make sure the condition in
> [HDFS-9379] is tested in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
