JiangHua Zhu created HDFS-16389:
-----------------------------------
Summary: Improve NNThroughputBenchmark test mkdirs
Key: HDFS-16389
URL: https://issues.apache.org/jira/browse/HDFS-16389
Project: Hadoop HDFS
Issue Type: Improvement
Components: benchmarks, namenode
Affects Versions: 2.9.2
Reporter: JiangHua Zhu
When using the NNThroughputBenchmark test to create a large number of
directories, some abnormal information will be prompted.
Here is the command:
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
hdfs://xxxx -op mkdirs -threads 30 -dirs 5000000
There are some exceptions here, such as:
21/12/20 10:25:00 INFO namenode.NNThroughputBenchmark: Starting benchmark:
mkdirs
21/12/20 10:25:01 INFO namenode.NNThroughputBenchmark: Generate 5000000 inputs
for mkdirs
21/12/20 10:25:08 ERROR namenode.NNThroughputBenchmark:
java.lang.ArrayIndexOutOfBoundsException: 20
at
org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextDirName(FileNameGenerator.java:65)
at
org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextFileName(FileNameGenerator.java:73)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$MkdirsStats.generateInputs(NNThroughputBenchmark.java:668)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 20
at
org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextDirName(FileNameGenerator.java:65)
at
org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextFileName(FileNameGenerator.java:73)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$MkdirsStats.generateInputs(NNThroughputBenchmark.java:668)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
at
org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
These messages appear because some parameters are incorrectly set, such as
dirsPerDir or filesPerDir.
When we see this log, this will make us have some questions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
