Marton Elek created HDDS-2325:
---------------------------------
Summary: BenchMarkDatanodeDispatcher genesis test is failing with
NPE
Key: HDDS-2325
URL: https://issues.apache.org/jira/browse/HDDS-2325
Project: Hadoop Distributed Data Store
Issue Type: New Feature
Reporter: Marton Elek
Assignee: Marton Elek
## What changes were proposed in this pull request?
Genesis is a microbenchmark tool for Ozone based on JMH
([https://openjdk.java.net/projects/code-tools/jmh/).]
Due to the recent Datanode changes the BenchMarkDatanodeDispatcher is failing
with NPE:
{code:java}
java.lang.NullPointerException
at
org.apache.hadoop.ozone.container.common.interfaces.Handler.<init>(Handler.java:69)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.<init>(KeyValueHandler.java:114)
at
org.apache.hadoop.ozone.container.common.interfaces.Handler.getHandlerForContainerType(Handler.java:78)
at
org.apache.hadoop.ozone.genesis.BenchMarkDatanodeDispatcher.initialize(BenchMarkDatanodeDispatcher.java:115)
at
org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest._jmh_tryInit_f_benchmarkdatanodedispatcher0_G(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:438)
at
org.apache.hadoop.ozone.genesis.generated.BenchMarkDatanodeDispatcher_createContainer_jmhTest.createContainer_Throughput(BenchMarkDatanodeDispatcher_createContainer_jmhTest.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
at
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
And this is the just the biggest problem there are a few other problems. I
propose the following fixes:
*fix 1*: NPE is thrown because the 'context' object is required by
KeyValueHandler/Handler classes.
In fact the context is not required, we need two functionalities/info from the
context: the ability to send icr (IncrementalContainerReport) and the ID of the
datanode.
Law of Demeter principle suggests to have only the minimum required information
from other classes.
For example instead of having context but using only
context.getParent().getDatanodeDetails().getUuidString() we can have only the
UUID string which makes more easy to test (unit and benchmark) the
Handler/KeyValueHandler.
This is the biggest (but still small change) in this patch: I started to use
the datanodeId and an icrSender instead of having the full context.
*fix 2,3:* There were a few other problems. The scmId was missing if the
writeChunk was called from Benchmark and and the Checksum was also missing.
*fix 4:* I also had a few other problems: very huge containers are used
(default 5G) and as the benchmark starts with creating 100 containers it
requires 500G space by default. I adjusted the container size to make it
possible to run on local machine.
## How this patch can be tested?
{code:java}
./ozone genesis -benchmark=BenchMarkDatanodeDispatcher.writeChunk{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
