Daryn Sharp created HDFS-10743:
----------------------------------
Summary: MiniDFSCluster test runtimes can be drastically reduce
Key: HDFS-10743
URL: https://issues.apache.org/jira/browse/HDFS-10743
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 2.0.0-alpha
Reporter: Daryn Sharp
{{MiniDFSCluster}} tests have excessive runtimes. The main problem appears to
be the heartbeat interval. The NN may have to wait up to 3s (default value)
for all DNs to heartbeat, triggering registration, so NN can go active. Tests
that repeatedly restart the NN are severely affected.
Example for varying heartbeat intervals for {{TestFSImageWithAcl}}:
* 3s = ~70s -- (disgusting, why I investigated)
* 1s = ~27s
* 500ms = ~17s -- (had to hack DNConf for millisecond precision)
That a 4x improvement in runtime.
17s is still excessively long for what the test does. Further areas to explore
when running tests:
* Reduce numerous sleeps intervals in DN's {{BPServiceActor}}.
* Ensure heartbeats and initial BR are sent immediately upon (re)registration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
