[
https://issues.apache.org/jira/browse/HBASE-23955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059164#comment-17059164
]
Duo Zhang commented on HBASE-23955:
-----------------------------------
This is easy to introduce too many open files in tests...
> Have test runs use less resources
> ---------------------------------
>
> Key: HBASE-23955
> URL: https://issues.apache.org/jira/browse/HBASE-23955
> Project: HBase
> Issue Type: Improvement
> Components: test
> Reporter: Michael Stack
> Priority: Major
>
> Our tests can create thousands of threads all up in the one JVM. Using less
> means less memory, less contention, and hopefully, likelier passes.
> I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to
> see what it does as it runs (this test puts up 4 clusters with replication
> between). It peaks at 2k threads. After some configuration and using less
> HDFS, can get it down to ~800 threads and about 1/2 the memory-used.
> (HDFS is a profligate offender. DataXceivers (Server and Client), jetty
> threads, Volume threads (async disk 'worker' then another for cleanup...),
> image savers, ipc clients -- new thread per incoming connection w/o bound (or
> reuse), block responder threads, anonymous threads, and so on. Many are not
> configurable or boundable or are hard-coded; e.g. each volume gets 4 workers.
> Biggest impact was to be had by downing the count of data nodes. TODO: a
> follow-on that turns down DN counts in all tests)
> I've been using Java Flight Recorder during this study. Here is how you get a
> flight recorder for the a single test run:
> {code:java}
> MAVEN_OPTS="
> -XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m"
> mvn test -Dtest=TestNamespaceReplicationWithBulkLoadedData
> -Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 {code}
> i.e. start recording on mvn launch, bound the size of the recording, and have
> the test run in the mvn context (DON'T fork). Useful is connecting to the
> running test at the same time from JDK Mission Control. We do the latter
> because the thread reporting screen is overwhelmed by the count of running
> threads and if you connect live, you can at least get a 'live threads' graph
> w/ count as the test progresses. Useful.
> When the test finishes, it dumps a .jfr file which can be opened in JDK MC.
> I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC
> Version 7, the non-commercial latest. Works pretty well.
> Let me put up a patch for tests that cuts down thread counts where we can.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
