[ https://issues.apache.org/jira/browse/HBASE-23956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057360#comment-17057360 ]
Hudson commented on HBASE-23956: -------------------------------- Results for branch branch-2 [build #2543 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2543/]: (x) *{color:red}-1 overall{color}* ---- details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2543//General_Nightly_Build_Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2543//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2543//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2543//JDK11_Nightly_Build_Report/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Use less resources running tests > -------------------------------- > > Key: HBASE-23956 > URL: https://issues.apache.org/jira/browse/HBASE-23956 > Project: HBase > Issue Type: Improvement > Components: test > Reporter: Michael Stack > Assignee: Michael Stack > Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: Screen Shot 2020-03-09 at 10.11.26 PM.png, Screen Shot > 2020-03-09 at 10.11.35 PM.png, Screen Shot 2020-03-09 at 10.20.08 PM.png, > Screen Shot 2020-03-10 at 7.10.15 AM.png, Screen Shot 2020-03-10 at 7.10.28 > AM.png > > > Our tests can create thousands of threads all up in the one JVM. Using less > means less memory, less contention, likelier passes, and later, more possible > parallelism. > I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to > see what it does as it runs (this test puts up 4 clusters with replication > between). It peaks at 2k threads. After some configuration and using less > HDFS, its possible to get it down to ~800 threads and about 1/2 the > memory-used. HDFS is a main offender. DataXceivers (Server and Client), jetty > threads, Volume threads (async disk 'worker' then another for cleanup...), > image savers, ipc clients -- new thread per incoming connection w/o bound (or > reuse), block responder threads, anonymous threads, and so on. Many are not > configurable or boundable or are hard-coded; e.g. each volume gets 4 workers > regardless. Biggest impact was just downing the count of data nodes. TODO: a > follow-on that turns down DN counts in all tests. > I've been using Java Flight Recorder during this study. Here is how you get a > flight recorder for the a single test run: \{code:java} MAVEN_OPTS=" > -XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m" > mvn test -Dtest=TestNamespaceReplicationWithBulkLoadedData > -Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 \{code} i.e. > start recording on mvn launch, bound the size of the recording, and have the > test run in the mvn context (DON'T fork). Useful is connecting to the running > test at the same time from JDK Mission Control. We do the latter because the > thread reporting screen is overwhelmed by the count of running threads and if > you connect live, you can at least get a 'live threads' graph w/ count as the > test progresses. Useful. When the test finishes, it dumps a .jfr file which > can be opened in JDK MC. > I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC > Version 7, the non-commercial latest. Works pretty well. Let me put up a > patch for tests that cuts down thread counts where we can. > Let me put up a patch that does first pass on curtailing resource usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)