[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083792#comment-17083792 ] Michael Stack commented on HBASE-24072: --- Found that HBASE-23956 Use less resources running tests (#1266) was missing off branch-2.3. Added it (and the correction HBASE-23987 NettyRpcClientConfigHelper will not share event loop by default which is incorrect (#1288)). Lets see if that helps. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Task > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > print_ulimit.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083492#comment-17083492 ] Michael Stack commented on HBASE-24072: --- Ran my little TestPrintEnv which outputs ulimits seen inside container and it seems to show right numbers... {code} core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 192962 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 12500 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited {code} > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Task > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > print_ulimit.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077700#comment-17077700 ] Hudson commented on HBASE-24072: Results for branch branch-2 [build #2589 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2589/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2589/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2589/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2589/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2589/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Task > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > print_ulimit.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077299#comment-17077299 ] Michael Stack commented on HBASE-24072: --- HBASE-24126 is about upping container ulimit. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > print_ulimit.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076651#comment-17076651 ] Michael Stack commented on HBASE-24072: --- Trying to up the container nproc limit on the container we use building in sub-task. Tried to do it as part of https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1426/14/ but the 'docker run --ulimit' didn't seem to take. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > print_ulimit.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076489#comment-17076489 ] Michael Stack commented on HBASE-24072: --- You can now see that INFRA-20074 has been implemented after HBASE-24122 went in; see the ulimit-a file under 'machine' in 'Build Artifacts'. Related, one thing I learned is that containers do not inherit the ulimit of their host. In particular, ulimit is per user, not per container. For nproc, the number of processes/threads, its interesting. Our containers said the max procs was 1. The hosts max before INFRA-20074 was 1 so if two containers were running on the one machine, this would cap our thread count. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > print_ulimit.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074985#comment-17074985 ] Michael Stack commented on HBASE-24072: --- Filed INFRA-20074 to see if can get process/threads upped on the hadoop build hosts. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > print_ulimit.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074882#comment-17074882 ] Michael Stack commented on HBASE-24072: --- Attached patch for printing out jenkins ulimit and for trying out how many threads we can create up to limit. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > print_ulimit.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074868#comment-17074868 ] Michael Stack commented on HBASE-24072: --- One particular test studied runs thread count up to ~1800. When 0.5C for fork count, on a jenkins box, that could be 8 tests running in parallel. If a bunch of heavy hitters launch around same time, could brush up against the 10k limit with 'OOME Can't create native thread' as the result. So tests parallelization is bounded by thread max and by container max memory. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > run_ulimit_a_and_see_how_many_threads_can_create.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074857#comment-17074857 ] Michael Stack commented on HBASE-24072: --- Was able to get ulimit and test how many threads I could create. We are bound to 10k processes/threads which is borne out by out thread creation script (it fails at 9753...). {code} core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 192962 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited opendfs: 81 maxfds: 1048576 New thread #1 New thread #2 New thread #3 New thread #4 ... New thread #9752 New thread #9753 Standard Error New thread #9753 > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > run_ulimit_a_and_see_how_many_threads_can_create.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074592#comment-17074592 ] Hudson commented on HBASE-24072: Results for branch master [build #1687 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1687/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1687/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1687/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1687/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://builds.apache.org/job/HBase%20Nightly/job/master/1687/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > run_ulimit_a_and_see_how_many_threads_can_create.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074559#comment-17074559 ] Hudson commented on HBASE-24072: Results for branch branch-2.3 [build #15 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/15/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/15/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/15/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/15/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/15/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > run_ulimit_a_and_see_how_many_threads_can_create.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074261#comment-17074261 ] Michael Stack commented on HBASE-24072: --- HBASE-23779 is the issue that changed the forkcount to 0.5C for first and second part (Duo says on end of issue: "...0.5C seems a bit aggresive? And I think we should also take care of memory?..."). Let me push the 0.25C everywhere since it calms things down, till fix the OOME can't create native threads. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > run_ulimit_a_and_see_how_many_threads_can_create.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074189#comment-17074189 ] Michael Stack commented on HBASE-24072: --- Put up a patch to see how many threads can create on hadoop host and to dump ulimit to see what it is like on the patch test machines. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch, > run_ulimit_a_and_see_how_many_threads_can_create.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074171#comment-17074171 ] Michael Stack commented on HBASE-24072: --- All tests passed w/ this in place. No OOME can't create native thread complaint. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073297#comment-17073297 ] Hudson commented on HBASE-24072: Results for branch branch-2 [build #2578 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2578/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2577/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (x) {color:red}-1 client integration test{color} --Failed when running client tests on top of Hadoop 2. [see log for details|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2578//artifact/output-integration/hadoop-2.log]. (note that this means we didn't run on Hadoop 3) > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073236#comment-17073236 ] Michael Stack commented on HBASE-24072: --- I pushed 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch on branch-2. Sets forkcount down from 0.5C to 0.25C for both first and second parts. Lets see how it does. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-24072-Nightlies-reporting-OutOfMemoryError-una.patch > > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073234#comment-17073234 ] Michael Stack commented on HBASE-24072: --- Let me down the parallelism and see if that helps; i.e. 0.25C vs 0.5C for fork count as we currently have. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071900#comment-17071900 ] Michael Stack commented on HBASE-24072: --- I see now that the jdk11 profile sets the forked heap to 3200. Need to try and rein this in too. Would be sweet to run test suite serially through a monitor to see when heap spikes and then work on cutting down resource usage in the offending test... > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071118#comment-17071118 ] Michael Stack commented on HBASE-24072: --- 2400 seems fine for local runs. Was going to push it unless objection. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24072) Nightlies reporting OutOfMemoryError: unable to create new native thread
[ https://issues.apache.org/jira/browse/HBASE-24072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069197#comment-17069197 ] Michael Stack commented on HBASE-24072: --- Dang. I set it 2000 and ran it locally and started to see timeouts I'd not seen before. Trying 2400. > Nightlies reporting OutOfMemoryError: unable to create new native thread > > > Key: HBASE-24072 > URL: https://issues.apache.org/jira/browse/HBASE-24072 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Priority: Major > > Seeing this kind of thing in nightly... > {code} > java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new > native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at > org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper.beforeClass(TestMultithreadedTableMapper.java:83) > {code} > Chatting w/ Nick and Huaxiang, doing the math, we are likely oversubscribing > our docker container. It is set to 20G (The hosts are 48G). Fork count is > 0.5C on a 16 CPU machine which is 8 *2.8G our current forked jvm size. Add > the maven 4G and we could be over the top. > Play w/ downing the fork size (in earlier study we didn't seem to need this > much RAM when running a fat long test). Let me also take th ms off the mvn > allocation to see if that helps. -- This message was sent by Atlassian Jira (v8.3.4#803005)