[ https://issues.apache.org/jira/browse/IOTDB-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haiming Zhu reassigned IOTDB-4986: ---------------------------------- Sprint: Catalyst-Backlog (was: 2023-1-Storage) Assignee: Xinyu Tan (was: Haiming Zhu) > Too many IoTDB-DataNodeInternalRPC-Processor threads are open > ------------------------------------------------------------- > > Key: IOTDB-4986 > URL: https://issues.apache.org/jira/browse/IOTDB-4986 > Project: Apache IoTDB > Issue Type: Improvement > Components: mpp-cluster > Affects Versions: 0.14.0-SNAPSHOT > Reporter: 刘珍 > Assignee: Xinyu Tan > Priority: Critical > > m_1118_3d5eeae > 1. 启动3副本3C21D 集群 > 2. 顺序启动7Benchmark > 3. 某个节点的datanode IoTDB-DataNodeInternalRPC-Processor 线程会开的很多,2k+ > (慢慢会降下来),但是会偶现OOM > 2022-11-18 14:26:48,320 > [pool-22-IoTDB-DataNodeInternalRPC-Processor-374$20221118_062422_29227_16.1.0] > ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:234 - write locally > failed. TSStatus: TSStatus(code:506, subStatus:[]), message: null > 2022-11-18 14:29:44,568 [DataNodeInternalRPC-Service]{color:red}* ERROR > o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread > DataNodeInternalRPC-Service-40 > java.lang.OutOfMemoryError: unable to create native thread: possibly out of > memory or process/resource limits reached*{color} > at java.base/java.lang.Thread.start0(Native Method) > at java.base/java.lang.Thread.start(Thread.java:803) > at > java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) > at > java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354) > at > org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:155) > at > org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139) > at > org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258) > 2022-11-18 14:29:53,751 [ClientRPC-Service] ERROR > o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread > ClientRPC-Service-42 > java.lang.OutOfMemoryError: unable to create native thread: possibly out of > memory or process/resource limits reached > at java.base/java.lang.Thread.start0(Native Method) > at java.base/java.lang.Thread.start(Thread.java:803) > at > java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) > at > java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354) > at > org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:155) > at > org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:139) > at > org.apache.iotdb.commons.service.AbstractThriftServiceThread.run(AbstractThriftServiceThread.java:258) > 2022-11-18 14:30:11,736 [pool-6-IoTDB-Flush-4] ERROR > o.a.i.d.e.s.TsFileProcessor:1095 - root.test.g_0-6: > /data/iotdb/m_1118_3d5eeae/sbin/../data/datanode/data/unsequence/root.test.g_0/6/2538/1668752675355-5-0-0.tsfile > meet error when flushing a memtable, change system mode to error > java.lang.OutOfMemoryError: unable to create native thread: possibly out of > memory or process/resource limits reached > at java.base/java.lang.Thread.start0(Native Method) > at java.base/java.lang.Thread.start(Thread.java:803) > at > java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937) > at > java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354) > at > java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) > at > org.apache.iotdb.db.rescon.AbstractPoolManager.submit(AbstractPoolManager.java:56) > at > org.apache.iotdb.db.engine.flush.MemTableFlushTask.<init>(MemTableFlushTask.java:88) > at > org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.flushOneMemTable(TsFileProcessor.java:1082) > at > org.apache.iotdb.db.engine.flush.FlushManager$FlushThread.runMayThrow(FlushManager.java:108) > at > org.apache.iotdb.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:29) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > 2022-11-18 14:30:11,736 [pool-6-IoTDB-Flush-4] ERROR > o.a.i.c.e.HandleSystemErrorStrategy:37 - Unrecoverable error occurs! Change > system status to read-only because handle_system_error is > CHANGE_TO_READ_ONLY. Only query statements are permitted! > java.lang.RuntimeException: System mode is set to READ_ONLY > at > org.apache.iotdb.commons.enums.HandleSystemErrorStrategy.handle(HandleSystemErrorStrategy.java:37) > at > org.apache.iotdb.commons.conf.CommonConfig.handleUnrecoverableError(CommonConfig.java:282) > at > org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.flushOneMemTable(TsFileProcessor.java:1100) > at > org.apache.iotdb.db.engine.flush.FlushManager$FlushThread.runMayThrow(FlushManager.java:108) > at > org.apache.iotdb.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:29) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > 查看哪个节点线程增多,可以通过日志,如 > ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:193 - can't connect to > node TEndPoint(ip:172.16.2.15, port:9003) > 就去ip15上去找。 > 测试环境 > 同https://issues.apache.org/jira/browse/IOTDB-4901 -- This message was sent by Atlassian Jira (v8.20.10#820010)