Tanya-W commented on issue #11319:
URL: https://github.com/apache/doris/issues/11319#issuecomment-1793997817
We still encountered this issue on **version 2.0.2-rc03.**
We encountered a prolonged "get tableList write lock timeout" error during
continuous load, and accompanied by a warnning of "the report queue size
exceeds the limit".
jsatck info:
- load txn thread waiting for the `StampedLock.readLock` in
TabTetInvertedindex
```
"Thrift-server-pool-22" #452 daemon prio=5 os_prio=0 tid=0x0000fff30002c800
nid=0x1770e waiting on condition [0x0000ffee805fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000fff48cb0c010> (a
java.util.concurrent.locks.StampedLock)
at
java.util.concurrent.locks.StampedLock.acquireread(StampedLock.java:1286)
at
java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
at
org.apache.doris.catalog.TabletinvertedIndex.readLock(TabTetInvertedindex.java:105)
at
org.apache.doris.catalog.TabletInvertedIndex.getTabletMetaList(TabletInvertedIndex.java:365)
at
org.apache.doris.transaction.DatabaseTransactiongr.checkcommitStatus(DatabaseTransactiongr.java:457)
at
org.apache.doris.transaction.DatabaseTransactiongr.preCommitTransaction2PC(DatabaseTransactiongr.java:423)
at
org.apache.doris.transaction.Globaltransactiongr.precommitTransaction2PC(GlobalTransactionMgr.java:219)
at
org.apache.doris.transaction.GlobalTransactiongr.preCommitTransaction2PC(GlobalTransactiongr.java:204)
at
org.apache.doris.service.Frontendserviceimpl.loadTxnPrecommitImpl(FrontendserviceImpl.java:1376)
at
org.apache.doris.service.FrontendServiceImpl.loadTxnPreCommit(FrontendserviceImp1.java:1294)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImp1.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.doris.service.Feserver.lambda$start$0(FeServer.java:59)
at
org.apache.doris.service.Feserver$$Lambda$126/969432090.invoke(Unknown Source)
at com.sun.proxy.$Proxy35.loadTxnPrecommit(Unknown Source)
at
org.apache.doris.thrift.Frontendservice$processor$1oadTxnprecommit.getResult(Frontendservice.java:3092)
at
org.apache.doris.thrift.Frontendservice$Processor$loadTxnPreCommit.getResult(Frontendservice.java:3072)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolserver$workerProcess.run(TThreadPoolserver.java:250)
at
java.til.concurrent.ThreadPoolexecutor.runworker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolexecutor$worker.run(ThreadPoolexecutor.java:
624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x0000fff483d426c0> (a
java.util.concurrent.locks.ReentrantReadwriteLock$Fairsync)
- <0x0000fff4c343f4b8> (a
java.util.concurrent.ThreadPoolExecutor$worker)
```
- the readLock hold by tabletReport, and there are two thread info
```
"ForkJoinPoo1-1-worker-93" #7584 daemon prio=5 os_prio=0
tid=0x000ffee300c000 nid=0x4e51 in obiect.wait() [0x0000ffee827fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.object.wait(Native method)
- waiting on <0x0000fff55aad70f8> (a
java.util.stream.ForEachOps$ForEachTask)
at
java.util.concurrent.ForkjoinTask.interna7wait(forkJoinTask.java:311)
- locked <0x0000fff55aad70f8> (a
java.util.stream.ForEachops$ForeachTask)
at java.util.concurrent.ForkJoinpoo.awaitjoin(ForkJoinPool.java:2058)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:404)
at java.uti1.concurrent.ForkJoinTask.invoke(ForkjoinTask.java:734)
at
java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
at
java.util.stream.ForEachOps$ForEachOp.$OfRef.evaluateParallel(Foreachops.java:174)
at
java.util.stream.Abstractpipeline.evaluate(Abstractpipeline.java:233)
at
java.util.stream.Referencepipeline.forEach(Referencepipeline.java:418)
at
java.util.stream.Referencepipeline$Head.forEach(Referencepipeline.java:583)
at
org.apache.doris.catalog.TabletInvertedindex.lambda$tabletReport$1(TabletInvertedindex.java:141)
at
org.apache.doris.catalog.Tabletinvertedindex$$Lambda$952/131508762.run(Unknown
Source)
at
java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJointask.java:1386)
at java.util.concurrent.ForkJoinTask.doExec(ForkJointask.java:289)
at
java.util.concurrent.ForkJoinPool$workQueue.runTask(ForkJoinPool.java:1056)
at
java.util.concurrent.ForkJoinPool.runworker(ForkJoinPool.java:1692)
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers:
- None
```
```
"Thread-54" #90 daemon prio=5 os_prio=0 tid=0x0000fffd3ad32800 nid=0x13061
in obiect.wait() [0x0000fff37d0fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.object.wait(Native method)
- waiting on <0x0000fff554117610> (a
java.util.concurrent.ForkJoinTask$AdaptedRunnableAction)
at
java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:334)
- locked <0x0000fff554117610> (a
java.util.concurrent.ForkJoinTask$AdaptedRunnableAction)
at java.util.concurrent.ForkjoinTask.doJoin(ForkjoinTask.java:391)
at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:719)
at
org.apache.doris.catalog.TabletInvertedindex.tabletReport(TabletInvertedindex.java:328)
at
org.apache.doris.master.ReportHandler.tabletReport(ReportHandler.java:436)
at
org.apache.doris.master.ReportHandler.access$600(ReportHandler.java:109)
at
org.apache.doris.master.ReportHandler$ReportTask.exec(ReportHandler.java:272)
at
org.apache.doris.master.ReportHandler.runOneCycle(ReportHandler.java:1311)
at org.apache.doris.common.util.Daemon.run(Daemon.java:116)
Locked ownable synchronizers:
- None
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
