liutang123 opened a new issue, #24982: URL: https://github.com/apache/doris/issues/24982
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? #24304 has supported group commit in stream load. the logic is as follows: ```GroupCommitMgr::group_commit_stream_load GroupCommitMgr::_group_commit_stream_load (async) GroupCommitMgr::_get_first_block_load_queue GroupCommitTable::get_first_block_load_queue GroupCommitTable::_create_group_commit_load GroupCommitTable::_exec_plan_fragment (The fragment will running asynchronized) ctx->promise.set_value // bug code ``` After `exec_plan_fragment`, the `promise` of `StreamLoadContext` will be set. And then, stream load will return. ```StreamLoadAction::handle StreamLoadAction::_handle RETURN_IF_ERROR(ctx->future.get()); if (ctx->group_commit) { LOG(INFO) << "skip commit because this is group commit, pipe_id=" << ctx->id.to_string(); return Status::OK(); } HttpChannel::send_reply ``` But, the fragment may not be over yet. Some logs: ``` I20230927 01:13:56.898559 152097 stream_load.cpp:201] new income streaming load request.id=0a47dd1e6d01557b-0b28a3db1d95718c, job_id=-1, txn_id=-1, label=, elapse(s)=0, db=regression_test_load_p0_stream_load, tbl=test_stream_load_lineorder I20230927 01:14:11.026175 152097 stream_load.cpp:158] skip commit because this is group commit, pipe_id=0a47dd1e6d01557b-0b28a3db1d95718c I20230927 01:14:16.507047 151039 vtablet_writer.cpp:1563] total mem_exceeded_block_ns=0, total queue_push_lock_ns=4300092, total actual_consume_ns=2225691016, load id=d34539309d7a549d-e4eb52f4536dac81 ``` Stream load finished in 01:14:11.026175, but the fragment close in 01:14:16.507047. This bug may cause p0 pipeline `test_group_commit_stream_load` fail. Some logs: ``` 2023-09-27 01:14:11.031 INFO [suite-thread-9] (null:-1) - Stream load 0, result: { "TxnId": 6939, "Label": "group_commit_d34539309d7a549d_e4eb52f4536dac81", "Comment": "", "GroupCommit": true, "Status": "Success", "Message": "OK", "NumberTotalRows": 596508, "NumberLoadedRows": 596508, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 19362295, "LoadTimeMs": 14128, "StreamLoadPutTimeMs": 5, "ReadDataTimeMs": 10523, "WriteDataTimeMs": 0 } 2023-09-27 01:14:11.037 ERROR [suite-thread-9] (ScriptContext.groovy:121) - Run test_group_commit_stream_load in /home/work/pipline/OpenSourceDoris/clusterRegressionCenter/P0/Cluster0/regression-test/suites/load_p0/stream_load/test_group_commit_stream_load.groovy failed org.opentest4j.AssertionFailedError: expected: <596508> but was: <600572> at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62) at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182) at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177) at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:551) at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1529) at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1003) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:180) at org.apache.doris.regression.suite.Suite.invokeMethod(Suite.groovy:650) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:397) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:339) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:819) at groovy.lang.GroovyObject.invokeMethod(GroovyObject.java:39) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:413) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:339) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:819) at groovy.lang.GroovyObject.invokeMethod(GroovyObject.java:39) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:397) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:337) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:194) at test_group_commit_stream_load$_run_closure1$_closure12$_closure16.doCall(test_group_commit_stream_load.groovy:296) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035) ``` ### What You Expected? Stream load wait fragment done. Set `promise` of `StreamLoadContext` in fragment callback like `StreamLoadExecutor`. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
