github-actions[bot] commented on code in PR #62282:
URL: https://github.com/apache/doris/pull/62282#discussion_r3063405800
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java:
##########
@@ -447,21 +447,31 @@ ExecutorFactory selectInsertExecutorFactory(
if (getLogicalQuery().containsType(InlineTable.class)) {
jobId = -1;
}
+ // Do not register internal group commit loads to
LoadManager.
+ // Internal group commit uses 'SELECT * FROM
group_commit(...)' as source,
+ // and the actual commit is managed by BE group commit
mechanism.
+ // These jobs will never transition to a completed state
through FE,
+ // causing a memory leak if registered.
+ if (getAllTVFRelation().stream().anyMatch(
+ rel ->
"group_commit".equals(rel.getFunctionName()))) {
+ jobId = -1;
+ }
if (targetTableIf instanceof RemoteDorisExternalTable) {
executorFactory = ExecutorFactory.from(
planner,
dataSink,
physicalSink,
() -> new RemoteOlapInsertExecutor(
- ctx, (RemoteOlapTable) olapTable,
label, planner, insertCtx, emptyInsert, jobId)
+ ctx, (RemoteOlapTable) olapTable,
label, planner, insertCtx, emptyInsert,
+ jobId, jobId != -1)
Review Comment:
`RemoteOlapInsertExecutor` still leaks `InsertLoadJob`s here. `needRegister
= (jobId != -1)` means the constructor will still call
`ctx.getEnv().getLoadManager().addLoadJob(...)` on the local FE, but the remote
executor only ever finishes jobs through
`remoteCatalog.getFeServiceClient().recordFinishedLoadJob(...)`. That RPC runs
on the remote FE, and `FrontendServiceImpl.recordFinishedLoadJobRequestImpl()`
currently records with `jobId = -1`, so it does not complete the locally
registered job. Also, `executeSingleInsert()` returns from the catch block
before `afterExec()`, so remote failures still never record completion at all.
As a result, remote OLAP inserts with `jobId != -1` remain stuck in `PENDING`,
so the leak is still present on this parallel OLAP path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
