yujun777 commented on code in PR #66488:
URL: https://github.com/apache/doris/pull/66488#discussion_r3780850267
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java:
##########
@@ -548,6 +550,13 @@ public void updateTaskStatus(AnalysisInfo info,
AnalysisState taskState, String
if (MetricRepo.isInit) {
MetricRepo.COUNTER_STATISTICS_FAILED_ANALYZE_JOB.increase(1L);
}
+ // The job reached a terminal state: all tasks share the
job's
+ // partitionUpdateRows map, so clearing it here releases
the memory
+ // retained by every task record in the history at once.
The success
+ // path clears it inside updateTableStats.
+ if (job.partitionUpdateRows != null) {
+ job.partitionUpdateRows.clear();
Review Comment:
Fixed. The task-side writers now check the killed flag before writing to the
shared partitionUpdateRows map (BaseAnalysisTask.doPartitionTable,
OlapAnalysisTask.deleteNotExistPartitionStats), so a cancelled task stops
writing. AnalysisJob.taskFailed now also clears the shared map in a finally
block after all task cancels have been issued, releasing it even when the
terminal-state clear in updateTaskStatus missed in-flight writes. Added
AnalysisJobTest.testTaskFailedClearsSharedPartitionUpdateRows.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java:
##########
@@ -992,11 +1006,17 @@ public void handleKillAnalyzeJob(KillAnalyzeJobCommand
killAnalyzeJobCommand) th
return;
}
checkPriv(anyTask);
- logKilled(analysisJobInfoMap.get(anyTask.getJobId()));
+ AnalysisInfo job = analysisJobInfoMap.get(anyTask.getJobId());
+ logKilled(job);
for (BaseAnalysisTask taskInfo : analysisTaskMap.values()) {
taskInfo.cancel();
logKilled(taskInfo.info);
}
+ // The job reached a terminal state: all tasks share the job's
partitionUpdateRows
+ // map, so clearing it here releases the memory retained by every task
record.
+ if (job.partitionUpdateRows != null) {
+ job.partitionUpdateRows.clear();
Review Comment:
Fixed. The kill path relies on the same killed-flag guards on the task-side
writers, so tasks stop writing to the shared map once cancelled, and the clear
in handleKillAnalyzeJob no longer races with in-flight writes.
handleKillAnalyzeJob also now tolerates a job record evicted from
analysisJobInfoMap while still running (previously it would NPE); its shared
map is already cleared at eviction time. Added
AnalysisManagerTest.testHandleKillAnalyzeJobWithEvictedJobRecord.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java:
##########
@@ -567,22 +576,27 @@ public void updateTaskStatus(AnalysisInfo info,
AnalysisState taskState, String
@VisibleForTesting
public void updateTableStats(AnalysisInfo jobInfo) {
- TableIf tbl = StatisticsUtil.findTable(jobInfo.catalogId,
jobInfo.dbId, jobInfo.tblId);
- TableStatsMeta tableStats = findTableStatsStatus(tbl.getId());
- if (tableStats == null) {
- updateTableStatsStatus(new TableStatsMeta(jobInfo.rowCount,
jobInfo, tbl));
- } else {
- tableStats.update(jobInfo, tbl);
- logCreateTableStats(tableStats);
- }
- if (jobInfo.jobColumns != null) {
- jobInfo.jobColumns.clear();
- }
- if (jobInfo.partitionNames != null) {
- jobInfo.partitionNames.clear();
- }
- if (jobInfo.partitionUpdateRows != null) {
- jobInfo.partitionUpdateRows.clear();
+ // Clear the shared maps even when the stats update fails, so the job
and all its
+ // task records release their memory once the job reaches a terminal
state.
+ try {
+ TableIf tbl = StatisticsUtil.findTable(jobInfo.catalogId,
jobInfo.dbId, jobInfo.tblId);
+ TableStatsMeta tableStats = findTableStatsStatus(tbl.getId());
+ if (tableStats == null) {
+ updateTableStatsStatus(new TableStatsMeta(jobInfo.rowCount,
jobInfo, tbl));
+ } else {
+ tableStats.update(jobInfo, tbl);
+ logCreateTableStats(tableStats);
+ }
+ } finally {
+ if (jobInfo.jobColumns != null) {
+ jobInfo.jobColumns.clear();
+ }
+ if (jobInfo.partitionNames != null) {
+ jobInfo.partitionNames.clear();
+ }
+ if (jobInfo.partitionUpdateRows != null) {
+ jobInfo.partitionUpdateRows.clear();
Review Comment:
Fixed. replayCreateAnalysisJob now clears the shared partitionUpdateRows map
of the evicted job entry, so a job evicted while still running (whose
updateTaskStatus returns early on the job == null guard and never reaches the
terminal-state clear) releases its memory at eviction time. Added
AnalysisManagerTest.testReplayCreateAnalysisJobEvictionClearsSharedMap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
