DanielLeens commented on issue #11806:
URL: https://github.com/apache/seatunnel/issues/11806#issuecomment-5295769212
I re-verified this against the current `dev` head (`b1e21631ff`), and I have
to correct my own report: the gap described above no longer exists. My analysis
was run on an outdated local checkout, and the cleanup path was already
reworked by #10757 (merged 2026-07-21). Apologies for the noise.
Current state on `dev`:
1. `JobMaster.removeMetricsContext(...)` covers the `FAILED` terminal state
in addition to `FINISHED` (non-savepoint end) and `CANCELED`, and delegates to
`SeaTunnelServer.removeMetrics(...)`.
2. `SubPlan.subPlanDone(...)` also calls
`JobMaster.enqueuePipelineCleanupIfNeeded(...)`, so a `FAILED` pipeline
additionally enters the pending-cleanup queue
(`IMAP_PENDING_PIPELINE_CLEANUP`). `CoordinatorService` drains that queue on a
fixed-rate scheduler and keeps the cleanup record until the pipeline's task
groups are actually cleaned, which also covers late metrics writes racing with
the removal.
3. `HazelcastMetricsSnapshotStateStore.removePipeline(...)` removes the
pipeline's `TaskLocation` entries per partition via `IMap.compute`, without the
old full-map scan.
4. The test coverage requested in this issue already exists:
-
`JobMasterTest#testFailedPipelineCleanupEnqueuesRecordAndRemovesMetrics` (seeds
metrics for a running pipeline, triggers the `FAILED` path, and asserts the
metrics map no longer contains the pipeline's entries)
-
`CoordinatorServicePipelineCleanupTest#testCleanupRemovesMetricsAndRecordForFailedPipeline`,
plus `FINISHED`/`CANCELED` and restore-invalidation variants
So on current `dev`, `FAILED` pipelines no longer retain entries in
`IMAP_RUNNING_JOB_METRICS`, and no additional fix PR is needed for this report.
Closing as already fixed by #10757. If anyone can still reproduce a leak on a
build that includes #10757, please reopen with the observed map contents and
the engine version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
