JayModha100 opened a new pull request, #13272:
URL: https://github.com/apache/seatunnel/pull/13272
### Purpose of this pull request
Related to #13137 (fixes the intermediate-queue metric ownership defect
reproduced there with FakeSource).
`getQueueCache` registers the queue counters on the metrics context of
whichever task of a group reaches the queue first. After a same-ID savepoint
restore, that owner can change, but the previous owner's context is reloaded
from the metrics snapshot with its old queue counters, so the group reports two
rows for one queue and `subtaskCount`, `queueCapacity` and the ratio
denominators drift upward.
This clears only the `IntermediateQueue*` metric keys when a task-local
context is restored in `TaskExecutionContext.getOrCreateMetricsContext` (via
`SeaTunnelMetricsContext.cleanIntermediateQueueMetrics`). Unrelated task
metrics and the snapshot store are untouched.
Out of scope: `--get_running_job_metrics` history still lists old rows and
is unchanged.
Not verified: the MySQL CDC variant in the issue (idle reader close), which
I did not run.
### Does this PR introduce _any_ user-facing change?
No API, config or documentation change. It does change
realtime-observability output: after a same-ID savepoint restore, the realtime
edges endpoint (`/metrics/realtime/jobs/<id>/edges`) no longer accumulates
stale queue rows.
- Previous behavior (current `dev`, df59bb4cd1): in my FakeSource run,
`subtaskCount` went 5 -> 9 and `queueCapacity` 10240 -> 18432 across 4 restores.
- New behavior: `subtaskCount` stays 5 and `queueCapacity` stays 10240
across 5 restores (table below).
- This is a fix against the unreleased `dev` branch; I did not test a
released version.
### How was this patch tested?
Unit tests:
- `SeaTunnelMetricsContextTest`: cleanup removes stale queue keys, preserves
unrelated ones, idempotent.
- `TaskExecutionContextMetricsRestoreTest`: ownership handoff across restore
and 5 alternating restores, BlockingQueue and Disruptor.
- `RealtimeMetricsServiceTest`: restore + aggregation end to end, including
a 5-group case (BlockingQueue only for the aggregation path).
Manual reproduction: FakeSource -> Sql -> Console, parallelism 5, streaming,
reused job ID, single node, JDK 17, same-ID savepoint stop/restore. Only
`subtaskCount` and `queueCapacity` are compared; queue fill and backpressure
depend on workload and are not compared across runs. Expected queue-bearing
groups: 5 (job config, parallelism 5; DAG metadata). The task-mapping endpoint
was not observed.
Setup: Before = runtime built from df59bb4cd1 (captured Nov 12). After = the
same runtime with only the `TaskExecutionContext` and `SeaTunnelMetricsContext`
classes replaced from this branch (captured Nov 13).
| Phase | Before subtaskCount / queueCapacity | After subtaskCount /
queueCapacity |
|---|---|---|
| Baseline | 5 / 10240 | 5 / 10240 |
| Restore 1 | 6 / 12288 | 5 / 10240 |
| Restore 2 | 8 / 16384 | 5 / 10240 |
| Restore 3 | 8 / 16384 | 5 / 10240 |
| Restore 4 | 9 / 18432 | 5 / 10240 |
| Restore 5 | not captured | 5 / 10240 |
<details><summary>Steps to reproduce</summary>
1. Start a local cluster with realtime observability enabled
(`engine.observability.enabled = true`, `bucket_ms = 5000`).
2. Submit with a fixed job ID:
`.\bin\seatunnel.cmd --config job.conf --async --set-job-id 13137`
3. Wait for RUNNING, then read
`http://localhost:8080/metrics/realtime/jobs/13137/edges`
4. Savepoint-stop: `.\bin\seatunnel.cmd -s 13137`
5. Restore: `.\bin\seatunnel.cmd --config job.conf --async --restore 13137`
6. Wait for RUNNING (about 20 s), read the edges endpoint again, and repeat
4-5.
Unit tests:
`./mvnw test -pl seatunnel-engine/seatunnel-engine-server
"-Dtest=TaskExecutionContextMetricsRestoreTest,RealtimeMetricsServiceTest,SeaTunnelMetricsContextTest"
-DfailIfNoTests=false`
</details>
<details><summary>Job config</summary>
```hocon
env {
job.mode = "STREAMING"
parallelism = 5
engine.observability.enabled = true
engine.observability.bucket_ms = 5000
engine.observability.retention_minutes = 3
}
source {
FakeSource {
parallelism = 5
split.num = 5
row.num = 100000000
split.read-interval = 100
schema {
fields {
id = "bigint"
name = "string"
}
}
}
}
transform {
Sql {
query = "SELECT id, name FROM dual"
}
}
sink {
Console {
parallelism = 5
}
}
```
</details>
<details><summary>Edges values, one bucket per phase (before / after), with
timestamps</summary>
BEFORE (runtime built from df59bb4cd1)
- Baseline (ts=1794462275000): subtaskCount 5, queueCapacity 10240,
queueSize 107, queueFillRatio 0.01044921875, bpRatio 0.0
- Restore 1 (ts=1794462490000): subtaskCount 6, queueCapacity 12288,
queueSize 10240, queueFillRatio 0.833333333333333, bpRatio 0.7893262
- Restore 2 (ts=1794462665000): subtaskCount 8, queueCapacity 16384,
queueSize 9799, queueFillRatio 0.59808349609375, bpRatio 0.466015115
- Restore 3 (ts=1794462730000): subtaskCount 8, queueCapacity 16384,
queueSize 10225, queueFillRatio 0.62408447265625, bpRatio 0.3460710625
- Restore 4 (ts=1794463325000): subtaskCount 9, queueCapacity 18432,
queueSize 10240, queueFillRatio not recorded, bpRatio 0.35648457555555557
AFTER (same runtime with the two patched classes)
- Baseline (ts=1794567570000): subtaskCount 5, queueCapacity 10240,
queueSize 10240, queueFillRatio 1.0, bpRatio 0.80672334
- Restore 1 (ts=1794567895000): subtaskCount 5, queueCapacity 10240,
queueSize 10240, queueFillRatio 1.0, bpRatio 0.665410488
- Restore 2 (ts=1794568210000): subtaskCount 5, queueCapacity 10240,
queueSize 9200, queueFillRatio 0.8984375, bpRatio 0.51802198
- Restore 3 (ts=1794568260000): subtaskCount 5, queueCapacity 10240,
queueSize 10240, queueFillRatio 1.0, bpRatio 0.455696756
- Restore 4 (ts=1794568315000): subtaskCount 5, queueCapacity 10240,
queueSize 10240, queueFillRatio 1.0, bpRatio 0.650740108
- Restore 5 (ts=1794568395000): subtaskCount 5, queueCapacity 10240,
queueSize 10184, queueFillRatio 0.99453125, bpRatio 0.386733668
</details>
<details><summary>Raw edges response examples</summary>
#### BEFORE, Restore 4
```json
{
"toMs": 1794414511414,
"fromMs": 1794414331414,
"edges": [
{
"queueId": -7,
"targetVertexId": 3,
"points": [
{
"queueSize": 10240,
"subtaskCount": 7,
"queueFillRatio": 0.7142857142857143,
"queueCapacity": 14336,
"emitBlockedNs": 0,
"ts": 1794414335000,
"bpRatio": 0.0
},
{
"queueSize": 0,
"subtaskCount": 8,
"queueFillRatio": 0.0,
"queueCapacity": 16384,
"emitBlockedNs": 74880170400,
"ts": 1794414385000,
"bpRatio": 1.87200426
},
{
"queueSize": 9238,
"subtaskCount": 8,
"queueFillRatio": 0.5638427734375,
"queueCapacity": 16384,
"emitBlockedNs": 8191725400,
"ts": 1794414390000,
"bpRatio": 0.204793135
},
{
"queueSize": 9584,
"subtaskCount": 8,
"queueFillRatio": 0.5849609375,
"queueCapacity": 16384,
"emitBlockedNs": 12229169800,
"ts": 1794414395000,
"bpRatio": 0.305729245
},
{
"queueSize": 10240,
"subtaskCount": 8,
"queueFillRatio": 0.625,
"queueCapacity": 16384,
"emitBlockedNs": 11967576400,
"ts": 1794414400000,
"bpRatio": 0.29918941
},
{
"queueSize": 9806,
"subtaskCount": 8,
"queueFillRatio": 0.5985107421875,
"queueCapacity": 16384,
"emitBlockedNs": 12620156700,
"ts": 1794414405000,
"bpRatio": 0.3155039175
},
{
"queueSize": 10062,
"subtaskCount": 8,
"queueFillRatio": 0.6141357421875,
"queueCapacity": 16384,
"emitBlockedNs": 12493219700,
"ts": 1794414410000,
"bpRatio": 0.3123304925
},
{
"queueSize": 9276,
"subtaskCount": 8,
"queueFillRatio": 0.566162109375,
"queueCapacity": 16384,
"emitBlockedNs": 12490559900,
"ts": 1794414415000,
"bpRatio": 0.3122639975
},
{
"queueSize": 9609,
"subtaskCount": 8,
"queueFillRatio": 0.58648681640625,
"queueCapacity": 16384,
"emitBlockedNs": 12588483800,
"ts": 1794414420000,
"bpRatio": 0.314712095
},
{
"queueSize": 1599,
"subtaskCount": 8,
"queueFillRatio": 0.09759521484375,
"queueCapacity": 16384,
"emitBlockedNs": 8815012300,
"ts": 1794414425000,
"bpRatio": 0.2203753075
},
{
"queueSize": 17,
"subtaskCount": 9,
"queueFillRatio": 0.0009223090277777778,
"queueCapacity": 18432,
"emitBlockedNs": 9912858900,
"ts": 1794414480000,
"bpRatio": 0.22028575333333333
},
{
"queueSize": 10073,
"subtaskCount": 9,
"queueFillRatio": 0.5464952256944444,
"queueCapacity": 18432,
"emitBlockedNs": 7131898400,
"ts": 1794414485000,
"bpRatio": 0.1584866311111111
},
{
"queueSize": 10240,
"subtaskCount": 9,
"queueFillRatio": 0.5555555555555556,
"queueCapacity": 18432,
"emitBlockedNs": 11912727200,
"ts": 1794414490000,
"bpRatio": 0.2647272711111111
},
{
"queueSize": 9746,
"subtaskCount": 9,
"queueFillRatio": 0.5287543402777778,
"queueCapacity": 18432,
"emitBlockedNs": 13606112500,
"ts": 1794414495000,
"bpRatio": 0.30235805555555556
},
{
"queueSize": 9529,
"subtaskCount": 9,
"queueFillRatio": 0.5169813368055556,
"queueCapacity": 18432,
"emitBlockedNs": 13152952300,
"ts": 1794414500000,
"bpRatio": 0.2922878288888889
},
{
"queueSize": 10186,
"subtaskCount": 9,
"queueFillRatio": 0.5526258680555556,
"queueCapacity": 18432,
"emitBlockedNs": 11203698500,
"ts": 1794414505000,
"bpRatio": 0.24897107777777777
},
{
"queueSize": 9918,
"subtaskCount": 9,
"queueFillRatio": 0.5380859375,
"queueCapacity": 18432,
"emitBlockedNs": 12515962100,
"ts": 1794414510000,
"bpRatio": 0.2781324911111111
}
]
}
],
"bucketMs": 5000
}
```
#### AFTER, Restore 5
```json
{
"toMs": 1794568401226,
"fromMs": 1794568371226,
"edges": [
{
"queueId": -7,
"targetVertexId": 3,
"points": [
{
"queueSize": 0,
"subtaskCount": 5,
"queueFillRatio": 0.0,
"queueCapacity": 10240,
"emitBlockedNs": 0,
"ts": 1794568375000,
"bpRatio": 0.0
},
{
"queueSize": 10013,
"subtaskCount": 5,
"queueFillRatio": 0.97783203125,
"queueCapacity": 10240,
"emitBlockedNs": 1241846000,
"ts": 1794568380000,
"bpRatio": 0.04967384
},
{
"queueSize": 9943,
"subtaskCount": 5,
"queueFillRatio": 0.97099609375,
"queueCapacity": 10240,
"emitBlockedNs": 9858882100,
"ts": 1794568385000,
"bpRatio": 0.394355284
},
{
"queueSize": 9864,
"subtaskCount": 5,
"queueFillRatio": 0.96328125,
"queueCapacity": 10240,
"emitBlockedNs": 9848875100,
"ts": 1794568390000,
"bpRatio": 0.393955004
},
{
"queueSize": 10184,
"subtaskCount": 5,
"queueFillRatio": 0.99453125,
"queueCapacity": 10240,
"emitBlockedNs": 9668341700,
"ts": 1794568395000,
"bpRatio": 0.386733668
}
]
}
],
"bucketMs": 5000
}
```
</details>
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar added)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
(not necessary: bug fix, no new feature or config)
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
(not necessary: no incompatible change)
* [ ] If you are contributing the connector code, please check that the
following files are updated:
(not applicable: not a connector PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]