SEZ9 opened a new pull request, #12688:
URL: https://github.com/apache/seatunnel/pull/12688
Closes #12685
### Benchmark
Per our convention of leading with benchmark numbers: this change cannot move
the L1 static smoke preset, and I would rather say so than report a
meaningless delta. `--diagnose` returns before any provider is constructed,
and
this PR adds one module plus one best-effort GET behind that same flag — no
prompt, no tool spec, no planner path is touched, so there is nothing for a
generation benchmark to measure. The run-to-run spread on identical code is
around ±5 tasks, which would swamp any effect even if one existed.
What this rests on instead is 27 new tests (271 total, up from 245 on the
parent branch), including mutation checks on the three assumptions below.
### Why
The rules added in #12631 all compare against zero or a fixed threshold. That
answers "did it break" and "is it stuck", but not the question people
actually
ask about a long-running CDC job — it was fine at first and now it lags.
Such a
job is `RUNNING`, both row counters keep moving and `restoreCount` is 0, so
every rule we have stays silent. Running `--diagnose` against exactly that
shape today produces one line:
```
Job 852 mysql-cdc-to-doris — RUNNING
[info] Running for 72h00m. (state=RUNNING)
```
Which restates what the user already knew.
### What
`GET /jobs/checkpoints/<jobId>` is the one place the engine publishes a
*series* rather than a current value, so the job can be compared against its
own recent past. Six rules read it:
| rule | severity | fires when |
|------|----------|-----------|
| `checkpoint_duration_growing` | warning | recent checkpoints take ≥2× what
earlier ones took |
| `checkpoint_state_growing` | info | recent state is ≥2× earlier state —
usually the cause of the above |
| `checkpoint_bound` | warning | a checkpoint takes ≥80% of the gap between
triggers |
| `checkpoint_never_completed` | warning | checkpoints are triggered but
`completed == 0` |
| `checkpoint_failing` | warning | `failed > 0`, with the engine's own
`failureReason` |
| `checkpoint_stalled` | error | one has been in progress > 120s, naming
unacknowledged subtasks |
Two of these answer different questions on purpose. A *trend* needs a
baseline,
so it only sees degradation still inside the retained window. A *ratio*
(`checkpoint_bound`) needs no baseline at all, which is what makes it work
for
the job that degraded yesterday and has been slow ever since — the common
case
by the time someone asks for help.
The same scenario now gets an actionable answer:
```
Job 852 mysql-cdc-to-doris — RUNNING
[warning] Pipeline 1 has 19 failed checkpoints. ... Last failure:
checkpoint expired before completing (failed=19)
[warning] Pipeline 1 checkpoints are getting slower: recently 48.0s
against 4.0s earlier in the retained
history. ... (median durationMillis 4000 -> 48000 over 16
checkpoints)
[warning] Pipeline 1 now spends almost all its time checkpointing: a
checkpoint takes 48.0s and one is
triggered every 1m00s. Raise checkpoint.interval, or find why
the snapshot is slow ...
(recent median duration 48000ms vs interval 60000ms)
[info] Running for 72h00m. (state=RUNNING)
[info] Pipeline 1 checkpoint state is growing: recently 120.0MiB against
12.0MiB earlier. ...
```
Still read-only, still no LLM call, so it works with no API key and no
provider
configured. The checkpoint read is best-effort on purpose: an engine without
the endpoint, or a cluster with the checkpoint monitor off, still gets the
rest
of the diagnosis rather than a failure.
### Three wire-format details the arithmetic depends on
Each of these fails *silently* — it yields a confident finding that is simply
backwards — so each has a dedicated regression test:
1. **`history` is newest-first.** `PipelineCheckpointOverview.addHistory`
uses
`history.addFirst(...)` and `CheckpointMonitorService.getHistory` sorts by
`triggerTimestamp` descending. Read in array order, every trend conclusion
inverts. Pinned from both directions: a history that is getting *faster*
must not be reported as slower.
2. **Failed checkpoints omit `durationMillis` entirely** (it is a `Long`, and
null fields are not serialized). Counted as zero they would halve the
recent
median and hide a real slowdown.
3. **Their `stateSize` is a primitive `long`**, so it arrives as a plausible
`0` rather than being absent. Counted in, growing state looks like
shrinking
state.
All three are handled by filtering to `status == "COMPLETED"` before any
duration or state arithmetic — but intervals are computed from *every* entry,
because a checkpoint that failed was still triggered on schedule.
### Honest limits
- `maxHistorySize` is 32 per pipeline (`SeaTunnelServer.java:202`), so at a
10s
interval the trend sees roughly five minutes, not three days. This is why
`checkpoint_bound` exists.
- Where the time *goes* (sink vs source) needs `/metrics/realtime`, which is
a
follow-up: `engine.observability.enabled` defaults to `false`, so that rule
has a prerequisite this one does not, and it deserves its own PR.
- The CDC snapshot→incremental transition is not exposed over REST at all
(`AssignerStatus` appears nowhere in `seatunnel-engine/`), so it can only
ever be a hint, never a rule. Not attempted here.
### Also in this PR
Printing findings with `markup=False`. This fixes a real display bug in
#12631: `"error"` is a style name in the CLI theme, so rich was consuming the
`[error]` severity prefix as a style tag. Severity reached the user by colour
alone — invisible in a pipe, in CI, or under `NO_COLOR`. It also stops an
engine failure message containing brackets from corrupting the output, which
is
newly reachable here because `failureReason` is passed through verbatim.
`_as_int` in `jobs.py` is renamed to `as_int` so a sibling module need not
import a private name, and `sort_findings` is extracted so findings from
several rule sets are ordered as one list — sorting each set separately would
put an info line from one above an error from another.
### Stack
This is third in a stack and the diff includes its parents:
- #12629 — one REST module (base)
- #12631 — the `--diagnose` flag (on #12629)
- **this PR** (on #12631)
Review only the last commit, `[Feature][seatunnel-cli] Diagnose a gradual
slowdown from checkpoint history`. I will rebase as the parents land.
### Tests
271 pass (245 on the parent branch). `ruff check .` — the exact command
`cli.yml` runs — passes. Mutation-verified that the ordering assumption is
caught by 5 tests, the `markup=False` fix by 3, and the "unknown `completed`
count is not zero" guard by 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
