goutamadwant opened a new issue, #12484:
URL: https://github.com/apache/seatunnel/issues/12484
### Search before asking
I searched open issues and PRs. #11513 lists "split docs, UI,
dependency-file, workflow-control-plane, API-contract, and engine flags".
#11515 does docs/UI/dependency but leaves `seatunnel-transforms-v2/**` in
`api`. No open PR changes transform routing.
### Summary
`seatunnel-transforms-v2/**` and
`seatunnel-e2e/seatunnel-transforms-v2-e2e/**` are part of the `api` glob
(`backend.yml:261` on dev). A PR that changes only a transform gets the full
matrix:
- unit tests on the whole reactor × 4 legs;
- all 7 `all-connectors-it` shards, all JDBC parts, every dedicated
connector job, engine-v2-it, engine-k8s-it, website and UI;
- each connector E2E on 7 engines.
That is 83 job legs and about 4,500–5,800 runner-minutes per push (5 sampled
full fork runs, 2026-09-22..24).
### Evidence
- Of 400 PRs merged 2026-07-23 → 2026-09-24, 90 took the `api` route. For 14
of those 90, transforms were the only reason: every `api`-matched file was
under the two transform paths, and no engine file changed. Examples: #12215,
#12193, #11994, #11951, #11937, #11920, #11918, #11855, #11763, #11724, #11712,
#11697. Most change 2–8 files.
- Transform coverage itself sits in `transform-v2-it-part-1/2` (and
`ExampleUdfIT`, which no job selects today; #12474 adds it). The connector
matrix exercises transforms only incidentally:
`ContainerUtil.copySeaTunnelStarterToContainer` copies the transforms jar into
every engine container.
- Modules that declare `seatunnel-transforms-v2` as a dependency: the
starters, `seatunnel-dist`, `seatunnel-e2e-common`, the engine E2E bases, and
some connector E2E modules (Kafka, RocketMQ, Elasticsearch, CDC).
### Proposal
1. Take the two transform paths out of the `api` glob. Add a `transforms`
flag for them.
2. When `transforms=true` (and `api`/`engine` are false), run:
- unit tests for `seatunnel-transforms-v2` and its reactor dependents
(`-pl seatunnel-transforms-v2 -amd`, which uses the #11515 impacted-module
path);
- `transform-v2-it-part-1`, `transform-v2-it-part-2` and the UDF E2E on
the PR engine set;
- `engine-v2-it` as the Zeta canary;
- one fixed connector canary shard that loads transforms through a real
source and sink. For example, the `connector-fake`/`connector-assert` E2E plus
one Kafka or JDBC transform pipeline; the exact set is to be agreed here.
3. Keep `api=true` for changes to `seatunnel-api`, `seatunnel-core`,
`seatunnel-translation`, `seatunnel-e2e-common`, and so on, exactly as today.
4. The nightly keeps the full matrix on all engines.
### Expected effect [estimate]
A transform-only PR goes from about 83 legs to about 15 legs (4 unit, 4
transform IT, 2 engine-v2-it, 2–4 canary, plus gates). Runner time drops from
about 5,000 to roughly 600–900 minutes per push. Over the sampled window, that
saves about 14 × (pushes per PR) × ~4,000 runner-minutes.
### Risk and backstop
- A transform change that breaks a connector-specific pipeline, such as a
CDC schema-change path through a transform, would surface in the nightly
instead of the PR. Mitigations:
- the canary shard;
- a `full-ci` label maintainers can apply;
- the nightly, once it can finish (#12467).
- Before switching, I can replay the last 6 months of transform-only PRs
through the routing model and list any whose PR run had a connector-shard
failure the reduced route would not have run.
### Coordination
- It builds on #11515 (impacted unit tests, per-path flags).
- It is independent of #9976.
- The dev@ thread requested on #11513 should confirm the canary set, because
it changes what "Build" guarantees for transform PRs.
### Questions for dev@
- Which connector canary is representative enough for transforms?
- Should transform changes also run the full JDBC parts? Many transform bugs
are type conversions that JDBC E2E exercises.
### Required checks and coordination
- `Build` stays the single required check. This only changes which legs run
inside it on ordinary PRs; the nightly (and a maintainer-applied full-CI
option) keep full coverage, per the boundary set in #11513.
- Complementary to #9900 (merged CI optimization) and #11074 (translation
test dependencies pulling unrelated connector modules into incremental IT
builds); it does not change either.
- This is a proposal for the dev@ discussion requested in #11513; no
implementation before there is agreement.
### Are you willing to submit a PR?
- [x] Yes, once the direction is agreed on dev@.
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
