sd4324530 opened a new pull request, #4623: URL: https://github.com/apache/flink-cdc/pull/4623
## Purpose This PR refactors Flink CDC's multi-flink-version support into the shape used by Paimon and Fluss — **a generation compat layer + per-version thin modules + per-version release artifacts** — and completes guava self-governance along the way. The goal is to reduce the marginal cost of adding a Flink version from "a full build configuration round plus profile surgery" to "one thin module plus one CI line", so the support matrix can widen again. Full motivation, target architecture, the evidence behind each decision, and the open decision points (D1-D3) are tracked in the umbrella ticket [FLINK-41452](https://issues.apache.org/jira/browse/FLINK-41452). Quick summary of the problems being solved: - The support matrix keeps shrinking (3.1 supported 4 Flink versions; 3.6/3.7 only 1.20 + 2.2) because every added version needs its own full build setup; - Generation switching was scattered across 7 module-local Maven profiles, `flink-shaded-guava` package names drift with Flink (guava31 -> guava33), 30+ `org.apache.flink.*` shadow classes were never linked against each target Flink version, and main code reflected into Flink internals (`SinkWriterOperator` constructor probing, the private `StreamExecutionEnvironment.configuration` field). ## Change Log The PR contains six phase commits. Each is a self-contained, behavior-preserving change unit that can be reviewed — and reverted — on its own. The entries below map 1:1, in order, to the PR's commits; the up-to-date hashes are always visible in the Commits tab (kept out of this description intentionally, so rebases never leave it stale). ### Commit 1 — Phase 1: Guava self-governance - New `flink-cdc-shaded-guava` module: guava relocated to `org.apache.flink.cdc.shaded.guava.*` with a **version-less prefix** (the guava31 -> guava33 prefix drift cannot repeat); - One-shot replacement of 89+ files' `org.apache.flink.shaded.guava31.*` imports; root pom global dependency switch; dist fat jars shade the owned guava; - NOTICE/LICENSE updated per release rules; - Removes the dependency basis of the flink2 `copy-flink2-guava` surefire classpath hack (the hack itself goes in Commit 2). ### Commit 2 — Phase 2: Build hygiene - The root pom becomes the **single generation switch point**: the `flink1` (default) and `flink2` profiles centralize all generation-scoped properties (`flink.version`, compat artifactId, external connector versions, generation gating); - Deletes the 7 modules' local flink1/flink2 profiles, the runtime `flink2-exclude-flink1` neutralization profile, the connect-pom neutralizing declarations and the `copy-flink2-extra-libs` classpath hack; - starrocks/hudi produce no fat jars under `-Pflink2` (`connector.generation.skip`), reflected in CI and the README; doris stays on the 1.20-line external dependency (rationale in FLINK-41452). ### Commit 3 — Phase 3: Compat layer reorg - New `flink-cdc-flink/` aggregate: `flink-cdc-flink1-common` / `flink-cdc-flink2-common` (the two compat modules migrated and renamed, **zero class-content change**) plus `flink-cdc-flink-1.20` / `flink-cdc-flink-2.2` version thin modules (skeletons + exact-version UTs; they will host intra-generation minor differences); - Review focus: pure migration/rename — the diff should only show path and GAV changes. ### Commit 4 — Phase 4: Per-version dist modules - `flink-cdc-dist` splits into `flink-cdc-dist-flink-1.20` / `flink-cdc-dist-flink-2.2` (gated by `dist.<ver>.skip`: the inactive line's module builds inert, nothing installed or deployed); - Yarn deployment jar lookup switches to exact `-flink-<ver>` suffix matching (also fixes `findFirst` picking the wrong jar when multiple lines sit in one lib directory); - Release script adapted for the two-line invocation + transition-period legacy-named artifacts; - `.gitignore` gains `release-artifacts/` (the release script's transition-period output directory). ### Commit 5 — Phase 5: Per-version connector fat jars + 2.x e2e - The 9 SQL source connector fat jars become per-version artifacts (aggregate turns into a pure code module + `-flink-1.20` / `-flink-2.2` child modules, gated by `source.fatjar.*`); - The 4 pipeline connectors whose shaded external dependency lines fork across generations (kafka, paimon, elasticsearch, iceberg) get the same treatment (gated by `pipeline.fatjar.*`) — the other 11 pipeline connector fat jars were verified cross-generation equivalent by jar-level comparison and keep their single GAV; - Shade include wildcards fixed for the thin compat modules, e2e dependency-copy sections parameterized (`pipeline.fatjar.suffix`), 2.x-line jdbc ecosystem coordinates adapted (`flink-connector-jdbc-core` + split dialect jar); - New CI job `source_e2e_2_x`. ### Commit 6 — Phase 6: Version-sensitive surface convergence + docs - Main-code reflection into `org.apache.flink` internals **eliminated**: the three-segment `SinkWriterOperator` reflection -> per-generation `SinkWriterOperatorAdapter`s; the `StreamExecutionEnvironment.configuration` reflection -> per-generation `StreamExecutionEnvironmentAdapter` (same-package field read); the dead `MySqlSource` metricGroup reflection and the `PreTransformOperator` `Class.forName` probe -> direct calls; - Legacy `SinkFunction` / `SourceFunction` call sites (moved into the `legacy` subpackage by 2.x) routed through `StreamSinkAdapter` and `StreamExecutionEnvironmentAdapter.addSource`; the legacy `RuntimeContextAdapter` signature aligned to the minimal common interface; - New bytecode-scan guard `ReflectionGuardTest` (asserts an empty violation set for main code; UDF-by-classname instantiation exempt); - Bilingual docs refresh: README / connector overviews / DataStream packaging guidance / FAQ — GAV examples and naming notes in both EN and zh; - Note for reviewers: `AGENTS.md` is updated across Commits 1/3/4/5 — the AI-agent guidance file's repository-structure sections (dist layout, compat layer names, fat-jar gating, guava guidance) are kept in sync with each phase's structural change. ## Testing - **Dual-generation builds**: full `mvn clean install -DskipTests` and `-Pflink2` builds green on JDK 11; - **Unit tests**: full UTs per module on both generations. Pre-existing environment-only failures unrelated to this refactor (`CliFrontendTest`, `AiFunctionParserTest` — Windows line-ending format variance; the ES `*ITCaseTest` trio — Docker dependency) reproduce identically on the base commit and are not affected on Linux CI; - **Artifact equivalence**: every per-version fat jar was compared class-by-class (name + CRC32) against the pre-refactor artifact of its Flink line — class content identical (new-GAV pom entries and a `DummyDocs` placeholder class aside). Intermediate modules' cross-generation differences were dissected via javap constant pools and reduce to Flink annotation `@Retention` differences (2-byte delta, zero runtime impact); - **E2E**: covered by CI (no Docker on the dev machine) — `source_e2e` (1.20), the new `source_e2e_2_x` (2.2), plus the existing dual-generation pipeline e2e matrix; - **Repository gates**: spotless and checkstyle pass repo-wide. ## Documentation Impact - README: artifact naming notes (artifactId-suffix scheme), pipeline connector table, version matrix annotations; - docs (EN + zh mirrors): source/pipeline connector overview download links, DataStream packaging pom examples, FAQ artifact name examples; - **Not in this PR**: the tutorial pages (mysql-to-kafka, migrate-paimon-cdc-action, ...) still carry download tables in the old naming scheme and need a separate docs refresh (tracked as a follow-up). ## Compatibility - `@Public` APIs unchanged; - DataStream users' dependency coordinates unchanged (thin single-GAV artifacts, verified usable on both generations); - Zero runtime behavior change (class-level artifact equivalence certified); - Download-link continuity: transition-period releases also emit legacy-named files, by default for one minor cycle. ## Review Guide - Review in Commit 1 -> 6 order; each stands alone; - Commit 3 is a pure migration (verify "zero class-content change"); Commits 2/4 are pom/shell-only (verify the gating property wiring); Commits 5/6 carry the most code (verify the Adapter signature contract: same FQCN, identical self-declared member signatures across generations); - The jar-level comparison tooling and evidence are available on request during review. --- **AI disclosure**: this change was produced with AI-assisted tooling. Per the ASF Generative Tooling Guidance, the `Generated-by` trailer is included in the PR template and this work is disclosed here. *Generated-by: CodeBuddy (GLM-5.3)* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
