nzw921rx commented on PR #13283:
URL: https://github.com/apache/seatunnel/pull/13283#issuecomment-6907642888
> Thanks for working on this. I reviewed the latest head `f1586236a2f3`.
>
> # What this PR fixes
> - User pain: the shaded `seatunnel-starter.jar` bundles a different
`netty-buffer`/`netty-common` depending on how it is built -- 4.1.118.Final in
the reactor (won through the *test-scoped* `seatunnel-transforms-v2` -> AWS SDK
2.31.30 path) versus 4.1.104.Final in a standalone build against installed
dependency-reduced POMs (won through `seatunnel-common` ->
`seatunnel-shade-arrow` / Arrow 15.0.1), so the starter jar is not reproducible
across build modes.
> - Fix approach: adds a module-local
`<netty.version>4.1.118.Final</netty.version>` property
(`seatunnel-core/seatunnel-starter/pom.xml:32`) and a `<dependencyManagement>`
block (`pom.xml:36-49`) that pins only `io.netty:netty-buffer` and
`io.netty:netty-common`; no root-POM, `seatunnel-dist`, or code change.
> - One-line summary: the pin does make the shaded starter jar
deterministic, but it is a hand-copied snapshot of what the AWS SDK brings
(root `pom.xml:207`), is decoupled from both producers (root `pom.xml:208`
Arrow 15.0.1), manages only two Netty modules, introduces a fourth differently
named property (`netty.version`) next to the three existing
`netty-buffer.version` pins (imap-storage-file 4.1.60, e2e-base 4.1.89,
`seatunnel-dist/pom.xml:149` 4.1.89), and is moot in the shipped CI
distribution where `assembly-seatunnel-bin-ci.xml:239-240` copies
netty-buffer/netty-common into `lib/`, which `seatunnel-cluster.sh:221`
(`CLASS_PATH=${APP_DIR}/lib/*:${APP_JAR}`) places ahead of the starter jar --
so Zeta still loads a different Netty than the one this PR pins, and nothing in
the build (`tools/dependencies/known-dependencies.txt` still lists both 4.1.104
and 4.1.118) guards the new invariant.
>
> # Runtime chain I rechecked
> ```text
> Maven dependency mediation for the seatunnel-starter shaded jar
(build-time only; no runtime code changed)
> seatunnel-core/seatunnel-starter/pom.xml:36-49 <dependencyManagement>
io.netty:netty-buffer / netty-common -> ${netty.version}
> seatunnel-core/seatunnel-starter/pom.xml:32
<netty.version>4.1.118.Final</netty.version>
> overrides nearest-wins resolution from:
> seatunnel-starter -> seatunnel-transforms-v2 (test scope,
pom.xml:103-108) -> software.amazon.awssdk:*
(seatunnel-transforms-v2/pom.xml:51-58; root pom.xml:207, 2.31.30) -> netty
4.1.118.Final [reactor winner]
> seatunnel-starter -> seatunnel-api/engine -> seatunnel-common
(seatunnel-common/pom.xml:64 seatunnel-shade-arrow) -> arrow-memory-netty (root
pom.xml:662, arrow.version 15.0.1 at root pom.xml:208) -> netty 4.1.104.Final
[standalone/dependency-reduced-POM winner]
> maven-shade-plugin (seatunnel-starter/pom.xml:128-170; no io.netty
exclude) bundles the mediated netty-buffer/netty-common into
starter/seatunnel-starter.jar
>
> Runtime classpath in the CI distribution (unchanged by this PR; what Zeta
actually loads)
> seatunnel-dist/pom.xml:149
<netty-buffer.version>4.1.89.Final</netty-buffer.version>
> seatunnel-dist/pom.xml:1162-1167 io.netty:netty-buffer (provided) ->
${netty-buffer.version}
> seatunnel-dist/src/main/assembly/assembly-seatunnel-bin-ci.xml:239-240
include io.netty:netty-buffer:jar, io.netty:netty-common:jar -> lib/
> seatunnel-core/seatunnel-starter/src/main/bin/seatunnel-cluster.sh:221
CLASS_PATH=${APP_DIR}/lib/*:${APP_JAR} (lib/ shadows the starter jar's
4.1.118.Final)
> seatunnel-core/seatunnel-starter/src/main/bin/seatunnel-cluster.sh:39
APP_JAR=${APP_DIR}/starter/seatunnel-starter.jar
> ```
>
> # Findings
> **Issue 1: Hard-coded 4.1.118.Final in the starter POM is a detached copy
of the version the AWS SDK (root `software.amazon.awssdk.version` = 2.31.30)
happens to bring; the root POM manages no `io.netty` artifact, so the next
SDK/Arrow bump silently produces mixed Netty module versions on the starter
classpath**
> - Location: `seatunnel-core/seatunnel-starter/pom.xml:32`
> - Why it matters: - Root cause: the value `4.1.118.Final` on
`seatunnel-core/seatunnel-starter/pom.xml:32` is not derived from anything; it
mirrors what `software.amazon.awssdk:*:2.31.30` (root `pom.xml:207`, pulled in
via the test-scoped `seatunnel-transforms-v2`,
`seatunnel-transforms-v2/pom.xml:51-58`) resolves today. The root
`<dependencyManagement>` has no `io.netty` entry at all (only
`arrow-memory-netty`, root `pom.xml:662`), and the three other Netty pins in
the repo are each module-local and different (`seatunnel-dist/pom.xml:149` =
4.1.89.Final, `imap-storage-file/pom.xml:38` = 4.1.60.Final,
`connector-seatunnel-e2e-base/pom.xml:33` = 4.1.89.Final).
> - Impact: the management block only covers `netty-buffer` and
`netty-common` (lines 38-47). When `software.amazon.awssdk.version` or
`arrow.version` is bumped, `netty-transport`/`netty-handler`/`netty-codec*` on
the starter test classpath (and any future compile-scope Netty consumer in
engine-server/engine-client) float to the new version while buffer/common are
forced back to 4.1.118 -- exactly the kind of cross-module Netty mismatch
(`NoSuchMethodError`/`AbstractMethodError` inside `io.netty.buffer`) that
surfaces at runtime, not compile time. Nothing in the root POM reveals that
this pin exists, so whoever does the bump has no reason to look here.
> - Fix: manage Netty once at the root instead of per module -- import
`io.netty:netty-bom:${netty.version}` (`<type>pom</type>`,
`<scope>import</scope>`) in the root `<dependencyManagement>` with a single
root `netty.version` property, and have seatunnel-dist, imap-storage-file and
e2e-base reference that property instead of their private
`netty-buffer.version`. If this change is deliberately starter-only, at minimum
import `netty-bom` here rather than listing two artifacts, and add a comment
tying the value to `software.amazon.awssdk.version` so the two are bumped
together.
> - Evidence: `seatunnel-core/seatunnel-starter/pom.xml:40
<artifactId>netty-buffer</artifactId>`;
`seatunnel-core/seatunnel-starter/pom.xml:45
<artifactId>netty-common</artifactId>`;
`seatunnel-core/seatunnel-starter/pom.xml:35 <!-- Use the same Netty version
in reactor and standalone builds. -->`
> - Suggested fix: This value is a hand-copied snapshot of what
`software.amazon.awssdk:*:${software.amazon.awssdk.version}` (2.31.30) resolves
today; the root POM has no `io.netty` management at all, and seatunnel-dist
(4.1.89), imap-storage-file (4.1.60) and e2e-base (4.1.89) each carry their own
`netty-buffer.version`. Managing only `netty-buffer`/`netty-common` here means
the next AWS SDK or Arrow bump will move
`netty-transport`/`netty-handler`/`netty-codec*` on this module's classpath
while pinning buffer/common back to 4.1.118 -- a mixed-version Netty that fails
at runtime. Could we instead import `io.netty:netty-bom` with a single
`netty.version` property in the root `<dependencyManagement>` and point the
other three modules at it? If this must stay starter-local, please at least
import `netty-bom` rather than two artifacts, and document that this value must
track `software.amazon.awssdk.version`.
> - Severity: Medium
>
> **Issue 2: The module-local pin only makes the shaded starter jar
deterministic; in the shipped distribution `lib/netty-buffer-4.1.89.Final.jar`
(from `seatunnel-dist`) is loaded ahead of `starter/seatunnel-starter.jar`, so
Zeta still runs a different Netty than the one this PR pins**
> - Location: `seatunnel-core/seatunnel-starter/pom.xml:36`
> - Why it matters: - Root cause: the new `<dependencyManagement>` block in
`seatunnel-core/seatunnel-starter/pom.xml:36-49` is scoped to this module only.
`seatunnel-dist/pom.xml:149` still declares
`<netty-buffer.version>4.1.89.Final</netty-buffer.version>` (used by the
`provided` `netty-buffer` dependency at `pom.xml:1162-1167`), and
`seatunnel-dist/src/main/assembly/assembly-seatunnel-bin-ci.xml:239-240` copies
`io.netty:netty-buffer:jar` and `io.netty:netty-common:jar` into `/lib`.
`seatunnel-cluster.sh:221` builds `CLASS_PATH=${APP_DIR}/lib/*:${APP_JAR}`, so
`lib/` wins over the shaded starter jar for every `io.netty.buffer.*` /
`io.netty.util.*` class.
> - Impact: for a Zeta server started from the binary distribution, the
Netty classes that actually execute are still `netty-buffer` 4.1.89 plus
whatever transitive `netty-common` `seatunnel-dist` resolves, not 4.1.118. The
4.1.118 classes bundled in the starter jar are dead weight at runtime, and
`imap-storage-file` (which compiles against `netty-buffer` 4.1.60.Final as
`provided`, see its `pom.xml:38,108-111`, and uses
`io.netty.buffer.ByteBuf`/`Unpooled` in `CloudWriter.java:33-34`) still runs on
a version nobody pinned deliberately. The PR title promises consistent Netty
resolution for the starter, but the user-visible artifact (the distribution)
keeps the same three-way mismatch.
> - Fix: either pin the version once in the root `pom.xml`
`<dependencyManagement>` (so `seatunnel-starter`, `seatunnel-dist` and
`imap-storage-file` all inherit it) and drop the per-module
`netty-buffer.version` properties, or, if the intent is that `lib/` is the
single source of Netty for Zeta, add `io.netty:*` to the starter's
`maven-shade-plugin` `<artifactSet><excludes>` (`pom.xml:132-156`) so the
starter jar does not carry a second copy at all. Please also state in the PR
which of the two layouts is intended.
> - Evidence: `seatunnel-core/seatunnel-starter/pom.xml:40
<artifactId>netty-buffer</artifactId>`;
`seatunnel-core/seatunnel-starter/pom.xml:45
<artifactId>netty-common</artifactId>`;
`seatunnel-core/seatunnel-starter/pom.xml:35 <!-- Use the same Netty version
in reactor and standalone builds. -->`
> - Suggested fix: This `<dependencyManagement>` only affects the shaded
`seatunnel-starter.jar`. In the binary distribution
`seatunnel-dist/pom.xml:149` still ships `netty-buffer` 4.1.89.Final (plus a
transitive `netty-common`) into `lib/` via
`assembly-seatunnel-bin-ci.xml:239-240`, and `seatunnel-cluster.sh` builds
`CLASS_PATH=${APP_DIR}/lib/*:${APP_JAR}`, so `lib/` wins over the starter jar
at runtime. The Zeta server will therefore keep running 4.1.89 regardless of
this pin, and `imap-storage-file` (compiled against 4.1.60 `provided`) is still
on an unpinned version. Could we move the pin to the root POM
`<dependencyManagement>` so `seatunnel-dist`/`imap-storage-file` inherit the
same value (and remove their local `netty-buffer.version` properties), or
alternatively exclude `io.netty:*` from the starter shade if `lib/` is meant to
be the single Netty source?
> - Severity: Medium
>
> **Issue 3: Pinning only `netty-buffer` and `netty-common` with a
hard-coded version decoupled from the AWS SDK / Arrow that actually produce
them will yield mixed Netty module versions on the next SDK or Arrow bump**
> - Location: `seatunnel-core/seatunnel-starter/pom.xml:40`
> - Why it matters: - Root cause: `netty.version` at
`seatunnel-core/seatunnel-starter/pom.xml:32` is a literal `4.1.118.Final` that
mirrors the version the AWS SDK path (`netty-nio-client-2.31.30` ->
`netty-transport/handler/codec-4.1.118.Final`, see
`tools/dependencies/known-dependencies.txt:109-121`) currently brings, while
Arrow 15.0.1 (`arrow.version`, root `pom.xml:208`) brings 4.1.104
(`known-dependencies.txt:83-84`). Nothing ties the three together, and the
management block covers just two of the ten `io.netty` artifacts that the AWS
SDK path places on the starter classpath.
> - Failure scenario: when the AWS SDK or Arrow is bumped (which is how
Netty normally moves in this repo), `netty-transport`, `netty-handler`,
`netty-codec*`, `netty-resolver` float to the new release but
`netty-buffer`/`netty-common` are forced back to 4.1.118. Netty modules are
only guaranteed compatible within one release;
`netty-transport`/`netty-handler` regularly call new
`netty-common`/`netty-buffer` methods, so the result is
`NoSuchMethodError`/`NoClassDefFoundError` at runtime in whatever path
exercises Netty (today the starter test classpath via
`seatunnel-transforms-v2`, tomorrow any compile-scope Netty user). Nothing in
the build fails fast on this, so it surfaces as a confusing runtime error
rather than a build error.
> - Fix: import `io.netty:netty-bom:${netty.version}` with
`<type>pom</type><scope>import</scope>` instead of listing two artifacts, so
every `io.netty:*` artifact resolves to the same release, and keep
`netty.version` next to `arrow.version`/`software.amazon.awssdk.version` in the
root POM with a comment stating which upstream it must track.
> - Evidence: `seatunnel-core/seatunnel-starter/pom.xml:41
<version>${netty.version}</version>`;
`seatunnel-core/seatunnel-starter/pom.xml:46
<version>${netty.version}</version>`;
`seatunnel-core/seatunnel-starter/pom.xml:40
<artifactId>netty-buffer</artifactId>`
> - Suggested fix: Managing only `netty-buffer` and `netty-common` leaves
`netty-transport`, `netty-handler`, `netty-codec*`, `netty-resolver` (all
currently 4.1.118 via the AWS SDK path, see
`tools/dependencies/known-dependencies.txt:112-121`) unmanaged. The moment the
AWS SDK or Arrow is bumped those will float to a newer release while these two
are forced back to 4.1.118, producing a mixed Netty set and
`NoSuchMethodError`s at runtime. Could we import
`io.netty:netty-bom:${netty.version}` (`<type>pom</type><scope>import</scope>`)
instead, so all `io.netty:*` artifacts are kept on a single release?
> - Severity: Medium
>
> # Review conclusion
> ### Conclusion: final approval remains pending on the previous comments
>
> **2. Remaining follow-ups**
> - Issue 1: Hard-coded 4.1.118.Final in the starter POM is a detached copy
of the version the AWS SDK (root `software.amazon.awssdk.version` = 2.31.30)
happens to bring; the root POM manages no `io.netty` artifact, so the next
SDK/Arrow bump silently produces mixed Netty module versions on the starter
classpath
> - Issue 2: The module-local pin only makes the shaded starter jar
deterministic; in the shipped distribution `lib/netty-buffer-4.1.89.Final.jar`
(from `seatunnel-dist`) is loaded ahead of `starter/seatunnel-starter.jar`, so
Zeta still runs a different Netty than the one this PR pins
> - Issue 3: Pinning only `netty-buffer` and `netty-common` with a
hard-coded version decoupled from the AWS SDK / Arrow that actually produce
them will yield mixed Netty module versions on the next SDK or Arrow bump
>
> Thanks for the updates so far. Happy to discuss the remaining points.
Thanks for the review. This PR intentionally addresses a specific build
inconsistency: reactor and standalone builds of `seatunnel-starter` resolve
different Netty versions.
The local validation covered dependency resolution and the actual shaded
JARs:
| Build | Before | After |
| --- | --- | --- |
| Reactor | 4.1.118.Final | 4.1.118.Final |
| Standalone with installed dependency-reduced POMs | 4.1.104.Final |
4.1.118.Final |
All **702 Netty class entries** are byte-for-byte identical between the
patched JARs and match the unmodified reactor JAR. The 42-module reactor
verification and all 94 existing starter tests also passed, with the validation
settings documented in the PR.
Regarding issues 1 and 3, broader Netty alignment is worth considering, but
the future SDK/Arrow upgrade scenario does not demonstrate a regression in this
patch. Introducing a root-level BOM and updating the dist, storage, and E2E
modules would expand the compatibility scope considerably. I would prefer to
handle that in a separate PR.
Regarding issue 2, the cited `assembly-seatunnel-bin-ci.xml` configuration
already exists before this change. The regular `assembly-seatunnel-bin.xml`
does not explicitly include these two standalone Netty JARs in the same way, so
the CI assembly alone does not establish which version the release distribution
loads. That runtime classpath concern deserves separate verification and does
not invalidate the starter artifact consistency demonstrated above.
I would like to keep this PR focused on making standalone builds match the
existing reactor output. If there is a concrete regression introduced by the
current patch, please provide the dependency tree, artifact comparison, or
reproducer so I can investigate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]