loustler opened a new issue, #11655: URL: https://github.com/apache/seatunnel/issues/11655
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened `org.apache.hadoop.util.JsonSerialization` — reached from `hadoop-aws` — cannot construct its Jackson `ObjectMapper` at runtime, because the shaded Hadoop uber jar relocates Jackson while `hadoop-aws` still references the original package. This is **pre-existing on `dev` with the current 3.1.4 uber jar**; it is not introduced by the Hadoop 3.4.3 upgrade. It was found while doing binary-compatibility analysis for #11648 and is out of scope there, so opening it separately as @DanielLeens suggested in the review of that PR. The blast radius is narrow. `JsonSerialization.getMapper()` is used by `org.apache.hadoop.fs.s3a.auth.RoleModel`, which serialises and deserialises assumed-role IAM policy JSON. So it only affects S3A configurations that use assumed roles (`fs.s3a.assumed.role.*`) or otherwise exercise `RoleModel`. Plain access-key and instance-profile credentials do not go through it. ### SeaTunnel Version dev ### SeaTunnel Config ```conf # Any S3 source/sink using S3A assumed-role credentials, e.g. env { parallelism = 1 job.mode = "BATCH" } source { S3File { path = "/data" bucket = "s3a://mybucket" fs.s3a.aws.credentials.provider = "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider" hadoop_s3_properties = { "fs.s3a.assumed.role.arn" = "arn:aws:iam::000000000000:role/example" } file_format_type = "json" } } sink { Console {} } ``` ### Running Command ```shell ./bin/seatunnel.sh --config ./config/s3-assumed-role.conf -e local ``` ### Error Exception ```log NoClassDefFoundError / ClassNotFoundException on the original (unrelocated) Jackson package when org.apache.hadoop.util.JsonSerialization initialises its ObjectMapper, surfacing from org.apache.hadoop.fs.s3a.auth.RoleModel ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version Java 8 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Additional context The mismatch is between two modules that are both shaded, but differently: - `seatunnel-shade/seatunnel-hadoop3-*-uber` relocates Jackson into `shade.hadoop.com.fasterxml.jackson`. - `seatunnel-shade/seatunnel-hadoop-aws` does not relocate Jackson, and the `hadoop-aws` bytecode inside it references `com.fasterxml.jackson.*` directly. Both jars land in `lib/` on the runtime classpath, so `JsonSerialization`'s references do not resolve to the relocated copy. Possible directions, in rough order of how contained they are — I have not yet measured which is best and would welcome guidance: 1. Add a matching Jackson relocation to `seatunnel-hadoop-aws` so both jars agree. 2. Stop relocating Jackson in the uber jar (larger blast radius; the relocation presumably exists to avoid a clash with the Jackson SeaTunnel itself ships). 3. Ship the unrelocated Jackson that `hadoop-aws` expects alongside it. Happy to put a PR together once there is a preferred direction. ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
