hansva opened a new pull request, #8997: URL: https://github.com/apache/hop/pull/8997
## Dependency upgrades and lib/core cleanup Upgrades third-party dependencies to their latest patch/minor releases, and removes duplicate and conflicting jars from the assemblies. ### Upgrades - Patch/minor upgrades across `lib/pom.xml`, `lib-jdbc` and the plugin poms, among them the AWS SDK (2.20 → 2.56, now one version through the AWS BOM), Netty, OpenTelemetry, Jetty, Jersey, SWT/RAP, the JDBC drivers, Parquet, Tika 3.3.3 and OpenLineage 1.54. - Driver download defaults in the database meta classes, the bundled-library versions in `LICENSE`, and the Snowflake docs page now match the bundled drivers. - Spark, and the libraries Beam/Spark depend on through `lib/core` (Jackson, protobuf/gRPC, Hadoop, Kafka), are unchanged. ZooKeeper is the exception: it moves to 3.9.6, replacing the end-of-life 3.6.3 that Beam's Spark runner brought. ### One version per library in lib/core `lib/core` is a single classloader, so two versions of a jar there means whichever is listed first wins. - `lib/pom.xml` now manages the libraries that used to land in `lib/core` in several versions: google-auth, api-common, proto-google-common-protos, animal-sniffer, failureaccess, httpcore5, jaxb-api, jcl-over-slf4j, snappy-java, Parquet, Log4j (through `log4j-bom`) and more. - Plugins no longer ship jars that `lib/core` already has. They declare them `provided` (JMS, Kafka, Lakehouse, OpenLineage, LanguageModelChat, Cassandra, Stanford NLP and others), or, for Spark, exclude them in the assembly. Spark keeps its own Spark 4 / Scala 2.13 stack. - Native Spark uses Log4j and lz4 from `lib/core`. Spark's old `org.lz4:lz4-java` is replaced by the maintained `at.yawk.lz4` fork. - Test jars no longer reach `lib/core`: JUnit 4.10 through `json-simple`, `reload4j` through Hadoop. Hamcrest stays: Beam's `PipelineOptionsFactory` needs it at startup. - Library versions are aligned across plugins: jgit, antlr-runtime, libphonenumber, commons-pool2, jts-core. ### Fixes found by the integration tests - **Beam on Spark:** OpenLineage 1.54 brought `jackson-module-blackbird` 2.18. The Beam fat jar bundles it, and it breaks on Spark 3.5 executors (jackson-databind 2.15), which shows up as a `serialVersionUID` mismatch. It is now pinned to 2.15.3. - **Gremlin:** TinkerPop stays on the 3.7 line (3.7.7), because there is no stable JanusGraph release for 3.8 yet. - **Tika:** `golden-apache-tika.csv` is updated in `mdi` and `transforms`. Tika 3.3.3 extracts three Office documents slightly differently: it keeps a soft hyphen, drops a trailing newline, and puts a space between hyperlinks. - **GCP / BigQuery:** `protobuf-javalite` is excluded from `grpc-protobuf-lite` in hop-core. It clashed with `protobuf-java` in `lib/core` and caused `NoSuchMethodError: DurationProto.getDescriptor()`. - **Mongo ITs on arm64:** the no-AVX `mongod` wrapper now only uses qemu on x86_64 hosts. ### New: GCS upload buffer size for Dataflow Beam stages the fat jar in 1 MB chunks by default, one request per chunk, which caps staging at about 1 MB/s from a laptop. The Dataflow run configuration has a new **GCS upload buffer size (MB)** field. When it's empty, 128 MB is used. It is documented in the Dataflow run configuration page. ### Testing - Unit tests: core, engine, Beam, Spark and all touched plugins. - Integration tests that pass: `spark`, `spark-native`, `mongo`, both Tika tests, `beam_directrunner` 0006, `samples` 0001, and `transforms` 0102/0109/0111/0112. - Not yet verified: the `protobuf-javalite` exclusion and the GCS upload buffer field. Both still need a GCP integration test run. The Google Sheets tests were skipped because there was no service-account key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Run `mvn clean install apache-rat:check` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using `git rebase -i`. - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) you have to acknowledge this by using the following check-box. - [ ] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) - [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
