NestDream opened a new pull request, #20776: URL: https://github.com/apache/hudi/pull/20776
### Describe the issue this Pull Request addresses Closes #20775. Flink 2.3.0 has been on Maven Central since June 2026. Hudi has adapter modules and bundles for Flink 1.18 to 2.2, so users on Flink 2.3 have no bundle that is compiled and tested against their Flink version. This PR adds Flink 2.3 the way #19655 added Flink 2.2. Two things need a maintainer: the bundle validation base image for Flink 2.3.0 has to be published (end of Risk Level), and there is a design choice about `ON CONFLICT` (end of Impact). ### Summary and Changelog - Add the `hudi-flink2.3.x` module and a `flink2.3` Maven profile based on Flink 2.3.0. The module is a copy of `hudi-flink2.2.x` and only its `pom.xml` differs. The adapter code needed no change for Flink 2.3.0. - Use Avro 1.11.4, Parquet 1.15.2 and ORC 1.5.6 for the profile (what Flink 2.3.0 builds with) and Kafka Connector 5.0.0-2.2 (no release for Flink 2.3 yet). - Make Flink 2.3 the default Flink build. The `flink2.2` profile and the older ones stay. - Set `table.exec.sink.require-on-conflict` to `false` in `ITTestHoodieDataSource`, `ITTestDynamicBucketStreamWrite`, `ITTestBucketStreamWrite`, the `HoodieFlinkQuickstart` example and `packaging/bundle-validation/flink/insert.sql`. - Update the README, the release guide, the release and build scripts and the CI configuration for the Flink 2.3 bundle, and add the bundle validation entry with the base image tag `flink230hive313spark359`. The first commit is the plain module copy, so the other four can be reviewed on their own. ### Impact The default Hudi Flink build and bundle now target Flink 2.3.0. Users on Flink 2.2 keep building with `-Dflink2.2`. The storage format and the public APIs do not change. One Flink 2.3 change affects Hudi users. Flink 2.3 adds the `ON CONFLICT` clause and the option `table.exec.sink.require-on-conflict`, `true` by default (FLIP-558, [FLINK-38927](https://issues.apache.org/jira/browse/FLINK-38927)). A streaming `INSERT INTO` a table with a primary key now fails at planning time when the upsert key of the query differs from the primary key and the statement has no `ON CONFLICT` clause. An append-only source has no upsert key and `HoodieTableSink` asks for an upsert stream, so Flink rejects a plain `INSERT INTO hudi_table SELECT ... FROM append_only_source`: ``` ValidationException: The query has an upsert key that differs from the primary key of the sink table 'default_catalog.default_database.t1'. Primary key: [uuid], upsert key: . [...] Please specify an ON CONFLICT clause to define how conflicts should be handled ``` Without the option, 95 of the 434 Flink integration tests fail with this error under `-Dflink2.3` (88 in `ITTestHoodieDataSource`, 7 in `ITTestBucketStreamWrite` and `ITTestDynamicBucketStreamWrite`), and so do the 4 tests of `TestHoodieFlinkQuickstart`. On Flink 2.3 a user can run such a statement in any of these ways. I ran each one on a Flink 2.3.0 cluster with this bundle: - append `ON CONFLICT DO DEDUPLICATE` - set `table.exec.sink.upsert-materialize` to `NONE` - set `table.exec.sink.require-on-conflict` to `false`. On 2.3.0 this adds a stateful `SinkMaterializer` in front of the writer, which [FLINK-40324](https://issues.apache.org/jira/browse/FLINK-40324) fixes for 2.3.1. The other two do not add it. Batch jobs and tables without a primary key are not affected. The `release-2.4` branch of Flink changes the default of the option to `false` ([FLINK-41478](https://issues.apache.org/jira/browse/FLINK-41478)). This PR leaves production code alone. The tests, the quickstart and the validation SQL set the option and not the clause because they run on every Flink profile: Flink 2.2 and earlier ignore the unknown key, and `ON CONFLICT` is new syntax in 2.3. The alternative is to let `HoodieTableSink` report insert-only for an insert-only input, as the Iceberg, Paimon and Fluss sinks do. A plain `INSERT` then needs neither the clause nor an option. The cost: Flink then rejects any `ON CONFLICT` clause on such a statement, and today `DO NOTHING` and `DO ERROR` are accepted there when the source has a watermark. I have that variant tested as a single commit. Let me know if you want it here or as a follow-up. Not covered here: the Flink 2.3.1 release candidate changes the `BinaryExternalSorter` constructor ([FLINK-40959](https://issues.apache.org/jira/browse/FLINK-40959)) that `bulk_insert` with sorted input and clustering with sort columns call. The 1.20.6 and 2.2.2 candidates carry the same change, so it needs its own fix across the adapters. ### Risk Level Medium, as for #19655: the default Flink dependency line changes and a version-specific module is added. What I ran on top of master `f383424d889`, on Java 17 unless noted. Commands and cluster cases are in the collapsed blocks below. | Check | Result | |---|---| | Build with `-Dflink2.3`: Flink modules, examples, bundle, `hudi-kafka-connect` and its bundle, with checkstyle and RAT | BUILD SUCCESS | | Unit tests of `hudi-flink2.3.x`, `hudi-flink-client`, `hudi-flink` and `TestHoodieFlinkQuickstart` | 55, 246, 2215 and 4 run, 0 failed, 2 skipped | | Integration tests, the two CI filters: `IT*` without `ITTestHoodieDataSource`, and `ITTestHoodieDataSource` | 154 and 280 run, 0 failed, 1 skipped | | Unit tests of `hudi-kafka-connect` | 28 run, 0 failed | | Java 11: bundle build, unit tests of `hudi-flink2.3.x` | BUILD SUCCESS, 55 run, 0 failed | | With `-Dflink2.2`: 20 integration tests of the three changed classes and `TestHoodieFlinkQuickstart` | 20 and 4 run, 0 failed | | Bundle build with `flink1.18` to `flink2.2` | BUILD SUCCESS, all six | | The new `SET` line of `insert.sql` in the SQL clients of Flink 1.20.5 and 2.2.1 | accepted | | Bundle validation (`ci_run.sh`) against a locally built base image, OpenJDK 11 and 17 | passed | Rows 2 and 3 match the [CI run](https://github.com/apache/hudi/actions/runs/44520972293) of the same master commit with `flink2.2`, so the new profile runs the same tests. I also ran the bundle on a standalone Flink 2.3.0 cluster: COPY_ON_WRITE and MERGE_ON_READ upserts, `bulk_insert`, async clustering and a streaming read. A separate batch job read each table back, and count, distinct keys, minimum, maximum and sum equal what was written. The `validate-bundles` job for `flink2.3` needs the base image `apachehudi/hudi-ci-bundle-validation-base:flink230hive313spark359`, which is not on Docker Hub yet. I built it locally with `packaging/bundle-validation/base/build_flink230hive313spark359.sh`, and `ci_run.sh` passes against it. I cannot publish it, so `CI (fork)` stays red on that job until then. Could someone with push rights build and publish it, as was done for #19655? <details> <summary>Cluster cases</summary> | Case | Written | Read back | |---|---|---| | COPY_ON_WRITE upsert, streaming, once with each of the three ways above | 20000 rows on 1000 keys | 1000 rows, the latest value of every key | | MERGE_ON_READ upsert with async compaction | 30000 rows on 1000 keys | 1000 rows, the latest value of every key, 7 deltacommits, 3 compactions | | `bulk_insert` in batch mode into a partitioned table | 10000 rows | 10000 rows, 2500 in each of the 4 partitions | | Append with async clustering and a sort column | 45000 rows | 45000 rows, 4 replacecommits | | Streaming read from one Hudi table into another | 30000 rows | 30000 rows in both tables | </details> <details> <summary>Commands</summary> ``` # build mvn clean install -T 4 -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 -Dparquet.version=1.15.2 -DskipTests=true \ -pl hudi-examples/hudi-examples-flink,hudi-examples/hudi-examples-k8s,packaging/hudi-flink-bundle -am mvn clean install -T 4 -Dscala-2.12 -Dspark3.5 -Dflink2.3 -DskipTests=true \ -pl hudi-kafka-connect,packaging/hudi-kafka-connect-bundle -am # unit tests mvn test -Punit-tests -Pwarn-log -fae -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 -Dparquet.version=1.15.2 \ -pl hudi-flink-datasource/hudi-flink2.3.x,hudi-client/hudi-flink-client,hudi-flink-datasource/hudi-flink \ -Dhudi.surefire.forkCount=2 mvn test -Punit-tests -Pwarn-log -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 -Dparquet.version=1.15.2 \ -pl hudi-examples/hudi-examples-flink -Dhudi.surefire.forkCount=1 mvn test -Punit-tests -Pwarn-log -Dscala-2.12 -Dspark3.5 -Dflink2.3 -pl hudi-kafka-connect # integration tests mvn verify -Pintegration-tests -Pwarn-log -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 -Dparquet.version=1.15.2 \ '-Dit.test=IT*,!ITTestHoodieDataSource' -pl hudi-flink-datasource/hudi-flink mvn verify -Pintegration-tests -Pwarn-log -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 -Dparquet.version=1.15.2 \ -Dit.test=ITTestHoodieDataSource -pl hudi-flink-datasource/hudi-flink # Java 11 mvn clean package -T 4 -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 -Dparquet.version=1.15.2 -DskipTests=true \ -pl hudi-examples/hudi-examples-flink,packaging/hudi-flink-bundle -am mvn test -Punit-tests -Pwarn-log -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 -Dparquet.version=1.15.2 \ -pl hudi-flink-datasource/hudi-flink2.3.x # the changed tests with flink2.2 mvn verify -Pintegration-tests -Pwarn-log -Dscala-2.12 -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 \ '-Dit.test=ITTestBucketStreamWrite#testRemotePartitioner,ITTestDynamicBucketStreamWrite,ITTestHoodieDataSource#testStreamWriteAndRead*' \ -Dfailsafe.failIfNoSpecifiedTests=false -pl hudi-flink-datasource/hudi-flink -am mvn test -Punit-tests -Pwarn-log -Dscala-2.12 -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 \ -Dtest=TestHoodieFlinkQuickstart -Dsurefire.failIfNoSpecifiedTests=false -pl hudi-examples/hudi-examples-flink -am # bundle validation, as the validate-bundles job runs it (FLINK_PROFILE=flink2.3, SPARK_PROFILE=spark3.5, # SPARK_RUNTIME=spark3.5.9, SCALA_PROFILE=scala-2.12), also with openjdk17 (cd packaging/bundle-validation/base && ./build_flink230hive313spark359.sh) ./packaging/bundle-validation/ci_run.sh hudi_docker_java11 1.3.0-SNAPSHOT openjdk11 # other profiles: flink2.2 and flink2.1 with -Dparquet.version=1.15.2, flink2.0 with 1.14.4, # flink1.20, flink1.19 and flink1.18 with 1.13.1 mvn clean package -T 4 -Dscala-2.12 -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 -DskipTests=true \ -pl packaging/hudi-flink-bundle -am ``` </details> ### Documentation Update The README and the release guide are updated for Flink 2.3. The Flink pages of the website should mention the `ON CONFLICT` behavior described under Impact, as a separate change. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
