NestDream opened a new pull request, #20776:
URL: https://github.com/apache/hudi/pull/20776

   ### Describe the issue this Pull Request addresses
   
   Closes #20775.
   
   Flink 2.3.0 has been on Maven Central since June 2026. Hudi has adapter 
modules and bundles for Flink 1.18 to 2.2, so users on Flink 2.3 have no bundle 
that is compiled and tested against their Flink version. This PR adds Flink 2.3 
the way #19655 added Flink 2.2.
   
   Two things need a maintainer: the bundle validation base image for Flink 
2.3.0 has to be published (end of Risk Level), and there is a design choice 
about `ON CONFLICT` (end of Impact).
   
   ### Summary and Changelog
   
   - Add the `hudi-flink2.3.x` module and a `flink2.3` Maven profile based on 
Flink 2.3.0. The module is a copy of `hudi-flink2.2.x` and only its `pom.xml` 
differs. The adapter code needed no change for Flink 2.3.0.
   - Use Avro 1.11.4, Parquet 1.15.2 and ORC 1.5.6 for the profile (what Flink 
2.3.0 builds with) and Kafka Connector 5.0.0-2.2 (no release for Flink 2.3 yet).
   - Make Flink 2.3 the default Flink build. The `flink2.2` profile and the 
older ones stay.
   - Set `table.exec.sink.require-on-conflict` to `false` in 
`ITTestHoodieDataSource`, `ITTestDynamicBucketStreamWrite`, 
`ITTestBucketStreamWrite`, the `HoodieFlinkQuickstart` example and 
`packaging/bundle-validation/flink/insert.sql`.
   - Update the README, the release guide, the release and build scripts and 
the CI configuration for the Flink 2.3 bundle, and add the bundle validation 
entry with the base image tag `flink230hive313spark359`.
   
   The first commit is the plain module copy, so the other four can be reviewed 
on their own.
   
   ### Impact
   
   The default Hudi Flink build and bundle now target Flink 2.3.0. Users on 
Flink 2.2 keep building with `-Dflink2.2`. The storage format and the public 
APIs do not change.
   
   One Flink 2.3 change affects Hudi users. Flink 2.3 adds the `ON CONFLICT` 
clause and the option `table.exec.sink.require-on-conflict`, `true` by default 
(FLIP-558, [FLINK-38927](https://issues.apache.org/jira/browse/FLINK-38927)). A 
streaming `INSERT INTO` a table with a primary key now fails at planning time 
when the upsert key of the query differs from the primary key and the statement 
has no `ON CONFLICT` clause. An append-only source has no upsert key and 
`HoodieTableSink` asks for an upsert stream, so Flink rejects a plain `INSERT 
INTO hudi_table SELECT ... FROM append_only_source`:
   
   ```
   ValidationException: The query has an upsert key that differs from the 
primary key of the sink table
   'default_catalog.default_database.t1'. Primary key: [uuid], upsert key: . 
[...]
   Please specify an ON CONFLICT clause to define how conflicts should be 
handled
   ```
   
   Without the option, 95 of the 434 Flink integration tests fail with this 
error under `-Dflink2.3` (88 in `ITTestHoodieDataSource`, 7 in 
`ITTestBucketStreamWrite` and `ITTestDynamicBucketStreamWrite`), and so do the 
4 tests of `TestHoodieFlinkQuickstart`.
   
   On Flink 2.3 a user can run such a statement in any of these ways. I ran 
each one on a Flink 2.3.0 cluster with this bundle:
   
   - append `ON CONFLICT DO DEDUPLICATE`
   - set `table.exec.sink.upsert-materialize` to `NONE`
   - set `table.exec.sink.require-on-conflict` to `false`. On 2.3.0 this adds a 
stateful `SinkMaterializer` in front of the writer, which 
[FLINK-40324](https://issues.apache.org/jira/browse/FLINK-40324) fixes for 
2.3.1. The other two do not add it.
   
   Batch jobs and tables without a primary key are not affected. The 
`release-2.4` branch of Flink changes the default of the option to `false` 
([FLINK-41478](https://issues.apache.org/jira/browse/FLINK-41478)).
   
   This PR leaves production code alone. The tests, the quickstart and the 
validation SQL set the option and not the clause because they run on every 
Flink profile: Flink 2.2 and earlier ignore the unknown key, and `ON CONFLICT` 
is new syntax in 2.3.
   
   The alternative is to let `HoodieTableSink` report insert-only for an 
insert-only input, as the Iceberg, Paimon and Fluss sinks do. A plain `INSERT` 
then needs neither the clause nor an option. The cost: Flink then rejects any 
`ON CONFLICT` clause on such a statement, and today `DO NOTHING` and `DO ERROR` 
are accepted there when the source has a watermark. I have that variant tested 
as a single commit. Let me know if you want it here or as a follow-up.
   
   Not covered here: the Flink 2.3.1 release candidate changes the 
`BinaryExternalSorter` constructor 
([FLINK-40959](https://issues.apache.org/jira/browse/FLINK-40959)) that 
`bulk_insert` with sorted input and clustering with sort columns call. The 
1.20.6 and 2.2.2 candidates carry the same change, so it needs its own fix 
across the adapters.
   
   ### Risk Level
   
   Medium, as for #19655: the default Flink dependency line changes and a 
version-specific module is added.
   
   What I ran on top of master `f383424d889`, on Java 17 unless noted. Commands 
and cluster cases are in the collapsed blocks below.
   
   | Check | Result |
   |---|---|
   | Build with `-Dflink2.3`: Flink modules, examples, bundle, 
`hudi-kafka-connect` and its bundle, with checkstyle and RAT | BUILD SUCCESS |
   | Unit tests of `hudi-flink2.3.x`, `hudi-flink-client`, `hudi-flink` and 
`TestHoodieFlinkQuickstart` | 55, 246, 2215 and 4 run, 0 failed, 2 skipped |
   | Integration tests, the two CI filters: `IT*` without 
`ITTestHoodieDataSource`, and `ITTestHoodieDataSource` | 154 and 280 run, 0 
failed, 1 skipped |
   | Unit tests of `hudi-kafka-connect` | 28 run, 0 failed |
   | Java 11: bundle build, unit tests of `hudi-flink2.3.x` | BUILD SUCCESS, 55 
run, 0 failed |
   | With `-Dflink2.2`: 20 integration tests of the three changed classes and 
`TestHoodieFlinkQuickstart` | 20 and 4 run, 0 failed |
   | Bundle build with `flink1.18` to `flink2.2` | BUILD SUCCESS, all six |
   | The new `SET` line of `insert.sql` in the SQL clients of Flink 1.20.5 and 
2.2.1 | accepted |
   | Bundle validation (`ci_run.sh`) against a locally built base image, 
OpenJDK 11 and 17 | passed |
   
   Rows 2 and 3 match the [CI 
run](https://github.com/apache/hudi/actions/runs/44520972293) of the same 
master commit with `flink2.2`, so the new profile runs the same tests.
   
   I also ran the bundle on a standalone Flink 2.3.0 cluster: COPY_ON_WRITE and 
MERGE_ON_READ upserts, `bulk_insert`, async clustering and a streaming read. A 
separate batch job read each table back, and count, distinct keys, minimum, 
maximum and sum equal what was written.
   
   The `validate-bundles` job for `flink2.3` needs the base image 
`apachehudi/hudi-ci-bundle-validation-base:flink230hive313spark359`, which is 
not on Docker Hub yet. I built it locally with 
`packaging/bundle-validation/base/build_flink230hive313spark359.sh`, and 
`ci_run.sh` passes against it. I cannot publish it, so `CI (fork)` stays red on 
that job until then. Could someone with push rights build and publish it, as 
was done for #19655?
   
   <details>
   <summary>Cluster cases</summary>
   
   | Case | Written | Read back |
   |---|---|---|
   | COPY_ON_WRITE upsert, streaming, once with each of the three ways above | 
20000 rows on 1000 keys | 1000 rows, the latest value of every key |
   | MERGE_ON_READ upsert with async compaction | 30000 rows on 1000 keys | 
1000 rows, the latest value of every key, 7 deltacommits, 3 compactions |
   | `bulk_insert` in batch mode into a partitioned table | 10000 rows | 10000 
rows, 2500 in each of the 4 partitions |
   | Append with async clustering and a sort column | 45000 rows | 45000 rows, 
4 replacecommits |
   | Streaming read from one Hudi table into another | 30000 rows | 30000 rows 
in both tables |
   
   </details>
   
   <details>
   <summary>Commands</summary>
   
   ```
   # build
   mvn clean install -T 4 -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 
-Dparquet.version=1.15.2 -DskipTests=true \
     -pl 
hudi-examples/hudi-examples-flink,hudi-examples/hudi-examples-k8s,packaging/hudi-flink-bundle
 -am
   mvn clean install -T 4 -Dscala-2.12 -Dspark3.5 -Dflink2.3 -DskipTests=true \
     -pl hudi-kafka-connect,packaging/hudi-kafka-connect-bundle -am
   
   # unit tests
   mvn test -Punit-tests -Pwarn-log -fae -Dscala-2.12 -Dflink2.3 
-Davro.version=1.11.4 -Dparquet.version=1.15.2 \
     -pl 
hudi-flink-datasource/hudi-flink2.3.x,hudi-client/hudi-flink-client,hudi-flink-datasource/hudi-flink
 \
     -Dhudi.surefire.forkCount=2
   mvn test -Punit-tests -Pwarn-log -Dscala-2.12 -Dflink2.3 
-Davro.version=1.11.4 -Dparquet.version=1.15.2 \
     -pl hudi-examples/hudi-examples-flink -Dhudi.surefire.forkCount=1
   mvn test -Punit-tests -Pwarn-log -Dscala-2.12 -Dspark3.5 -Dflink2.3 -pl 
hudi-kafka-connect
   
   # integration tests
   mvn verify -Pintegration-tests -Pwarn-log -Dscala-2.12 -Dflink2.3 
-Davro.version=1.11.4 -Dparquet.version=1.15.2 \
     '-Dit.test=IT*,!ITTestHoodieDataSource' -pl 
hudi-flink-datasource/hudi-flink
   mvn verify -Pintegration-tests -Pwarn-log -Dscala-2.12 -Dflink2.3 
-Davro.version=1.11.4 -Dparquet.version=1.15.2 \
     -Dit.test=ITTestHoodieDataSource -pl hudi-flink-datasource/hudi-flink
   
   # Java 11
   mvn clean package -T 4 -Dscala-2.12 -Dflink2.3 -Davro.version=1.11.4 
-Dparquet.version=1.15.2 -DskipTests=true \
     -pl hudi-examples/hudi-examples-flink,packaging/hudi-flink-bundle -am
   mvn test -Punit-tests -Pwarn-log -Dscala-2.12 -Dflink2.3 
-Davro.version=1.11.4 -Dparquet.version=1.15.2 \
     -pl hudi-flink-datasource/hudi-flink2.3.x
   
   # the changed tests with flink2.2
   mvn verify -Pintegration-tests -Pwarn-log -Dscala-2.12 -Dflink2.2 
-Davro.version=1.11.4 -Dparquet.version=1.15.2 \
     
'-Dit.test=ITTestBucketStreamWrite#testRemotePartitioner,ITTestDynamicBucketStreamWrite,ITTestHoodieDataSource#testStreamWriteAndRead*'
 \
     -Dfailsafe.failIfNoSpecifiedTests=false -pl 
hudi-flink-datasource/hudi-flink -am
   mvn test -Punit-tests -Pwarn-log -Dscala-2.12 -Dflink2.2 
-Davro.version=1.11.4 -Dparquet.version=1.15.2 \
     -Dtest=TestHoodieFlinkQuickstart -Dsurefire.failIfNoSpecifiedTests=false 
-pl hudi-examples/hudi-examples-flink -am
   
   # bundle validation, as the validate-bundles job runs it 
(FLINK_PROFILE=flink2.3, SPARK_PROFILE=spark3.5,
   # SPARK_RUNTIME=spark3.5.9, SCALA_PROFILE=scala-2.12), also with openjdk17
   (cd packaging/bundle-validation/base && ./build_flink230hive313spark359.sh)
   ./packaging/bundle-validation/ci_run.sh hudi_docker_java11 1.3.0-SNAPSHOT 
openjdk11
   
   # other profiles: flink2.2 and flink2.1 with -Dparquet.version=1.15.2, 
flink2.0 with 1.14.4,
   # flink1.20, flink1.19 and flink1.18 with 1.13.1
   mvn clean package -T 4 -Dscala-2.12 -Dflink2.2 -Davro.version=1.11.4 
-Dparquet.version=1.15.2 -DskipTests=true \
     -pl packaging/hudi-flink-bundle -am
   ```
   
   </details>
   
   ### Documentation Update
   
   The README and the release guide are updated for Flink 2.3. The Flink pages 
of the website should mention the `ON CONFLICT` behavior described under 
Impact, as a separate change.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to