danny0405 opened a new pull request, #19131: URL: https://github.com/apache/hudi/pull/19131
### Describe the issue this Pull Request addresses This closes #10937 #16504 #7381 The Flink datasource currently depends on `flink-table-planner` for planner-side sort code generation and managed-memory helper utilities. That makes Hudi's Flink runtime modules depend on planner internals even though the affected paths only need runtime generated comparator/key-computer classes and transformation managed-memory declaration. This PR removes the direct `flink-table-planner` dependency from the Flink datasource and examples modules, and replaces the required planner usages with local implementations in the Hudi Flink codebase. There are no storage format, public API, or breaking configuration changes. ### Summary and Changelog This change keeps the existing Flink write, bulk insert, append sort, and clustering sort paths working without requiring `flink-table-planner` on the compile classpath. #### Commit 1: fix: remove the dependency to flink-table-planner (`393332d408a0`) - Removed `flink-table-planner` dependencies from `hudi-flink`, version-specific Flink modules (`hudi-flink1.18.x`, `hudi-flink1.19.x`, `hudi-flink1.20.x`, `hudi-flink2.0.x`, `hudi-flink2.1.x`), and `hudi-examples-flink`. - Removed the root `flink.table.planner.artifactId` property and updated the Scala `_2.12` enforcer whitelist comment accordingly. - Replaced `SortCodeGenerator` usage in `SortOperatorGen`, `StreamWriteFunction`, append-sort write functions, and `ClusteringOperator` with local generation of `GeneratedNormalizedKeyComputer` and `GeneratedRecordComparator`. - Added local comparator source generation for primitive, string, binary, decimal, and timestamp sort fields, with fallback field-getter comparison for unsupported logical types. - Added `FlinkTransformationUtils#setManagedMemoryWeight` to replace planner `ExecNodeUtil.setManagedMemoryWeight` in `Pipelines`, `PipelinesV2`, `HoodieFlinkClusteringJob`, and related tests. - Replaced test-only usages of Flink calcite-shaded Guava `Lists` in catalog tests with JDK collections. - Added `TestSortOperatorGen` to instantiate generated comparator and normalized-key computer classes through Flink's generated-class runtime path. ### Impact This affects Flink datasource build dependencies and internal Flink write/sort/clustering execution paths. Users should not see public API, table format, configuration, or behavior changes. The main compatibility benefit is that Hudi no longer needs the Flink planner jar for these runtime paths. Sort behavior remains ascending with nulls first, matching the previous local usage of `SortSpec` in `SortOperatorGen`. ### Risk Level medium The touched code includes Flink write buffering, append sort, bulk insert sort, and clustering sort paths, so the blast radius is broader than a dependency-only cleanup. The risk is mitigated by adding `TestSortOperatorGen` for generated comparator/key-computer instantiation and by running: - `mvn -pl hudi-flink-datasource/hudi-flink -am -Dtest=TestSortOperatorGen -DskipITs -Dsurefire.failIfNoSpecifiedTests=false -Dmaven.compiler.proc=full test` - `mvn -pl hudi-flink-datasource/hudi-flink -am -DskipTests -Dmaven.compiler.proc=full compile` - `mvn -pl hudi-flink-datasource/hudi-flink -DskipTests -DincludeScope=compile -Dincludes=org.apache.flink:flink-table-planner_2.12 dependency:tree` ### Documentation Update none This is an internal dependency and implementation cleanup. It does not add user-facing options, change defaults, or alter documented Flink table/write behavior. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
