ericyuan915 opened a new pull request, #19982:
URL: https://github.com/apache/hudi/pull/19982
HoodieSink builds its whole write pipeline in addPreWriteTopology, which
Flink only invokes for a sink implementing the pre-write-topology interface of
the running version. The Flink 1.18 SupportsPreWriteTopologyAdapter extended
nothing -- it existed only to keep the shared hudi-flink module compiling -- so
on 1.18 the hook was never called and PipelinesV2.sink fenced Sink V2 off
behind a runtime version check.
Flink 1.18 does have the capability, named WithPreWriteTopology, which
SinkTransformationTranslator dispatches on there. Flink 1.19 split the
identical method out into SupportsPreWriteTopology and deprecated
WithPreWriteTopology, so only the type name and its Sink supertype differ.
Extend the real interface in the 1.18 adapter and inherit
addPreWriteTopology instead of redeclaring it. HoodieSink then reaches
Sink<RowData> through both SinkAdapter and WithPreWriteTopology, which is
legal: same type argument, and SinkAdapter's createWriter(InitContext) default
overrides the abstract declaration inherited from Sink. Drop the version guard
accordingly -- hudi-flink1.18.x is the oldest Flink profile on master, so it is
unreachable.
Add TestPipelinesV2#testSinkLetsFlinkExpandTheHudiWritePipeline, which runs
PipelinesV2.sink through Flink's own SinkTransformationTranslator by generating
the StreamGraph, then asserts the Hudi write operator is in the graph with the
sink uid prefixed to its own. It names no version-specific interface, so it
covers the pre-write hook on every Flink profile.
closes #19966
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the
changes
(or) link to an issue by including `Closes #<issue-number>` for
context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in
behavior.
Followed by a detailed log of all the changes. Highlight if any code
was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any
performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain
the risk.
If medium or high, explain what verification was done to mitigate the
risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new
feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the
default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website.
Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website)
to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
