rangareddy opened a new pull request, #19435:
URL: https://github.com/apache/hudi/pull/19435
### Describe the issue this Pull Request addresses
Closes #15279 (HUDI-4386).
Flink supports syncing metadata to AWS Glue: `HiveSyncMode` has four values
(`HMS`, `GLUE`, `HIVEQL`,
`JDBC`) and `HiveSyncContext#hiveSyncTool` loads
`org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool` when the
mode is `GLUE` (added in #5202). But both places that describe the Flink
option advertise only three values:
- `FlinkOptions.HIVE_SYNC_MODE` — "Mode to choose for Hive ops. Valid values
are hms, jdbc and hiveql, default 'hms'"
- `FlinkStreamerConfig --hive-sync-mode` — same list, default 'jdbc'
So the supported way to enable Glue sync from Flink is invisible in the
config that controls it. The
reporter went looking for a `hive_sync.catalog.sync.tool.class` key instead
— a key that does not exist
anywhere in the codebase — and hit a class-not-found failure.
### Summary and Changelog
Adds `glue` to the list of valid values in the two Flink descriptions. No
code or behavior change.
Deliberately scoped to Flink. `glue` is **not** a valid value for the shared
`hoodie.datasource.hive_sync.mode`:
- `HoodieHiveSyncClient` switches on `HMS`, `HIVEQL` and `JDBC`, and throws
`HoodieHiveSyncException("Invalid sync mode given …")` for anything else —
including `GLUE`.
- On Spark the Glue tool is selected through the sync tool class, not the
mode:
`HoodieSparkSqlWriter` adds
`org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool` to the sync client tool
classes.
I started by adding `glue` to `HiveSyncConfigHolder.HIVE_SYNC_MODE` and
`HoodieDropPartitionsTool --hive-sync-mode` as well, then reverted both once
I traced those paths — it would
have documented a value that throws at runtime.
Two related things I noticed but left alone, happy to fold in if you'd like:
- `HiveSyncConfig`'s `--sync-mode` description already lists `hms,glue,jdbc
and hiveql`, which looks like the
same over-claim in the opposite direction, since that path also ends in
`HoodieHiveSyncClient`.
- `rfc/rfc-55.md` refers to the class as `AWSGlueCatalogSyncTool`, while the
class is
`AwsGlueCatalogSyncTool` — the capitalisation the reporter used, which may
be where their config came from.
### Impact
None at runtime. Configuration description text only, in the Flink module.
### Risk Level
none
### Documentation Update
The change is itself the documentation fix. `FlinkOptions` descriptions
surface in Hudi's generated Flink
configuration reference, so the published list of valid `hive_sync.mode`
values gains `glue`. No new config
and no default value change.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
- [x] CI passes on my PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
