yihua opened a new pull request, #20247: URL: https://github.com/apache/hudi/pull/20247
### Describe the issue this Pull Request addresses part of #16933, follows #20240 `hoodie.database.name` in `hoodie.properties` is a copy of an identity the catalog owns, and the Spark read path used it, or the session's current database when absent, as a key into the catalog for two things: the simple bucket index options and the catalog-backed partition listing. Both read the wrong entry, or none, once that copy is stale or the table is registered under another name. The catalog-backed listing also returned stale results for partitions meta sync had not yet recorded. ### Summary and Changelog The catalog integration now passes the identity Spark resolved down to the read path as two internal options, `hoodie.datasource.read.catalog.table.database` and `hoodie.datasource.read.catalog.table.name`, set by `HoodieInternalV2Table.v1TableWrapper` (storage properties, which Spark forwards as options) and by each Spark version's V2-to-V1 fallback; `HoodieCatalogTable.catalogProperties` strips them so they never reach write configs or TBLPROPERTIES. The bucket index lookup in `HoodieFileIndex` keys on that identity and skips the merge when it is absent; `getDatabaseName` and the session fallback are gone. `CatalogBackedTableMetadata` keys on the same identity, fails closed with a clear error when the listing option is on and no identity is present, and uses the catalog's partitions only while the entry's `last_commit_time_sync` (and completion time, when recorded) match the latest completed commit, otherwise it warns and lists the file system. `upgrade_table` and `downgrade_tab le` pass a database only for a qualified table name instead of stamping the session's current one. Unread copies of the database name in `HoodieCatalogTable` and `FileSystemBackedTableMetadata` are removed. Tests: `TestHoodieFileIndex` (identity used, no-identity skip, failure path), `TestCatalogBackedTableMetadata` (fail closed, fresh entry used), `TestPartitionPushDownWhenListingPaths` (stale catalog falls back, which failed before with a dropped partition), `TestUpgradeOrDowngradeProcedure` (no session database in the write config). All fallbacks compile on Spark 3.3 through 4.2. This is step one of #16933: the config stays, but nothing keys a catalog operation on it any more. Step two removes it from the table config. ### Impact A read with the catalog partition listing option enabled but no catalog identity (a path read, or a session catalog without Hudi's catalog plugin) now fails with an explicit error instead of guessing a database. Reads through `HoodieCatalog` are unaffected. No other user-facing change. ### Risk Level low. The identity options are internal and stripped from table properties; the fail-closed case only affects an opt-in option used without a catalog. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
