yyanyy opened a new pull request, #57865: URL: https://github.com/apache/spark/pull/57865
### What changes were proposed in this pull request? This PR follows up on #57582 and forwards all user-provided options when Spark loads a DataSource V2 table as a batch write target. It: * adds `CatalogV2Util.getTableForWrite` to load a table with both required write privileges and options; * forwards options in `DataFrameWriter` paths backed by a catalog, including `SupportsCatalogOptions`, `insertInto`, and `saveAsTable`; * attaches options to the unresolved target created by `DataFrameWriterV2` for `append`, `overwrite`, and `overwritePartitions`; * forwards options when schema evolution reloads the target; * forwards options in staging and non-staging CTAS/RTAS fallback loads when the catalog returns `null` from create/stage operations; * keeps the options on the resolved `DataSourceV2Relation` so the same options reach the actual V2 write; * preserves the original option key casing while adding or removing Spark's internal write-privilege marker; and * updates the relevant API documentation. Streaming writes are intentionally out of scope and remain unchanged. ### Why are the changes needed? Before this change, batch write options could reach `LogicalWriteInfo` but be omitted from the earlier `TableCatalog.loadTable` call used to resolve the write target. A catalog whose loaded `Table` depends on those options could therefore authorize or construct the wrong write target even though the connector later received the options for the write itself. The missing paths include the batch writer APIs called out during review of #57582, as well as schema-evolution and CTAS/RTAS reloads. Forwarding options through `DataFrameWriterV2` also exposed an existing key-casing issue: copying a `CaseInsensitiveStringMap` through its `Map` view lowercases its keys. That made command options and relation options differ for mixed-case keys and could fail `V2Writes.mergeOptions`. This PR copies through `asCaseSensitiveMap()` so adding and clearing the internal privilege marker preserves the user's original key casing. ### Does this PR introduce _any_ user-facing change? Yes. For catalog-backed DataSource V2 batch writes, user-provided write options are now also passed to `TableCatalog.loadTable(Identifier, TableContext, CaseInsensitiveStringMap)`. The same options continue to reach the actual V2 write. Streaming behavior is unchanged. ### How was this patch tested? Tests use distinct load and write options and verify both the catalog `loadTable` call and the actual V2 connector write callback. They also assert that the write is backed by a catalog V2 table rather than a V1 fallback. The following full Scala suites were run with `-Dtest=none -DwildcardSuites=<suite>`: * `CatalogV2UtilSuite` (12 tests) * `DataSourceV2OptionSuite` (33 tests) * `SupportsCatalogOptionsSuite` (19 tests) * `AppendDataTransactionSuite` (17 tests) * all 4 DELETE row-level suites (176 tests) * all 5 UPDATE row-level suites (249 tests) * all 5 MERGE row-level suites (396 tests) In total, 902 targeted Scala tests passed with 0 failures and 0 errors. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex CLI 0.145.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
