haoyangeng-db opened a new pull request, #58951:
URL: https://github.com/apache/spark/pull/58951
### What changes were proposed in this pull request?
This PR makes Spark Connect's Python `DataFrameReader.option` handling
case-insensitive.
When a new option matches an existing key ignoring case, it removes the
existing entry and stores
the newest spelling and value.
It also adds plan serialization coverage for both named-table and
data-source reads.
### Why are the changes needed?
Spark reader option keys are case-insensitive and later calls should
override earlier values.
Spark Connect currently stores options in a regular Python dictionary, so
differently cased forms
of the same logical key can both reach the protobuf map. The server then
resolves those duplicates
from an unordered map, which can select a value that does not match the
user's call order.
For example, this should deterministically produce one `versionAsOf` option
with value `2`:
```python
spark.read.option("versionAsOf", 0).option("versionasof",
1).option("versionAsOf", 2)
```
The fix preserves the spelling of the most recent key instead of lowercasing
keys, so vendor option
names such as `oracle.jdbc.timezoneAsRegion` remain unchanged.
### Does this PR introduce _any_ user-facing change?
Yes. Spark Connect Python reader options now follow case-insensitive,
last-call-wins semantics,
matching the classic DataFrameReader behavior. Workloads that supplied the
same logical option
using multiple casing variants now deterministically use the most recently
supplied value.
### How was this patch tested?
- Added regression coverage in `SparkConnectPlanTests` for mixed-case
named-table options and
repeated data-source options.
- Ran `pyspark.sql.tests.connect.test_connect_plan`: 83 tests passed before
the clean rebase onto
the latest `upstream/master`; the rebase did not change the patch.
- Ran Python compilation checks for both modified files after rebasing.
- Ran `git diff --check` and line-length checks after rebasing.
- A post-rebase suite rerun was blocked because the fresh worktree lacks the
optional Connect test
dependencies, and installing them was blocked by DNS resolution to PyPI.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex 0.154.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
