brijrajk opened a new pull request, #56154:
URL: https://github.com/apache/spark/pull/56154
### What changes were proposed in this pull request?
Clarify the Scaladoc and PySpark docstring for three public `RuntimeConfig`
methods whose descriptions did not accurately explain the interaction between
an explicitly-set value and the key's built-in (`ConfigEntry`) default value.
**`get(key: String)` / `get(key)`**
- Before: *"If the key is not set yet, return its default value if possible"*
- After: *"If the key is not explicitly set, return its built-in default
value if one exists"*
**`get(key: String, default: String)` / `get(key, default)`**
- Before: *"If the key is not set yet, return the user given `default`"* —
silent about overriding the ConfigEntry default
- After: explicitly states the user-supplied `default` is returned *instead
of* the key's built-in default value
**`getOption(key: String)`**
- Before: *"return its default value if possible, otherwise `None`"* — vague
- After: *"return `Some` of its built-in default value if one exists,
otherwise `None`"*
**PySpark `RuntimeConfig.get()`**
- Removed the misleading phrase *"assuming it is set"*
- Rewrote the description to explain both calling forms separately
- Added a `Raises` section documenting `SparkNoSuchElementException`
- Fixed the `Returns` section to follow NumPy docstring format
- Added four illustrative doctest examples using
`spark.sql.sources.partitionOverwriteMode` that demonstrate the
built-in-default behaviour
---
### Why are the changes needed?
The documentation was misleading in a subtle but important way. Users
reading `get(key)` would think it throws whenever the key is *unset*, but in
fact it returns the ConfigEntry's built-in default silently. Similarly, users
reading `get(key, default)` had no way to know the supplied default overrides
the built-in default — not just fills a gap when no default exists at all.
A concrete example (from SPARK-49798):
```python
spark.conf.unset("spark.sql.session.timeZone")
# Old doc implied this would throw — it does NOT; returns built-in default
"Etc/UTC"
spark.conf.get("spark.sql.session.timeZone")
# => "Etc/UTC"
# Old doc did not mention this ignores the built-in default "Etc/UTC"
spark.conf.get("spark.sql.session.timeZone", "Europe/Berlin")
# => "Europe/Berlin"
```
---
### Does this PR introduce _any_ user-facing change?
No.
---
### How was this patch tested?
No new tests added. The described behaviour is already fully covered by
existing tests:
- **Scala:** `RuntimeConfigSuite` — *"set and get a config with
defaultValue"* tests all three cases (`get(key)` returning built-in default,
`getOption(key)` returning `Some(builtInDefault)`, and `get(key, userDefault)`
overriding the built-in default)
- **Python:** `pyspark/sql/tests/test_conf.py` — `test_conf` (lines 38–45)
asserts the same behaviour using `spark.sql.sources.partitionOverwriteMode`
Four new doctest examples were added to `pyspark/sql/conf.py`. They are
executable via `python python/pyspark/sql/conf.py` and assert the same cases
the unit tests cover.
---
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
