dejankrak-db opened a new pull request, #58767:
URL: https://github.com/apache/spark/pull/58767
### What changes were proposed in this pull request?
Adds a per-write CSV option `treatNullAsEmptyString` that overrides the
session-level `spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv`
config
for a single write.
- When set to `false`, nulls are written as a bare, unquoted empty token
while
empty strings stay quoted (`""`), so downstream consumers can tell a null
apart from an actual empty string.
- When set to `true`, nulls are written through the `emptyValue` (a quoted
empty string `""` by default).
- When unset (including an explicit `null` value), the session config decides
the behavior, so existing writes are unaffected.
The option is parsed via the existing `getBool` helper, so a non-boolean
value
raises the structured `paramIsNotBooleanValueError` naming the option rather
than a bare error. It only takes effect when `nullValue` is left at its
default
(empty string); a non-empty `nullValue` is written verbatim, making the
option
a no-op, matching how the SQL config it overrides already composes with
`nullValue`.
### Why are the changes needed?
The behavior of writing a null vs. an empty string in CSV is currently only
controllable through the session-level
`spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` config. A workload
that needs to differentiate null from an empty string in its output cannot do
so per-write without changing the session/workspace default, which is not
always settable. A per-write option lets a single write opt into the
differentiation without affecting other writes.
### Does this PR introduce _any_ user-facing change?
Yes, it adds a new CSV write option `treatNullAsEmptyString`. Behavior is
unchanged when the option is not set (the session config continues to
decide),
so existing writes are unaffected.
### How was this patch tested?
Added unit tests in `CSVSuite` covering: the option winning over the session
config for both `true`/`false`, the no-op behavior when a non-empty
`nullValue`
is set, explicit-null handling (falls back to the session config), and the
structured error raised for a non-boolean value. Also updated the
`validate CSV Options` test for the new option.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
