pan3793 opened a new pull request, #58646:
URL: https://github.com/apache/spark/pull/58646
### What changes were proposed in this pull request?
SPARK-59273 widened the fill-value type match in
`DataFrameNaFunctions.fillValue` from the exact `StringType` to the whole
string family:
```scala
- case (StringType, dt) => dt == StringType
+ case (StringType, _: StringType) => true
```
That pattern is no longer exhaustive for a string fill value: when the
schema contains any non-string column, that column falls through to the
throwing `case _` and `df.na.fill(<string>)` fails with
`IllegalArgumentException: StringType is not matched at fillValue`. This patch
keeps the string-family match but restores the skip for non-string columns:
```scala
case (StringType, dt) => dt.isInstanceOf[StringType]
```
which mirrors the `NumericType` case above it.
### Why are the changes needed?
`na.fill` with a string value must fill only string-family columns and leave
other columns untouched; a schema with any non-string column must not throw.
The regression breaks basic usage such as:
```scala
Seq[(String, Integer)]((null, null)).toDF("name", "age").na.fill("unknown")
```
and reddened master CI in `DataFrameNaFunctionsSuite`,
`SparkConnectProtoSuite`, and the PySpark `DataFrameStatTests.test_fillna` /
`DataFrameStatParityTests.test_fillna`. SPARK-59273 has not shipped in a
release.
### Does this PR introduce _any_ user-facing change?
No. It restores on master the pre-SPARK-59273 behavior of
`DataFrameNaFunctions.fill` with a string value.
### How was this patch tested?
Existing tests already exercise mixed-type schemas and both behaviors, and
failed on master after SPARK-59273:
- `DataFrameNaFunctionsSuite`: `fill`, `fill with col(*)`, `fill with nested
columns`
- `SparkConnectProtoSuite`: `SPARK-41128: Test fill na`
- PySpark `DataFrameStatTests.test_fillna`,
`DataFrameStatParityTests.test_fillna`
Ran locally:
```
build/sbt 'sql/testOnly *DataFrameNaFunctionsSuite'
build/sbt 'connect/testOnly *SparkConnectProtoSuite'
```
Both suites pass; no new tests added because the failing cases above cover
the regression.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: DeepSeek V4 Flash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
