This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b1fc6eb16620 [SPARK-58104][SQL] Make XML schema inference honor the
nullValue option, consistent with CSV
b1fc6eb16620 is described below
commit b1fc6eb166206df6babb52cb9ed61726122b493d
Author: Wenchen Fan <[email protected]>
AuthorDate: Tue Jul 14 08:46:18 2026 +0200
[SPARK-58104][SQL] Make XML schema inference honor the nullValue option,
consistent with CSV
### What changes were proposed in this pull request?
Gate XML schema inference on the `nullValue` option. In
`XmlInferSchema.inferFrom(datum, typeSoFar)`, a value equal to
`options.nullValue` now preserves `typeSoFar` (is skipped for type inference)
instead of being inferred by its string content:
```scala
if (value == null || value.isEmpty || value == options.nullValue) {
return typeSoFar
}
```
This matches `CSVInferSchema.inferField`, which already gates on
`nullValue`:
```scala
if (field == null || field.isEmpty || field == options.nullValue) {
typeSoFar
} else { ... }
```
### Why are the changes needed?
The XML **parser** already treats a value equal to `nullValue` as null when
reading (see the `options.nullValue` checks in `StaxXmlParser`), but schema
**inference** did not skip it — so a value equal to `nullValue` was inferred
from its string content. This produces an inference-vs-parse inconsistency: the
inferred type does not reflect how the data is actually read.
Concretely, with `nullValue=1` and input `<ROW>1</ROW>`, inference produced
`LongType` for the value tag while the parser read the value as `null`. It also
diverges from the CSV datasource, whose `inferField` gates date/numeric/etc.
inference on `nullValue`. XML inference is meant to mirror CSV (the two text
datasources share the same `tryParse*` cascade design), so this aligns the two.
### Does this PR introduce _any_ user-facing change?
Yes. With a non-default `nullValue`, XML schema inference no longer infers
a type from values equal to `nullValue`. For example, with `nullValue=1`,
`<ROW>1</ROW>` previously inferred the value tag as `LongType` (while the data
read as null); it now infers `StringType` (the canonicalized type of an
all-null field) and still reads null. With the default `nullValue` (unset),
behavior is unchanged.
### How was this patch tested?
- Added a unit test in `XmlInferSchemaTypeCastingSuite` ("A value matching
the nullValue option keeps the type inferred so far") asserting that a
`nullValue`-matching value preserves `typeSoFar` from
`NullType`/`LongType`/`TimestampType`, that a non-null value is still inferred
normally, and that a numeric-looking `nullValue` token (`nullValue=0`, value
`0`) is treated as null rather than inferred as `LongType`.
- Updated the existing `XmlInferSchemaSuite` end-to-end test "value tag -
equals to null value" to the corrected schema (`StringType` instead of
`LongType`), reflecting that the all-null value tag now canonicalizes to
`StringType`.
- Existing XML inference suites pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic Claude Opus)
Closes #57229 from cloud-fan/xml-infer-nullvalue.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
.../spark/sql/catalyst/xml/XmlInferSchema.scala | 7 ++++++-
.../xml/XmlInferSchemaTypeCastingSuite.scala | 20 ++++++++++++++++++--
.../datasources/xml/XmlInferSchemaSuite.scala | 7 ++++---
3 files changed, 28 insertions(+), 6 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala
index 234de9160421..9df0ebcec7ae 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala
@@ -366,7 +366,12 @@ class XmlInferSchema(private val options: XmlOptions,
private val caseSensitive:
if (!options.inferSchema) {
return StringType
}
- if (value == null || value.isEmpty) {
+ // A value matching the `nullValue` option carries no type information, so
it preserves the
+ // type inferred so far -- matching `CSVInferSchema.inferField`. The XML
parser already reads
+ // such values as null (see `StaxXmlParser`), so inference must skip them
too; otherwise a
+ // column that is entirely `nullValue`s (or mixes them with a typed value)
would infer the
+ // string content of the token instead of ignoring it.
+ if (value == null || value.isEmpty || value == options.nullValue) {
return typeSoFar
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchemaTypeCastingSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchemaTypeCastingSuite.scala
index e5570ddde88f..94cd261ae185 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchemaTypeCastingSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchemaTypeCastingSuite.scala
@@ -104,14 +104,30 @@ class XmlInferSchemaTypeCastingSuite extends
SparkFunSuite with SQLHelper {
val inferSchema = newInferSchema(Map.empty[String, String])
// A genuinely empty/null value carries no type information, so
`typeSoFar` is preserved.
- // (Unlike CSV, XML inference does not treat the `nullValue` option as
null here; a value
- // equal to `nullValue` is inferred by its content, matching the
pre-existing behavior.)
assert(inferSchema.inferFrom("", NullType) == NullType)
assert(inferSchema.inferFrom("", LongType) == LongType)
assert(inferSchema.inferFrom(null, DoubleType) == DoubleType)
assert(inferSchema.inferFrom(null, TimestampType) == TimestampType)
}
+ test("A value matching the nullValue option keeps the type inferred so far")
{
+ // A token equal to the `nullValue` option is read as null by the parser,
so inference must
+ // ignore it and preserve `typeSoFar` -- matching
`CSVInferSchema.inferField`. Otherwise a
+ // `nullValue` token would be inferred by its string content and
(incorrectly) widen the field.
+ val inferSchema = newInferSchema(Map("nullValue" -> "NA"))
+ assert(inferSchema.inferFrom("NA", NullType) == NullType)
+ assert(inferSchema.inferFrom("NA", LongType) == LongType)
+ assert(inferSchema.inferFrom("NA", TimestampType) == TimestampType)
+ // A non-null value is still inferred by content as usual.
+ assert(inferSchema.inferFrom("5", NullType) == LongType)
+
+ // The nullValue token itself, when it happens to look like another type,
is still treated as
+ // null rather than inferred as that type.
+ val inferSchemaNumericNull = newInferSchema(Map("nullValue" -> "0"))
+ assert(inferSchemaNumericNull.inferFrom("0", NullType) == NullType)
+ assert(inferSchemaNumericNull.inferFrom("1", NullType) == LongType)
+ }
+
test("Refining from a wider type never narrows and is consistent with fresh
inference") {
val inferSchema = newInferSchema(Map.empty[String, String])
// Once a field is Double, an integral value keeps it Double (does not
narrow to Long).
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlInferSchemaSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlInferSchemaSuite.scala
index 18cfe17d433e..e1579a53b9dd 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlInferSchemaSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlInferSchemaSuite.scala
@@ -641,12 +641,13 @@ class XmlInferSchemaSuite
}
test("value tag - equals to null value") {
- // we don't consider options.nullValue during schema inference
+ // A value tag equal to the nullValue option is treated as null during
inference (matching
+ // CSVInferSchema and the parser, which reads it as null), so it carries
no type and the
+ // all-null field canonicalizes to StringType rather than being inferred
as LongType.
val xmlDF = readData(valueTagIsNullValue, Map("nullValue" -> "1"))
val expectedSchema = new StructType()
- .add(valueTagName, LongType)
+ .add(valueTagName, StringType)
val expectedAns = Seq(Row(null))
- // nullValue option is used during parsing
assert(xmlDF.schema === expectedSchema)
checkAnswer(xmlDF, expectedAns)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
