cloud-fan opened a new pull request, #57265:
URL: https://github.com/apache/spark/pull/57265
### What changes were proposed in this pull request?
In `XmlInferSchema.inferFrom`, stop skipping empty values during schema
inference. Previously the guard was:
```scala
if (value == null || value.isEmpty || value == options.nullValue) {
return typeSoFar
}
```
This PR drops the `value.isEmpty` term:
```scala
if (value == null || value == options.nullValue) {
return typeSoFar
}
```
so an empty value now falls through the type cascade to `StringType`. As a
result, a field mixing empty and numeric values widens to `StringType` instead
of being inferred as the numeric type.
### Why are the changes needed?
XML schema inference was copied from `CSVInferSchema.inferField`, which does
early-return on `field.isEmpty`. But the two datasources have a crucial
difference at parse time:
- CSV defaults `nullValue` to `""`, so an empty field is read as null by the
parser (`UnivocityParser.nullSafeDatum`). Its inference can safely skip empty
values because they never reach a numeric converter.
- XML defaults `nullValue` to `null`, so an empty value is **not** read as
null. If a column mixing empty and numeric values inferred `LongType`,
`StaxXmlParser` would call `convertTo("", LongType)`, which throws
`NumberFormatException`. In `PERMISSIVE` mode (the default), the
malformed-record error recovery can then silently consume sibling XML events,
**dropping records** — a silent data-loss bug.
Concretely, for input
```xml
<ROW><entry Code="001">first</entry><entry Code="002">second</entry></ROW>
<ROW><entry Code="">third</entry><entry Code="003">fourth</entry></ROW>
<ROW><entry Code="004">fifth</entry><entry Code="">sixth</entry></ROW>
```
`entry._Code` was inferred as `LongType` (from the numeric `Code` values,
empty skipped), and reading then dropped records when it hit the empty
`Code=""`. After this change `_Code` infers as `StringType` and all rows are
read.
By letting empty values fall through to `StringType`, inference stays
consistent with how the XML parser actually reads the data.
### Does this PR introduce _any_ user-facing change?
Yes. With the default `nullValue` (unset), a field that mixes empty and
typed (e.g. numeric) values is now inferred as `StringType` instead of the
typed type. This prevents the silent record-dropping described above. Fields
that are entirely typed, or that use a non-default `nullValue` covering the
empty token, are unaffected. Empty *elements* (`<c/>`) are still read as
`NullType` by the parser and are unaffected.
### How was this patch tested?
Added two tests to `XmlInferSchemaSuite`:
- `SPARK-58133: empty values are not skipped during inference (no data
loss)` — asserts that an attribute mixing empty and numeric values infers
`StringType` and that all rows/entries survive (no PERMISSIVE-mode record
dropping).
- `SPARK-58133: empty and numeric element values still infer via NullType` —
documents that empty elements keep inferring `LongType` (via the parser's
`NullType` path), pinning the element/attribute asymmetry.
Existing XML inference suites pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic Claude Opus)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
