cloud-fan opened a new pull request, #56919:
URL: https://github.com/apache/spark/pull/56919
### What changes were proposed in this pull request?
This refactors XML primitive-type schema inference in `XmlInferSchema` from
a stateless "probe every candidate type from scratch for each value" cascade
into a `typeSoFar`-driven refinement, matching the design already used by the
CSV datasource (`CSVInferSchema.inferField`) and, conceptually, JSON.
Before, `inferFrom(datum)` evaluated an independent chain of `is*`
predicates (`isLong`, `isDouble`, `isBoolean`, `isDate`, `isTimestamp`, ...) on
every value from `NullType`, regardless of what had already been inferred for
the field. Now:
- `inferFrom(datum)` seeds inference with `inferFrom(datum, NullType)`.
- `inferFrom(datum, typeSoFar)` dispatches on the type accumulated so far
and only ever attempts to *widen* it. Given a field already known to be
`Double`, there is no point re-checking whether the value is a `Long`; the
merged type can only be `Double` or wider. `compatibleType` reconciles the
newly inferred type with `typeSoFar`, falling back to `StringType` (the top
type) when they are incompatible.
- The `is*` predicates are replaced by a `tryParse*` cascade (`tryParseLong
-> tryParseDecimal -> tryParseDouble -> tryParseTime -> tryParseDate ->
tryParseTimestampNTZ -> tryParseTimestamp -> tryParseBoolean -> String`), where
each parser falls through to the next-wider type on failure. Entering the
cascade at the parser matching `typeSoFar` keeps the type lattice monotonic.
This is a behavior-preserving refactor: the type lattice, the ordering of
inference attempts, and the results are unchanged. Notably, date inference
remains unconditional (the `preferDate` option continues to only select which
date formatter is used, not whether date inference is attempted), and
nanosecond timestamp inference (SPARK-57458) is preserved.
### Why are the changes needed?
The three text-based datasources — CSV, JSON, and XML — all perform the same
job (per-field type inference merged across rows), but XML was the only one
using the stateless "try every type from scratch" approach. Aligning XML with
the CSV/JSON `typeSoFar` model:
- Removes redundant work: once a field has widened to (say) `Double`,
subsequent values no longer re-run the narrower `Long` checks.
- Makes the inference monotonic and easier to reason about (a field's type
can only stay or widen, never narrow).
- Reduces gratuitous divergence between the three inference implementations,
making them easier to maintain together.
### Does this PR introduce _any_ user-facing change?
No. This is an internal refactor with no change to inferred schemas.
### How was this patch tested?
- Existing `XmlInferSchemaSuite` and `XmlSuite` pass unchanged (202 tests).
- Added two regression tests to `XmlInferSchemaSuite`:
- `incremental inference widens the type across rows monotonically` —
verifies `Long -> Double` does not narrow back to `Long`, an incompatible value
widens to `String`, and the merge is order-independent.
- `date is inferred regardless of preferDate` — guards the invariant that
`preferDate` selects the date formatter rather than gating date inference.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic Claude Opus)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
