matthewbayer opened a new pull request, #59404: URL: https://github.com/apache/spark/pull/59404
### What changes were proposed in this pull request? Restore the `spark-xml` timestamp parser for XML `TIMESTAMP` reads and inference under `spark.sql.legacy.timeParserPolicy=LEGACY`. The XML-specific helper tries ISO-8601 formats before an optional custom format, as `spark-xml` did. `CORRECTED`, `EXCEPTION`, `TIMESTAMP_NTZ`, and nanosecond timestamp parsing continue to use their existing formatters. No shared timestamp formatter or session configuration is changed. ### Why are the changes needed? [SPARK-60199](https://issues.apache.org/jira/browse/SPARK-60199): the built-in XML reader applies a legacy `SSS` pattern to longer fractional seconds, accepting the input but changing its value. For example, `2024-02-29T12:34:56.123456Z` becomes `2024-02-29T12:36:59.456Z`. Both `PERMISSIVE` and `FAILFAST` return the wrong value because parsing succeeds. The previous `spark-xml` connector parsed these timestamps independently of Spark's legacy formatter. Changing the session-wide policy is not an XML-only workaround. ### Does this PR introduce _any_ user-facing change? Yes. XML reads under `LEGACY` preserve fractional seconds and restore the connector's ISO timestamp handling: optional fractions, offsets, UTC for unzoned ISO timestamps, and ISO-before-custom-format precedence. Historical timestamps retain the connector's `java.sql.Timestamp` conversion. Malformed ISO input is rejected instead of normalized (null in `PERMISSIVE`, an error in `FAILFAST`). ### How was this patch tested? Added regression coverage for file reads with explicit and inferred schemas, both XML parser implementations, `PERMISSIVE`/`FAILFAST`, custom formats, historical dates, and `from_xml`. Added checks that modern and nanosecond parser behavior remains unchanged. The regression suite reproduces the corruption on unpatched upstream master. All 40 focused tests pass with the fix: `XmlLegacyTimestampSuite`, `XmlTimestampParserSuite`, `XmlInferSchemaTypeCastingSuite`, and the timestamp tests in `XmlSuite` and `XmlSuiteWithLegacyParser`. Catalyst production/test and SQL test Scalastyle checks pass. The full Spark suite was not run locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) This contribution is licensed under the Apache License, Version 2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
