dwsmith1983 opened a new pull request, #59275: URL: https://github.com/apache/spark/pull/59275
### What changes were proposed in this pull request? `OrcUtils.requestedColumnIds` runs once per file split in the V1 and V2 ORC readers. To reject timestamp family or precision mismatches with a clear error, it called `checkTimestampCompatibility(toCatalystSchema(orcSchema), dataSchema)`, which converts the whole file schema and runs `CatalystSqlParser.parseDataType` for every primitive column. This PR converts and checks only the file columns whose read type is, or contains, a timestamp type. `checkTypeCompatibility` can only fail on a timestamp in the read type, so the skipped columns could never fail the check. The type conversion helpers move out of `toCatalystSchema` into private members of `OrcUtils` so a single column can be converted; their logic is unchanged. ### Why are the changes needed? Performance on wide ORC files. The conversion cost grows with the number of columns in the file, not with the columns the query reads, and it is paid on every split. On 400 ORC files with 1000 INT columns (local[1], JDK 17, best of 3): | | Before | After | |---|---|---| | `toCatalystSchema` on the 1000-column file schema, per call | 0.75 ms | not called | | `SELECT c0` | 715 ms | 384 ms | | `SELECT *` | 12425 ms | 12199 ms | ### Does this PR introduce _any_ user-facing change? Errors for timestamp mismatches are the same. One case now succeeds: a file column whose ORC type the SQL type parser cannot handle (for example a Hive `uniontype`) used to fail every read of the file in this check, even when its read type had no timestamp. It is now only converted, and so only rejected, when its read type contains a timestamp. Reads with pushed-down filters still convert the full file schema for predicate pushdown, so they behave as before; that path is left for a separate change. ### How was this patch tested? New `OrcUtilsSuite` calling `requestedColumnIds` directly: - a `uniontype` file column read as `STRING` (fails on master with `PARSE_SYNTAX_ERROR`, passes now) - timestamp mismatches still raise `UNSUPPORTED_FEATURE.ORC_TYPE_CAST` when the column comes after non-timestamp columns, sits in a map value or a nested struct, or is next to an unconverted `uniontype` column - same-kind nanos timestamps at a different precision are accepted Existing suites pass: `OrcV1QuerySuite`, `OrcV2QuerySuite`, `OrcSourceV1Suite`, `OrcSourceV2Suite`, `OrcV1SchemaPruningSuite`, `OrcV2SchemaPruningSuite`, `OrcPartitionReaderFactorySuite`, `OrcColumnarBatchReaderSuite`, and the ORC timestamp tests in `QueryExecutionErrorsSuite`. The numbers above come from a local benchmark run before and after the change on the same commit. ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
