Raviston6296 commented on PR #60204:
URL: https://github.com/apache/spark/pull/60204#issuecomment-6711918544
The fix looks right to me. On master, the native reader builds
`orc.mapred.input.schema` from
`getOrcSchemaString(orcReadSchemaWithoutCharVarchar(readSchema))`, and a
collated `StringType` falls through to `dt.catalogString`, i.e. `string collate
UTF8_LCASE`, which `TypeDescription.fromString` rejects. The write side already
works, because `orcTypeDescription` builds a `STRING` `TypeDescription`
directly for every `StringType`, and that is why `INSERT` succeeds and only the
read fails. `CHAR`/`VARCHAR` keep producing the same `char(n)`/`varchar(n)`
strings as before. On the read path they're already turned into `StringType` by
`orcReadSchemaWithoutCharVarchar`, so the new case only changes plain collated
strings there.
Two suggestions for the tests:
1. **Make the predicate check sensitive to pushdown.** `k = 'a'` returning
`a` and `A` is right today because `DataSourceStrategy` translates it into
`CollatedEqualTo`, which `OrcFilters` doesn't push. But with `('a'), ('A'),
('b')` in one file, the stripe stats are `min = 'A'`, `max = 'b'`, so even a
wrongly pushed binary `k = 'a'` would keep that stripe, and the test would
still pass. Writing `'A'` in its own `INSERT` gives it a file whose stats are
`min = max = 'A'`. A binary `= 'a'` would then skip that file, so a regression
that pushes collated predicates to ORC would make the test fail.
2. **Cover more collations.** #59887 (SPARK-60642, the Hive ORC writer fix)
tested `UTF8_BINARY`, `UTF8_LCASE` and `UNICODE_CI`. Explicit `UTF8_BINARY`
matters because `StringType("UTF8_BINARY")` prints `string collate UTF8_BINARY`
too, unlike the default `StringType`. `UNICODE_CI` is an ICU collation rather
than a Spark-native one. Adding both here, or making the test loop over the
collations, would cover the same cases for the native reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
