voonhous opened a new issue, #19826:
URL: https://github.com/apache/hudi/issues/19826
### Bug Description
`HoodieSchemaUtils.generateProjectionSchema` matches field names
case-insensitively by lowercasing both the schema's field names and the
requested names with the default locale (`HoodieSchemaUtils.java:475` and
`:478`, `toLowerCase()`). The caller that relies on the case-insensitive
contract, `HiveHoodieReaderContext.java:236`, pre-lowercases the requested
names with `Locale.ROOT`.
Under a Turkish or Azeri default JVM locale the two disagree for any column
whose name contains an upper-case `I`: the schema-side key becomes dotless-i
(`toLowerCase()` maps `I` to U+0131), the request-side name is `id`, the map
lookup misses and the read fails with `Field id not found in log schema. Query
cannot proceed!`. The other callers (`HoodieFileGroupReaderBasedRecordReader`,
`HoodieAvroParquetReader`, `MergeIntoHoodieTableCommand`,
`BaseBootstrapMetadataHandler`) lowercase both sides through the same call, so
they stay consistent and are not affected.
Pre-existing; #19809 only documents the case-insensitive matching and adds a
test for it (`_ROW_KEY`, which has no `I`, so the test cannot expose this). The
repo has locale precedent in `6629bf7287fd` (#7215).
Fix direction: `toLowerCase(Locale.ROOT)` on both calls in
`generateProjectionSchema`, and use a name with an `I` (`PII_COL` against
`EXAMPLE_SCHEMA`) in `TestHoodieSchemaUtils#testGenerateProjectionSchema`.
Worth noting in the same javadoc that the lowercase `toMap` throws
`IllegalStateException: Duplicate key` for two fields differing only in case.
### Environment
- Hudi master (`c59987a024cd`), Hive reads through
`HiveHoodieReaderContext`, JVM default locale `tr_TR` or `az_AZ`
### Logs and Stack Trace
```
org.apache.hudi.exception.HoodieException: Field id not found in log schema.
Query cannot proceed! Derived Schema Fields: [...]
at
org.apache.hudi.common.schema.HoodieSchemaUtils.generateProjectionSchema(HoodieSchemaUtils.java:481)
at
org.apache.hudi.hadoop.HiveHoodieReaderContext.<init>(HiveHoodieReaderContext.java:241)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
