rangareddy commented on issue #17314:
URL: https://github.com/apache/hudi/issues/17314#issuecomment-5365821530
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-8585).
**Findings: confirmed on `master`, with a version nuance worth recording.**
Two in-tree markers name this ticket. The load-bearing one is in
`hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:1533`,
inside `isColumnTypeSupportedV1`:
```java
// if record Type is not set or if recordType is SPARK then we cannot
support AVRO, MAP, ARRAY, RECORD, ENUM and FIXED and BYTES type as well.
// HUDI-8585 will add support for BYTES and FIXED
return type != HoodieSchemaType.RECORD && type != HoodieSchemaType.ARRAY &&
type != HoodieSchemaType.MAP
&& type != HoodieSchemaType.ENUM && type != HoodieSchemaType.BYTES &&
type != HoodieSchemaType.FIXED
&& type != HoodieSchemaType.DECIMAL // DECIMAL's underlying type is BYTES
...
```
and a second at `:1291` - *"Currently only avro is fully supported for
extracting column ranges (see HUDI-8585)"*.
**Nuance that changes the scope.** `isColumnTypeSupportedV2` (`:1542`) does
**not** exclude `BYTES` or `FIXED`, and `HoodieIndexVersion.getCurrentVersion`
returns `V2` for `COLUMN_STATS` / `PARTITION_STATS` at table version 9 and
above. So the type gap is specific to **index version V1**, i.e. table version
8 and below. Tables on version 9+ already index these types.
That does not close the ticket, because the ask here is a diagnostics one -
*"lets fix the user-exp by adding these as unsupported types upfront, rather
than throwing random errors"* - and that is still true on the V1 path: an
unsupported type is discovered mid-write rather than rejected up front with a
clear message.
Also for the record, PR #12331 (`[DNM] Enable MDT on reader and col stats by
default`), referenced in the description as where this surfaced, was **closed
without merging**.
Keeping this open, with the scope now: fail fast with a clear message for
BYTES/FIXED on V1, and note the V2 upgrade as the real fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
