voonhous opened a new issue, #19279:
URL: https://github.com/apache/hudi/issues/19279
Parent: #18780 (RFC-105 hudi-trino connector migration)
## Summary
With the connector default `hudi.metadata-enabled=true`, any query against a
Hudi 1.x table whose metadata table (MDT) has uncompacted delta commits fails
outright:
```
Query ... failed: Failed to generate splits for default.<table>
Caused by: java.lang.UnsupportedOperationException:
Native HFILE log files are not supported by the Hudi Trino connector
```
Since MDT compaction only runs every
`hoodie.metadata.compact.max.delta.commits` (default 10) delta commits, an
actively written table spends most of its life in this state. In other words:
default Spark writer + default connector config = every read fails.
## Root cause
1. `HudiBackgroundSplitLoader.getPartitionInfos` invokes partition-stats
index pruning (`HudiPartitionStatsIndexSupport.prunePartitions`) whenever the
MDT is enabled and the index exists.
2. The pruning read goes through `HoodieBackedTableMetadata` ->
`HoodieMergedLogRecordReader`, which must scan the MDT partition's log files.
Fresh (uncompacted) MDT deltas are log files with HFILE data blocks.
3. `HudiTrinoIOFactory` rejects HFILE log blocks
(`UnsupportedOperationException`). HFile *base* files are fine (read via
hudi-io's native Java HFile reader); only the log-block path is unimplemented.
4. Unlike the MDT partition-listing path right above it (which catches
exceptions and falls back to metastore-based listing), the pruning call has
**no guard**, so the exception propagates and fails the whole query. The
MDT-backed file listing path has the same exposure once pruning is fixed.
## Why the existing hudi-trino tests do not catch this
The zip fixtures under `hudi-trino/src/test/resources/hudi-testing-data/`
were generated with:
```
SET hoodie.metadata.compact.max.delta.commits=1
```
so every MDT partition is always freshly compacted and all MDT reads are
served from HFile base files, never from HFILE log blocks. The tests exercise
MDT reads only in the one state the connector can read.
## Reproduction
1. Create any Hudi table with Spark using default configs and write 1-2
commits (MDT gets HFILE log deltas, no compaction yet).
2. Query it through the trino-hudi connector with default catalog config.
3. Every query on the table fails with the stack above.
Found via the RFC-105 E2E testcontainers pipeline (Trino 481, `hudi-trino`
at current HEAD): freshly seeded COW/MOR tables failed 25/29 E2E assertions
until `hudi.metadata-enabled=false` was set in the catalog.
## Proposed fix
Two levels, not mutually exclusive:
1. **Guard/degrade (small, immediate):** wrap the pruning call in
`getPartitionInfos` with the same catch-and-fallback treatment as the
partition-listing path (index pruning is an optimization and must not fail the
query). Consider disabling MDT usage for the remainder of the query on the
first MDT read failure, since the directory lister will hit the same HFILE wall.
2. **Support HFILE log data blocks (proper):** wire hudi-io's native HFile
reader (already used for HFile base files) into the log-block path in
`HudiTrinoIOFactory`, analogous to the RFC-103 native parquet log-block
support. This makes MDT reads work in all compaction states and lets the
connector keep `hudi.metadata-enabled=true` as a safe default.
## Workaround
Set `hudi.metadata-enabled=false` in the catalog properties. This bypasses
all MDT reads (metastore partition listing + filesystem-based file listing) at
the cost of losing MDT-driven listing and index pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
