voonhous commented on issue #20628: URL: https://github.com/apache/hudi/issues/20628#issuecomment-6822382673
Findings so far (local; fix is on a local branch, PR to follow). **Per-split cost on master, beyond the issue description.** - The schema resolver's operation-field probe reads the schema of a file from the latest commit: for a parquet base file that is an exception and an INFO log line per split with no I/O; for a legacy Avro log file it opens the file and walks every block header. - With `hudi.table.resolve-column-name-casing.enabled=true` the per-split schema work goes away but the per-split meta client stays. On tables before version 8, log block commit validation then loads the commits timeline per split through the meta-client-backed `TableState`, so the flag does not remove the timeline listing there. - That flag also changes predicate key mapping in index pruning (`IndexSupportFactory`), so it is not a pure performance switch. **Fix measured.** The coordinator resolves the table schema once per query for MERGE_ON_READ handles and ships it in the handle along with the table config map (without the create schema) and, for tables before version 8, the committed instants. Workers build a `TableState` from it and no longer build a meta client. Index pruning still checks the casing flag, so its behavior is unchanged. `HudiMorReadBenchmark` scan `_rt`, default session, default latency profile (20/15/40 ms), 48 partitions: | | master | fix | |---|---|---| | `hoodie.properties` reads | 50 | 2 | | timeline reads | 97 | 2 | | log file GETs | 432 | 385 | | total storage requests | 723 | 533 | | wall | 4.22 s | 3.59 s | The request counts are identical at 96 partitions. "Fetched table schema" is logged once per query, on a coordinator thread. Handle size on a 40-column table: about 1.2 KB for the config map, plus the schema JSON (about 5.9 KB), which was already shipped when casing resolution was on. Behavior changes to review: schema resolution now runs once per MOR snapshot query on the coordinator even when no split has log files, and a resolution failure surfaces from handle serialization instead of from each split. The existing file-operation tests only query read-optimized tables, which never reach the merge path, so they had no per-split reads to remove; snapshot-read cases were added instead (version 8 table with 3 file groups: `TABLE_PROPERTIES` 5 to 2, log opens 9 to 7). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
