voonhous opened a new issue, #20628: URL: https://github.com/apache/hudi/issues/20628
Part of #20625. Line references are against master `db90a0da`. ### Problem For every split with log files, `HudiPageSourceProvider.createPageSource`: - builds a new `HoodieTableMetaClient` (`HudiPageSourceProvider.java:232-234`, has a TODO), which reads `hoodie.properties`, plus the index definitions file when one is registered; - unless `hudi.table.resolve-column-name-casing.enabled=true` (default false), runs `TableSchemaResolver.getTableSchema()`: a recursive timeline listing, at least one instant file read with full commit metadata parsing, and an operation-field probe that can open a log file. Nothing is cached across splits. Measured locally with `HudiMorReadBenchmark` scan (48 splits): `_rt` does 50 `hoodie.properties` reads and 97 timeline reads, against 2 and 1 for `_ro`. ### Direction Resolve the schema, and what the readers need from the table config, once on the coordinator and carry it in `HudiTableHandle` or the split always, not only when casing resolution is on. Stop building a meta client per split; `TableState` is serializable and `HoodieFileGroupReader` accepts it in place of a meta client. ### Exit criteria - `HudiMorReadBenchmark` scan `_rt` with the default session: timeline reads <= 2 and `hoodie.properties` reads <= 2 per query, identical at `-Dbench.partitions=48` and `96`. - No "Fetched table schema" log line on workers during a MOR query. - `TestHudiNoCacheFileOperations` and the cache variants updated: the per-split `TABLE_PROPERTIES` reads disappear from the expected multisets. - Schema-evolution and column-casing tests green: a schema change between queries is picked up by the next query. - Scan `_rt` wall time at the default latency profile improves against master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
