prathik-onehouse opened a new issue, #20603: URL: https://github.com/apache/hudi/issues/20603
On a fresh table, the metadata table and the record index are initialized at the start of the first write, when the table holds no data. The record count estimate is 0, so the record index file group count is clamped to the configured minimum and stays there for the life of the table. The partitioned record index gets the minimum for every data partition, including partitions added later. `hoodie.metadata.record.level.index.defer.init` only waits for the second commit, which for a streaming writer is usually another small micro-batch. Proposal: defer the record index until the latest base files hold a configurable number of records, then build it sized from them, and size new partitions of the partitioned record index from the existing partitions and the records the commit inserts. Writes before that use the existing fallback index (GLOBAL_SIMPLE or SIMPLE), as they do today while the index is missing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
