deepakpanda93 commented on issue #11917: URL: https://github.com/apache/hudi/issues/11917#issuecomment-4850647052
@kishoreraj05 This is fixed in Hudi 1.x. There are three things happening here: - Block has already been inflated (what actually fails the job): when reading a metadata-table log block hits an I/O error, the block's content wasn't reset, so a retry tried to re-inflate it. As @yihua noted, this is fixed by HUDI-5240 / [#7434](https://github.com/apache/hudi/pull/7434) ("Reset the content during error handling in the log block reading"), which is in 1.0.0 and all later 1.x releases. - Could not read commit details from …replacecommit.requested: this comes from the failed-write rollback path reading a pending replacecommit instant that a concurrent rollback / delete_partition had already removed. It's handled by [#18288](https://github.com/apache/hudi/pull/18288) (the clustering-plan lookup now tolerates an instant rolled back by a concurrent writer), in 1.2.0. - Timeout waiting for connection from pool: this is the underlying S3 trigger — increase fs.s3a.connection.maximum (and related S3A pool settings) so the log-block reads don't time out. That's what surfaced (1) on 0.14.0. You were on 0.14.0, which predates the fixes. Please move to 1.2.0+ (which includes both [#7434](https://github.com/apache/hudi/pull/7434) and [#18288](https://github.com/apache/hudi/pull/18288)) and bump the S3A connection-pool size. If you still hit Block has already been inflated or the commit-details read error on 1.2.0+, please share the timeline. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
