Hans-Raintree commented on issue #10334: URL: https://github.com/apache/hudi/issues/10334#issuecomment-5725930454
I'm having a similar issue on Hudi 1.1.0. (S3, EMR, Metadata table off) I had cleaner.commits.retained and hoodie.keep.max.commits and hoodie.keep.min.commits too low. I was using hoodie.datasource.read.incr.path.glob to read a single partition and it took 105 seconds, when I turned fallback off it produced the same result in 2.7 seconds. As this option is now true by default, I think this code path should be optimized. Some options would be: 1. Filter affected files before checking existence (when using hoodie.datasource.read.incr.path.glob) 2. Check only file slices actually needed by the query 3. Run existence checks concurrently (currently it's sequential) 4. Use Hudi cleaner metadata instead of probing S3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
