varun-lakhyani commented on issue #15287: URL: https://github.com/apache/iceberg/issues/15287#issuecomment-3946017120
@RussellSpitzer I have completed rough desgin/POC along with test implementation and Benchmarking of new async v/s exisitng sync. **Benchmarking details:** Basically little latency overhead is created manually using LockSupport.parkNanos(1_000_000) in open () function in org/apache/iceberg/spark/source/BatchDataReader.java to stimulate real IO overhead caused by cloud storages. ( used @Warmup(iterations = 5) @Measurement(iterations = 15) for benchmarking) Result for 1000 files - 15-20 Kb each compaction (rewrite_data_files) for various cases: <meta charset="utf-8"><b style="font-weight:normal;" id="docs-internal-guid-465efd34-7fff-cd51-7fd1-124fe6f531f9"><div dir="ltr" style="margin-left:0pt;" align="left"> Overhead (ms) | Async (s) | Sync (s) (existing) | % Improvement -- | -- | -- | -- No manual Overhead | 0.765 | 0.932 | 17.9% 1 | 0.772 | 2.881 | 73.2% 5 | 1.778 | 8.512 | 79.1% 10 | 3.284 | 15.159 | 78.3% 15 | 4.709 | 21.260 | 77.8% </div></b> Detailed results for 100, 500, 1000 files for all overheads are mentioned in reference design document: https://docs.google.com/document/d/17vBz5t-gSDdmB0S40MYRceyvmcBSzw9Gii-FcU97Lds/edit?usp=sharing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
