varun-lakhyani commented on PR #17284: URL: https://github.com/apache/iceberg/pull/17284#issuecomment-5024166803
### Benchmark Setup (JMH) **Machine:** AWS EC2 (same region as the S3 bucket to minimize network latency) | Property | Value | |----------|-------| | Instance type | r5.4xlarge | | vCPUs | 16 | | Memory | 128 GB | | Network | Up to 10 Gbps | | Storage | 50 GB gp3 EBS | | AMI | Amazon Linux 2023 | | Region | ap-south-1 (Mumbai) | **Benchmark:** [ManifestBenchmark](https://github.com/apache/iceberg/blob/1ec15051dc09ef6a0d61fa332eb8f2503f3170f8/core/src/jmh/java/org/apache/iceberg/ManifestBenchmark.java#L86) - Warmup iterations: **6** - Measurement iterations: **10** - Entry Counts: **37500** ### Results #### Parquet v4 (Non-Partitioned) | Number of Columns | Default (s/op) | Eager Fetch (s/op) | Latency Reduction | |------------------:|---------------:|-------------------:|------------------:| | 10 | 0.142 | 0.100 | 29.6% | | 50 | 0.134 | 0.100 | 25.4% | | 100 | 0.138 | 0.101 | 26.8% | #### Parquet v4 (Partitioned) | Number of Columns | Default (s/op) | Eager Fetch (s/op) | Latency Reduction | |------------------:|---------------:|-------------------:|------------------:| | 10 | 0.161 | 0.094 | 41.6% | | 50 | 0.170 | 0.100 | 41.2% | | 100 | 0.134 | 0.096 | 28.4% | #### Graphical Comparison <img width="1800" height="1280" alt="manifest_grouped_bars" src="https://github.com/user-attachments/assets/22458c7a-5b55-484b-b504-ee9bf22524c5"; /> ### S3 Request Analysis The number of S3 GET requests per manifest read is computed as: (Used S3 bucket logs) > `Total GET Requests / Total Benchmark Iterations` | Configuration | GET Requests / Read | Request Reduction | |--------------|--------------------:|------------------:| | Default | 3 | — | | Eager Fetch | 1 | 67% | <p></p> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
