Phoenix500526 commented on PR #10985: URL: https://github.com/apache/arrow-rs/pull/10985#issuecomment-5642914637
Here are the results from another local run of the six-case benchmark, including the `StreamWriter` baseline. The earlier run remains in the PR description for comparison. Times are per benchmark iteration (10 batches), with Criterion point estimates and 95% confidence intervals. Each case used 100 measurements. | Mode | Benchmark | Estimate | 95% confidence interval | Outliers | |---|---|---:|---:|---:| | Uncompressed | `StreamWriter/write_10` | 96.898 µs | 95.591 to 98.173 µs | 4/100 | | Uncompressed | `StreamEncoder/encode_and_write_10` | 106.21 µs | 105.25 to 107.18 µs | 3/100 | | Uncompressed | `StreamEncoder/encode_to_10` | 96.818 µs | 95.635 to 98.041 µs | 4/100 | | ZSTD | `StreamWriter/write_10/zstd` | 5.9798 ms | 5.9083 to 6.0800 ms | 5/100 | | ZSTD | `StreamEncoder/encode_and_write_10/zstd` | 6.0074 ms | 5.9671 to 6.0488 ms | 3/100 | | ZSTD | `StreamEncoder/encode_to_10/zstd` | 6.3504 ms | 6.1927 to 6.5342 ms | 11/100 | Without compression, `encode_to` takes about 8.8% less time than `encode + write` based on the point estimates. Its estimate is within 0.1% of `StreamWriter`, and their intervals almost coincide, so this run provides no evidence of a meaningful advantage over `StreamWriter`. With ZSTD, `encode_to` takes about 5.7% more time than `encode + write` and 6.2% more than `StreamWriter`. Its interval is above both alternatives in this run. It also has 11 outliers, including 7 high severe outliers, so repeated runs would help assess how stable this difference is. These percentages describe this run; the cause of the ZSTD slowdown has not been established. This comparison measures synchronous writes into a `Vec<u8>`. It does not measure allocations, peak memory, async behavior, or backpressure. Given these results, is the synchronous `Write` API useful enough to justify the additional overlap with `StreamWriter`? Would allocation and peak-memory measurements, or an ownership-taking sink prototype as discussed in #10445, be the most useful next experiment? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
