alamb commented on issue #8783:
URL: https://github.com/apache/arrow-rs/issues/8783#issuecomment-3487778391
I think I have isolated the commit that caused the slow down 🥁
Specifically it appears to be the d5df352f18e50664605aa3ea598643e19a0b1647
/ #8530 (thrift remodel)
I am testing using this crazy command
```shell
cargo build --release && hyperfine --warmup=1 --runs=5 'rm -f
lineitem.parquet && ./target/release/tpchgen-cli --scale-factor=10
--tables=lineitem --format=parquet'
```
Here is the result at that commit:
```
HEAD is now at d5df352f18 Use custom thrift parser for parquet metadata
(phase 1 of Thrift remodel) (#8530)
Benchmark 1: rm -f lineitem.parquet && ./target/release/tpchgen-cli
--scale-factor=10 --tables=lineitem --format=parquet
Time (mean ± σ): 3.606 s ± 0.057 s [User: 38.845 s, System: 0.925
s]
Range (min … max): 3.527 s … 3.670 s 5 runs
```
- Here is the result at the next 521f219e308613811aeae11300bf7a7b0fb5ec29
/ (#8545)
```
HEAD is now at 521f219e30 Migrate `arrow-avro` to Rust 2024 (#8545)
Benchmark 1: rm -f lineitem.parquet && ./target/release/tpchgen-cli
--scale-factor=10 --tables=lineitem --format=parquet
Time (mean ± σ): 3.333 s ± 0.034 s [User: 34.432 s, System: 0.897
s]
Range (min … max): 3.299 s … 3.385 s 5 runs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
