alamb commented on pull request #8553:
URL: https://github.com/apache/arrow/pull/8553#issuecomment-722690860
@jorgecarleitao and @andygrove -- I did some profiling of the tpch
benchmark (SF10) locally on my laptop this afternoon. I was not able to
reproduce a slowdown on this branch
Here is my data from on my laptop (a Macbook Pro):
Test command was: `cargo run --release --bin tpch -- --iterations 3 --path
/Users/alamb/Software/tpch_data/SF10-parquet-64 --format parquet --query 1
--batch-size 4096`
When run against @ d4121d8a17d9e53ad4421960e357dd2f89771603
```
Running benchmarks with the following options: TpchOpt { query: 1, debug:
false, iterations: 3, concurrency: 2, batch_size: 4096, path:
"/Users/alamb/Software/tpch_data/SF10-parquet-64", file_format: "parquet",
mem_table: false }
Query 1 iteration 0 took 8913 ms
Query 1 iteration 1 took 10533 ms
Query 1 iteration 2 took 10633 ms
```
When run against this branch @ 802868617777022bd0945f98f987993ca7267eb6
(after rebasing against d4121d8a17d9e53ad4421960e357dd2f89771603):
```
Running benchmarks with the following options: TpchOpt { query: 1, debug:
false, iterations: 3, concurrency: 2, batch_size: 4096, path:
"/Users/alamb/Software/tpch_data/SF10-parquet-64", file_format: "parquet",
mem_table: false }
Query 1 iteration 0 took 9168 ms
Query 1 iteration 1 took 10097 ms
Query 1 iteration 2 took 10199 ms
```
So in other words I didn't measure a noticeable change on my machine
Qualatatively both this branch and master While kept my CPUs ~ 100% busy and
the memory used was fairly consistent
I then spent some time poking around with a profile from XCode Instruments
to see where the time was going. Unsurprisingly it is going to memory
management (this is the story of my professional life):
I am going to poke some more time seeing if I can figure out any easy to
recycle buffers or find a smoking gun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
