Thanks, Will! That's an interesting insight.
Also, fixing the title so it's more easily searchable in the archive 😅 On Dec 18, 2023, at 7:11 PM, Will Jones <will.jones...@gmail.com> wrote:
 Hi Dane,
I haven’t benchmarked the Arrow C++ ones, but it might be worth looking into. I recently was benchmarking the Rust object store crate (part of the arrow-rs) codebase and found it was limited in throughput because we were defaulting to HTTP/2 [1]. So you might discover something similar in Arrow C++.
Best, Will Jones  Hello,
Arrow has some small benchmarking tests for S3[1] and Local FS. Does anyone know if there is any benchmarking done for GCS? I noticed it uses storage-testbench[2] for testing, but it doesn't look like it produces any profiling data.
I realize these are trivial benchmarks using stubbed file systems, so I'd be curious if anyone has done their own benchmarking of filesystems with Arrow against the actual storage systems themselves.
|
