Hi Dane, I haven’t benchmarked the Arrow C++ ones, but it might be worth looking into. I recently was benchmarking the Rust object store crate (part of the arrow-rs) codebase and found it was limited in throughput because we were defaulting to HTTP/2 [1]. So you might discover something similar in Arrow C++.
[1] https://github.com/apache/arrow-rs/issues/5194 Best, Will Jones On Mon, Dec 18, 2023 at 14:36 Dane Pitkin <[email protected]> wrote: > Hello, > > Arrow has some small benchmarking tests for S3[1] and Local FS. Does > anyone know if there is any benchmarking done for GCS? I noticed it uses > storage-testbench[2] for testing, but it doesn't look like it produces any > profiling data. > > I realize these are trivial benchmarks using stubbed file systems, so I'd > be curious if anyone has done their own benchmarking of filesystems with > Arrow against the actual storage systems themselves. > > > [1] > https://github.com/apache/arrow/blob/main/cpp/src/arrow/filesystem/s3fs_benchmark.cc > [2] https://github.com/googleapis/storage-testbench >
