Re: [C++] arrow compute and tbb

2023-04-21 Thread Surya Kiran Gullapalli
Hi, Thanks for the reply and the suggestion on custom executor. I'll take a look at it. I was profiling my application (windows 10) which is dealing with millions of rows (roughly 10M) of data and I found some places where only a few cpus were engaged and the rest of them were sitting idle. On

Re: [C++] arrow compute and tbb

2023-04-21 Thread Weston Pace
No, there's no build-time configuration settings to enable TBB specifically. You can, at runtime, specify a custom executor to use for most operations. We use one thread pool for CPU tasks and one for I/O tasks. You could replace either or both with a TBB-based executor. For example, the method

[C++] arrow compute and tbb

2023-04-21 Thread Surya Kiran Gullapalli
Hello, I'm curious to know if c++ sdk of arrow compute functions can use tbb parallelization underneath ? The documentation mentions that arrow uses a threadpool for parallelization. Does compute functions also use threadpool and parallelize computation ? Looking at the .so file created I do not