nirandaperera commented on pull request #10410: URL: https://github.com/apache/arrow/pull/10410#issuecomment-853004003
> I did a quick comparison with `np.where` (using plain float64 arrays without nulls to have it comparable), and it seems this implementation is already doing quite good compared to numpy (using a release build): > > ``` > In [19]: N = 10_000_000 > ...: arr1 = np.random.randn(N) > ...: arr2 = np.random.randn(N) > ...: mask = np.random.randint(0, 2, N).astype(bool) > ...: > ...: pa_arr1 = pa.array(arr1) > ...: pa_arr2 = pa.array(arr2) > ...: pa_mask = pa.array(mask) > ...: > > In [20]: %timeit np.where(mask, arr1, arr2) > 82.3 ms ± 8.76 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) > > In [21]: %timeit pc.if_else(pa_mask, pa_arr1, pa_arr2) > 50.4 ms ± 6.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) > ``` We may be able to further improve this if we directly use vector operations inside the kernel (I haven't checked the compiled code yet, may be compiler does that already), because if-else use case directly map to `mask_move` operations in AVX512 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
