dbtsai commented on PR #50327: URL: https://github.com/apache/arrow/pull/50327#issuecomment-4920289691
**Follow-up ideas: further performance opportunities** The structural win here (no per-element Scalar/Array allocation + child-array caching) is solid. A few places that still leave performance on the table, roughly by impact: 1. **Hoist the numeric type-ID dispatch out of the per-element loop (biggest).** `NumericArray._getitem_py` recomputes `type_id()` and walks an `if/elif` ladder (up to 10 branches) on *every* element, even though the type is invariant across the array. For an int8 column that's ~6 failed comparisons per value. Resolving the type once and running a monomorphic loop over the buffer turns the "no Scalar" win into a "vectorized" win. 2. **Share the fast path with `as_py()` / `__iter__`.** `_getitem_py` is private and only called from `to_pylist` (when `maps_as_pydicts is None`). `arr[i].as_py()`, iteration, and `ChunkedArray` element access still allocate Scalars. Wiring those to reuse `_getitem_py` would extend the win to the other common access patterns. 3. **Null fast path.** Every accessor calls `IsNull(i)` per element even when `null_count == 0`. Checking `null_count == 0` once and skipping the per-element null test helps the very common all-non-null column. 4. **Offset reads in list/map paths.** `value_offset(i)` / `value_offset(i+1)` are two virtual calls per row; reading the offsets buffer pointer once and indexing avoids that on long arrays. Also worth confirming `for j in range(start, end)` compiles to a C loop rather than allocating a `range` per row. 5. **Missing specializations that still fall back to Scalars:** temporal (date/timestamp/time/duration), decimal, `DictionaryArray`, and string_view/binary_view have no `_getitem_py`. Timestamps/dates and dictionaries are common enough to be worth a fast path (dictionary especially — see the separate value-reuse comment). None of these are blocking — the PR is a clear improvement as-is. #1 and #2 are the two with real leverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
