viirya commented on PR #50327: URL: https://github.com/apache/arrow/pull/50327#issuecomment-4917636970
Reworked as suggested — the PR now adds a scalar-free `cdef _getitem_py(self, int64_t i)` on `Array` and the baseline `to_pylist` is a single loop over it. The base implementation is `GetScalar + Scalar.as_py`, so unspecialized types (dates/times/timestamps/decimals/dictionary/extension/...) behave exactly as before, and specializations exist for integers, floats, boolean, string/binary (+ large variants), list/large_list/fixed_size_list, map and struct. Nested types compose: a list row is built directly from the child's `_getitem_py` over its offset range, with the child wrapper cached on the parent array. `maps_as_pydicts != None` keeps the Scalar path since map→dict conversion has per-entry duplicate-key semantics. Your `int64` example: 4M `int64` with nulls goes from ~0.39 s to **0.028 s (~7 ns/element — on par with `ndarray.tolist`)**. Other numbers (M4 Max): flat string 4M 0.83→0.06 s; `list<string>` 2M 1.93→0.46 s; `list<list<int32>>` 1M 2.10→0.40 s; `struct<int64,string>` 1M 0.91→0.07 s; `map<string,int64>` 1M 2.77→0.74 s. Net diff is smaller than the previous approach (−104 lines) since the per-type `to_pylist` overrides are gone. Differential tests against the Scalar path (exact type equality, incl. slices, all-null, duplicate struct field names raising `ValueError`, strict-mode maps) pass, as does the pytest suite (1295 passed). PR description updated accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
