pitrou commented on PR #50327: URL: https://github.com/apache/arrow/pull/50327#issuecomment-4912480927
> The cause is the per-element conversion loop (`[x.as_py() for x in self]`): every row allocates a C++ Scalar (`Array::GetScalar`), a Python Scalar wrapper and, for list types, a Python Array wrapper for the row's values slice plus a fresh generator before recursing per element. I agree this is extremely wasteful, but the approach here seems very ad hoc and also solves the performance issue for a very limited set of types, while the performance problem is more general: ```python In [5]: a = np.arange(10_0000) In [6]: %timeit a.tolist() 722 μs ± 43.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) In [7]: b = pa.array(a) In [8]: %timeit b.to_pylist() 17.5 ms ± 147 μs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [9]: b.to_pylist() == a.tolist() Out[9]: True ``` How about we do something like the following: 1) Add an `Array` method that indexes and returns a Python value (e.g. an `int`) directly instead of a PyArrow scalar (e.g. an Int64Scalar) 2) Use that new method in the baseline implementation of `Array::to_pylist`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
