Gil Forsyth created ARROW-17582:
-----------------------------------
Summary: Relax / extend type checking for pyarrow array creation
Key: ARROW-17582
URL: https://issues.apache.org/jira/browse/ARROW-17582
Project: Apache Arrow
Issue Type: Improvement
Components: Python
Reporter: Gil Forsyth
in ibis we're interested in offering query results as a record batch – some of
the data we're starting with is coming back from a `sqlalchemy.cursor` which
_look_ like `tuple`s and `dict`s but are actually
`sqlalchemy.engine.row.LegacyRow` and `sqlalchemy.engine.row.RowMapping`,
respectively.
The checks in `python_to_arrow.cc` are strict enough that these can't be
readily dumped into an `array` without first calling, e.g. `tuple` on the
individual rows of the results.
{code:java}
In [168]: batch[:5]
Out[168]: [(1, 2173), (1, 943), (1, 892), (1, 30), (1, 337)]
In [169]: pa_schema = pa.struct([("l_orderkey", pa.int32()), ("l_partkey",
pa.int32())])
In [170]: pa.array(batch[:5], type=pa_schema)
---------------------------------------------------------------------------
ArrowTypeError Traceback (most recent call last)
Input In [170], in <cell line: 1>()
----> 1 pa.array(batch[:5], type=pa_schema)
File
/nix/store/z9qn3g22d8nx1x4mgzq3497iy8ji5h8x-python3-3.10.6-env/lib/python3.10/site-packages/pyarrow/array.pxi:317,
in pyarrow.lib.array()
File
/nix/store/z9qn3g22d8nx1x4mgzq3497iy8ji5h8x-python3-3.10.6-env/lib/python3.10/site-packages/pyarrow/array.pxi:39,
in pyarrow.lib._sequence_to_array()
File
/nix/store/z9qn3g22d8nx1x4mgzq3497iy8ji5h8x-python3-3.10.6-env/lib/python3.10/site-packages/pyarrow/error.pxi:144,
in pyarrow.lib.pyarrow_internal_check_status()
File
/nix/store/z9qn3g22d8nx1x4mgzq3497iy8ji5h8x-python3-3.10.6-env/lib/python3.10/site-packages/pyarrow/error.pxi:123,
in pyarrow.lib.check_status()
ArrowTypeError: Could not convert 1 with type int: was expecting tuple of (key,
value) pair
/build/apache-arrow-9.0.0/cpp/src/arrow/python/python_to_arrow.cc:938
GetKeyValuePair(items, i)
/build/apache-arrow-9.0.0/cpp/src/arrow/python/python_to_arrow.cc:1010
InferKeyKind(items)
/build/apache-arrow-9.0.0/cpp/src/arrow/python/iterators.h:73 func(value,
static_cast<int64_t>(i), &keep_going)
/build/apache-arrow-9.0.0/cpp/src/arrow/python/python_to_arrow.cc:1182
converter->Extend(seq, size)
{code}
vs
{{{{ }}}}
{code:java}
In [171]: pa.array(map(tuple, batch[:5]), type=pa_schema)
Out[171]:
<pyarrow.lib.StructArray object at 0x7fd4fb52d660>
-- is_valid: all not null
-- child 0 type: int32
[
1,
1,
1,
1,
1
]
-- child 1 type: int32
[
2173,
943,
892,
30,
337
]{code}
{{{{}}}}
To avoid the overhead of this extra conversion, maybe there are some checks
that aren't explicit python type-checks that we can rely on?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
