[
https://issues.apache.org/jira/browse/ARROW-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252710#comment-17252710
]
Joris Van den Bossche commented on ARROW-10955:
-----------------------------------------------
[~pgoldsborough] Thanks for the report!
This seems a problem in the JSON reader in the type it creates for empty list
arrays. With the json reader, we get:
{code}
In [10]: pyarrow.json.read_json("input.json")["foo"].type
Out[10]: ListType(list<item: null not null>)
{code}
while when creating the same empty list array manually, we get:
{code}
In [6]: pa.array([[]]).type
Out[6]: ListType(list<item: null>)
{code}
So in the case of the JSON array, the value item is "null not null", which is a
bit contradictory, and the Parquet writer complains about this.
Manually trying to create such a contradictory field actually raises an error:
{code}
In [13]: pa.field("test", pa.null(), nullable=False)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-13-e28fa0cf2870> in <module>
----> 1 pa.field("test", pa.null(), nullable=False)
~/scipy/repos/arrow/python/pyarrow/types.pxi in pyarrow.lib.field()
ValueError: A null type field may not be non-nullable
{code}
> [Python] Cannot read empty json lists and write them as parquet
> ---------------------------------------------------------------
>
> Key: ARROW-10955
> URL: https://issues.apache.org/jira/browse/ARROW-10955
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
> Affects Versions: 0.17.0, 0.17.1, 1.0.0, 2.0.0
> Environment: linux and mac
> Reporter: Peter Goldsborough
> Priority: Major
>
> We're using Arrow to convert from JSON to Parquet and occasionally have empty
> lists in our json. Reading such JSON into an Arrow table and writing it to
> Parquet currently fails. We noticed this issue in our C++ Arrow code, but it
> also happens from Python.
> Minimal repro:
> input.json:
> {"foo": []}
>
> convert.py:
> import pyarrow.json
> import pyarrow.parquet
> t = pyarrow.json.read_json("input.json")
> pyarrow.parquet.write_table(t, "out.parquet")
>
> Produces:
> Traceback (most recent call last):
> File "repro.py", line 5, in <module>
> pyarrow.parquet.write_table(t, "out.parquet")
> env/lib/python3.8/site-packages/pyarrow/parquet.py", line 1717, in write_table
> with ParquetWriter(
> File "env/lib/python3.8/site-packages/pyarrow/parquet.py", line 554, in
> __init__
> self.writer = _parquet.ParquetWriter(
> File "pyarrow/_parquet.pyx", line 1409, in
> pyarrow._parquet.ParquetWriter.__cinit__
> File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: NullType Arrow field must be nullable
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
