[
https://issues.apache.org/jira/browse/ARROW-17349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613030#comment-17613030
]
Will Jones commented on ARROW-17349:
------------------------------------
Yes, I've updated the title. Casting lists only was broken if it was inside a
map. The only reason casting maps looked as if it was working was because of
the early return if types are "equal" (and maps are "equal" even if they have
different field names).
> [C++] Add casting support for map type
> --------------------------------------
>
> Key: ARROW-17349
> URL: https://issues.apache.org/jira/browse/ARROW-17349
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Affects Versions: 9.0.0
> Reporter: Will Jones
> Assignee: Will Jones
> Priority: Major
> Labels: good-first-issue, kernel, pull-request-available
> Fix For: 10.0.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Different parquet implementations use different field names for internal
> fields of ListType and MapType, which can sometimes cause silly conflicts.
> For example, we use {{item}} as the field name for list, but Spark uses
> {{element}}. Fortunately, we can automatically cast between List and Map
> Types with different field names. Unfortunately, it only works at the top
> level. We should get it to work at arbitrary levels of nesting.
> This was discovered in delta-rs:
> https://github.com/delta-io/delta-rs/pull/684#discussion_r935099285
> Here's a reproduction in Python:
> {code:Python}
> import pyarrow as pa
> import pyarrow.parquet as pq
> import pyarrow.dataset as ds
> def roundtrip_scanner(in_arr, out_type):
> table = pa.table({"arr": in_arr})
> pq.write_table(table, "test.parquet")
> schema = pa.schema({"arr": out_type})
> ds.dataset("test.parquet", schema=schema).to_table()
> # MapType
> ty_named = pa.map_(pa.field("x", pa.int32(), nullable=False), pa.int32())
> ty = pa.map_(pa.int32(), pa.int32())
> arr_named = pa.array([[(1, 2), (2, 4)]], type=ty_named)
> roundtrip_scanner(arr_named, ty)
> # ListType
> ty_named = pa.list_(pa.field("x", pa.int32(), nullable=False))
> ty = pa.list_(pa.int32())
> arr_named = pa.array([[1, 2, 4]], type=ty_named)
> roundtrip_scanner(arr_named, ty)
> # Combination MapType and ListType
> ty_named = pa.map_(pa.string(), pa.field("x", pa.list_(pa.field("x",
> pa.int32(), nullable=True)), nullable=False))
> ty = pa.map_(pa.string(), pa.list_(pa.int32()))
> arr_named = pa.array([[("string", [1, 2, 3])]], type=ty_named)
> roundtrip_scanner(arr_named, ty)
> # Traceback (most recent call last):
> # File "<stdin>", line 1, in <module>
> # File "<stdin>", line 5, in roundtrip_scanner
> # File "pyarrow/_dataset.pyx", line 331, in
> pyarrow._dataset.Dataset.to_table
> # File "pyarrow/_dataset.pyx", line 2577, in
> pyarrow._dataset.Scanner.to_table
> # File "pyarrow/error.pxi", line 144, in
> pyarrow.lib.pyarrow_internal_check_status
> # File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
> # pyarrow.lib.ArrowNotImplementedError: Unsupported cast to map<string,
> list<item: int32>> from map<string, list<x: int32> ('arr')>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
