[
https://issues.apache.org/jira/browse/ARROW-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153232#comment-16153232
]
Jonas Amrich commented on ARROW-1459:
-------------------------------------
I'll try to look deeper into this. However I'm not so familiar with Arrow's
internals, so I expect it will take some time..
> [Python] PyArrow fails to load partitioned parquet files with non-primitive
> types
> ---------------------------------------------------------------------------------
>
> Key: ARROW-1459
> URL: https://issues.apache.org/jira/browse/ARROW-1459
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.6.0
> Reporter: Jonas Amrich
> Fix For: 0.7.0
>
>
> When reading partitioned parquet files (tested with those produced by Spark),
> that contain lists, the resulting table seems to contain data loaded only
> from one partition. Primitive types seems to be loaded correctly.
> It can be reproduced using following code (arrow 0.6.0, spark 2.1.1):
> {noformat}
> >>> df = spark.createDataFrame(list(zip(np.arange(10).tolist(),
> >>> np.arange(20).reshape((10,2)).tolist())))
> >>> df.toPandas()
> _1 _2
> 0 0 [0, 1]
> 1 1 [2, 3]
> 2 2 [4, 5]
> 3 3 [6, 7]
> 4 4 [8, 9]
> 5 5 [10, 11]
> 6 6 [12, 13]
> 7 7 [14, 15]
> 8 8 [16, 17]
> 9 9 [18, 19]
> >>> df.repartition(2).write.parquet('df_parts.parquet')
> >>> pq.read_table('df_parts.parquet').to_pandas()
> _1 _2
> 0 0 [0, 1]
> 1 2 [4, 5]
> 2 4 [8, 9]
> 3 6 [12, 13]
> 4 8 [16, 17]
> 5 1 [0, 1]
> 6 3 [4, 5]
> 7 5 [8, 9]
> 8 7 [12, 13]
> 9 9 [16, 17]
> {noformat}
> When the data is loaded using Spark or coalesced into one partition,
> everything works as expected:
> {noformat}
> >>> spark.read.parquet('df_parts.parquet').toPandas()
> _1 _2
> 0 1 [2, 3]
> 1 3 [6, 7]
> 2 5 [10, 11]
> 3 7 [14, 15]
> 4 9 [18, 19]
> 5 0 [0, 1]
> 6 2 [4, 5]
> 7 4 [8, 9]
> 8 6 [12, 13]
> 9 8 [16, 17]
> >>> df.coalesce(1).write.parquet('df_single.parquet')
> >>> pq.read_table('df_single.parquet').to_pandas()
> _1 _2
> 0 0 [0, 1]
> 1 1 [2, 3]
> 2 2 [4, 5]
> 3 3 [6, 7]
> 4 4 [8, 9]
> 5 5 [10, 11]
> 6 6 [12, 13]
> 7 7 [14, 15]
> 8 8 [16, 17]
> 9 9 [18, 19]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
