[
https://issues.apache.org/jira/browse/ARROW-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872692#comment-16872692
]
Julian de Ruiter commented on ARROW-5666:
-----------------------------------------
Ideally you would want to preserve the data type of the partition columns, but
that's going to be hard to do properly if you also want to include other types
such as dates, etc. Maybe it would be most consistent to just read partitioned
columns as a categorical type and let the user handle the details, as this
makes the code less magic than what happens now.
> [Python] Underscores in partition (string) values are dropped when reading
> dataset
> ----------------------------------------------------------------------------------
>
> Key: ARROW-5666
> URL: https://issues.apache.org/jira/browse/ARROW-5666
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.13.0
> Reporter: Julian de Ruiter
> Priority: Major
> Labels: parquet
>
> When reading a partitioned dataset, in which the partition column contains
> string values with underscores, pyarrow seems to be ignoring the underscores
> in the resulting values.
> For example if I write and then read a dataset as follows:
> {code:java}
> import pyarrow as pa
> import pandas as pd
> df = pd.DataFrame({
> "year_week": ["2019_2", "2019_3"],
> "value": [1, 2]
> })
> table = pa.Table.from_pandas(df.head())
> pq.write_to_dataset(table, 'test', partition_cols=["year_week"])
> table2 = pq.ParquetDataset('test').read()
> {code}
> The resulting 'year_week' column in table 2 has lost the underscores:
> {code:java}
> table2[1] # Gives:
> <Column name='year_week' type=DictionaryType(dictionary<values=int64,
> indices=int32, ordered=0>)>
> [
> -- dictionary:
> [
> 20192,
> 20193
> ]
> -- indices:
> [
> 0
> ],
> -- dictionary:
> [
> 20192,
> 20193
> ]
> -- indices:
> [
> 1
> ]
> ]
> {code}
> Is this intentional behaviour or is this a bug in arrow?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
