[
https://issues.apache.org/jira/browse/ARROW-9215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277915#comment-17277915
]
Joris Van den Bossche commented on ARROW-9215:
----------------------------------------------
Yeah, I understand that for uint32 there is a "safe" alternative while for
uint64 that doesn't exist (so it's certainly a trade-off). But I assume the
reason we use this safe alternative (with {{version="1.0"}}) is to avoid old
readers to interpret the uint32 data wrongly as its physical int32 type (which
would silently give wrong numbers for values outside of the int32 range but
within the uint32 range). But, the same thing currently can happen for uint64
columns, as far as I understand it correctly?
So if we deem it OK for uint64 to write it in a way that potentially cannot be
read by old readers, why not be consistent for uint32?
> pyarrow parquet writer converts uint32 columns to int64
> -------------------------------------------------------
>
> Key: ARROW-9215
> URL: https://issues.apache.org/jira/browse/ARROW-9215
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Devavret Makkar
> Assignee: Uwe Korn
> Priority: Major
>
> pyarrow parquet writer changes uint32 columns to int64. This change is not
> made for other types and uint8, uint16, and uint64 columns retain their type.
> {code:python}
> In [1]: import pandas as pd
> In [2]: import pyarrow as pa
> In [3]: import pyarrow.parquet as pq
> In [5]: df = pd.DataFrame({'a':pd.Series([1,2,3], dtype='uint32')})
> In [6]: padf = pa.Table.from_pandas(df)
> In [7]: padf
> Out[7]:
> pyarrow.Table
> a: uint32
> In [8]: pq.write_table(padf, 'pa.parquet')
> In [9]: pq.read_table('pa.parquet')
> Out[9]:
> pyarrow.Table
> a: int64
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
