[ https://issues.apache.org/jira/browse/ARROW-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441223#comment-17441223 ]
Joris Van den Bossche commented on ARROW-14564: ----------------------------------------------- This is a limitation of the Parquet format itself. Pyarrow defaults to writing version 1.0 parquet files, which don't yet have support for uint32 type. However, you can specify {{version="2.4"}} in the {{write_table}} call, and then it should preserve both unsigned integer types. We plan to switch to the newer parquet version by default in the near future (see ARROW-12203 and the linked issues) > [python] uint32 incorrectly saves to Parquet as int64 > ----------------------------------------------------- > > Key: ARROW-14564 > URL: https://issues.apache.org/jira/browse/ARROW-14564 > Project: Apache Arrow > Issue Type: Bug > Components: Python > Affects Versions: 6.0.0 > Environment: Ubuntu 20.10, Python 3.8.10 > Reporter: Bruce Allen > Priority: Major > Attachments: test_u32.py > > > Function pyarrow.parquet.write_table() incorrectly saves data of type > unsigned int32 as signed int64. Code test_u32.py showing failure is attached. > Output from running test_u32.py indicating faulty retyping: > pyarrow version: 6.0.0 > numpy data: > [(1, 2) (3, 4)] > [('my_u2', '<u2'), ('my_u4', '<u4')] > result: > my_u2 my_u4 > 0 1 2 > 1 3 4 > my_u2 uint16 > my_u4 int64 > dtype: object > > We can also observe that the incorrect int64 type is in the Parquet file by > using the "parq" tool: > $ parq _test_u32_pq --schema > # Schema > <pyarrow._parquet.ParquetSchema object at 0x7ff2e40b2a40> > required group field_id=-1 schema { > optional int32 field_id=-1 my_u2 (Int(bitWidth=16, isSigned=false)); > optional int64 field_id=-1 my_u4; > } -- This message was sent by Atlassian Jira (v8.20.1#820001)