[ https://issues.apache.org/jira/browse/ARROW-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe L. Korn resolved ARROW-2891. -------------------------------- Resolution: Fixed Fix Version/s: 0.10.0 Issue resolved by pull request 2302 [https://github.com/apache/arrow/pull/2302] > Preserve schema in write_to_dataset > ----------------------------------- > > Key: ARROW-2891 > URL: https://issues.apache.org/jira/browse/ARROW-2891 > Project: Apache Arrow > Issue Type: Bug > Components: Python > Affects Versions: 0.9.0 > Reporter: Jonathan Kulzick > Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When using `pyarrow.parquet.write_to_dataset` with `partition_cols` set, the > schema of the `table` passed into the function is not enforced when iterating > over the `subgroup` to create the `subtable`. See > [here|[https://github.com/apache/arrow/blob/master/python/pyarrow/parquet.py#L1146]]. > Since pandas is used to generate the subtables, there is a risk that some > specificity is lost from the original `table.schema` due to the data types > supported by pandas and some of the internal type conversions pandas > performs. It would be ideal if a `subschema` was generated from > `table.schema` and passed to `Table` when instantiating the `subtable` to > allow the user to enforce the original schema. > Here is a simple example of where we are running into issues while trying to > preserve a valid schema. This use case is more likely to occur when working > with sparse data sets. > > {code:java} > >>> from io import StringIO > >>> import pandas as pd > >>> import numpy as np > >>> import pyarrow as pa > >>> import parquet as pq > >>> import pyarrow.parquet as pq > # in csv col2 has no NaNs and in csv_nan col2 only has NaNs > >>> csv = StringIO('"1","10","100"') > >>> csv_nan = StringIO('"2","","200"') > # read in col2 as a float since pandas does not support NaNs in ints > >>> pd_dtype = {'col1': np.int32, 'col2': np.float32, 'col3': np.int32} > >>> df = pd.read_csv(csv, header=None, names=['col1', 'col2', 'col3'], > >>> dtype=pd_dtype) > >>> df_nan = pd.read_csv(csv_nan, header=None, names=['col1', 'col2', > >>> 'col3'], dtype=pd_dtype) > # verify both dfs and their dtypes > >>> df > col1 col2 col3 > 0 1 10.0 100 > >>> df.dtypes > col1 int32 > col2 float32 > col3 int32 > dtype: object > >>> df_nan > col1 col2 col3 > 0 2 NaN 200 > >>> df_nan.dtypes > col1 int32 > col2 float32 > col3 int32 > dtype: object > # define col2 as an int32 since pyarrow does support NaNs in ints > # we want to preserve the original schema we started with and not > # upcast just because we're using pandas to go from csv to pyarrow > >>> schema = pa.schema([pa.field('col1', type=pa.int32()), > pa.field('col2', type=pa.int32()), > pa.field('col3', type=pa.int32())]) > # verify schema > >>> schema > col1: int32 > col2: int32 > col3: int32 > # create tables > >>> table = pa.Table.from_pandas(df, schema=schema, preserve_index=False) > >>> table_nan = pa.Table.from_pandas(df_nan, schema=schema, > >>> preserve_index=False) > # verify table schemas and metadata > # col2 has pandas_type int32 and numpy_type float32 in both tables > >>> table > pyarrow.Table > col1: int32 > col2: int32 > col3: int32 > metadata > -------- > {b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": > [{"name":' > b' "col1", "field_name": "col1", "pandas_type": "int32", "numpy_ty' > b'pe": "int32", "metadata": null}, {"name": "col2", "field_name": ' > b'"col2", "pandas_type": "int32", "numpy_type": "float32", "metada' > b'ta": null}, {"name": "col3", "field_name": "col3", "pandas_type"' > b': "int32", "numpy_type": "int32", "metadata": null}], "pandas_ve' > b'rsion": "0.22.0"}'} > >>> table_nan > pyarrow.Table > col1: int32 > col2: int32 > col3: int32 > metadata > -------- > {b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": > [{"name":' > b' "col1", "field_name": "col1", "pandas_type": "int32", "numpy_ty' > b'pe": "int32", "metadata": null}, {"name": "col2", "field_name": ' > b'"col2", "pandas_type": "int32", "numpy_type": "float32", "metada' > b'ta": null}, {"name": "col3", "field_name": "col3", "pandas_type"' > b': "int32", "numpy_type": "int32", "metadata": null}], "pandas_ve' > b'rsion": "0.22.0"}'} > # write both tables to local filesystem > >>> pq.write_to_dataset(table, '/Users/jkulzick/pyarrow_example', > partition_cols=['col1'], > preserve_index=False) > >>> pq.write_to_dataset(table_nan, '/Users/jkulzick/pyarrow_example', > partition_cols=['col1'], > preserve_index=False) > # read parquet files into a ParquetDataset to validate the schemas > # the metadata and schemas for both files is different from their original > tables > # table now has pandas_type int32 and numpy_type int32 (was float32) for col2 > # table_nan now has pandas_type float64 (was int32) and numpy_type int64 (was > float32) for col2 > >>> ds = pq.ParquetDataset('/Users/jkulzick/pyarrow_example') > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File > "/Users/jkulzick/miniconda3/envs/bowerbird/lib/python3.6/site-packages/pyarrow/parquet.py", > line 745, in __init__ > self.validate_schemas() > File > "/Users/jkulzick/miniconda3/envs/bowerbird/lib/python3.6/site-packages/pyarrow/parquet.py", > line 775, in validate_schemas > dataset_schema)) > ValueError: Schema in partition[col1=1] > /Users/jkulzick/pyarrow_example/col1=2/b7b42ce9de6a46a786a5361c42d28731.parquet > was different. > col2: double > col3: int32 > metadata > -------- > {b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": > [{"name":' > b' "col2", "field_name": "col2", "pandas_type": "float64", "numpy_' > b'type": "float64", "metadata": null}, {"name": "col3", "field_nam' > b'e": "col3", "pandas_type": "int32", "numpy_type": "int32", "meta' > b'data": null}], "pandas_version": "0.22.0"}'} > vs > col2: int32 > col3: int32 > metadata > -------- > {b'pandas': b'{"index_columns": [], "column_indexes": [], "columns": > [{"name":' > b' "col2", "field_name": "col2", "pandas_type": "int32", "numpy_ty' > b'pe": "int32", "metadata": null}, {"name": "col3", "field_name": ' > b'"col3", "pandas_type": "int32", "numpy_type": "int32", "metadata' > b'": null}], "pandas_version": "0.22.0"}'}{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)