[jira] [Created] (ARROW-14488) [Python] Incorrect inferred schema from pandas dataframe with length 0.

[Yuan Zhou (Jira)]

Yuan Zhou created ARROW-14488:
---------------------------------

             Summary: [Python] Incorrect inferred schema from pandas dataframe 
with length 0.
                 Key: ARROW-14488
                 URL: https://issues.apache.org/jira/browse/ARROW-14488
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 5.0.0
         Environment: OS: Windows 10, CentOS 7
            Reporter: Yuan Zhou


We use pandas(with pyarrow engine) to write out parquet files and those outputs 
will be consumed by other applications such as Java apps using 
org.apache.parquet.hadoop.ParquetFileReader. We found that some empty 
dataframes would get incorrect schema for string columns in other applications. 
After some investigation, we narrow down the issue to the schema inference by 
pyarrow:

{{In [1]: import pandas as pd}}

{{In [2]: df = pd.DataFrame([['a', 1, 1.0]], columns=['a', 'b', 'c'])}}

{{In [3]: import pyarrow as pa}}

{{In [4]: pa.Schema.from_pandas(df)}}
{{Out[4]:}}
{{a: string}}
{{b: int64}}
{{c: double}}
{{-- schema metadata --}}
{{pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 
562}}

{{In [5]: pa.Schema.from_pandas(df.head(0))}}
{{Out[5]:}}
{{a: null}}
{{b: int64}}
{{c: double}}
{{-- schema metadata --}}
{{pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 
560}}

{{In [6]: pa.__version__}}
{{Out[6]: '5.0.0'}}

 

Is this an expected behavior? Or do we have any workaround for this issue? 
Could anyone take a look please. Thanks!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)