Uwe L. Korn created ARROW-1967:
----------------------------------
Summary: Python: AssertionError w.r.t Pandas conversion on Parquet
files in 0.8.0 dev version
Key: ARROW-1967
URL: https://issues.apache.org/jira/browse/ARROW-1967
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Uwe L. Korn
I have some Parquet files generated a few commit before 0.8.0 that I cannot
load in 0.8.0.
They have the following Pandas metadata:
{code}
{u'pandas_version': u'0.20.3', u'index_columns': [u'__index_level_0__'],
u'columns': [{u'pandas_type': u'unicode', u'name': u'col1', u'numpy_type':
u'object', u'metadata': None}, {u'pandas_type': u'list[unicode]', u'name':
u'partition', u'numpy_type': u'object', u'metadata': None}, {u'pandas_type':
u'int64', u'name': None, u'numpy_type': u'int64', u'metadata': None}],
u'column_indexes': [{u'pandas_type': u'unicode', u'name': None, u'numpy_type':
u'object', u'metadata': None}]}
{code}
And I get the following exception on loading the file:
{code}
df = table.to_pandas()
File "pyarrow/table.pxi", line 1043, in pyarrow.lib.Table.to_pandas
(/home/jenkins/.conan/data/pyarrow/0.8.0/by/stable/build/0fda246af0df114df08c06ca4fdd2cac7d7696fd/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:46331)
File
"/io/venv_deb8/local/lib/python2.7/site-packages/pyarrow/pandas_compat.py",
line 498, in table_to_blockmanager
assert len(logical_index_names) == len(index_columns_set)
AssertionError
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
