[ https://issues.apache.org/jira/browse/ARROW-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated ARROW-6678: ---------------------------------- Labels: pull-request-available (was: ) > [C++] Regression in Parquet file compatibility introduced by ARROW-3246 > ----------------------------------------------------------------------- > > Key: ARROW-6678 > URL: https://issues.apache.org/jira/browse/ARROW-6678 > Project: Apache Arrow > Issue Type: Bug > Components: C++ > Reporter: Wes McKinney > Priority: Blocker > Labels: pull-request-available > Fix For: 0.15.0 > > > I randomly discovered that this script fails after applying the patch for > ARROW-3246 > https://github.com/apache/arrow/commit/2ba0566b29312e84fafc987fd8dc9664748be96a > {code} > import pandas as pd > import pyarrow as pa > import pyarrow.parquet as pq > import fastparquet as fp > df = pd.util.testing.makeDataFrame() > pq.write_table(pa.table(df), 'test.parquet') > fp.ParquetFile('test.parquet') > {code} > with > {code} > Traceback (most recent call last): > File > "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", > line 110, in __init__ > with open_with(fn2, 'rb') as f: > File > "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/util.py", > line 38, in default_open > return open(f, mode) > NotADirectoryError: [Errno 20] Not a directory: 'test.parquet/_metadata' > During handling of the above exception, another exception occurred: > Traceback (most recent call last): > File "test.py", line 10, in <module> > fp.ParquetFile('test.parquet') > File > "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", > line 116, in __init__ > self._parse_header(f, verify) > File > "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", > line 135, in _parse_header > fmd = read_thrift(f, parquet_thrift.FileMetaData) > File > "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/thrift_structures.py", > line 25, in read_thrift > obj.read(pin) > File > "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py", > line 1929, in read > iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) > UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: > invalid start byte > {code} > I don't recall making any metadata-related changes but I'm going to review > the patch to see if I can narrow down where the problem is to see whether > it's a bug with Arrow/parquet-cpp or with the third party library -- This message was sent by Atlassian Jira (v8.3.4#803005)