Frédérique Vanneste created ARROW-3139: ------------------------------------------
Summary: ArrowIOError: Arrow error: Capacity error during read Key: ARROW-3139 URL: https://issues.apache.org/jira/browse/ARROW-3139 Project: Apache Arrow Issue Type: Bug Components: Python Affects Versions: 0.10.0 Environment: pandas=0.23.1=py36h637b7d7_0 pyarrow==0.10.0 Reporter: Frédérique Vanneste My assumption: the problem is caused by a large object column containing strings up to 27 characters long. (so that column is much larger than 2GB of strings, chunking issue) looks similar as https://issues.apache.org/jira/browse/ARROW-2227?focusedCommentId=16379574&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16379574 Code * basket_plateau= pq.read_table("basket_plateau.parquet") * basket_plateau = pd.read_parquet("basket_plateau.parquet") Error produced * ArrowIOError: Arrow error: Capacity error: BinaryArray cannot contain more than 2147483646 bytes, have 2147483655 Dataset * Pandas dataframe (pandas=0.23.1=py36h637b7d7_0) * 2.7 billion record, 4 columns ( int64/object/datetime64/float64) * aprox 90GB in memory * example of object col: "Fresh Vegetables", "Alcohol Beers", ... (thing food retail categories) History to bug: * was using older version of pyarrow * tried writing dataset to disk (parquet) and failed * stumbled on https://issues.apache.org/jira/browse/ARROW-2227 * upgraded to 0.10 * tried writing dataset to disk (parquet) and succeeded * tried reading dataset and failed * looks like a similar case as: https://issues.apache.org/jira/browse/ARROW-2227?focusedCommentId=16379574&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16379574 Code outputs -- This message was sent by Atlassian JIRA (v7.6.3#76005)