hlongmore commented on issue #39927: URL: https://github.com/apache/arrow/issues/39927#issuecomment-4978899846
IDK if this would help with implementing, but I was running into this issue because I have categorical data that I saved to parquet using pandas, with at least one of the categories as a partition column; then reading the parquet using `dataframe = pd.read_parquet(filepath, engine='pyarrow', dtype_backend='pyarrow')`. The read happens just fine, but when I try to e.g. create a mask or a column based on `isna()` and then try to use that to filter the dataframe, I get the above error. Poking around in the partition directories, I found that the partition directories for NA values for the categorical were written as `column_name=__HIVE_DEFAULT_PARTITION__`, and these values did not behave well on read (I couldn't even convert the column to string type to try to convert it to category after). I bulk renamed the directories with `find . -type d | grep "HIVE_DEFAULT_PARTITION" | sed 'p;s/__HIVE_DEFAULT_PARTITION__/NA_unknown/' | xargs -n2 mv`, reloaded the data, and was able to proceed just fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
