Bryan Cutler created SPARK-31964: ------------------------------------ Summary: Avoid Pandas import for CategoricalDtype with Arrow conversion Key: SPARK-31964 URL: https://issues.apache.org/jira/browse/SPARK-31964 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 3.1.0 Reporter: Bryan Cutler
The import for CategoricalDtype changed in Pandas from 0.23 to 1.0 and currently pyspark checks 2 places to import. It would be better check the type as a string and avoid any imports. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org