[ https://issues.apache.org/jira/browse/SPARK-21766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Cutler updated SPARK-21766: --------------------------------- Summary: DataFrame toPandas() raises ValueError with nullable int columns (was: DataFrame toPandas() ) > DataFrame toPandas() raises ValueError with nullable int columns > ---------------------------------------------------------------- > > Key: SPARK-21766 > URL: https://issues.apache.org/jira/browse/SPARK-21766 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.3.0 > Reporter: Bryan Cutler > > When calling {{DataFrame.toPandas()}} (without Arrow enabled), if there is a > IntegerType column that has null values the following exception is thrown: > {noformat} > ValueError: Cannot convert non-finite values (NA or inf) to integer > {noformat} > This is because the null values first get converted to float NaN during the > construction of the Pandas DataFrame in {{from_records}}, and then it is > attempted to be converted back to to an integer where it fails. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org