Phillip Cloud created SPARK-43194: ------------------------------------- Summary: PySpark 3.4.0 cannot convert timestamp-typed objects to pandas with pandas 2.0 Key: SPARK-43194 URL: https://issues.apache.org/jira/browse/SPARK-43194 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 3.4.0 Environment: {code} In [4]: import pandas as pd
In [5]: pd.__version__ Out[5]: '2.0.0' In [6]: import pyspark as ps In [7]: ps.__version__ Out[7]: '3.4.0' {code} Reporter: Phillip Cloud {code} In [1]: from pyspark.sql import SparkSession In [2]: session = SparkSession.builder.appName("test").getOrCreate() 23/04/19 09:21:42 WARN Utils: Your hostname, albatross resolves to a loopback address: 127.0.0.2; using 192.168.1.170 instead (on interface enp5s0) 23/04/19 09:21:42 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/04/19 09:21:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable In [3]: session.sql("select now()").toPandas() {code} Results in: {code} ... TypeError: Casting to unit-less dtype 'datetime64' is not supported. Pass e.g. 'datetime64[ns]' instead. {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org