[ https://issues.apache.org/jira/browse/SPARK-42458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-42458. ---------------------------------- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40240 [https://github.com/apache/spark/pull/40240] > createDataFrame should support DDL string as schema > --------------------------------------------------- > > Key: SPARK-42458 > URL: https://issues.apache.org/jira/browse/SPARK-42458 > Project: Spark > Issue Type: Sub-task > Components: Connect > Affects Versions: 3.4.0 > Reporter: Takuya Ueshin > Assignee: Takuya Ueshin > Priority: Major > Fix For: 3.4.0 > > > {code:python} > File "/.../python/pyspark/sql/connect/readwriter.py", line 393, in > pyspark.sql.connect.readwriter.DataFrameWriter.option > Failed example: > with tempfile.TemporaryDirectory() as d: > # Write a DataFrame into a CSV file with 'nullValue' option set to > 'Hyukjin Kwon'. > df = spark.createDataFrame([(100, None)], "age INT, name STRING") > df.write.option("nullValue", "Hyukjin > Kwon").mode("overwrite").format("csv").save(d) > # Read the CSV file as a DataFrame. > spark.read.schema(df.schema).format('csv').load(d).show() > Exception raised: > Traceback (most recent call last): > File "/.../lib/python3.9/doctest.py", line 1334, in __run > exec(compile(example.source, filename, "single", > File "<doctest > pyspark.sql.connect.readwriter.DataFrameWriter.option[2]>", line 3, in > <module> > df = spark.createDataFrame([(100, None)], "age INT, name STRING") > File "/.../python/pyspark/sql/connect/session.py", line 312, in > createDataFrame > raise ValueError( > ValueError: Some of types cannot be determined after inferring, a > StructType Schema is required in this case > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org