Takuya Ueshin created SPARK-42458:
-------------------------------------
Summary: createDataFrame should support DDL string as schema
Key: SPARK-42458
URL: https://issues.apache.org/jira/browse/SPARK-42458
Project: Spark
Issue Type: Sub-task
Components: Connect
Affects Versions: 3.4.0
Reporter: Takuya Ueshin
{code:python}
File "/.../python/pyspark/sql/connect/readwriter.py", line 393, in
pyspark.sql.connect.readwriter.DataFrameWriter.option
Failed example:
with tempfile.TemporaryDirectory() as d:
# Write a DataFrame into a CSV file with 'nullValue' option set to
'Hyukjin Kwon'.
df = spark.createDataFrame([(100, None)], "age INT, name STRING")
df.write.option("nullValue", "Hyukjin
Kwon").mode("overwrite").format("csv").save(d)
# Read the CSV file as a DataFrame.
spark.read.schema(df.schema).format('csv').load(d).show()
Exception raised:
Traceback (most recent call last):
File "/.../lib/python3.9/doctest.py", line 1334, in __run
exec(compile(example.source, filename, "single",
File "<doctest
pyspark.sql.connect.readwriter.DataFrameWriter.option[2]>", line 3, in <module>
df = spark.createDataFrame([(100, None)], "age INT, name STRING")
File "/.../python/pyspark/sql/connect/session.py", line 312, in
createDataFrame
raise ValueError(
ValueError: Some of types cannot be determined after inferring, a
StructType Schema is required in this case
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
