Uros Stankovic created SPARK-49893:
--------------------------------------
Summary: File data sources don't respect provided user schema
nullability for all APIs
Key: SPARK-49893
URL: https://issues.apache.org/jira/browse/SPARK-49893
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Uros Stankovic
DataFrameReader has 3 APIs for JSON reading
- json(DataSet[String])
- json(Rdd)
- json(filePath)
First two APIs respects provided user schema nullability when spark flag
`spark.sql.legacy.respectNullabilityInTextDatasetConversion` is set to true,
but third one does not respect and provided schema nullability is always
overriden to true.
Some users just want to have a validation of data and to get exception when
some field is nullable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
