[ https://issues.apache.org/jira/browse/SPARK-42679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-42679: ------------------------------------ Assignee: Apache Spark > createDataFrame doesn't work with non-nullable schema. > ------------------------------------------------------ > > Key: SPARK-42679 > URL: https://issues.apache.org/jira/browse/SPARK-42679 > Project: Spark > Issue Type: Sub-task > Components: Connect > Affects Versions: 3.4.0 > Reporter: Haejoon Lee > Assignee: Apache Spark > Priority: Major > > spark.createDataFrame won't work with non-nullable schema as below: > {code:java} > from pyspark.sql.types import * > schema_false = StructType([StructField("id", IntegerType(), False)]) > spark.createDataFrame([[1]], schema=schema_false) > Traceback (most recent call last): > ... > pyspark.errors.exceptions.connect.AnalysisException: > [NULLABLE_COLUMN_OR_FIELD] Column or field `id` is nullable while it's > required to be non-nullable.{code} > whereas it works fine with nullable schema: > {code:java} > schema_true = StructType([StructField("id", IntegerType(), True)]) > spark.createDataFrame([[1]], schema=schema_true) > DataFrame[id: int]{code} > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org