Haejoon Lee created SPARK-42679: ----------------------------------- Summary: createDataFrame doesn't work with non-nullable schema. Key: SPARK-42679 URL: https://issues.apache.org/jira/browse/SPARK-42679 Project: Spark Issue Type: Bug Components: Connect Affects Versions: 3.4.0 Reporter: Haejoon Lee
spark.createDataFrame won't work with non-nullable schema as below: {code:java} from pyspark.sql.types import * schema_false = StructType([StructField("id", IntegerType(), False)]) spark.createDataFrame([[1]], schema=schema_false) Traceback (most recent call last): ... pyspark.errors.exceptions.connect.AnalysisException: [NULLABLE_COLUMN_OR_FIELD] Column or field `id` is nullable while it's required to be non-nullable.{code} whereas it works fine with nullable schema: {code:java} schema_true = StructType([StructField("id", IntegerType(), True)]) spark.createDataFrame([[1]], schema=schema_true) DataFrame[id: int]{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org