[
https://issues.apache.org/jira/browse/SPARK-27233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798852#comment-16798852
]
Sandeep Katta commented on SPARK-27233:
---------------------------------------
{{asNullable}} should return it as nullable,it cannot be decided based on the
current DF.
for e.g below use case will fail if we don't it return as {{true}}
{{val df = Seq(Some(Seq[Long](1L, 2L,3L)), None).toDF("seq")}}
{{ df.write.format("parquet").saveAsTable("testSchema"); }}
{{val df1 = Seq(Some(Seq(null)), None).toDF("seq")
df1.write.format("parquet").mode("append").saveAsTable("testSchema")}}
{{So it is not an issue, if you use other datatype for e.g. String instead Long
then schema will be same.In spark asNullable is generic, so it returns true }}
> Schema of ArrayType change after saveAsTable and read
> ------------------------------------------------------
>
> Key: SPARK-27233
> URL: https://issues.apache.org/jira/browse/SPARK-27233
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Environment: Spark 2.4
> Scala 2.11
> Reporter: Kritsada Limpawatkul
> Priority: Major
>
> This is code for reproducing.
> {code:java}
> val testTable = "testseq"
> val df = Seq(Some(Seq(1L,2L,3L)), None).toDF("seq")
> sqlContext.sql(s"DROP TABLE IF EXISTS $testTable")
> df.write.format("parquet").saveAsTable(testTable)
> val res = sqlContext.table(testTable)
> assert(df.schema === res.schema){code}
> My code is trying to save dataframe with array type and read from that table.
> After checking schema, I found that schema has change.
> This is the assertion message.
> {code:java}
> Expected :StructType(StructField(seq,ArrayType(LongType,true),true))
> Actual :StructType(StructField(seq,ArrayType(LongType,false),true)){code}
> containsNull in ArrayType changes from false to true after reading from the
> table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
