Kritsada Limpawatkul created SPARK-27233:
--------------------------------------------
Summary: Schema of ArrayType change after saveAsTable and read
Key: SPARK-27233
URL: https://issues.apache.org/jira/browse/SPARK-27233
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.0
Environment: Spark 2.4
Scala 2.11
Reporter: Kritsada Limpawatkul
This is code for reproducing.
{code:java}
val testTable = "testseq"
val df = Seq(Some(Seq(1L,2L,3L)), None).toDF("seq")
sqlContext.sql(s"DROP TABLE IF EXISTS $testTable")
df.write.format("parquet").saveAsTable(testTable)
val res = sqlContext.table(testTable)
assert(df.schema === res.schema){code}
My code is trying to save dataframe with array type and read from that table.
After checking schema, I found that schema has change.
This is the assertion message.
{code:java}
Expected :StructType(StructField(seq,ArrayType(LongType,true),true))
Actual :StructType(StructField(seq,ArrayType(LongType,false),true)){code}
containsNull in ArrayType changes from false to true after reading from the
table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
