[
https://issues.apache.org/jira/browse/SPARK-27233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kritsada Limpawatkul updated SPARK-27233:
-----------------------------------------
Description:
This is code for reproducing.
{code:java}
val testTable = "testseq"
val df = Seq(Some(Seq(1L,2L,3L)), None).toDF("seq")
sqlContext.sql(s"DROP TABLE IF EXISTS $testTable")
df.write.format("parquet").saveAsTable(testTable)
val res = sqlContext.table(testTable)
assert(df.schema === res.schema){code}
My code is trying to save dataframe with array type and read from that table.
After checking schema, I found that schema has change.
This is the assertion message.
{code:java}
Expected :StructType(StructField(seq,ArrayType(LongType,true),true))
Actual :StructType(StructField(seq,ArrayType(LongType,false),true)){code}
containsNull in ArrayType changes from false to true after reading from the
table.
was:
This is code for reproducing.
{code:java}
val testTable = "testseq"
val df = Seq(Some(Seq(1L,2L,3L)), None).toDF("seq")
sqlContext.sql(s"DROP TABLE IF EXISTS $testTable")
df.write.format("parquet").saveAsTable(testTable)
val res = sqlContext.table(testTable)
assert(df.schema === res.schema){code}
My code is trying to save dataframe with array type and read from that table.
After checking schema, I found that schema has change.
This is the assertion message.
{code:java}
Expected :StructType(StructField(seq,ArrayType(LongType,true),true))
Actual :StructType(StructField(seq,ArrayType(LongType,false),true)){code}
containsNull in ArrayType changes from false to true after reading from the
table.
> Schema of ArrayType change after saveAsTable and read
> ------------------------------------------------------
>
> Key: SPARK-27233
> URL: https://issues.apache.org/jira/browse/SPARK-27233
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Environment: Spark 2.4
> Scala 2.11
> Reporter: Kritsada Limpawatkul
> Priority: Minor
>
> This is code for reproducing.
> {code:java}
> val testTable = "testseq"
> val df = Seq(Some(Seq(1L,2L,3L)), None).toDF("seq")
> sqlContext.sql(s"DROP TABLE IF EXISTS $testTable")
> df.write.format("parquet").saveAsTable(testTable)
> val res = sqlContext.table(testTable)
> assert(df.schema === res.schema){code}
> My code is trying to save dataframe with array type and read from that table.
> After checking schema, I found that schema has change.
> This is the assertion message.
> {code:java}
> Expected :StructType(StructField(seq,ArrayType(LongType,true),true))
> Actual :StructType(StructField(seq,ArrayType(LongType,false),true)){code}
> containsNull in ArrayType changes from false to true after reading from the
> table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
