[jira] [Commented] (SPARK-10301) For struct type, if parquet's global schema has less fields than a file's schema, data reading will fail
[
https://issues.apache.org/jira/browse/SPARK-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716125#comment-14716125
]
Yin Huai commented on SPARK-10301:
--
Seems this one is hard because at the executor side, we are actually using
parquet file's schema to read data and parquet file's schema contains struct
fields that do not appear in the global schema.
For now, the workaround is to enable schema merge (set {{mergeSchema}} to true
when load a parquet dataset), so the global schema is always the superset of
the local schema.
> For struct type, if parquet's global schema has less fields than a file's
> schema, data reading will fail
>
>
> Key: SPARK-10301
> URL: https://issues.apache.org/jira/browse/SPARK-10301
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.5.0
>Reporter: Yin Huai
>Assignee: Yin Huai
>Priority: Critical
>
> When parquet's global schema has less number of fields than the local schema
> of a file, the data reading path will fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10301) For struct type, if parquet's global schema has less fields than a file's schema, data reading will fail
[ https://issues.apache.org/jira/browse/SPARK-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720112#comment-14720112 ] Apache Spark commented on SPARK-10301: -- User 'liancheng' has created a pull request for this issue: https://github.com/apache/spark/pull/8509 > For struct type, if parquet's global schema has less fields than a file's > schema, data reading will fail > > > Key: SPARK-10301 > URL: https://issues.apache.org/jira/browse/SPARK-10301 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.0 >Reporter: Yin Huai >Assignee: Yin Huai >Priority: Critical > > When parquet's global schema has less number of fields than the local schema > of a file, the data reading path will fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10301) For struct type, if parquet's global schema has less fields than a file's schema, data reading will fail
[ https://issues.apache.org/jira/browse/SPARK-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14721187#comment-14721187 ] Apache Spark commented on SPARK-10301: -- User 'yhuai' has created a pull request for this issue: https://github.com/apache/spark/pull/8515 > For struct type, if parquet's global schema has less fields than a file's > schema, data reading will fail > > > Key: SPARK-10301 > URL: https://issues.apache.org/jira/browse/SPARK-10301 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.0 >Reporter: Yin Huai >Assignee: Yin Huai >Priority: Critical > > When parquet's global schema has less number of fields than the local schema > of a file, the data reading path will fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10301) For struct type, if parquet's global schema has less fields than a file's schema, data reading will fail
[ https://issues.apache.org/jira/browse/SPARK-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14721312#comment-14721312 ] Yin Huai commented on SPARK-10301: -- https://github.com/apache/spark/pull/8515 has been merged. It is not the fix for this issue but will give users a nice error message when the global schema as less struct fields than local parquet file schema (it will ask users to enable schema merging). I am re-targeting this issue to 1.6 for the proper fix (https://github.com/apache/spark/pull/8509). > For struct type, if parquet's global schema has less fields than a file's > schema, data reading will fail > > > Key: SPARK-10301 > URL: https://issues.apache.org/jira/browse/SPARK-10301 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.0 >Reporter: Yin Huai >Assignee: Yin Huai >Priority: Critical > > When parquet's global schema has less number of fields than the local schema > of a file, the data reading path will fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10301) For struct type, if parquet's global schema has less fields than a file's schema, data reading will fail
[
https://issues.apache.org/jira/browse/SPARK-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14721447#comment-14721447
]
Cheng Lian commented on SPARK-10301:
Updated ticket description to provide a more general view of this issue. Would
also be helpful for reviewing https://github.com/apache/spark/pull/8509
> For struct type, if parquet's global schema has less fields than a file's
> schema, data reading will fail
>
>
> Key: SPARK-10301
> URL: https://issues.apache.org/jira/browse/SPARK-10301
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.5.0
>Reporter: Yin Huai
>Assignee: Cheng Lian
>Priority: Critical
>
> We hit this issue when reading a complex Parquet dateset without turning on
> schema merging. The data set consists of Parquet files with different but
> compatible schemas. In this way, the schema of the dataset is defined by
> either a summary file or a random physical Parquet file if no summary files
> are available. Apparently, this schema may not containing all fields
> appeared in all physicla files.
> Parquet was designed with schema evolution and column pruning in mind, so it
> should be legal for a user to use a tailored schema to read the dataset to
> save disk IO. For example, say we have a Parquet dataset consisting of two
> physical Parquet files with the following two schemas:
> {noformat}
> message m0 {
> optional group f0 {
> optional int64 f00;
> optional int64 f01;
> }
> }
> message m1 {
> optional group f0 {
> optional int64 f01;
> optional int64 f01;
> optional int64 f02;
> }
> optional double f1;
> }
> {noformat}
> Users should be allowed to read the dataset with the following schema:
> {noformat}
> message m1 {
> optional group f0 {
> optional int64 f01;
> optional int64 f02;
> }
> }
> {noformat}
> so that {{f0.f00}} and {{f1}} are never touched. The above case can be
> expressed by the following {{spark-shell}} snippet:
> {noformat}
> import sqlContext._
> import sqlContext.implicits._
> import org.apache.spark.sql.types.{LongType, StructType}
> val path = "/tmp/spark/parquet"
> range(3).selectExpr("NAMED_STRUCT('f00', id, 'f01', id) AS f0").coalesce(1)
> .write.mode("overwrite").parquet(path)
> range(3).selectExpr("NAMED_STRUCT('f00', id, 'f01', id, 'f02', id) AS f0",
> "CAST(id AS DOUBLE) AS f1").coalesce(1)
> .write.mode("append").parquet(path)
> val tailoredSchema =
> new StructType()
> .add(
> "f0",
> new StructType()
> .add("f01", LongType, nullable = true)
> .add("f02", LongType, nullable = true),
> nullable = true)
> read.schema(tailoredSchema).parquet(path).show()
> {noformat}
> Expected output should be:
> {noformat}
> ++
> | f0|
> ++
> |[0,null]|
> |[1,null]|
> |[2,null]|
> | [0,0]|
> | [1,1]|
> | [2,2]|
> ++
> {noformat}
> However, current 1.5-SNAPSHOT version throws the following exception:
> {noformat}
> org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in
> block -1 in file
> hdfs://localhost:9000/tmp/spark/parquet/part-r-0-56c4604e-c546-4f97-a316-05da8ab1a0bf.gz.parquet
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201)
> at
> org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:168)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
> at
> scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
> at
> org.apache.spark.sql.execution.Spark
[jira] [Commented] (SPARK-10301) For struct type, if parquet's global schema has less fields than a file's schema, data reading will fail
[
https://issues.apache.org/jira/browse/SPARK-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728807#comment-14728807
]
Apache Spark commented on SPARK-10301:
--
User 'liancheng' has created a pull request for this issue:
https://github.com/apache/spark/pull/8583
> For struct type, if parquet's global schema has less fields than a file's
> schema, data reading will fail
>
>
> Key: SPARK-10301
> URL: https://issues.apache.org/jira/browse/SPARK-10301
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.5.0
>Reporter: Yin Huai
>Assignee: Cheng Lian
>Priority: Critical
> Fix For: 1.6.0
>
>
> We hit this issue when reading a complex Parquet dateset without turning on
> schema merging. The data set consists of Parquet files with different but
> compatible schemas. In this way, the schema of the dataset is defined by
> either a summary file or a random physical Parquet file if no summary files
> are available. Apparently, this schema may not containing all fields
> appeared in all physicla files.
> Parquet was designed with schema evolution and column pruning in mind, so it
> should be legal for a user to use a tailored schema to read the dataset to
> save disk IO. For example, say we have a Parquet dataset consisting of two
> physical Parquet files with the following two schemas:
> {noformat}
> message m0 {
> optional group f0 {
> optional int64 f00;
> optional int64 f01;
> }
> }
> message m1 {
> optional group f0 {
> optional int64 f01;
> optional int64 f01;
> optional int64 f02;
> }
> optional double f1;
> }
> {noformat}
> Users should be allowed to read the dataset with the following schema:
> {noformat}
> message m1 {
> optional group f0 {
> optional int64 f01;
> optional int64 f02;
> }
> }
> {noformat}
> so that {{f0.f00}} and {{f1}} are never touched. The above case can be
> expressed by the following {{spark-shell}} snippet:
> {noformat}
> import sqlContext._
> import sqlContext.implicits._
> import org.apache.spark.sql.types.{LongType, StructType}
> val path = "/tmp/spark/parquet"
> range(3).selectExpr("NAMED_STRUCT('f00', id, 'f01', id) AS f0").coalesce(1)
> .write.mode("overwrite").parquet(path)
> range(3).selectExpr("NAMED_STRUCT('f00', id, 'f01', id, 'f02', id) AS f0",
> "CAST(id AS DOUBLE) AS f1").coalesce(1)
> .write.mode("append").parquet(path)
> val tailoredSchema =
> new StructType()
> .add(
> "f0",
> new StructType()
> .add("f01", LongType, nullable = true)
> .add("f02", LongType, nullable = true),
> nullable = true)
> read.schema(tailoredSchema).parquet(path).show()
> {noformat}
> Expected output should be:
> {noformat}
> ++
> | f0|
> ++
> |[0,null]|
> |[1,null]|
> |[2,null]|
> | [0,0]|
> | [1,1]|
> | [2,2]|
> ++
> {noformat}
> However, current 1.5-SNAPSHOT version throws the following exception:
> {noformat}
> org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in
> block -1 in file
> hdfs://localhost:9000/tmp/spark/parquet/part-r-0-56c4604e-c546-4f97-a316-05da8ab1a0bf.gz.parquet
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201)
> at
> org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:168)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
> at
> scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5
[jira] [Commented] (SPARK-10301) For struct type, if parquet's global schema has less fields than a file's schema, data reading will fail
[
https://issues.apache.org/jira/browse/SPARK-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735735#comment-14735735
]
Yin Huai commented on SPARK-10301:
--
[~lian cheng] Let's also have a follow-up pr for the master branch to address
post-hoc review comments.
> For struct type, if parquet's global schema has less fields than a file's
> schema, data reading will fail
>
>
> Key: SPARK-10301
> URL: https://issues.apache.org/jira/browse/SPARK-10301
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.5.0
>Reporter: Yin Huai
>Assignee: Cheng Lian
>Priority: Critical
> Labels: backport-needed
> Fix For: 1.6.0
>
>
> We hit this issue when reading a complex Parquet dateset without turning on
> schema merging. The data set consists of Parquet files with different but
> compatible schemas. In this way, the schema of the dataset is defined by
> either a summary file or a random physical Parquet file if no summary files
> are available. Apparently, this schema may not containing all fields
> appeared in all physicla files.
> Parquet was designed with schema evolution and column pruning in mind, so it
> should be legal for a user to use a tailored schema to read the dataset to
> save disk IO. For example, say we have a Parquet dataset consisting of two
> physical Parquet files with the following two schemas:
> {noformat}
> message m0 {
> optional group f0 {
> optional int64 f00;
> optional int64 f01;
> }
> }
> message m1 {
> optional group f0 {
> optional int64 f01;
> optional int64 f01;
> optional int64 f02;
> }
> optional double f1;
> }
> {noformat}
> Users should be allowed to read the dataset with the following schema:
> {noformat}
> message m1 {
> optional group f0 {
> optional int64 f01;
> optional int64 f02;
> }
> }
> {noformat}
> so that {{f0.f00}} and {{f1}} are never touched. The above case can be
> expressed by the following {{spark-shell}} snippet:
> {noformat}
> import sqlContext._
> import sqlContext.implicits._
> import org.apache.spark.sql.types.{LongType, StructType}
> val path = "/tmp/spark/parquet"
> range(3).selectExpr("NAMED_STRUCT('f00', id, 'f01', id) AS f0").coalesce(1)
> .write.mode("overwrite").parquet(path)
> range(3).selectExpr("NAMED_STRUCT('f00', id, 'f01', id, 'f02', id) AS f0",
> "CAST(id AS DOUBLE) AS f1").coalesce(1)
> .write.mode("append").parquet(path)
> val tailoredSchema =
> new StructType()
> .add(
> "f0",
> new StructType()
> .add("f01", LongType, nullable = true)
> .add("f02", LongType, nullable = true),
> nullable = true)
> read.schema(tailoredSchema).parquet(path).show()
> {noformat}
> Expected output should be:
> {noformat}
> ++
> | f0|
> ++
> |[0,null]|
> |[1,null]|
> |[2,null]|
> | [0,0]|
> | [1,1]|
> | [2,2]|
> ++
> {noformat}
> However, current 1.5-SNAPSHOT version throws the following exception:
> {noformat}
> org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in
> block -1 in file
> hdfs://localhost:9000/tmp/spark/parquet/part-r-0-56c4604e-c546-4f97-a316-05da8ab1a0bf.gz.parquet
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201)
> at
> org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:168)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
> at
> scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
> at
> org.apache.spark.
[jira] [Commented] (SPARK-10301) For struct type, if parquet's global schema has less fields than a file's schema, data reading will fail
[
https://issues.apache.org/jira/browse/SPARK-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736947#comment-14736947
]
Apache Spark commented on SPARK-10301:
--
User 'liancheng' has created a pull request for this issue:
https://github.com/apache/spark/pull/8670
> For struct type, if parquet's global schema has less fields than a file's
> schema, data reading will fail
>
>
> Key: SPARK-10301
> URL: https://issues.apache.org/jira/browse/SPARK-10301
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.5.0
>Reporter: Yin Huai
>Assignee: Cheng Lian
>Priority: Critical
> Fix For: 1.6.0, 1.5.1
>
>
> We hit this issue when reading a complex Parquet dateset without turning on
> schema merging. The data set consists of Parquet files with different but
> compatible schemas. In this way, the schema of the dataset is defined by
> either a summary file or a random physical Parquet file if no summary files
> are available. Apparently, this schema may not containing all fields
> appeared in all physicla files.
> Parquet was designed with schema evolution and column pruning in mind, so it
> should be legal for a user to use a tailored schema to read the dataset to
> save disk IO. For example, say we have a Parquet dataset consisting of two
> physical Parquet files with the following two schemas:
> {noformat}
> message m0 {
> optional group f0 {
> optional int64 f00;
> optional int64 f01;
> }
> }
> message m1 {
> optional group f0 {
> optional int64 f01;
> optional int64 f01;
> optional int64 f02;
> }
> optional double f1;
> }
> {noformat}
> Users should be allowed to read the dataset with the following schema:
> {noformat}
> message m1 {
> optional group f0 {
> optional int64 f01;
> optional int64 f02;
> }
> }
> {noformat}
> so that {{f0.f00}} and {{f1}} are never touched. The above case can be
> expressed by the following {{spark-shell}} snippet:
> {noformat}
> import sqlContext._
> import sqlContext.implicits._
> import org.apache.spark.sql.types.{LongType, StructType}
> val path = "/tmp/spark/parquet"
> range(3).selectExpr("NAMED_STRUCT('f00', id, 'f01', id) AS f0").coalesce(1)
> .write.mode("overwrite").parquet(path)
> range(3).selectExpr("NAMED_STRUCT('f00', id, 'f01', id, 'f02', id) AS f0",
> "CAST(id AS DOUBLE) AS f1").coalesce(1)
> .write.mode("append").parquet(path)
> val tailoredSchema =
> new StructType()
> .add(
> "f0",
> new StructType()
> .add("f01", LongType, nullable = true)
> .add("f02", LongType, nullable = true),
> nullable = true)
> read.schema(tailoredSchema).parquet(path).show()
> {noformat}
> Expected output should be:
> {noformat}
> ++
> | f0|
> ++
> |[0,null]|
> |[1,null]|
> |[2,null]|
> | [0,0]|
> | [1,1]|
> | [2,2]|
> ++
> {noformat}
> However, current 1.5-SNAPSHOT version throws the following exception:
> {noformat}
> org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in
> block -1 in file
> hdfs://localhost:9000/tmp/spark/parquet/part-r-0-56c4604e-c546-4f97-a316-05da8ab1a0bf.gz.parquet
> at
> org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228)
> at
> org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201)
> at
> org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.hasNext(SqlNewHadoopRDD.scala:168)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:308)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
> at
> scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:215)
> at
> org.apache.spark.sql.execution.SparkPlan$$an
