Andrew Ash created SPARK-22725:
----------------------------------
Summary: df.select on a Stream is broken, vs a List
Key: SPARK-22725
URL: https://issues.apache.org/jira/browse/SPARK-22725
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.3.0
Reporter: Andrew Ash
See failing test at https://github.com/apache/spark/pull/19917
Failing:
{noformat}
test("SPARK-ABC123: support select with a splatted stream") {
val df = spark.createDataFrame(sparkContext.emptyRDD[Row],
StructType(List("bar", "foo").map {
StructField(_, StringType, false)
}))
val allColumns = Stream(df.col("bar"), col("foo"))
val result = df.select(allColumns : _*)
}
{noformat}
Succeeds:
{noformat}
test("SPARK-ABC123: support select with a splatted stream") {
val df = spark.createDataFrame(sparkContext.emptyRDD[Row],
StructType(List("bar", "foo").map {
StructField(_, StringType, false)
}))
val allColumns = Seq(df.col("bar"), col("foo"))
val result = df.select(allColumns : _*)
}
{noformat}
After stepping through in a debugger, the difference manifests at
https://github.com/apache/spark/blob/8ae004b4602266d1f210e4c1564246d590412c06/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala#L120
Changing {{seq.map}} to {{seq.toList.map}} causes the test to pass.
I think there's a very subtle bug here where the {{Seq}} of column names passed
into {{select}} is expected to eagerly evaluate when {{.map}} is called on it,
even though that's not part of the {{Seq}} contract.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
