[ https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-22725. ------------------------------- Resolution: Won't Fix No follow up and not clear this is expected to work > df.select on a Stream is broken, vs a List > ------------------------------------------ > > Key: SPARK-22725 > URL: https://issues.apache.org/jira/browse/SPARK-22725 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Andrew Ash > Priority: Minor > > See failing test at https://github.com/apache/spark/pull/19917 > Failing: > {noformat} > test("SPARK-ABC123: support select with a splatted stream") { > val df = spark.createDataFrame(sparkContext.emptyRDD[Row], > StructType(List("bar", "foo").map { > StructField(_, StringType, false) > })) > val allColumns = Stream(df.col("bar"), col("foo")) > val result = df.select(allColumns : _*) > } > {noformat} > Succeeds: > {noformat} > test("SPARK-ABC123: support select with a splatted stream") { > val df = spark.createDataFrame(sparkContext.emptyRDD[Row], > StructType(List("bar", "foo").map { > StructField(_, StringType, false) > })) > val allColumns = Seq(df.col("bar"), col("foo")) > val result = df.select(allColumns : _*) > } > {noformat} > After stepping through in a debugger, the difference manifests at > https://github.com/apache/spark/blob/8ae004b4602266d1f210e4c1564246d590412c06/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala#L120 > Changing {{seq.map}} to {{seq.toList.map}} causes the test to pass. > I think there's a very subtle bug here where the {{Seq}} of column names > passed into {{select}} is expected to eagerly evaluate when {{.map}} is > called on it, even though that's not part of the {{Seq}} contract. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org