Xiao Li created SPARK-24696: ------------------------------- Summary: ColumnPruning rule fails to remove extra Project Key: SPARK-24696 URL: https://issues.apache.org/jira/browse/SPARK-24696 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.3.1 Reporter: Xiao Li
{code} test("optimization infinite loop") { withTempDir { dir => val path = dir.getCanonicalPath import testImplicits._ Seq((3, "a")).toDF("id", "value").write.format("parquet").mode("overwrite").save(path) import org.apache.spark.sql.functions.udf val filterIt = udf((value: Int) => value > 0).asNondeterministic spark.read.load(path) .where(filterIt('id)) .where('id < 0) .select('id) .collect } } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org