Cheng Hao created SPARK-10829: --------------------------------- Summary: Scan DataSource with predicate expression combine partition key and attributes doesn't work Key: SPARK-10829 URL: https://issues.apache.org/jira/browse/SPARK-10829 Project: Spark Issue Type: Bug Components: SQL Reporter: Cheng Hao Priority: Blocker
To reproduce that with the code: {code} withSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") { withTempPath { dir => val path = s"${dir.getCanonicalPath}/part=1" (1 to 3).map(i => (i, i.toString)).toDF("a", "b").write.parquet(path) // If the "part = 1" filter gets pushed down, this query will throw an exception since // "part" is not a valid column in the actual Parquet file checkAnswer( sqlContext.read.parquet(path).filter("a > 0 and (part = 0 or a > 1)"), (2 to 3).map(i => Row(i, i.toString, 1))) } } {code} We expect the result as: {code} 2, 1 3, 1 {code} But we got: {code} 1, 1 2, 1 3, 1 {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org