Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/21603#discussion_r198124578
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
---
@@ -270,6 +270,11 @@ private[parquet] class ParquetFilters(pushDownDate:
Boolean) {
case sources.Not(pred) =>
createFilter(schema, pred).map(FilterApi.not)
+ case sources.In(name, values) if canMakeFilterOn(name) &&
values.length < 20 =>
--- End diff --
It mainly depends on how many row groups can skip. for small table
(assuming only one row group). There is no obvious difference.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
