Yash Datta created SPARK-3711: --------------------------------- Summary: Optimize where in clause filter queries Key: SPARK-3711 URL: https://issues.apache.org/jira/browse/SPARK-3711 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 1.1.0 Reporter: Yash Datta Priority: Minor Fix For: 1.1.1
The In case class is replaced by a InSet class in case all the filters are literals, which uses a hashset instead of Sequence, thereby giving significant performance improvement. Maximum improvement should be visible in case small percentage of large data matches the filter list -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org