[ https://issues.apache.org/jira/browse/SPARK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195063#comment-16195063 ]
Apache Spark commented on SPARK-22181: -------------------------------------- User 'sathiyapk' has created a pull request for this issue: https://github.com/apache/spark/pull/19451 > ReplaceExceptWithNotFilter if one or both of the datasets are fully derived > out of Filters from a same parent > ------------------------------------------------------------------------------------------------------------- > > Key: SPARK-22181 > URL: https://issues.apache.org/jira/browse/SPARK-22181 > Project: Spark > Issue Type: New Feature > Components: Optimizer, SQL > Affects Versions: 2.1.1, 2.2.0 > Reporter: Sathiya Kumar > Priority: Minor > > While applying Except operator between two datasets, if one or both of the > datasets are purely transformed using filter operations, then instead of > rewriting the Except operator using expensive join operation, we can rewrite > it using filter operation by flipping the filter condition of the right node. > ![Case-1](https://github.com/sathiyapk/Blog-Posts/blob/master/images/spark-optimizer/ReplaceExceptWithNotFilter-case1.png) > ![Case-2](https://github.com/sathiyapk/Blog-Posts/blob/master/images/spark-optimizer/ReplaceExceptWithNotFilter-case2.png) -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org