Jacek, Thanks for your help. I didn’t want to write a bug/enhancement unless warranted.
~ Shawn From: Jacek Laskowski [mailto:ja...@japila.pl] Sent: Thursday, April 27, 2017 8:39 AM To: Lavelle, Shawn <shawn.lave...@osii.com> Cc: user <user@spark.apache.org> Subject: Re: Spark-SQL Query Optimization: overlapping ranges Hi Shawn, If you're asking me if Spark SQL should optimize such queries, I don't know. If you're asking me if it's possible to convince Spark SQL to do so, I'd say, sure, it is. Write your optimization rule and attach it to Optimizer (using extraOptimizations extension point). Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Thu, Apr 27, 2017 at 3:22 PM, Lavelle, Shawn <shawn.lave...@osii.com<mailto:shawn.lave...@osii.com>> wrote: Hi Jacek, I know that it is not currently doing so, but should it be? The algorithm isn’t complicated and could be applied to both OR and AND logical operators with comparison operators as children. My users write programs to generate queries that aren’t checked for this sort of thing. We’re probably going to write our own org.apache.spark.sql.catalyst.rules.Rule to handle it. ~ Shawn From: Jacek Laskowski [mailto:ja...@japila.pl<mailto:ja...@japila.pl>] Sent: Wednesday, April 26, 2017 2:55 AM To: Lavelle, Shawn <shawn.lave...@osii.com<mailto:shawn.lave...@osii.com>> Cc: user <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Spark-SQL Query Optimization: overlapping ranges explain it and you'll know what happens under the covers. i.e. Use explain on the Dataset. Jacek On 25 Apr 2017 12:46 a.m., "Lavelle, Shawn" <shawn.lave...@osii.com<mailto:shawn.lave...@osii.com>> wrote: Hello Spark Users! Does the Spark Optimization engine reduce overlapping column ranges? If so, should it push this down to a Data Source? Example, This: Select * from table where col between 3 and 7 OR col between 5 and 9 Reduces to: Select * from table where col between 3 and 9 Thanks for your insight! ~ Shawn M Lavelle [cid:image001.png@01D2C298.8343C580] Shawn Lavelle Software Development 4101 Arrowhead Drive Medina, Minnesota 55340-9457 Phone: 763 551 0559<tel:(763)%20551-0559> Fax: 763 551 0750<tel:(763)%20551-0750> Email: shawn.lave...@osii.com<mailto:shawn.lave...@osii.com> Website: www.osii.com<http://www.osii.com>
