RE: Spark-SQL Query Optimization: overlapping ranges

2017-05-01 Thread Lavelle, Shawn
com<mailto:shawn.lave...@osii.com>> Cc: user <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Spark-SQL Query Optimization: overlapping ranges explain it and you'll know what happens under the covers. i.e. Use explain on the Dataset. Jacek On 25 Apr 2017

Re: Spark-SQL Query Optimization: overlapping ranges

2017-04-27 Thread Jacek Laskowski
e probably going to write our own > org.apache.spark.sql.catalyst.rules.Rule to handle it. > > ~ Shawn > > > > *From:* Jacek Laskowski [mailto:ja...@japila.pl] > *Sent:* Wednesday, April 26, 2017 2:55 AM > *To:* Lavelle, Shawn <shawn.lave...@osii.com> > *Cc:* user

RE: Spark-SQL Query Optimization: overlapping ranges

2017-04-27 Thread Lavelle, Shawn
of thing. We’re probably going to write our own org.apache.spark.sql.catalyst.rules.Rule to handle it. ~ Shawn From: Jacek Laskowski [mailto:ja...@japila.pl] Sent: Wednesday, April 26, 2017 2:55 AM To: Lavelle, Shawn <shawn.lave...@osii.com> Cc: user <user@spark.apache.org> Subject: Re: Spa

Re: Spark-SQL Query Optimization: overlapping ranges

2017-04-26 Thread Jacek Laskowski
explain it and you'll know what happens under the covers. i.e. Use explain on the Dataset. Jacek On 25 Apr 2017 12:46 a.m., "Lavelle, Shawn" wrote: > Hello Spark Users! > >Does the Spark Optimization engine reduce overlapping column ranges? > If so, should it push