Re: Spark SQL 2.1 Complex SQL - Query Planning Issue

Please let me know if anybody has any thoughts on this issue? On Thu, Mar 30, 2017 at 10:37 PM Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Also, is it possible to cache logical plan and parsed query so that in > subsequent executions it can be reused. It would improve overall

Re: Spark SQL 2.1 Complex SQL - Query Planning Issue

Also, is it possible to cache logical plan and parsed query so that in subsequent executions it can be reused. It would improve overall query performance particularly in streaming jobs On Thu, Mar 30, 2017 at 10:06 PM Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Hi Ayan, > > I

Re: Spark SQL 2.1 Complex SQL - Query Planning Issue

Hi Ayan, I have searched Spark configuration options but couldn't find one to pin execution plans in memory. Can you please help? Thanks Sathish On Thu, Mar 30, 2017 at 9:30 PM ayan guha wrote: > I think there is an option of pinning execution plans in memory to avoid >

Re: Spark SQL 2.1 Complex SQL - Query Planning Issue

I think there is an option of pinning execution plans in memory to avoid such scenarios On Fri, Mar 31, 2017 at 1:25 PM, Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Hi Everyone, > > I have complex SQL with approx 2000 lines of code and works with 50+ > tables with 50+

Spark SQL 2.1 Complex SQL - Query Planning Issue

Hi Everyone, I have complex SQL with approx 2000 lines of code and works with 50+ tables with 50+ left joins and transformations. All the tables are fully cached in Memory with sufficient storage memory and working memory. The issue is after the launch of the query for the execution; the query