Re: Physical plan for windows and joins - how to know which is faster?

2016-11-09 Thread Silvio Fiorito
Hi Jacek, I haven't played with 2.1.0 yet, so not sure how much more optimized Window functions are compared to 1.6 and 2.0. However, one thing I do see in the self-join is a broadcast. So there's going to be a need broadcast the results of the groupBy out to the executors before it can do

Physical plan for windows and joins - how to know which is faster?

2016-11-09 Thread Jacek Laskowski
Hi, While playing around with Spark 2.1.0-SNAPSHOT (built today) and explain'ing two queries with WindowSpec and inner join I found the following plans and am wondering if you could help me to judge which query could be faster. What else would you ask for to be able to answer the question of one