Hi, I would like to know: Is there any optimization has been done for window functions in Spark SQL?
For example. select key, max(value1) over(partition by key) as m1, max(value2) over(partition by key) as m2, max(value3) over(partition by key) as m3 from table The query above creates 3 fields based on the same partition rule. The question is: Will spark-sql partition the table 3 times in the same way to get the three max values ? or just partition once if it finds the partition rule is the same ? It would be nice if someone could point out some lines of code on it. Thank you. Hao -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-SQL-Window-Functions-optimization-tp23796.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
