[ https://issues.apache.org/jira/browse/SPARK-40176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-40176: --------------------------------- Target Version/s: (was: 3.3.1) > Enhance collapse window optimization to work in case partition or order by > keys are expressions > ----------------------------------------------------------------------------------------------- > > Key: SPARK-40176 > URL: https://issues.apache.org/jira/browse/SPARK-40176 > Project: Spark > Issue Type: Task > Components: SQL > Affects Versions: 3.2.0, 3.2.1, 3.3.0 > Reporter: Ayushi Agarwal > Priority: Major > Fix For: 3.3.1 > > > In window operator with multiple window functions, if any expression is > present in partition by or sort order columns, windows are not collapsed even > if partition and order by expression is same for all those window functions. > E.g. query: > val w = > Window.{_}partitionBy{_}("key").orderBy({_}lower{_}({_}col{_}("value"))) > df.select({_}lead{_}("key", 1).over(w), {_}lead{_}("value", 1).over(w)) > Current Plan: > -Window(lead(value,1), key, _w1) -------------- W1 > - Sort (key, _w1) > -Project (lower(“value”) as _w1) --------- P1 > -Window(lead(key,1), key, _w0) ---------------- W2 > -Sort(key, _w0) > -Exchange(key) > -Project (lower(“value”) as _w0) ---- P2 > -Scan > > W1 and W2 can be merged in single window -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org