Sunninsky commented on PR #27410:
URL: https://github.com/apache/datafusion/pull/27410#issuecomment-6923095214

   Thanks @alamb, that is fair: 4.7k lines is too much to review, and the 
description starts with the implementation instead of the problem. I will split 
this up and rewrite the descriptions.
   
   The problem, from a user's side: window queries with an `ORDER BY` take far 
more time per row than the arithmetic needs. On `main`, with 60,000-row 
partitions, `SUM(v) OVER (PARTITION BY k ORDER BY ts)` costs 286 ns of CPU per 
input row (3.8 ns of that is the scan and the final aggregate), and with `RANGE 
BETWEEN 5 PRECEDING AND 5 FOLLOWING` it costs 779 ns. With 10-row partitions 
even `row_number()` costs 92 ns. The prototype here brings these to 15, 21 and 
6 ns. The reproducer is in #27400.
   
   The design adds no new operator and changes no plans or SQL semantics: the 
frames and results of a run of rows are computed in one call instead of one row 
at a time, through optional methods whose default is today's per-row code, so a 
function that does not implement them runs exactly as it does today.
   
   Plan: a first PR that only adds these shapes to the `window_query_sql` 
benchmark (about 130 lines), so the cost is visible on `main`; then one 
optimization per PR, a few hundred lines each, with its own tests, a 
before/after number, and a description that starts from the query. Whatever 
overlaps #27466 will wait for it. I will move this PR back to draft and keep it 
as the reference for the whole picture. If you would prefer a different first 
step, tell me and I will change the order.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to