Sunninsky commented on issue #26868: URL: https://github.com/apache/datafusion/issues/26868#issuecomment-6770738385
A candidate for 55.3.0: #25472 (fix for #25471, `LEAD/LAG ... IGNORE NULLS` returning NULL / the default across NULL gaps). 55.2.0 still has the bug that #25472 fixes on `main`. In `WindowShiftEvaluator::evaluate` (`datafusion-functions-window/src/lead_lag.rs`), the NULL branch of the LEAD lookahead refill can never run, so `LEAD(v, N) IGNORE NULLS` with N >= 2 returns the wrong row once a NULL falls inside the lookahead. The repro in #25471 shows it: `LEAD(v, 2) IGNORE NULLS` for `id = 3` returns NULL instead of 70. This is a wrong result, not a slowdown. We found it in DataFusion Comet, which runs Spark's `lead` / `lag ... IGNORE NULLS` through this evaluator: a differential run against Spark on real rows returns different rows for LEAD with an offset of 2 or more. As far as I can see #25472 is only on `main` (labelled `v56.0.0`), not on `branch-55`, and it is not in the "Potential Content" list above. Could it be added? A related but separate case in the same evaluator is #27284: with offset 0, `IGNORE NULLS` returns the default for a row that is itself NULL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
