namanjain24-sudo commented on issue #25049: URL: https://github.com/apache/datafusion/issues/25049#issuecomment-5641220872
@alamb answered this in full on #25090, since you asked for it there: https://github.com/apache/datafusion/pull/25090#issuecomment-5640878571 — the engine, the two source lines it fails on, and a before/after run. One thing worth adding to @alexandrefimov's reply, because it matters if you try to reproduce it. The error is exactly right for a minimal hand-built `sum:i64` plan, but a plan that DataFusion actually exports never reaches that check. Handing the output of `to_substrait_plan` straight to substrait-java 0.103.0 fails earlier, at extension resolution: ``` IllegalStateException: Function 'sum' references URN anchor -1, but no URN is registered at that anchor ``` We write `extension_urn_reference: u32::MAX` for every function declaration (`extensions.rs:120`), which is #11545. Only once a URN is registered on the plan and the compound key `sum:i64` is used does it get as far as `output_type` and throw `UnsupportedOperationException: Type is not set`. With this change and that same workaround the plan is accepted and comes back as nullable `i64`, which is what our own optimized plan says `sum(t.i)` is. So it is a real compatibility bug and it is the second of two gates rather than the only one. #11545 is the first. Fixing either alone still leaves our aggregate plans unreadable by substrait-java. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
