namanjain24-sudo commented on PR #25527: URL: https://github.com/apache/datafusion/pull/25527#issuecomment-6416006211
Thanks for running it — both reproduce here. **Old plans.** Confirmed, and pinned in a test. `datafusion_grouping_id_bitmask.json` is main's own export of `GROUPING(a), GROUPING(b)` over `GROUPING SETS ((a), (b))`: it returns `(0,1)` / `(1,0)` on main and `(0,0)` / `(0,1)` here. Nothing in the plan records whether that trailing column is the bitmask or the index, so a consumer cannot honour both, and honouring the old one is the bug. Added the test plus a 56.0.0 upgrade-guide note that such plans need re-exporting. **The `4294967295`.** I think the diagnosis is slightly off, and it changes what the fix should be. The `equal` is registered — it comes out as `functionAnchor: 0`. The `4294967295` is the `extensionUrnReference`, which DataFusion hardcodes to `u32::MAX` for every function it emits (`extensions.rs`, #11545). On main: - `SELECT a + 1 FROM t` → `add`, urn `4294967295` - `SELECT a, b, count(*) ... GROUP BY GROUPING SETS ((a), (b))` → `count`, same urn - the `GROUPING(a)` plan above → `bitwise_and` and `bitwise_shift_right`, same urn So I would expect Isthmus to reject those on main too — does it? If it does, what is new here is narrower than it looks: your example emitted zero functions before and emits one now, so it is the first time that shape reaches the gap. Worth avoiding either way. The map is a switch over literals, which is what Substrait's `SwitchExpression` is for — no function reference at all, so the export drops back to zero declarations. Our consumer would need `consume_switch`, which is `not_impl` today. Does Isthmus 0.107 read `SwitchExpression`? If it does I will move the producer over to it; if not, this is #11545 and I would rather not take that on in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
