andygrove opened a new issue, #2368: URL: https://github.com/apache/datafusion-ballista/issues/2368
Both `ballista-core` and `ballista-executor` declare a feature named `arrow-ipc-optimizations`, and both have it on by default: - `ballista/core/Cargo.toml:38`, defaulted at line 40 - `ballista/executor/Cargo.toml:36`, defaulted at line 46 They are unrelated features that happen to share a name. The executor's gates exactly one call site, `flight_service.rs:159`. Core's gates the rest of the shuffle read path. The executor's does not forward to core's, unlike `spark-compat` right next to it at `ballista/executor/Cargo.toml:47`. This makes `ballista/client/Cargo.toml:33-35` misleading. It pulls the executor with `default-features = false, features = ["arrow-ipc-optimizations"]`, which reads as "keep the IPC optimization on" but only enables the executor's half. Core's happens to be on anyway through the client's direct dependency on `ballista-core` at line 32, so nothing is broken today, but it works by accident. Anyone building the executor standalone with `--no-default-features --features arrow-ipc-optimizations` gets one of the four decode sites optimized and the other three validating, with nothing to indicate it. Suggested fix, in `ballista/executor/Cargo.toml`: ```toml arrow-ipc-optimizations = ["ballista-core/arrow-ipc-optimizations"] ``` Default behavior is unchanged since both are already on by default. Worth noting in the fix that this still does not give a full off switch, because Cargo features are additive and anything else in the graph enabling core's default turns it back on. Related: `examples/Cargo.toml:61` sets `default-features = false` on `ballista-core` to get `build-binary` and silently loses `arrow-ipc-optimizations` as a side effect. Same root cause, may be worth fixing in the same PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
