andygrove opened a new issue, #2410:
URL: https://github.com/apache/datafusion-ballista/issues/2410
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
We do not currently check for unused dependencies anywhere in CI. Our Rust
lint surface is `cargo fmt`, `cargo clippy`, `cargo tomlfmt`, plus the
docs/proto/config-docs checks, and the `Dependencies` workflow only covers the
MSRV check and the `python/Cargo.lock` sync check.
As a result, dependencies that are no longer used stay in our manifests.
They still get resolved, downloaded, and (in many cases) compiled, which slows
down builds, adds noise to `Cargo.lock`, and grows the surface we have to audit
for supply chain and license reasons.
Running `cargo machete` locally today reports quite a few:
```
ballista-core -- ./ballista/core/Cargo.toml:
aws-config
aws-credential-types
md-5
prost-types
ballista-scheduler -- ./ballista/scheduler/Cargo.toml:
insta
prost-types
tonic-prost
ballista-executor -- ./ballista/executor/Cargo.toml:
arrow
tracing
pyballista -- ./python/Cargo.toml:
async-trait
msrvcheck -- ./dev/msrvcheck/Cargo.toml:
time
ballista-cli -- ./ballista-cli/Cargo.toml:
critical-section
tracing-appender
```
Some of these are likely false positives (for example deps that are only
referenced through macros or through generated code), but several look like
genuinely dead entries.
**Describe the solution you'd like**
1. Triage the list above, remove the dependencies that really are unused,
and add the rest to `[package.metadata.cargo-machete] ignored = [...]` with a
short comment explaining why each one is needed.
2. Add a `ci/scripts/rust_machete.sh` that runs `cargo machete`, wire it
into `dev/rust_lint.sh` alongside the other lint steps, and add a job or step
in CI so regressions are caught on PRs.
**Describe alternatives you've considered**
- `cargo udeps`, which is more accurate because it uses real build output,
but it requires a nightly toolchain and is considerably slower. `cargo machete`
is a source scan, so it is fast enough to sit next to clippy, at the cost of
needing an ignore list for the false positives.
- Doing nothing and cleaning up dependencies opportunistically. That is what
we do today, and it is how the list above accumulated.
**Additional context**
Both DataFusion and several other Arrow subprojects already run `cargo
machete` in CI, so this would bring Ballista in line with them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
