benispeti commented on issue #43002: URL: https://github.com/apache/superset/issues/43002#issuecomment-5240639098
> Hi [@benispeti](https://github.com/benispeti)! I'm [Dosu](https://go.dosu.dev/dosubot) and I'm helping the Apache Superset team. > > Good catch. The Linux `-dev` packages you identified are indeed not strictly required for runtime in the production (`lean`) image. Here's what's happening: > > **Root cause:** The `python-common` stage (shared by all final images) explicitly installs `libsasl2-dev`, `libpq-dev`, `libecpg-dev`, and `libldap2-dev` [[1]](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/Dockerfile#L191-L197). These `-dev` packages pull in the other development packages you found (`libc6-dev`, `libpython3.11-dev`, `python3.11-dev`, `zlib1g-dev`, `linux-libc-dev`, etc.) as transitive APT dependencies. They are never cleaned up. > > **Why `pip-install.sh` doesn't help:** The `--requires-build-essential` flag in [`pip-install.sh`](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/docker/pip-install.sh#L56-L62) correctly installs and removes `build-essential` around pip installs, but it doesn't touch these other `-dev` packages since they were explicitly installed in a prior stage. > > **Are they needed at runtime?** No — none of the packages in `requirements/base.txt` (the lean image's dependency set) actually link against these libraries at runtime. `psycopg2-binary` (PostgreSQL) is only in the optional `postgres` extra, `python-ldap` is in the `development` extra, and `thrift_sasl` is in optional `hive`/`spark` extras. These `-dev` packages could be replaced with their runtime-only equivalents (e.g., `libpq5` instead of `libpq-dev`, `libsasl2-2` instead of `libsasl2-dev`) or removed entirely from the base image and added only where needed. > > **Frontend packages (`devlop`, `preact-devtools`):** The `devlop` npm package is a transitive dependency in `package-lock.json` but should not end up in the compiled production bundle — the frontend build runs with `NODE_ENV=production` [[2]](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/Dockerfile#L41) and webpack tree-shaking should exclude dev-only code from the final assets in `/app/superset/static/assets`. `preact-devtools` does not appear in the codebase source, so it may be a transitive dev dependency similarly excluded from the production bundle. These would show up in an SBOM scan of the full image filesystem (since `node_modules` from the build stage might be copied), but shouldn't be in the actual served JavaScript. > > In summary, this looks like an optimization opportunity — the `-dev` packages in `python-common` appear to be there for convenience (supporting optional database driver compilation) rather than by strict necessity for the lean image. > > _To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fsuperset)._ > > Docs are dead. Just use [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-tagline&utm_term=apache%2Fsuperset). > > [](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-feedback&utm_term=apache%2Fsuperset&message_id=730abf11-4515-40e9-8f98-e9ad049dcde9) [](https://github.dosu.com/apache/superset?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-ask-repo&utm_term=apache%2Fsuperset) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=static-docs-share-team&utm_term=apache%2Fsuperset) **The Problem with linux dev packages: Build Tools Leaking into the Final Image** The lean stage calls pip-install.sh --requires-build-essential, which installs build-essential (GCC, G++, make, binutils, etc.) into that layer to compile native Python extensions from source. Once compilation finishes, those tools have no runtime purpose — but because the install happens inside the lean stage itself, they are baked permanently into the final image. The multi-stage build already has a python-base and python-common lineage, but no dedicated "build" stage for Python dependencies. The correct pattern would be: * Build stage — inherits python-common, installs build-essential, runs pip-install.sh, compiles all native wheels, populates /app/.venv * Lean stage — starts fresh from python-common (no build tools), and COPY --from=build /app/.venv /app/.venv -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
