YAshhh29 opened a new pull request, #69408:
URL: https://github.com/apache/airflow/pull/69408
The `apache-airflow-providers-openai` package pulls in the `[datalib]`
extra of the `openai` SDK — which transitively installs `numpy`, `pandas`,
and `pandas-stubs` — for every user of this provider. That extra has no
functional use inside this provider today, so it is pure bloat and CVE
noise for anyone installing the provider.
### Investigation
I audited every code path that could depend on `numpy`/`pandas` reaching
users through this provider:
| Check | Location | Result |
|---|---|---|
| Provider source | `providers/openai/src/**/*.py` | **0** references to
numpy/pandas/DataFrame |
| Provider unit tests | `providers/openai/tests/unit/**` | **0** references |
| Provider example DAGs / system tests |
`providers/openai/tests/system/openai/example_*.py` | **0** references — only
`pendulum` + Airflow imports |
| Provider docs | `providers/openai/docs/**` | Only auto-generated dep
tables |
| Operator return types | `operators/openai.py` | `list[float]`, `str`, `str
\| None` — no DataFrames |
| Cross-provider consumers (RAG examples) |
`providers/{pgvector,pinecone,weaviate}/tests/system/**/example_*_openai.py` |
pgvector & pinecone: **0** references. weaviate has its **own** pandas dep
declared in `providers/weaviate/pyproject.toml` |
| Historical usage | `git log providers/openai/pyproject.toml` | `[datalib]`
was carried forward across every major version bump (openai 0.x → 1.x → 2.x);
recent PR #69068 just bumped `openai[datalib]>=1.66.0` →
`openai[datalib]>=2.37.0` without justifying the extra |
| Upstream SDK docs | `openai/openai-python` README (SDK 2.x) | `[datalib]`
is not mentioned anywhere in the modern SDK docs (`[aiohttp]`, `[bedrock]`,
`[realtime]` are — `[datalib]` is not) |
### Why it was there in the first place
`[datalib]` was the extra used by OpenAI SDK 0.x's legacy CLI
(`openai tools fine_tunes.prepare_data`), which processed JSONL
fine-tuning files via pandas. That CLI was removed when OpenAI SDK 1.x
shipped in 2023, but the extra remained in the package for backward
compatibility. This Airflow provider carried the extra forward across
every subsequent version bump without re-evaluating whether it was
still needed — a classic transitive fossil dependency.
### What changes
- `providers/openai/pyproject.toml`: base dep is now `openai>=2.37.0` (no
`[datalib]`); inline comment records the rationale so it doesn't get re-added
by the next version bump.
- `providers/openai/README.rst` and `providers/openai/docs/index.rst`: dep
table updated to reflect the base dep.
- `providers/openai/docs/changelog.rst`: behavior note under the `Changelog`
header explaining the change and pointing users at `pip install
'openai[datalib]'` if they had unknowingly relied on the transitive
`numpy`/`pandas` install for their DAG code.
### Impact for users
- **Everyone** who installs the OpenAI provider stops pulling in `numpy`,
`pandas`, and `pandas-stubs` transitively — smaller Docker images, smaller
`pip-audit` surfaces.
- Users of `OpenAIHook`, `OpenAIEmbeddingOperator`,
`OpenAIResponseOperator`, and the batch operators are unaffected — none of
their code paths use pandas/numpy.
- The one edge case is a user whose *DAG code* (not the provider) does
`import pandas` and was relying on this provider to bring pandas along
transitively. The changelog note points those users at the exact one-line fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
