Dev-iL opened a new pull request, #74556: URL: https://github.com/apache/airflow/pull/74556
related: #74555 `uv sync --all-packages` installs every workspace member, including providers whose `requires-python` excludes the image's Python minor. Their third-party dependencies can then fail to build on newer interpreters, blocking the image even though those providers are unsupported there. This PR adds a utility to detect excluded workspace members and resolve only dependencies needed by the remaining members. This is especially useful when adding support for a new python version (e.g. 3.15) close to its release, when many dependencies haven't caught up yet. --- <details><summary>Before & after comparison</summary> <p> This compares the packages that the CI image installs before and after this PR. Before the change, the installer runs `uv sync --all-packages`, which installs every workspace member and the third-party packages those members reach, including members that are excluded for the image's Python minor. After the change, `scripts/docker/filter_ci_install_set.py` installs only the members that are not excluded, plus the third-party pins that the remaining members reach from the lock. The numbers below come from resolution and dry runs. An amd64 image build from an earlier attempt (`ci-image-315-amd64-r9`, exit 0) printed the same 3.15 counts, and `uv pip check` reported 562 packages compatible. That build ran the same helper file, but it used a locally built PyYAML wheel passed through `docker-context-files`, which is not part of this PR, and the working-tree installer script. The amended helper has not been image-built on 3.14. On merge, the only CI image whose install path changes is 3.14. Cassandra is the only provider excluded for an interpreter that upstream CI builds (3.11 to 3.14), so every 3.14 image moves from `uv sync --frozen` to the filtered install. CI is the first run of that path. ## Example: Kafka provider and confluent-kafka skipped on Python 3.15 `apache-airflow-providers-apache-kafka` lists `"3.15"` in `excluded-python-versions`. Its dependency `confluent-kafka` is reached by no other member, so the helper drops both. | Entry | Before (3.15 dry run) | After (3.15 helper) | | --- | --- | --- | | `apache-airflow-providers-apache-kafka` | `+ apache-airflow-providers-apache-kafka @ file:///…/providers/apache/kafka` | not in the editable list | | `confluent-kafka` | `+ confluent-kafka==2.15.1` | not in the requirements | The Kafka member requires `confluent-kafka>=2.13.2; python_version >= '3.14'` (`providers/apache/kafka/pyproject.toml`), so the unfiltered install reaches it on 3.15. `confluent-kafka` 2.15.1 has no 3.15 wheel for amd64 or arm64, so it compiles from source. Earlier 3.15 image attempts failed with `error: Failed to build confluent-kafka==2.15.1` (missing `librdkafka/rdkafka.h`). The `excluded-python-versions` entry for Kafka belongs to the 3.15 provider change, not to this PR. On upstream main no provider lists 3.15, so this PR removes nothing for 3.15 until that change lands. On 3.14 the same provider and dependency are kept in both runs. ## Measured counts | Python | Item | Before | After | | --- | --- | ---: | ---: | | 3.15 | Workspace members | 141 | 99 | | 3.15 | Third-party packages | 764 | 463 | | 3.15 | Total | 905 | 562 | | 3.14 | Workspace members | 141 | 140 | | 3.14 | Third-party packages | 782 | 782 | | 3.14 | Total | 923 | 922 | - On 3.15, 42 workspace members and 301 third-party packages are dropped. No package is added, and no pinned version changes. - On 3.14, only the Cassandra provider is dropped. No third-party package changes. ### Why 3.14 has no dependency example The Cassandra provider declares `cassandra-driver` only for Python below 3.14 (`providers/apache/cassandra/pyproject.toml`), and the lock pins it with `python_full_version < '3.14'`. Neither 3.14 install contains it, so on 3.14 the change drops only the Cassandra member. ## Dropped on Python 3.15 <details> <summary>42 workspace members excluded for 3.15</summary> - `providers/alibaba` - `providers/apache/arrow` - `providers/apache/beam` - `providers/apache/cassandra` - `providers/apache/drill` - `providers/apache/hdfs` - `providers/apache/hive` - `providers/apache/iceberg` - `providers/apache/impala` - `providers/apache/kafka` - `providers/apache/pinot` - `providers/apache/spark` - `providers/apache/tinkerpop` - `providers/clickhousedb` - `providers/cohere` - `providers/common/ai` - `providers/databricks` - `providers/duckdb` - `providers/exasol` - `providers/google` - `providers/ibm/db2` - `providers/influxdb` - `providers/jdbc` - `providers/microsoft/azure` - `providers/microsoft/psrp` - `providers/microsoft/winrm` - `providers/modal` - `providers/mongo` - `providers/mysql` - `providers/odbc` - `providers/papermill` - `providers/pgvector` - `providers/postgres` - `providers/presto` - `providers/qdrant` - `providers/salesforce` - `providers/samba` - `providers/snowflake` - `providers/trino` - `providers/vespa` - `providers/weaviate` - `providers/yandex` </details> <details> <summary>301 third-party packages dropped for 3.15</summary> ```text adal adbc-driver-manager adlfs aenum aiofile aiomysql alibabacloud-adb20211201 alibabacloud-credentials alibabacloud-credentials-api alibabacloud-gateway-spi alibabacloud-oss-v2 alibabacloud-tea alibabacloud-tea-openapi alibabacloud-tea-util ansicolors apache-beam apscheduler arro3-core asyncpg async-timeout azure-ai-projects azure-batch azure-common azure-core azure-cosmos azure-datalake-store azure-identity azure-keyvault-secrets azure-kusto-data azure-mgmt-compute azure-mgmt-containerinstance azure-mgmt-containerregistry azure-mgmt-core azure-mgmt-cosmosdb azure-mgmt-datafactory azure-mgmt-datalake-nspkg azure-mgmt-datalake-store azure-mgmt-nspkg azure-mgmt-resource azure-mgmt-storage azure-nspkg azure-servicebus azure-storage-blob azure-storage-file-datalake azure-storage-file-share azure-synapse-artifacts azure-synapse-spark banks beartype bitarray bleach caio cattrs cbor2 chardet ciso8601 clickhouse-connect cohere comm confluent-kafka crcmod-plus darabonba-core databricks-sqlalchemy databricks-sql-connector dataclasses-json db-dtypes debugpy deltalake dirtyjson docopt duckdb entrypoints et-xmlfile exceptiongroup fastavro fastcore fasteners fastjsonschema fastmcp-slim fastuuid filetype future gcloud-aio-auth gcloud-aio-bigquery gcloud-aio-storage gcsfs genai-prices google-ads google-analytics-admin google-cloud-aiplatform google-cloud-alloydb google-cloud-appengine-logging google-cloud-audit-log google-cloud-automl google-cloud-batch google-cloud-bigquery google-cloud-bigquery-datatransfer google-cloud-bigquery-storage google-cloud-bigtable google-cloud-build google-cloud-compute google-cloud-container google-cloud-core google-cloud-datacatalog google-cloud-dataflow-client google-cloud-dataform google-cloud-dataplex google-cloud-dataproc google-cloud-dataproc-metastore google-cloud-dlp google-cloud-kms google-cloud-language google-cloud-logging google-cloud-managedkafka google-cloud-memcache google-cloud-monitoring google-cloud-orchestration-airflow google-cloud-os-login google-cloud-pubsub google-cloud-redis google-cloud-resource-manager google-cloud-run google-cloud-secret-manager google-cloud-spanner google-cloud-speech google-cloud-storage google-cloud-storage-control google-cloud-storage-transfer google-cloud-tasks google-cloud-texttospeech google-cloud-translate google-cloud-videointelligence google-cloud-vision google-cloud-workflows google-crc32c google-genai google-resumable-media gremlinpython griffe griffecli griffelib grpc-google-iam-v1 grpc-interceptor grpcio-gcp grpcio-health-checking grpcio-status grpcio-tools grpclib gssapi hdfs hf-xet hmsclient httpr httpx-sse huggingface-hub ibm-db ibm-db-sa ijson immutabledict impyla influxdb3-python influxdb-client ipykernel isodate jaydebeapi joblib jpype1 jsonpatch jsonpickle jsonpointer jupyter-client jupyter-core jupyterlab-pygments kerberos krb5 langchain langchain-core langchain-protocol langgraph langgraph-checkpoint langgraph-prebuilt langgraph-sdk langsmith litellm llama-index-core llama-index-embeddings-openai llama-index-instrumentation llama-index-llms-openai llama-index-workflows logfire-api looker-sdk lz4 mcp mcp-types mergedeep microsoft-kiota-abstractions microsoft-kiota-authentication-azure microsoft-kiota-http microsoft-kiota-serialization-json microsoft-kiota-serialization-text mistune mmh3 modal msal msal-extensions msgraph-core msgraphfs msrest msrestazure mysqlclient mysql-connector-python narwhals nbclient nbconvert nbformat nest-asyncio nest-asyncio2 networkx nltk objsize openpyxl opensandbox opentelemetry-resourcedetector-gcp ormsgpack pandas-gbq pandocfilters papermill pgvector pillow pinotdb portalocker presto-python-client psycopg psycopg2-binary psycopg-binary pure-sasl py4j pybreaker pydantic-ai-skills pydantic-ai-slim pydantic-graph pydata-google-auth pyexasol pyhive pyiceberg py-key-value-aio pymongo pymysql pyodbc pyodps pyopenssl pypsrp pyroaring pyspark pyspark-client pyspnego pyvespa pywinrm pyzmq qdrant-client reactivex regex requests-file requests-kerberos requests-ntlm rsa ruamel-yaml scikit-learn scipy scrapbook simple-salesforce smbprotocol snowflake-connector-python snowflake-sqlalchemy sortedcontainers sqlalchemy-bigquery sqlalchemy-drill sqlalchemy-spanner sse-starlette std-uritemplate strictyaml synchronicity threadpoolctl thrift thrift-sasl tiktoken tinycss2 tinytag tokenizers toml trino typing-inspect uuid-utils validators weaviate-client webencodings xxhash yandexcloud yandex-query-client zeep zstandard ``` </details> ## Method - Pins: `uv export --frozen --all-packages --group ci-image --no-hashes --no-emit-workspace --output-file <pins>`. - Before: `uv sync --frozen --all-packages --group ci-image --python <minor> --no-python-downloads --dry-run`. Members are the entries with a `file://` location. Third-party packages are the other entries, compared by name and by exact `==` version. - After: `uv run --project scripts python scripts/docker/filter_ci_install_set.py --root <tree> --constraints <pins> --python-minor <minor> --platform x86_64 --workdir <absolute directory> --output <requirements> --editables <editables>`. - Trees: 3.14 uses upstream/main (`7749ecdb97`) and its lock, with the helper file from `17500e9b03`. The PR head, `b0a9abd867`, adds absolute-path resolution on top of that commit and does not change these counts (rechecked for 3.13). 3.15 uses the working tree that carries the 3.15 changes, where the root, `airflow-core`, `task-sdk` and `dev/mypy` declare `requires-python = ">=3.11,!=3.16"` and the provider exclusions, Kafka included. Both 3.15 runs use the same lock. ## Limits - x86_64 only. arm64 was not measured. - Resolution and dry runs only. `uv pip check` on the filtered set was not run. - The upgrade path (`UPGRADE_RANDOM_INDICATOR_STRING`) still runs `uv sync --resolution highest` and installs every member. - On upstream main alone, the helper cannot filter 3.15. The root, `airflow-core`, `task-sdk` and `dev/mypy` declare `!=3.15`, so the helper excludes them, but kept providers still depend on `airflow-core`. The helper then stops with `Resolver returned members that were not kept` and writes no output. The installer calls the helper whenever some member's `requires-python` or some provider's `excluded-python-versions` excludes the minor. On upstream main, 3.15 is not in `ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS` (3.11 to 3.14), so CI never reaches that path there. </p> </details> --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [x] Yes (please specify the tool below) Generated-by: Claude following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
