kaxil commented on code in PR #74557:
URL: https://github.com/apache/airflow/pull/74557#discussion_r4304698249
##########
pyproject.toml:
##########
@@ -1478,13 +1479,28 @@ no-build-isolation-package = ["sphinx-redoc"]
# move. Published wheels are unaffected: overrides only shape uv.lock and the
CI image. Drop this
# once run-llama/llama_index#22939 and BerriAI/litellm#40317 land and
google-cloud-aiplatform
# lifts its litellm cap (<1.86 below Python 3.14) far enough to reach the
fixed litellm.
-override-dependencies = ["openai>=2.47.0,<3"]
+override-dependencies = [
+ "openai>=2.47.0,<3",
+]
# The override replaces every package's openai requirement, not only
pydantic-ai-slim's, so the
# resolver no longer rejects a release that needs openai>=3 unconditionally,
and only `pip check`
# in the image build catches it. Cap each such package here at its last openai
2.x release, which
# is what the resolver would pick without the override. azure-ai-projects
2.5.0 moved to
# openai>=3.0.0; the Azure provider needs >=2.2.0.
-constraint-dependencies = ["azure-ai-projects<2.5"]
+constraint-dependencies = [
+ "azure-ai-projects<2.5",
+ # apache-beam 2.76.0 and 2.77.0 cap pyarrow below 26 and numpy below 2.5.
Beam is excluded on 3.15, but the universal
+ # lock still resolves one version of each for every interpreter. These
marker-scoped constraints fork the lock so 3.15
+ # gets the first releases with wheels (numpy 2.5.2, pyarrow 26.0.0). They
are constraints rather than overrides because
+ # an override is a direct requirement for lowest-direct, which then
installs its floor on every interpreter.
+ "numpy>=2.5.2; python_version == '3.15'",
+ "pyarrow>=26.0.0; python_version == '3.15'",
+ # Keep the other interpreters on the releases they locked before 3.15
support: pydantic 2.14 needs a typing fix in
Review Comment:
This comment says the other interpreters stay on what they locked before
3.15, but `uv.lock` moves them too. Against the base of this stack (and main),
3.11-3.14 now get sqlalchemy 2.1.4 (was 2.0.54), pendulum 3.3.0, apprise 2.0.1,
polars 2.0.0, huggingface-hub 2.2.0, pyathena 4.0.0 and anthropic 1.13.0.
sqlalchemy and pendulum have one lock entry each, so the 3.15-only floors
(`>=2.1.4`, `>=3.3.0`) lift every interpreter, which is what the numpy/pyarrow
constraints above avoid. For the other five I couldn't find a requirement
change that forces the move, so it looks like an upgrade relock. Could you
relock without upgrading and add `< '3.15'` constraints for sqlalchemy and
pendulum? If SQLAlchemy 2.1 everywhere is intended, I'd do that in its own PR,
otherwise a 3.14 regression from it will read as 3.15 fallout.
##########
providers/pinecone/pyproject.toml:
##########
@@ -59,7 +60,7 @@ requires-python = ">=3.11"
# After you modify the dependencies, and rebuild your Breeze CI image with
``breeze ci-image build``
dependencies = [
"apache-airflow>=2.11.0",
- "apache-airflow-providers-common-compat>=1.8.0",
+ "apache-airflow-providers-common-compat>=1.8.0", # use next version
"pinecone>=7.0.0",
Review Comment:
The `pinecone<10` cap for 3.15 only lives in the root
`constraint-dependencies`, which don't ship with the provider. On PyPI pinecone
10.0.0 requires `msgspec<0.22,>=0.19`, and msgspec 0.21.1 has no cp315 wheels,
so installing `apache-airflow-providers-pinecone` with pip on 3.15 picks
pinecone 10 and has to build msgspec from source. That's the failure the other
3.15 exclusions avoid. Should the cap be declared here (`"pinecone>=7.0.0,<10;
python_version >= \"3.15\""`), or pinecone excluded on 3.15?
##########
dev/breeze/src/airflow_breeze/utils/packages.py:
##########
@@ -991,12 +1017,15 @@ def regenerate_pyproject_toml(
f"[info]Provider {provider_id} in suspended list, skipping
adding to optional dependencies.\n"
)
continue
- cross_provider_dependencies.append(f'
"{get_pip_package_name(provider_id)}",')
+ requirement = get_pip_package_name(provider_id) + get_exclusion_marker(
+ all_provider_dependencies.get(provider_id, {})
+ )
+ cross_provider_dependencies.append(f' "{requirement}",')
if f'"{provider_id}" = [' not in optional_dependencies and
get_pip_package_name(
Review Comment:
The marker only reaches extras this loop creates. Extras that already exist
keep their unmarked requirement, so postgres's `"microsoft.azure"` and mysql's
`"presto"` / `"trino"` still point at providers that now declare `!=3.15.*`. On
3.15 pip would backtrack to an older presto/trino/azure release that still
allows 3.15 rather than skip the extra. Amazon's extras were fixed by hand;
could the generator add the marker to existing extras too, so this doesn't rely
on catching each one?
##########
airflow-core/pyproject.toml:
##########
@@ -182,8 +195,11 @@ dependencies = [
"async" = [
"eventlet>=0.37.0",
"gevent>=25.4.1",
+ "gevent>=26.5.0; python_version >= \"3.15\"",
"greenlet>=3.1.0; python_version < '3.14'",
"greenlet>=3.3.2; python_version >= '3.14'",
+ # 3.5.1-3.5.3 raise "greenlet context must be a contextvars.Context or
None" with SQLAlchemy async engines on 3.15.
+ "greenlet>=3.5.4; python_version >= '3.15'",
Review Comment:
The comment ties this floor to SQLAlchemy async engines, but `settings.py`
creates one on every install from the base `sqlalchemy[asyncio]` dependency,
while this sits in the `async` extra (the gevent/eventlet one). Without that
extra a 3.15 install can still land on greenlet 3.5.1-3.5.3. Should it move
next to `sqlalchemy[asyncio]` in the base dependencies?
##########
providers/common/sql/tests/unit/common/sql/datafusion/test_format_handlers.py:
##########
@@ -20,14 +20,19 @@
from unittest.mock import MagicMock, patch
import pytest
-from pyiceberg.table import Table
-from airflow.providers.common.sql.config import DataSourceConfig, FormatType
-from airflow.providers.common.sql.datafusion.exceptions import (
+from tests_common.test_utils.markers import skip_module_on_python_315_without
+
+skip_module_on_python_315_without("pyiceberg")
Review Comment:
Same here: `Table` is only used as a spec in the two iceberg tests, but the
module skip also drops the CSV, Parquet and Avro handler tests and the
validation tests on 3.15. Could the `importorskip` move into those two tests?
##########
providers/vespa/README.rst:
##########
@@ -45,18 +45,19 @@ You can install this package on top of an existing Airflow
installation (see ``R
for the minimum Airflow version supported) via
``pip install apache-airflow-providers-vespa``
-The package supports the following python versions: 3.11,3.12,3.13
+The package supports the following python versions: 3.11,3.12,3.13,3.14
Review Comment:
Vespa now has the 3.15 classifier, but this still lists 3.11-3.14. The
postgres, mysql and pgvector READMEs have the same stale line, so it looks like
the README regeneration didn't pick up every provider.
##########
providers/influxdb/provider.yaml:
##########
@@ -68,6 +68,9 @@ versions:
- 1.1.0
- 1.0.0
+excluded-python-versions:
Review Comment:
What blocks influxdb on 3.15? `influxdb-client` and `influxdb3-python` are
pure Python, and pyarrow and pandas already resolve for 3.15 for other
providers. Same question for microsoft.winrm and microsoft.psrp. If they
install fine, excluding them just costs 3.15 users three providers.
##########
providers/amazon/pyproject.toml:
##########
@@ -59,7 +60,7 @@ requires-python = ">=3.11"
# After you modify the dependencies, and rebuild your Breeze CI image with
``breeze ci-image build``
dependencies = [
"apache-airflow>=2.11.0",
- "apache-airflow-providers-common-compat>=1.22.0",
+ "apache-airflow-providers-common-compat>=1.22.0", # use next version
Review Comment:
This `# use next version` (and the same comment on 100 other providers) is
only here to satisfy the common.compat selective check, which fires because
common.compat's pyproject gained the 3.15 classifier. At release,
`update_providers_with_next_version_comment` raises all 101 floors to a
common.compat release whose only change is that classifier, so upgrading any of
these providers would force a common.compat upgrade for nothing. Could you drop
them and use the `skip common compat check` label instead?
##########
pyproject.toml:
##########
@@ -1478,13 +1479,28 @@ no-build-isolation-package = ["sphinx-redoc"]
# move. Published wheels are unaffected: overrides only shape uv.lock and the
CI image. Drop this
# once run-llama/llama_index#22939 and BerriAI/litellm#40317 land and
google-cloud-aiplatform
# lifts its litellm cap (<1.86 below Python 3.14) far enough to reach the
fixed litellm.
-override-dependencies = ["openai>=2.47.0,<3"]
+override-dependencies = [
+ "openai>=2.47.0,<3",
+]
# The override replaces every package's openai requirement, not only
pydantic-ai-slim's, so the
# resolver no longer rejects a release that needs openai>=3 unconditionally,
and only `pip check`
# in the image build catches it. Cap each such package here at its last openai
2.x release, which
# is what the resolver would pick without the override. azure-ai-projects
2.5.0 moved to
# openai>=3.0.0; the Azure provider needs >=2.2.0.
-constraint-dependencies = ["azure-ai-projects<2.5"]
+constraint-dependencies = [
+ "azure-ai-projects<2.5",
+ # apache-beam 2.76.0 and 2.77.0 cap pyarrow below 26 and numpy below 2.5.
Beam is excluded on 3.15, but the universal
+ # lock still resolves one version of each for every interpreter. These
marker-scoped constraints fork the lock so 3.15
+ # gets the first releases with wheels (numpy 2.5.2, pyarrow 26.0.0). They
are constraints rather than overrides because
+ # an override is a direct requirement for lowest-direct, which then
installs its floor on every interpreter.
+ "numpy>=2.5.2; python_version == '3.15'",
+ "pyarrow>=26.0.0; python_version == '3.15'",
+ # Keep the other interpreters on the releases they locked before 3.15
support: pydantic 2.14 needs a typing fix in
+ # lang_sdk_processor.py, and pinecone 10 caps msgspec<0.22 while 3.15
needs msgspec>=0.22 for wheels.
+ "pydantic<2.14; python_version < '3.15'",
Review Comment:
The pydantic 2.14 typing fix in `lang_sdk_processor.py` is already on main
(#74628), so is this constraint still needed? As it is, 3.11-3.14 CI tests
pydantic 2.13.5 while a plain pip install picks up 2.14.x.
##########
providers/common/sql/tests/unit/common/sql/hooks/test_dbapi.py:
##########
@@ -22,13 +22,18 @@
from unittest import mock
import pytest
-from pyodbc import Cursor
-from airflow.models import Connection
-from airflow.providers.common.compat.sdk import BaseHook
-from airflow.providers.common.sql.dialects.dialect import Dialect
-from airflow.providers.common.sql.hooks.handlers import fetch_all_handler,
fetch_one_handler
-from airflow.providers.common.sql.hooks.sql import DbApiHook
+from tests_common.test_utils.markers import skip_module_on_python_315_without
+
+skip_module_on_python_315_without("pyodbc")
Review Comment:
This skips the whole of `test_dbapi.py` on 3.15 for one `spec=Cursor` (line
52), and `DbApiHook` is the base of every SQL hook. A small local stand-in
class with the attributes the hook touches (`fast_executemany`, `execute`,
`executemany`, `fetchall`, ...) would keep these tests running on 3.15 without
pyodbc.
##########
airflow-core/tests/unit/cli/commands/test_connection_command.py:
##########
@@ -643,6 +643,9 @@ def setup_method(self):
"extra": None,
},
id="individual-parts",
+ marks=xfail_on_python_315(
Review Comment:
These two cases test the CLI's argument parsing, not hive or google.
Switching their conn types to ones whose providers are installed on 3.15
(`postgres`, say) would keep them running there instead of xfailing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
