This is an automated email from the ASF dual-hosted git repository.
eladkal pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new b21e4299268 Migrate Airbyte provider to `airbyte-api` library 1.0 and
`httpx` (#68882)
b21e4299268 is described below
commit b21e42992682fec16d072ff25010128046d58d9c
Author: Aaron ("AJ") Steers <[email protected]>
AuthorDate: Wed Jul 8 06:05:54 2026 -0700
Migrate Airbyte provider to `airbyte-api` library 1.0 and `httpx` (#68882)
* fix(providers/airbyte): migrate from requests.Session to httpx.Client for
airbyte-api>=1.0
airbyte-api 1.0.0rc2 switched its HTTP layer from requests to httpx.
The client parameter now requires an httpx-compatible HttpClient protocol
and no longer accepts requests.Session.
Changes:
- Replace requests.Session proxy setup with httpx.Client(mounts=...)
- Update pyproject.toml: airbyte-api>=1.0.0rc2, httpx>=0.28.1, drop requests
- Update test to assert isinstance(client, httpx.Client) instead of
checking .proxies attribute (not available on httpx.Client)
Co-Authored-By: AJ Steers <[email protected]>
* fix: move httpx import to top-level in test file
Co-Authored-By: AJ Steers <[email protected]>
* fix: strengthen proxy test and bump airbyte-api lower bound to >=1.0.0
- Test now verifies proxy transport mounts are configured per-scheme
- Bump airbyte-api lower bound from >=1.0.0rc2 to >=1.0.0 (GA released)
Co-Authored-By: AJ Steers <[email protected]>
* Fix test compatibility with airbyte-api 1.0 Pydantic models
Co-Authored-By: AJ Steers <[email protected]>
* Regenerate provider docs index and add 1.0 migration changelog note
Co-Authored-By: AJ Steers <[email protected]>
* Regenerate airbyte provider docs index after merging main
Co-Authored-By: AJ Steers <[email protected]>
* Bump airbyte provider to 6.0.0 for breaking airbyte-api 1.0 / httpx
migration
Co-Authored-By: AJ Steers <[email protected]>
* Revert manual airbyte version bump and reframe changelog note as
non-breaking
Per reviewer feedback, this is not an Airflow-code breaking change, so the
version stays release-manager-managed (5.5.2) rather than a hardcoded 6.0.0
bump. The manual 6.0.0 entry in provider.yaml also broke the
prepare-provider-documentation CI step, which diffs against the previous
version tag (providers-airbyte/5.5.2) that is not yet released.
Retitle the changelog section from 'Breaking changes' to 'Transitive
dependency changes' and drop the 6.0.0 header.
Co-Authored-By: AJ Steers <[email protected]>
---------
Co-authored-by: Devin AI
<158243242+devin-ai-integration[bot]@users.noreply.github.com>
---
providers/airbyte/README.rst | 10 ++---
providers/airbyte/docs/changelog.rst | 20 ++++++++++
providers/airbyte/docs/connections.rst | 5 +--
providers/airbyte/docs/index.rst | 10 ++---
providers/airbyte/pyproject.toml | 7 +---
.../src/airflow/providers/airbyte/hooks/airbyte.py | 17 +++++++--
.../tests/unit/airbyte/hooks/test_airbyte.py | 44 +++++++++++++++-------
uv.lock | 38 +++++--------------
8 files changed, 87 insertions(+), 64 deletions(-)
diff --git a/providers/airbyte/README.rst b/providers/airbyte/README.rst
index c677b6bb862..86980548567 100644
--- a/providers/airbyte/README.rst
+++ b/providers/airbyte/README.rst
@@ -50,14 +50,14 @@ The package supports the following python versions:
3.10,3.11,3.12,3.13,3.14
Requirements
------------
-========================================== ===================
+========================================== ==================
PIP package Version required
-========================================== ===================
+========================================== ==================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
-``airbyte-api`` ``>=0.52.0,<1.0.0``
-``requests`` ``>=2.32.0``
-========================================== ===================
+``airbyte-api`` ``>=1.0.0,<2.0``
+``httpx`` ``>=0.28.1``
+========================================== ==================
The changelog for the provider package can be found in the
`changelog
<https://airflow.apache.org/docs/apache-airflow-providers-airbyte/5.5.2/changelog.html>`_.
diff --git a/providers/airbyte/docs/changelog.rst
b/providers/airbyte/docs/changelog.rst
index ead7d2efb28..84701e9ced3 100644
--- a/providers/airbyte/docs/changelog.rst
+++ b/providers/airbyte/docs/changelog.rst
@@ -27,6 +27,26 @@ Changelog
---------
+Transitive dependency changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note::
+ This release upgrades the provider's ``airbyte-api`` dependency to the
``1.x`` series and switches
+ the underlying HTTP client from ``requests`` to ``httpx``.
+
+ No changes are required for typical Dag authors: the ``AirbyteHook``,
``AirbyteTriggerSyncOperator``,
+ and ``AirbyteJobSensor`` public interfaces are unchanged, and the Airbyte
connection configuration
+ (including the ``proxies`` extra) keeps the same format.
+
+ Action is required only if your environment relies on the provider's
transitive dependencies:
+
+ * ``airbyte-api`` is now ``>=1.0.0,<2.0`` (previously ``>=0.52.0,<1.0.0``).
The ``1.x`` SDK is built
+ on Pydantic models, so any code importing ``airbyte_api`` directly must
pass request objects as
+ keyword arguments (e.g. ``GetJobRequest(job_id=...)``) and handle its
stricter response validation.
+ * ``requests`` is no longer installed by this provider. If your code relied
on it being pulled in
+ transitively, declare ``requests`` as an explicit dependency of your own
project.
+
+
5.5.2
.....
diff --git a/providers/airbyte/docs/connections.rst
b/providers/airbyte/docs/connections.rst
index 5d41d2096ea..63733b08bbc 100644
--- a/providers/airbyte/docs/connections.rst
+++ b/providers/airbyte/docs/connections.rst
@@ -57,9 +57,8 @@ Client Secret (optional)
Leave blank for Airbyte OSS deployments without auth enabled.
Extra (optional)
- Specify custom proxies in JSON format.
- Following default requests parameters are taken into account:
+ Specify the ``proxies`` key in JSON format to route traffic through an
HTTP proxy.
* ``proxies``
- Example: ``{"http": "http://proxy.example.com:8080";, "https":
"http://proxy.example.com:8080"}``
+ Example: ``{"proxies": {"http": "http://proxy.example.com:8080";, "https":
"http://proxy.example.com:8080"}}``
diff --git a/providers/airbyte/docs/index.rst b/providers/airbyte/docs/index.rst
index 3c54af9f6e2..fd9c3b41aa6 100644
--- a/providers/airbyte/docs/index.rst
+++ b/providers/airbyte/docs/index.rst
@@ -96,14 +96,14 @@ Requirements
The minimum Apache Airflow version supported by this provider distribution is
``2.11.0``.
-========================================== ===================
+========================================== ==================
PIP package Version required
-========================================== ===================
+========================================== ==================
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.12.0``
-``airbyte-api`` ``>=0.52.0,<1.0.0``
-``requests`` ``>=2.32.0``
-========================================== ===================
+``airbyte-api`` ``>=1.0.0,<2.0``
+``httpx`` ``>=0.28.1``
+========================================== ==================
Downloading official packages
-----------------------------
diff --git a/providers/airbyte/pyproject.toml b/providers/airbyte/pyproject.toml
index 76e1a6ed330..880e72d2c60 100644
--- a/providers/airbyte/pyproject.toml
+++ b/providers/airbyte/pyproject.toml
@@ -61,11 +61,8 @@ requires-python = ">=3.10"
dependencies = [
"apache-airflow>=2.11.0",
"apache-airflow-providers-common-compat>=1.12.0",
- # airbyte-api 1.0.0 migrated from requests to httpx; passing a
requests.Session
- # as the client now fails its HttpClient protocol check. Cap until the
httpx
- # migration lands; tracked at https://github.com/apache/airflow/pull/68882
- "airbyte-api>=0.52.0,<1.0.0",
- "requests>=2.32.0",
+ "airbyte-api>=1.0.0,<2.0",
+ "httpx>=0.28.1",
]
[dependency-groups]
diff --git a/providers/airbyte/src/airflow/providers/airbyte/hooks/airbyte.py
b/providers/airbyte/src/airflow/providers/airbyte/hooks/airbyte.py
index a30012d8394..dd7cc045626 100644
--- a/providers/airbyte/src/airflow/providers/airbyte/hooks/airbyte.py
+++ b/providers/airbyte/src/airflow/providers/airbyte/hooks/airbyte.py
@@ -20,10 +20,10 @@ from __future__ import annotations
import time
from typing import Any, TypeVar
+import httpx
from airbyte_api import AirbyteAPI
from airbyte_api.api import CancelJobRequest, GetJobRequest
from airbyte_api.models import JobCreateRequest, JobStatusEnum, JobTypeEnum,
SchemeClientCredentials, Security
-from requests import Session
from airflow.providers.common.compat.sdk import AirflowException, BaseHook
@@ -110,8 +110,19 @@ class AirbyteHook(BaseHook):
client = None
if self.conn["proxies"]:
self.log.debug("Creating client proxy...")
- client = Session()
- client.proxies = self.conn["proxies"]
+ proxies = self.conn["proxies"]
+ mounts = {}
+ if isinstance(proxies, dict):
+ for scheme, proxy_url in proxies.items():
+ # httpx mount keys require a "://" suffix
+ key = scheme if "://" in scheme else f"{scheme}://"
+ mounts[key] = httpx.HTTPTransport(proxy=proxy_url)
+ else:
+ mounts = {
+ "http://": httpx.HTTPTransport(proxy=proxies),
+ "https://": httpx.HTTPTransport(proxy=proxies),
+ }
+ client = httpx.Client(mounts=mounts)
return AirbyteAPI(
server_url=self.conn["host"],
diff --git a/providers/airbyte/tests/unit/airbyte/hooks/test_airbyte.py
b/providers/airbyte/tests/unit/airbyte/hooks/test_airbyte.py
index 030c5de5bdb..f837a8bf77e 100644
--- a/providers/airbyte/tests/unit/airbyte/hooks/test_airbyte.py
+++ b/providers/airbyte/tests/unit/airbyte/hooks/test_airbyte.py
@@ -19,6 +19,7 @@ from __future__ import annotations
from unittest import mock
+import httpx
import pytest
from airbyte_api.api import CancelJobRequest, GetJobRequest
from airbyte_api.models import JobResponse, JobStatusEnum, JobTypeEnum
@@ -86,7 +87,7 @@ class TestAirbyteHook:
def return_value_get_job(self, status):
response = mock.Mock()
- response.job_response = JobResponse(
+ response.job_response = JobResponse.model_construct(
connection_id="connection-mock",
job_id=self.job_id,
start_time="today",
@@ -146,7 +147,7 @@ class TestAirbyteHook:
def test_wait_for_job_succeeded(self, mock_get_job):
mock_get_job.side_effect =
[self.return_value_get_job(JobStatusEnum.SUCCEEDED)]
self.hook.wait_for_job(job_id=self.job_id, wait_seconds=0)
-
mock_get_job.assert_called_once_with(request=GetJobRequest(self.job_id))
+
mock_get_job.assert_called_once_with(request=GetJobRequest(job_id=self.job_id))
@mock.patch("airbyte_api.jobs.Jobs.get_job")
def test_wait_for_job_error(self, mock_get_job):
@@ -157,7 +158,10 @@ class TestAirbyteHook:
with pytest.raises(AirflowException, match="Job failed"):
self.hook.wait_for_job(job_id=self.job_id, wait_seconds=0)
- calls = [mock.call(request=GetJobRequest(self.job_id)),
mock.call(request=GetJobRequest(self.job_id))]
+ calls = [
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
+ ]
mock_get_job.assert_has_calls(calls)
@mock.patch("airbyte_api.jobs.Jobs.get_job")
@@ -168,7 +172,10 @@ class TestAirbyteHook:
]
self.hook.wait_for_job(job_id=self.job_id, wait_seconds=0)
- calls = [mock.call(request=GetJobRequest(self.job_id)),
mock.call(request=GetJobRequest(self.job_id))]
+ calls = [
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
+ ]
mock_get_job.assert_has_calls(calls)
@mock.patch("airbyte_api.jobs.Jobs.get_job")
@@ -182,9 +189,9 @@ class TestAirbyteHook:
self.hook.wait_for_job(job_id=self.job_id, wait_seconds=2,
timeout=1)
get_calls = [
- mock.call(request=GetJobRequest(self.job_id)),
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
]
- cancel_calls = [mock.call(request=CancelJobRequest(self.job_id))]
+ cancel_calls =
[mock.call(request=CancelJobRequest(job_id=self.job_id))]
mock_get_job.assert_has_calls(get_calls)
mock_cancel_job.assert_has_calls(cancel_calls)
assert mock_get_job.mock_calls == get_calls
@@ -199,7 +206,10 @@ class TestAirbyteHook:
with pytest.raises(AirflowException, match="unexpected state"):
self.hook.wait_for_job(job_id=self.job_id, wait_seconds=0)
- calls = [mock.call(request=GetJobRequest(self.job_id)),
mock.call(request=GetJobRequest(self.job_id))]
+ calls = [
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
+ ]
mock_get_job.assert_has_calls(calls)
@mock.patch("airbyte_api.jobs.Jobs.get_job")
@@ -211,7 +221,10 @@ class TestAirbyteHook:
with pytest.raises(AirflowException, match="Job was cancelled"):
self.hook.wait_for_job(job_id=self.job_id, wait_seconds=0)
- calls = [mock.call(request=GetJobRequest(self.job_id)),
mock.call(request=GetJobRequest(self.job_id))]
+ calls = [
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
+ mock.call(request=GetJobRequest(job_id=self.job_id)),
+ ]
mock_get_job.assert_has_calls(calls)
@mock.patch("airbyte_api.health.Health.get_health_check")
@@ -236,15 +249,18 @@ class TestAirbyteHook:
assert msg == '{"message": "internal server error"}'
def test_create_api_session_with_proxy(self):
- """
- Test the creation of the API session with proxy settings.
- """
- # Create a new AirbyteHook instance
+ """Test that proxy settings produce an httpx.Client with per-scheme
transport mounts."""
hook = AirbyteHook(airbyte_conn_id=self.airbyte_conn_id_with_proxy)
- # Check if the session is created correctly
assert hook.airbyte_api is not None
- assert hook.airbyte_api.sdk_configuration.client.proxies ==
self._mock_proxy["proxies"]
+ client = hook.airbyte_api.sdk_configuration.client
+ assert isinstance(client, httpx.Client)
+
+ default_transport = client._transport
+ for scheme in self._mock_proxy["proxies"]:
+ url = httpx.URL(f"{scheme}://example.com")
+ transport = client._transport_for_url(url)
+ assert transport is not default_transport, f"Expected proxy
transport for {scheme}"
def test_create_api_session_without_credentials(self):
"""Test that a session without OAuth credentials creates an
unauthenticated client."""
diff --git a/uv.lock b/uv.lock
index 1ff0a8117e7..e603933fffc 100644
--- a/uv.lock
+++ b/uv.lock
@@ -708,27 +708,16 @@ wheels = [
[[package]]
name = "airbyte-api"
-version = "0.53.0"
+version = "1.0.1"
source = { registry = "https://pypi.org/simple"; }
dependencies = [
- { name = "certifi" },
- { name = "charset-normalizer" },
- { name = "dataclasses-json" },
- { name = "idna" },
- { name = "jsonpath-python" },
- { name = "marshmallow" },
- { name = "mypy-extensions" },
- { name = "packaging" },
- { name = "python-dateutil" },
- { name = "requests" },
- { name = "six" },
- { name = "typing-extensions" },
- { name = "typing-inspect" },
- { name = "urllib3" },
+ { name = "httpcore" },
+ { name = "httpx" },
+ { name = "pydantic" },
]
-sdist = { url =
"https://files.pythonhosted.org/packages/ad/23/8debf9c4ca8652c9ceb60b8074e654191ba2053001616151993b39d6cdcb/airbyte-api-0.53.0.tar.gz";,
hash =
"sha256:f054ed170f9a691c3304e93a5212670fd2a38e5debb667c72d5ef8eb89cf7e9d", size
= 330090, upload-time = "2025-10-02T19:55:43.378Z" }
+sdist = { url =
"https://files.pythonhosted.org/packages/0a/94/9dee92bac34428e28e403f0863e1fd2db14450dc72ad9a58294c94af71e4/airbyte_api-1.0.1.tar.gz";,
hash =
"sha256:4c40e49b94a62b21836aea126d373365e1002b424022c8e7e262fd0a3d7a4fc6", size
= 500468, upload-time = "2026-06-23T07:02:39.109Z" }
wheels = [
- { url =
"https://files.pythonhosted.org/packages/7a/f5/ca44b6f3919f049a4441755958c042fcea09a44e876352a6005f5e5a9e46/airbyte_api-0.53.0-py3-none-any.whl";,
hash =
"sha256:0bd86d5122789a7c97115a4b52ce492b013d0ea8eeab597b6495b0b310332f28", size
= 773825, upload-time = "2025-10-02T19:55:42.043Z" },
+ { url =
"https://files.pythonhosted.org/packages/24/fa/472753b8a2d38264a1643c27f10bd0b4d0a38f4912918ee943b1e69c7ea7/airbyte_api-1.0.1-py3-none-any.whl";,
hash =
"sha256:edaad71bb9ed123f15cf9f91f78704b88d449584c734b14c4ec8c13ad435516f", size
= 1060335, upload-time = "2026-06-23T07:02:37.503Z" },
]
[[package]]
@@ -2917,7 +2906,7 @@ dependencies = [
{ name = "airbyte-api" },
{ name = "apache-airflow" },
{ name = "apache-airflow-providers-common-compat" },
- { name = "requests" },
+ { name = "httpx" },
]
[package.dev-dependencies]
@@ -2933,10 +2922,10 @@ docs = [
[package.metadata]
requires-dist = [
- { name = "airbyte-api", specifier = ">=0.52.0,<1.0.0" },
+ { name = "airbyte-api", specifier = ">=1.0.0,<2.0" },
{ name = "apache-airflow", editable = "." },
{ name = "apache-airflow-providers-common-compat", editable =
"providers/common/compat" },
- { name = "requests", specifier = ">=2.32.0" },
+ { name = "httpx", specifier = ">=0.28.1" },
]
[package.metadata.requires-dev]
@@ -14995,15 +14984,6 @@ wheels = [
{ url =
"https://files.pythonhosted.org/packages/03/99/33c7d78a3fb70d545fd5411ac67a651c81602cc09c9cf0df383733f068c5/jsonpath_ng-1.8.0-py3-none-any.whl";,
hash =
"sha256:b8dde192f8af58d646fc031fac9c99fe4d00326afc4148f1f043c601a8cfe138", size
= 67844, upload-time = "2026-02-28T00:53:19.637Z" },
]
-[[package]]
-name = "jsonpath-python"
-version = "1.1.6"
-source = { registry = "https://pypi.org/simple"; }
-sdist = { url =
"https://files.pythonhosted.org/packages/98/18/4ca8742534a5993ff383f7602e325ce2d5d7cc93d72ac5e1cdedbea8a458/jsonpath_python-1.1.6.tar.gz";,
hash =
"sha256:dded9932b4ec41fb8726e09c83afa4e6be618f938c2db287cc2a81723c639671", size
= 88178, upload-time = "2026-05-07T01:26:34.482Z" }
-wheels = [
- { url =
"https://files.pythonhosted.org/packages/55/8a/1270a6803bd821cbfcdda387eaa13cb41a7b1f7b9bd145979b3bfb9d6cb7/jsonpath_python-1.1.6-py3-none-any.whl";,
hash =
"sha256:a1c50afd8d3fbbaf47a4873bc890dcb3c15da96f5c020327977d844d8731a2d4", size
= 14453, upload-time = "2026-05-07T01:26:33.306Z" },
-]
-
[[package]]
name = "jsonpointer"
version = "3.1.1"
