potiuk commented on code in PR #71941:
URL: https://github.com/apache/airflow/pull/71941#discussion_r4057802302
##########
providers/google/docs/connections/gcp.rst:
##########
@@ -393,6 +393,92 @@ Using a quota project affects where API usage is billed,
which quotas are applie
usage is reported for monitoring and auditing.
+.. _howto/connection:google_cloud_platform:corporate_proxy:
+
+Using the Google Cloud Connection Behind a Corporate Proxy
+----------------------------------------------------------
+
+If Airflow workers are deployed behind a corporate HTTP proxy, two things are
required for
+Google API calls to reach the internet.
+
+**1. Set the standard proxy environment variables on each worker:**
+
+.. code-block:: bash
+
+ export HTTPS_PROXY=http://<proxy-host>:<port>
+ export HTTP_PROXY=http://<proxy-host>:<port>
+ export NO_PROXY=localhost,127.0.0.1,.cluster.local
+
+In a Kubernetes / Helm deployment add them to ``values.yaml``:
+
+.. code-block:: yaml
+
+ env:
+ - name: HTTPS_PROXY
+ value: "http://<proxy-host>:<port>"
+ - name: HTTP_PROXY
+ value: "http://<proxy-host>:<port>"
+ - name: NO_PROXY
+ value: "localhost,127.0.0.1,.cluster.local"
+
+**2. Install the** ``PySocks`` **package in the worker image — this is
mandatory.**
+
+For a bare-metal or custom Docker image, add it at build time:
+
+.. code-block:: dockerfile
+
+ RUN pip install pysocks
+
+In a Kubernetes / Helm deployment, bake ``pysocks`` into your custom worker
image the same way as
+above, then point the chart at that image in ``values.yaml``:
+
+.. code-block:: yaml
+
+ images:
+ airflow:
+ repository: your-registry/airflow-with-pysocks
+ tag: "3.x.x"
+
+Why PySocks is required even for an HTTP proxy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All Google API endpoints (``oauth2.googleapis.com``,
``bigquery.googleapis.com``, etc.) are
+HTTPS. To route an HTTPS request through an HTTP proxy, the client must first
open an
+``HTTP CONNECT`` tunnel and then do TLS end-to-end inside it.
+
+The ``httplib2`` library — used by ``google-api-python-client`` and the
+``_authorize()`` path of ``GoogleBaseHook`` for services such as BigQuery Jobs
API,
+Dataflow, Compute, Cloud SQL, Datastore, Cloud Functions, and Marketing
Platform — does
+**not** implement ``CONNECT`` tunneling itself. It bundles a minimal socks
adapter but
Review Comment:
This one is factually wrong rather than a style point, so worth applying
before merge.
httplib2 on Python 3 ships **no** bundled socks module. In httplib2 0.32.0
(what this repo resolves) the package contains only `__init__.py, auth.py,
cacerts.txt, certs.py, decode.py, error.py, iri2uri.py`, and the import is:
```python
try:
import socks
except ImportError:
socks = None
```
with no `from . import socks` fallback — so `socks` resolves *only* to
external PySocks. The bundled `socks.py` belonged to the Python-2 branch of
httplib2, which Airflow does not use.
```suggestion
**not** implement ``CONNECT`` tunneling itself. It delegates proxying to the
external
`PySocks <https://pypi.org/project/PySocks/>`_ package, which must be
importable at runtime
```
(The following line already reads "importable at runtime to activate its
HTTP ``CONNECT`` tunnel path via a ``socks.socksocket``", so it flows straight
on — you may want to trim the duplicated "importable at runtime".)
##########
providers/google/docs/connections/gcp.rst:
##########
@@ -393,6 +393,92 @@ Using a quota project affects where API usage is billed,
which quotas are applie
usage is reported for monitoring and auditing.
+.. _howto/connection:google_cloud_platform:corporate_proxy:
+
+Using the Google Cloud Connection Behind a Corporate Proxy
+----------------------------------------------------------
+
+If Airflow workers are deployed behind a corporate HTTP proxy, two things are
required for
+Google API calls to reach the internet.
+
+**1. Set the standard proxy environment variables on each worker:**
+
+.. code-block:: bash
+
+ export HTTPS_PROXY=http://<proxy-host>:<port>
+ export HTTP_PROXY=http://<proxy-host>:<port>
+ export NO_PROXY=localhost,127.0.0.1,.cluster.local
+
+In a Kubernetes / Helm deployment add them to ``values.yaml``:
+
+.. code-block:: yaml
+
+ env:
+ - name: HTTPS_PROXY
+ value: "http://<proxy-host>:<port>"
+ - name: HTTP_PROXY
+ value: "http://<proxy-host>:<port>"
+ - name: NO_PROXY
+ value: "localhost,127.0.0.1,.cluster.local"
+
+**2. Install the** ``PySocks`` **package in the worker image — this is
mandatory.**
+
+For a bare-metal or custom Docker image, add it at build time:
+
+.. code-block:: dockerfile
+
+ RUN pip install pysocks
+
+In a Kubernetes / Helm deployment, bake ``pysocks`` into your custom worker
image the same way as
+above, then point the chart at that image in ``values.yaml``:
+
+.. code-block:: yaml
+
+ images:
+ airflow:
+ repository: your-registry/airflow-with-pysocks
+ tag: "3.x.x"
+
+Why PySocks is required even for an HTTP proxy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All Google API endpoints (``oauth2.googleapis.com``,
``bigquery.googleapis.com``, etc.) are
+HTTPS. To route an HTTPS request through an HTTP proxy, the client must first
open an
+``HTTP CONNECT`` tunnel and then do TLS end-to-end inside it.
+
+The ``httplib2`` library — used by ``google-api-python-client`` and the
+``_authorize()`` path of ``GoogleBaseHook`` for services such as BigQuery Jobs
API,
+Dataflow, Compute, Cloud SQL, Datastore, Cloud Functions, and Marketing
Platform — does
+**not** implement ``CONNECT`` tunneling itself. It bundles a minimal socks
adapter but
+relies on the external `PySocks <https://pypi.org/project/PySocks/>`_ package
being
+importable at runtime to activate its HTTP ``CONNECT`` tunnel path via a
``socks.socksocket``.
+If PySocks is not installed, ``httplib2`` silently falls back to a direct
connection even
+when ``HTTPS_PROXY`` is set, and the worker attempts a direct DNS lookup of
the Google
+endpoint — which fails in a network-restricted environment.
+
+The symptom is a ``socket.gaierror: [Errno -2] Name or service not known`` (or
+``[Errno 11001] getaddrinfo failed`` on Windows) when a task first tries to
authenticate
+or call a Google service.
+
+.. note::
+ Hooks backed by the newer ``google-cloud-*`` client libraries (Cloud
Storage / GCS,
+ Pub/Sub, Spanner, BigQuery Storage API, etc.) use ``google-auth`` with the
``requests``
+ transport, which has native ``CONNECT``-tunnel support and does **not**
need PySocks.
Review Comment:
Nit, and the advice itself is right — only the stated reason is off. Of the
four examples, only GCS is requests/REST-based. Pub/Sub, Spanner and BigQuery
Storage default to the **gRPC** transport: the generated clients register
`grpc, grpc_asyncio, rest` in that order, and `get_transport_class()` documents
"If none is provided, then the first transport in the registry is used". Only
their credential refresh goes through `google-auth` + requests.
```suggestion
Hooks backed by the newer ``google-cloud-*`` client libraries do not use
``httplib2``:
REST-based clients (Cloud Storage / GCS) use ``google-auth`` with the
``requests``
transport, and gRPC-based clients (Pub/Sub, Spanner, BigQuery Storage
API) use gRPC's
own proxy support. Neither needs PySocks.
```
One caveat if you take this: I could not verify gRPC's proxy env-var
handling from this checkout (it lives in the C core), and my understanding is
that gRPC prefers `no_grpc_proxy` over `NO_PROXY` — so the `NO_PROXY` guidance
earlier in the section may not fully apply to the gRPC-backed hooks. Happy to
leave the wording as-is if you would rather not assert that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
