wolfdn opened a new pull request, #65361:
URL: https://github.com/apache/airflow/pull/65361
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
## Problem
Building Airflow Docker images fails in environments where GPG's `dirmngr`
cannot reach keyservers — most commonly behind corporate HTTP proxies. This
makes commands like `breeze start-airflow` fail, if it is attempted to build a
new Docker image - even if the Docker daemon and client have correct HTTP proxy
configuration.
The issue is that `dirmngr` (GnuPG's network daemon, used for `gpg
--recv-keys`) resolves DNS **locally** before connecting, even when
`--keyserver-options http-proxy=...` is configured. In corporate networks where
an intercepting DNS server returns NXDOMAIN for external domains (routing all
internet access exclusively through an HTTP proxy), `dirmngr` fails with
`keyserver receive failed: No keyserver available` before it ever attempts a
connection.
Tools like `wget` and `curl` work in these environments because they
delegate DNS resolution to the proxy via the `CONNECT` method when
`http_proxy`/`https_proxy` environment variables are set.
This affects two code paths:
1. **Python source verification** (`install_os_dependencies.sh`) — fetching
the Python release manager's GPG key from `keys.openpgp.org` when building
Python 3.10 from source.
2. **APT repository key import** (`common.sh` →
`common::import_trusted_gpg`) — fetching GPG keys for MariaDB, MySQL,
PostgreSQL, and MSSQL APT repositories from `keyserver.ubuntu.com` /
`pgp.surf.nl`.
I'm not sure how many developers of Airflow it would help to fix this - so
how many developers are (forced to) work behind a corporate HTTP proxy. At
least for us it would be nice to have this fixed, because it would make setting
up a development environment a bit more straightforward.
## Proposed Solution
Add a `wget` fallback after each `gpg --recv-keys` call. If GPG fails to
fetch the key natively, the same key is fetched via HTTPS using `wget` and
imported with `gpg --import`. This keeps the existing GPG-native path as the
primary method (no changes for environments without proxy issues) while
providing a working fallback for proxy-constrained environments.
### Why the wget command differs between the two code paths
The fallback uses different URLs depending on the keyserver API:
- **`keys.openpgp.org`** (Python key) uses the [VKS
API](https://keys.openpgp.org/about/api), which has a clean endpoint:
```
https://keys.openpgp.org/vks/v1/by-fingerprint/<fingerprint>
```
The key fingerprint is used directly — no prefix manipulation needed.
- **`keyserver.ubuntu.com` / `pgp.surf.nl`** (APT repo keys) use the
[HKP](https://datatracker.ietf.org/doc/html/draft-shaw-openpgp-hkp-00)
protocol's `/pks/lookup` endpoint:
```
https://<host>/pks/lookup?op=get&options=mr&search=<keyid>
```
This requires:
- Extracting the hostname from the `hkps://` or `hkp://` URL scheme
- Stripping the `0x` prefix from the key ID (some callers pass a `0x`
prefix for the key IDs, others don't) and re-adding it as a query parameter to
avoid a double-`0x` bug
- Verifying the import succeeded with `gpg --list-keys` before breaking
out of the keyserver loop, because `gpg --import` exits 0 even on empty input
## Changes
- `scripts/docker/common.sh` — `common::import_trusted_gpg()`: added wget
fallback with HKP lookup after `gpg --recv-keys` in the keyserver loop
- `scripts/docker/install_os_dependencies.sh` — `install_python()`: added
wget fallback with VKS lookup after `gpg --recv-keys` for the Python GPG key
- `Dockerfile.ci`, `Dockerfile` — updated inlined copies of both scripts
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [x] Yes (please specify the tool below)
GitHub Copilot - Claude Opus 4.6
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
