dabla opened a new pull request, #75819:
URL: https://github.com/apache/airflow/pull/75819
The Amazon provider has `AwsDuckDBHook` (#73301): a `duckdb_aws` connection
type that brokers an AWS connection into a DuckDB S3 secret, so
`DuckDBExecuteQueryOperator` reads and writes S3 with no credentials in the
Dag. Azure storage has no equivalent. A task that reads `abfss://` paths with
DuckDB, Microsoft Fabric OneLake included, has to build the DuckDB Azure secret
by hand from the connection.
This adds `AzureDuckDBHook` to the Microsoft Azure provider, the Azure
counterpart of `AwsDuckDBHook`, with the same shape:
- **A `duckdb_azure` connection type** that `DuckDBExecuteQueryOperator`
resolves from the connection. Its extra points at an ADLS connection through
`azure_conn_id` (default `adls_default`), and every hook parameter can be set
there or in `hook_params`.
- **Credential strategies:**
- `config` (the default) builds the secret from the ADLS connection: its
`connection_string` (`PROVIDER config`), else an Active Directory service
principal (`PROVIDER service_principal`, login, password, `tenant_id`).
- `credential_chain` lets the Azure SDK inside DuckDB resolve credentials
(CLI, managed identity, workload identity, env). The ADLS connection is
optional then.
- `none` creates no secret.
- **Account and proxy:** the storage account comes from `account_name`, else
from the ADLS connection's `account_url` (`onelake` for
`https://onelake.dfs.fabric.microsoft.com`) or host. The connection's
`proxies["https"]`, as `AzureDataLakeStorageV2Hook` uses it, becomes the
secret's `HTTP_PROXY`.
- **TLS transport option:** `transport_option_type` sets DuckDB's
`azure_transport_option_type`. `curl` resolves the SSL CA certificate errors
the default transport hits on Linux images whose CA bundle is not where it
looks (see the DuckDB Azure extension docs). It is set `GLOBAL` because
`DbApiHook` runs statements on `conn.cursor()`, a connection that does not see
a session setting.
- **Packaging:** the hook installs the `azure` extension when it is missing,
like the AWS hook does with its extensions, and `autoinstall_extensions=False`
forbids that. DuckDB support is an opt-in extra
(`apache-airflow-providers-microsoft-azure[duckdb]`), as on the Amazon side.
Also: the `duckdb.rst` how-to, the provider.yaml/`get_provider_info.py`
entries, the extra in `pyproject.toml`, the docs index tables and `uv.lock`.
We have run this hook as a plugin in production Dags, reading and writing
Fabric OneLake (Parquet, JSON Lines, Delta) through a service principal behind
a proxy.
Tests: `test_duckdb.py` (27 tests: connection type and operator resolution,
extensions, each strategy and its SQL, quoting, account and proxy derivation,
transport, parameters from the connection extra). Locally `test_duckdb.py` and
`test_data_lake.py` pass, `prek run --files` on the changed files passes, and
`dev/breeze/tests/test_selective_checks.py` passes with the regenerated
provider dependencies (no expectation change needed).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
