GitHub user Enucatl created a discussion: Hashicorp Vault Secrets Backend not used in PythonVirtualenvOperator
### Apache Airflow Provider(s) hashicorp ### Versions of Apache Airflow Providers `FROM docker.io/apache/airflow:3.0.4@sha256:6c07868196d1880d956f4b5ddf50244bc281faab39704a8940b1c1f3159a145e` ### Apache Airflow version 3.0.4 ### Operating System Ubuntu 22.04.5 LTS ### Deployment Docker-Compose ### Deployment details Full docker-compose if needed: https://github.com/Enucatl/docker-airflow/blob/main/docker-compose.yml ### What happened Connections defined in Hashicorp Vault are not available within `PythonVirtualenvOperators` ### What you think should happen instead test_connection log (no virtualenv = successful) ```log Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_connection/attempt=1.log"] ::group::Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_connection/attempt=1.log"] [2025-08-19, 08:42:12] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager" [2025-08-19, 08:42:12] INFO - Filling up the DagBag from /opt/airflow/dags/test_vault_virtualenv.py: source="airflow.models.dagbag.DagBag" [2025-08-19, 08:42:12] INFO - Connection Retrieved 'stva': source="airflow.hooks.base" [2025-08-19, 08:42:12] INFO - Done. Returned value was: None: source="airflow.task.operators.airflow.providers.standard.decorators.python._PythonDecoratedOperator" [2025-08-19, 08:42:12] INFO - https://portal.stva.zh.ch/ecari-dispoweb/ui/app/init/#/conduite/prive/login: chan="stdout": source="task" ``` test_virtualenv log (fails) ```log ::group::Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_virtualenv/attempt=1.log"] [2025-08-19, 08:42:12] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager" [2025-08-19, 08:42:12] INFO - Filling up the DagBag from /opt/airflow/dags/test_vault_virtualenv.py: source="airflow.models.dagbag.DagBag" [2025-08-19, 08:42:12] WARNING - /home/airflow/.local/lib/python3.12/site-packages/airflow/models/variable.py:147: DeprecationWarning: Using Variable.get from `airflow.models` is deprecated.Please use `from airflow.sdk import Variable` instead warnings.warn( : source="py.warnings" [2025-08-19, 08:42:12] INFO - Python virtual environment will be cached in /opt/airflow/venv/venv-aacc1679: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator" [2025-08-19, 08:42:12] INFO - Executing cmd: uv venv --allow-existing --seed --python python --system-site-packages /opt/airflow/venv/venv-aacc1679: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Using CPython 3.12.11 interpreter at: /usr/local/bin/python: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Creating virtual environment with seed packages at: venv/venv-aacc1679: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - If the cache and target directories are on different filesystems, hardlinking may not be supported.: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - + pip==25.2: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Executing cmd: uv pip install --python /opt/airflow/venv/venv-aacc1679/bin/python -r /opt/airflow/venv/venv-aacc1679/requirements.txt: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - warning: Requirements file `venv/venv-aacc1679/requirements.txt` does not contain any dependencies: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Using Python 3.12.11 environment at: venv/venv-aacc1679: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Audited in 62ms: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - New Python virtual environment created in /opt/airflow/venv/venv-aacc1679: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator" [2025-08-19, 08:42:12] INFO - Use 'pickle' as serializer.: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator" [2025-08-19, 08:42:12] INFO - Executing cmd: /opt/airflow/venv/venv-aacc1679/bin/python /tmp/venv-callpa921oow/script.py /tmp/venv-callpa921oow/script.in /tmp/venv-callpa921oow/script.out /tmp/venv-callpa921oow/string_args.txt /tmp/venv-callpa921oow/termination.log /tmp/venv-callpa921oow/airflow_context.json: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - Traceback (most recent call last):: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - File "/tmp/venv-callpa921oow/script.py", line 54, in <module>: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - res = test_virtualenv(*arg_dict["args"], **arg_dict["kwargs"]): source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - File "/tmp/venv-callpa921oow/script.py", line 26, in test_virtualenv: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - connection = BaseHook.get_connection("stva"): source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - File "/home/airflow/.local/lib/python3.12/site-packages/airflow/hooks/base.py", line 64, in get_connection: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - conn = Connection.get_connection_from_secrets(conn_id): source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 507, in get_connection_from_secrets: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined"): source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - airflow.exceptions.AirflowNotFoundException: The conn_id `stva` isn't defined: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] ERROR - Task failed with exception: source="task" AirflowException: Process returned non-zero exit status 1. The conn_id `stva` isn't defined File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 867 in run File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1159 in _execute_task File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 251 in execute File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 490 in execute File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 216 in execute File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 844 in execute_callable File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 576 in _execute_python_callable_in_subprocess ``` ### How to reproduce Minimum reproducible DAG (requires the Vault Secrets backend to be configured, with an example "stva" connection): ```python from datetime import datetime from airflow import DAG from airflow.decorators import task from airflow.hooks.base import BaseHook default_args = { "depends_on_past": False, "retries": 0, "venv_cache_path": "/opt/airflow/venv", } with DAG( "test_vault_virtualenv", default_args=default_args, description="Test vault access for connections", schedule="@once", start_date=datetime(2025, 8, 17), ) as dag: @task def test_connection(): connection = BaseHook.get_connection("stva") print(connection.host) @task.virtualenv( requirements=[], ) def test_virtualenv(logical_date): from airflow.hooks.base import BaseHook connection = BaseHook.get_connection("stva") print(connection.host) test_connection() test_virtualenv() ``` ### Anything else _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) GitHub link: https://github.com/apache/airflow/discussions/55066 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
