This is an automated email from the ASF dual-hosted git repository.
ferruzzi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 95b5d83eaaa Document that DuckDB extension installs need a HOME
directory (#74301)
95b5d83eaaa is described below
commit 95b5d83eaaa7b1cefdaddec31a9c76c4901bf927
Author: D. Ferruzzi <[email protected]>
AuthorDate: Tue Oct 6 11:40:57 2026 -0700
Document that DuckDB extension installs need a HOME directory (#74301)
* Document that DuckDB extension installs need a HOME directory
DuckDB installs extensions under the user's home directory and fails with
"IO Error: Can't find the home directory" when HOME is unset or empty,
which is the default state of an AWS Lambda function. Providing a writable
home directory is part of configuring the environment rather than something
the provider can guess, since any directory it picked would be discarded
when the task exits and so would re-download every extension.
Document the requirement in the DuckDB connection docs, in the Amazon
DuckDB operator docs where the two S3 extensions are installed by default,
and in the Lambda executor docs. Set HOME in the Lambda reference
Dockerfile beside the existing AIRFLOW_HOME, which is the same /tmp
constraint.
Checked against duckdb 1.2.0, 1.3.0, 1.3.2, 1.4.0, 1.4.1, 1.5.0 and 1.5.5:
an install fails on every version without a usable HOME, extension_directory
stops covering for a missing HOME at 1.5.0, and loading a pre-populated
extension_directory with autoinstall_extensions disabled works everywhere.
---
providers/amazon/docs/executors/lambda-executor.rst | 20 ++++++++++++++++++++
providers/amazon/docs/operators/duckdb.rst | 12 ++++++++++++
.../aws/executors/aws_lambda/docker/Dockerfile | 4 ++++
.../amazon/aws/executors/aws_lambda/docker/app.py | 7 +++++++
providers/duckdb/docs/connections/duckdb.rst | 18 ++++++++++++++++++
5 files changed, 61 insertions(+)
diff --git a/providers/amazon/docs/executors/lambda-executor.rst
b/providers/amazon/docs/executors/lambda-executor.rst
index 0a68b2defd5..123501e8915 100644
--- a/providers/amazon/docs/executors/lambda-executor.rst
+++ b/providers/amazon/docs/executors/lambda-executor.rst
@@ -168,6 +168,26 @@ needed for your use case.
It is also possible to build the image based of ``apache/airflow:latest``
and the Lambda runtime can be included separately (follow steps `here
<https://docs.aws.amazon.com/lambda/latest/dg/images-create.html#images-ric>`__).
+Writable paths and ``HOME``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A Lambda function's file system is read-only apart from ``/tmp``, which is why
the provided
+Dockerfile sets ``AIRFLOW_HOME=/tmp/airflow``. Unless you attach an Amazon EFS
file system to the
+function, anything that has to write at runtime has to write under ``/tmp``.
+
+A Lambda function also starts with no ``HOME`` set. Airflow itself does not
need one, but third
+party libraries commonly write caches, credentials or downloaded components
under the user's home
+directory and fail outright when it is missing. DuckDB is one example:
installing an extension
+fails with ``IO Error: Can't find the home directory``. The provided
Dockerfile sets ``HOME=/tmp/home``,
+and the provided ``app.py`` creates that directory when the function starts.
+
+It has to be created at runtime rather than in the image, because each
execution environment gets a
+fresh ``/tmp`` and anything written there at build time is gone by the time
the function runs.
+
+If you build your own image or supply your own handler, keep both halves.
``HOME`` has to point
+somewhere writable, which on Lambda means ``/tmp``, a directory below it, or a
path on an attached
+EFS file system, and whatever you point it at has to exist before a task runs.
+
.. include:: general.rst
:start-after: .. BEGIN LOADING_DAGS_OVERVIEW
diff --git a/providers/amazon/docs/operators/duckdb.rst
b/providers/amazon/docs/operators/duckdb.rst
index 07a75cd35f0..94edba3e95a 100644
--- a/providers/amazon/docs/operators/duckdb.rst
+++ b/providers/amazon/docs/operators/duckdb.rst
@@ -49,6 +49,18 @@ DuckDB support is an optional extra
``autoinstall_extensions=False`` and make the extensions available
yourself, either
in an ``extension_directory`` or baked into your image.
+.. warning::
+
+ Because this provider installs those two extensions by default, it needs
the environment to
+ have a usable ``HOME``. DuckDB installs extensions under the user's home
directory and fails
+ with ``IO Error: Can't find the home directory`` when there is none, which
is the default state
+ of an AWS Lambda function. Configuring a writable home directory is the
deployment
+ administrator's job; see
+ :doc:`the DuckDB connection docs
<apache-airflow-providers-duckdb:connections/duckdb>` for the
+ details and for the alternative that needs no home directory at all. If
you run tasks on the
+ :doc:`Lambda executor <../executors/lambda-executor>`, set ``HOME`` in the
function's image or
+ environment variables.
+
.. _howto/connection:duckdb_aws:
Run a DuckDB query against Amazon S3
diff --git
a/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/Dockerfile
b/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/Dockerfile
index 4520bf27f67..1769a262a54 100644
---
a/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/Dockerfile
+++
b/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/Dockerfile
@@ -43,6 +43,10 @@ RUN curl
"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"; -o "awscliv2
RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir
apache-airflow[amazon,postgres]
# /tmp is the only writable directory in Lambda, so we need to set the
AIRFLOW_HOME there.
ENV AIRFLOW_HOME=/tmp/airflow
+# Lambda does not set HOME and many third party libraries write caches or
downloaded components under
+# the user's home directory, so give it one of its own under /tmp. It cannot
be created here: each
+# execution environment gets a fresh /tmp, so app.py creates it when the
function starts.
+ENV HOME=/tmp/home
# Dags are read-only, so they can be stored in opt (or another path provided).
ARG container_dag_path=/opt/airflow/dags
ENV AIRFLOW__CORE__DAGS_FOLDER=$container_dag_path
diff --git
a/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/app.py
b/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/app.py
index 7c77e6c4c7d..f429f40edd9 100644
---
a/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/app.py
+++
b/providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/app.py
@@ -32,6 +32,13 @@ log = logging.getLogger()
log.setLevel(logging.INFO)
+# Lambda gives each execution environment a fresh /tmp, so a home directory
below it cannot exist
+# in the image and is created here instead. HOME is set in the Dockerfile used
to build the image.
+HOME_DIR = os.environ.get("HOME")
+if HOME_DIR:
+ os.makedirs(HOME_DIR, exist_ok=True)
+
+
# Get the S3 URI from the environment variable. Set either on the Lambda
function or in the
# docker image used for the lambda invocations.
S3_URI = os.environ.get("S3_URI", None)
diff --git a/providers/duckdb/docs/connections/duckdb.rst
b/providers/duckdb/docs/connections/duckdb.rst
index e67408c5094..5ffc5834e2d 100644
--- a/providers/duckdb/docs/connections/duckdb.rst
+++ b/providers/duckdb/docs/connections/duckdb.rst
@@ -96,6 +96,24 @@ Extra (JSON)
``autoinstall_extensions=True`` if downloading on demand is acceptable.
Loading an extension that
is already present needs neither setting.
+.. warning:: **Installing an extension needs a home directory**
+
+ DuckDB installs extensions under the Airflow user's home directory, in
+ ``~/.duckdb/extensions/<duckdb_version>/<platform>/``, so installing one
requires the
+ environment to provide a home directory that exists and is writable. When
``HOME`` is unset,
+ empty, or points at a directory that does not exist, the install fails with
+ ``IO Error: Can't find the home directory``.
+
+ Providing a writable home directory is part of configuring the
environment, and is the
+ deployment administrator's responsibility.
+
+ Setting ``extension_directory`` is not a reliable substitute. On DuckDB
1.5.0 and later an
+ install still resolves the home directory when ``HOME`` is unset or empty,
even with
+ ``extension_directory`` set, so it fails anyway. What does work on every
supported DuckDB
+ version is not installing at task runtime at all: pre-populate
``extension_directory`` and set
+ ``autoinstall_extensions=False``. Loading an extension that is already
present needs no home
+ directory.
+
.. warning:: **Extensions are native code**
A DuckDB extension is a shared library loaded into the task process.
Community extensions come
