o-nikolas commented on code in PR #74301: URL: https://github.com/apache/airflow/pull/74301#discussion_r4188981627
########## providers/amazon/docs/executors/lambda-executor.rst: ########## @@ -168,6 +168,30 @@ needed for your use case. It is also possible to build the image based of ``apache/airflow:latest`` and the Lambda runtime can be included separately (follow steps `here <https://docs.aws.amazon.com/lambda/latest/dg/images-create.html#images-ric>`__). +Writable paths and ``HOME`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A Lambda function's file system is read-only apart from ``/tmp``, which is why the provided +Dockerfile sets ``AIRFLOW_HOME=/tmp/airflow``. Unless you attach an Amazon EFS file system to the +function, anything that has to write at runtime has to write under ``/tmp``. + +A Lambda function also starts with no ``HOME`` set. Airflow itself does not need one, but third +party libraries commonly write caches, credentials or downloaded components under the user's home +directory and fail outright when it is missing. DuckDB is one example: installing an extension +fails with ``IO Error: Can't find the home directory``. The provided Dockerfile therefore sets +``HOME=/tmp`` as well. Review Comment: Can we contain it to a directory like /tmp/home? We can create it at runtime before werun the airflow task, it's a pretty simple operation. ########## providers/duckdb/docs/connections/duckdb.rst: ########## @@ -96,6 +96,26 @@ Extra (JSON) ``autoinstall_extensions=True`` if downloading on demand is acceptable. Loading an extension that is already present needs neither setting. +.. warning:: **Installing an extension needs a home directory** + + DuckDB installs extensions under the operating system user's home directory, in + ``~/.duckdb/extensions/<duckdb_version>/<platform>/``, so installing one requires the + environment to provide a home directory that exists and is writable. When ``HOME`` is unset, + empty, or points at a directory that does not exist, the install fails with + ``IO Error: Can't find the home directory``. + + Providing a writable home directory is part of configuring the environment, and is the + deployment administrator's responsibility. This provider does not invent one, because a Review Comment: "This provider does not invent one" sounds kinda AI. Maybe word it more human like here? ########## providers/duckdb/docs/connections/duckdb.rst: ########## @@ -96,6 +96,26 @@ Extra (JSON) ``autoinstall_extensions=True`` if downloading on demand is acceptable. Loading an extension that is already present needs neither setting. +.. warning:: **Installing an extension needs a home directory** + + DuckDB installs extensions under the operating system user's home directory, in + ``~/.duckdb/extensions/<duckdb_version>/<platform>/``, so installing one requires the + environment to provide a home directory that exists and is writable. When ``HOME`` is unset, + empty, or points at a directory that does not exist, the install fails with + ``IO Error: Can't find the home directory``. + + Providing a writable home directory is part of configuring the environment, and is the + deployment administrator's responsibility. This provider does not invent one, because a + directory it picked would be discarded when the task exits and would re-download every Review Comment: This isn't necessarily true, Claude assumed the Lambda-like usecase that it was working from. ########## providers/duckdb/docs/connections/duckdb.rst: ########## @@ -96,6 +96,26 @@ Extra (JSON) ``autoinstall_extensions=True`` if downloading on demand is acceptable. Loading an extension that is already present needs neither setting. +.. warning:: **Installing an extension needs a home directory** + + DuckDB installs extensions under the operating system user's home directory, in Review Comment: I'd say the airflow user's home directory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
