potiuk commented on a change in pull request #13660: URL: https://github.com/apache/airflow/pull/13660#discussion_r557043752
########## File path: docs/apache-airflow/start/docker-compose.yaml ########## @@ -0,0 +1,101 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +version: '3' +x-airflow-common: + &airflow-common + image: apache/airflow:2.0.0 + environment: + - AIRFLOW__CORE__EXECUTOR=CeleryExecutor + - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow + - AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:airflow@postgres/airflow + - AIRFLOW__CELERY__BROKER_URL=redis://:@redis:6379/0 + - AIRFLOW__CORE__FERNET_KEY= + - AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=True + - AIRFLOW__CORE__LOAD_EXAMPLES=True + volumes: + - ./dags:/opt/airflow/dags + - ./logs:/opt/airflow/logs Review comment: Unfortunately this does not work on linux. When you run the "db init' command you get this error: ``` Unable to load the config, contains a configuration error. Traceback (most recent call last): File "/usr/local/lib/python3.6/pathlib.py", line 1248, in mkdir self._accessor.mkdir(self, mode) File "/usr/local/lib/python3.6/pathlib.py", line 387, in wrapped return strfunc(str(pathobj), *args) FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/logs/scheduler/2021-01-14' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/logging/config.py", line 565, in configure handler = self.configure_handler(handlers[name]) File "/usr/local/lib/python3.6/logging/config.py", line 738, in configure_handler result = factory(**kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 46, in __init__ Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True) File "/usr/local/lib/python3.6/pathlib.py", line 1252, in mkdir self.parent.mkdir(parents=True, exist_ok=True) File "/usr/local/lib/python3.6/pathlib.py", line 1248, in mkdir self._accessor.mkdir(self, mode) File "/usr/local/lib/python3.6/pathlib.py", line 387, in wrapped return strfunc(str(pathobj), *args) PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/bin/airflow", line 5, in <module> from airflow.__main__ import main File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 46, in <module> settings.initialize() File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 432, in initialize LOGGING_CLASS_PATH = configure_logging() File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 62, in configure_logging raise e File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 57, in configure_logging dictConfig(logging_config) File "/usr/local/lib/python3.6/logging/config.py", line 802, in dictConfig dictConfigClass(config).configure() File "/usr/local/lib/python3.6/logging/config.py", line 573, in configure '%r: %s' % (name, e)) ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler' ``` The reason is that the "dags" "logs" and "plugins" folder are created as "root" owned automatically and they seem to be not writeable for airflow process. Also after running it "dags", "logs" and "plugins" folder are created with the "root" user which makes it difiicult for the user to manage it (they have to run `sudo` to delete the folders and files created. Idealy those folders should be created with the host user or there should be an easy way to delete them. ``` quick-start-docker+ 2 ± ls -la total 80 drwxrwxr-x 5 jarek jarek 4096 sty 14 05:39 . drwxr-xr-x 10 jarek jarek 12288 sty 14 05:30 .. drwxr-xr-x 2 root root 4096 sty 14 05:39 dags -rw-rw-r-- 1 jarek jarek 2767 sty 14 05:30 docker-compose.yaml -rw-rw-r-- 1 jarek jarek 4889 sty 14 05:30 docker.rst -rw-rw-r-- 1 jarek jarek 991 sty 14 05:30 index.rst -rw-rw-r-- 1 jarek jarek 3809 sty 14 05:30 local.rst drwxr-xr-x 2 root root 4096 sty 14 05:39 logs drwxr-xr-x 2 root root 4096 sty 14 05:39 plugins ``` In order to fix it, you can utillise the fact that airflow is openshift compatible. Airflow inside production docker file can be run as "any" user providing that group of that user is "0". This should allowe you to run all commands as "HOST" user with group "0" but I think it also requires to make sure that the folders are created with appropriate HOST user automatically. See https://docs.openshift.com/container-platform/4.1/openshift_images/create-images.html "support arbitrary user ids" chapter. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
