This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push: new d8ae7df Add extra information about time synchronization needed (#21685) d8ae7df is described below commit d8ae7df08168fd3ab92ed0d917f9b5dd34d1354d Author: Jarek Potiuk <ja...@potiuk.com> AuthorDate: Sat Feb 19 21:47:14 2022 +0100 Add extra information about time synchronization needed (#21685) When you have several machines running Airflow and their time is not synchronized, you might get very weird behaviour - some of the log retrieval actions might randomly return "forbidden" error because of expiring token. This is extremely difficult to diagnose and figure out (some of our users spent days on investigating that). Explicitly stating the requirement in the forbidden error and whenever secret_key parameter is mentioned, should help our users to diagnose it more easily (and save maintainers from unnecessary questions and discussions in Slack :)) --- airflow/config_templates/config.yml | 4 ++++ airflow/config_templates/default_airflow.cfg | 4 ++++ airflow/utils/log/file_task_handler.py | 5 +++-- docs/apache-airflow/configurations-ref.rst | 5 +++++ docs/apache-airflow/howto/set-config.rst | 5 +++++ docs/apache-airflow/upgrading-from-1-10/index.rst | 5 +++++ docs/helm-chart/production-guide.rst | 5 +++++ docs/spelling_wordlist.txt | 1 + 8 files changed, 32 insertions(+), 2 deletions(-) diff --git a/airflow/config_templates/config.yml b/airflow/config_templates/config.yml index 63da2c2..0eb771e 100644 --- a/airflow/config_templates/config.yml +++ b/airflow/config_templates/config.yml @@ -1085,6 +1085,10 @@ Secret key used to run your flask app. It should be as random as possible. However, when running more than 1 instances of webserver, make sure all of them use the same ``secret_key`` otherwise one of them will error with "CSRF session token is missing". + The webserver key is also used to authorize requests to Celery workers when logs are retrieved. + The token generated using the secret key has a short expiry time though - make sure that time on + ALL the machines that you run airflow components on is synchronized (for example using ntpd) + otherwise you might get "forbidden" errors when the logs are accessed. version_added: ~ type: string sensitive: true diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg index 1b27d9b..6d69578 100644 --- a/airflow/config_templates/default_airflow.cfg +++ b/airflow/config_templates/default_airflow.cfg @@ -559,6 +559,10 @@ reload_on_plugin_change = False # Secret key used to run your flask app. It should be as random as possible. However, when running # more than 1 instances of webserver, make sure all of them use the same ``secret_key`` otherwise # one of them will error with "CSRF session token is missing". +# The webserver key is also used to authorize requests to Celery workers when logs are retrieved. +# The token generated using the secret key has a short expiry time though - make sure that time on +# ALL the machines that you run airflow components on is synchronized (for example using ntpd) +# otherwise you might get "forbidden" errors when the logs are accessed. secret_key = {SECRET_KEY} # Number of workers to run the Gunicorn web server diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py index 08755bb..e056199 100644 --- a/airflow/utils/log/file_task_handler.py +++ b/airflow/utils/log/file_task_handler.py @@ -207,8 +207,9 @@ class FileTaskHandler(logging.Handler): if response.status_code == 403: log += ( "*** !!!! Please make sure that all your Airflow components (e.g. " - "schedulers, webservers and workers) have" - " the same 'secret_key' configured in 'webserver' section !!!!!\n***" + "schedulers, webservers and workers) have " + "the same 'secret_key' configured in 'webserver' section and " + "time is synchronized on all your machines (for example with ntpd) !!!!!\n***" ) log += ( "*** See more at https://airflow.apache.org/docs/apache-airflow/" diff --git a/docs/apache-airflow/configurations-ref.rst b/docs/apache-airflow/configurations-ref.rst index 63c5c96..c77a33c 100644 --- a/docs/apache-airflow/configurations-ref.rst +++ b/docs/apache-airflow/configurations-ref.rst @@ -27,6 +27,11 @@ does not require all, some configurations need to be same otherwise they would n work as expected. A good example for that is :ref:`secret_key<config:webserver__secret_key>` which should be same on the Webserver and Worker to allow Webserver to fetch logs from Worker. +The webserver key is also used to authorize requests to Celery workers when logs are retrieved. The token +generated using the secret key has a short expiry time though - make sure that time on ALL the machines +that you run airflow components on is synchronized (for example using ntpd) otherwise you might get +"forbidden" errors when the logs are accessed. + .. note:: For more information on setting the configuration, see :doc:`howto/set-config` diff --git a/docs/apache-airflow/howto/set-config.rst b/docs/apache-airflow/howto/set-config.rst index 03cf0de..b0702af 100644 --- a/docs/apache-airflow/howto/set-config.rst +++ b/docs/apache-airflow/howto/set-config.rst @@ -121,3 +121,8 @@ the example below. does not require all, some configurations need to be same otherwise they would not work as expected. A good example for that is :ref:`secret_key<config:webserver__secret_key>` which should be same on the Webserver and Worker to allow Webserver to fetch logs from Worker. + + The webserver key is also used to authorize requests to Celery workers when logs are retrieved. The token + generated using the secret key has a short expiry time though - make sure that time on ALL the machines + that you run airflow components on is synchronized (for example using ntpd) otherwise you might get + "forbidden" errors when the logs are accessed. diff --git a/docs/apache-airflow/upgrading-from-1-10/index.rst b/docs/apache-airflow/upgrading-from-1-10/index.rst index 85c04a5..c9cfd8d 100644 --- a/docs/apache-airflow/upgrading-from-1-10/index.rst +++ b/docs/apache-airflow/upgrading-from-1-10/index.rst @@ -339,6 +339,11 @@ the only supported UI. this via any configuration mechanism. The 1.10.15 bridge-release modifies this feature to use randomly generated secret keys instead of an insecure default and may break existing deployments that rely on the default. + The webserver key is also used to authorize requests to Celery workers when logs are retrieved. The token + generated using the secret key has a short expiry time though - make sure that time on ALL the machines + that you run airflow components on is synchronized (for example using ntpd) otherwise you might get + "forbidden" errors when the logs are accessed. + The ``flask-oauthlib`` has been replaced with ``authlib`` because ``flask-oauthlib`` has been deprecated in favor of ``authlib``. diff --git a/docs/helm-chart/production-guide.rst b/docs/helm-chart/production-guide.rst index f700f14..fb996e5 100644 --- a/docs/helm-chart/production-guide.rst +++ b/docs/helm-chart/production-guide.rst @@ -111,6 +111,11 @@ Example to create a Kubernetes Secret from ``kubectl``: kubectl create secret generic my-webserver-secret --from-literal="webserver-secret-key=$(python3 -c 'import secrets; print(secrets.token_hex(16))')" +The webserver key is also used to authorize requests to Celery workers when logs are retrieved. The token +generated using the secret key has a short expiry time though - make sure that time on ALL the machines +that you run airflow components on is synchronized (for example using ntpd) otherwise you might get +"forbidden" errors when the logs are accessed. + Extending and customizing Airflow Image --------------------------------------- diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt index e9adb37..f0fc45b 100644 --- a/docs/spelling_wordlist.txt +++ b/docs/spelling_wordlist.txt @@ -1060,6 +1060,7 @@ nosasl notificationChannels npm ntlm +ntpd nullable num oauth