This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new d8ae7df Add extra information about time synchronization needed
(#21685)
d8ae7df is described below
commit d8ae7df08168fd3ab92ed0d917f9b5dd34d1354d
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Feb 19 21:47:14 2022 +0100
Add extra information about time synchronization needed (#21685)
When you have several machines running Airflow and their time
is not synchronized, you might get very weird behaviour - some of
the log retrieval actions might randomly return "forbidden" error
because of expiring token. This is extremely difficult to
diagnose and figure out (some of our users spent days on
investigating that). Explicitly stating the requirement in the
forbidden error and whenever secret_key parameter is
mentioned, should help our users to diagnose it more easily
(and save maintainers from unnecessary questions and discussions
in Slack :))
---
airflow/config_templates/config.yml | 4 ++++
airflow/config_templates/default_airflow.cfg | 4 ++++
airflow/utils/log/file_task_handler.py | 5 +++--
docs/apache-airflow/configurations-ref.rst | 5 +++++
docs/apache-airflow/howto/set-config.rst | 5 +++++
docs/apache-airflow/upgrading-from-1-10/index.rst | 5 +++++
docs/helm-chart/production-guide.rst | 5 +++++
docs/spelling_wordlist.txt | 1 +
8 files changed, 32 insertions(+), 2 deletions(-)
diff --git a/airflow/config_templates/config.yml
b/airflow/config_templates/config.yml
index 63da2c2..0eb771e 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -1085,6 +1085,10 @@
Secret key used to run your flask app. It should be as random as
possible. However, when running
more than 1 instances of webserver, make sure all of them use the same
``secret_key`` otherwise
one of them will error with "CSRF session token is missing".
+ The webserver key is also used to authorize requests to Celery workers
when logs are retrieved.
+ The token generated using the secret key has a short expiry time
though - make sure that time on
+ ALL the machines that you run airflow components on is synchronized
(for example using ntpd)
+ otherwise you might get "forbidden" errors when the logs are accessed.
version_added: ~
type: string
sensitive: true
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 1b27d9b..6d69578 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -559,6 +559,10 @@ reload_on_plugin_change = False
# Secret key used to run your flask app. It should be as random as possible.
However, when running
# more than 1 instances of webserver, make sure all of them use the same
``secret_key`` otherwise
# one of them will error with "CSRF session token is missing".
+# The webserver key is also used to authorize requests to Celery workers when
logs are retrieved.
+# The token generated using the secret key has a short expiry time though -
make sure that time on
+# ALL the machines that you run airflow components on is synchronized (for
example using ntpd)
+# otherwise you might get "forbidden" errors when the logs are accessed.
secret_key = {SECRET_KEY}
# Number of workers to run the Gunicorn web server
diff --git a/airflow/utils/log/file_task_handler.py
b/airflow/utils/log/file_task_handler.py
index 08755bb..e056199 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -207,8 +207,9 @@ class FileTaskHandler(logging.Handler):
if response.status_code == 403:
log += (
"*** !!!! Please make sure that all your Airflow
components (e.g. "
- "schedulers, webservers and workers) have"
- " the same 'secret_key' configured in 'webserver'
section !!!!!\n***"
+ "schedulers, webservers and workers) have "
+ "the same 'secret_key' configured in 'webserver'
section and "
+ "time is synchronized on all your machines (for
example with ntpd) !!!!!\n***"
)
log += (
"*** See more at
https://airflow.apache.org/docs/apache-airflow/";
diff --git a/docs/apache-airflow/configurations-ref.rst
b/docs/apache-airflow/configurations-ref.rst
index 63c5c96..c77a33c 100644
--- a/docs/apache-airflow/configurations-ref.rst
+++ b/docs/apache-airflow/configurations-ref.rst
@@ -27,6 +27,11 @@ does not require all, some configurations need to be same
otherwise they would n
work as expected. A good example for that is
:ref:`secret_key<config:webserver__secret_key>` which
should be same on the Webserver and Worker to allow Webserver to fetch logs
from Worker.
+The webserver key is also used to authorize requests to Celery workers when
logs are retrieved. The token
+generated using the secret key has a short expiry time though - make sure that
time on ALL the machines
+that you run airflow components on is synchronized (for example using ntpd)
otherwise you might get
+"forbidden" errors when the logs are accessed.
+
.. note::
For more information on setting the configuration, see
:doc:`howto/set-config`
diff --git a/docs/apache-airflow/howto/set-config.rst
b/docs/apache-airflow/howto/set-config.rst
index 03cf0de..b0702af 100644
--- a/docs/apache-airflow/howto/set-config.rst
+++ b/docs/apache-airflow/howto/set-config.rst
@@ -121,3 +121,8 @@ the example below.
does not require all, some configurations need to be same otherwise they
would not
work as expected. A good example for that is
:ref:`secret_key<config:webserver__secret_key>` which
should be same on the Webserver and Worker to allow Webserver to fetch
logs from Worker.
+
+ The webserver key is also used to authorize requests to Celery workers
when logs are retrieved. The token
+ generated using the secret key has a short expiry time though - make sure
that time on ALL the machines
+ that you run airflow components on is synchronized (for example using
ntpd) otherwise you might get
+ "forbidden" errors when the logs are accessed.
diff --git a/docs/apache-airflow/upgrading-from-1-10/index.rst
b/docs/apache-airflow/upgrading-from-1-10/index.rst
index 85c04a5..c9cfd8d 100644
--- a/docs/apache-airflow/upgrading-from-1-10/index.rst
+++ b/docs/apache-airflow/upgrading-from-1-10/index.rst
@@ -339,6 +339,11 @@ the only supported UI.
this via any configuration mechanism. The 1.10.15 bridge-release modifies
this feature
to use randomly generated secret keys instead of an insecure default and
may break existing
deployments that rely on the default.
+ The webserver key is also used to authorize requests to Celery workers
when logs are retrieved. The token
+ generated using the secret key has a short expiry time though - make sure
that time on ALL the machines
+ that you run airflow components on is synchronized (for example using
ntpd) otherwise you might get
+ "forbidden" errors when the logs are accessed.
+
The ``flask-oauthlib`` has been replaced with ``authlib`` because
``flask-oauthlib`` has
been deprecated in favor of ``authlib``.
diff --git a/docs/helm-chart/production-guide.rst
b/docs/helm-chart/production-guide.rst
index f700f14..fb996e5 100644
--- a/docs/helm-chart/production-guide.rst
+++ b/docs/helm-chart/production-guide.rst
@@ -111,6 +111,11 @@ Example to create a Kubernetes Secret from ``kubectl``:
kubectl create secret generic my-webserver-secret
--from-literal="webserver-secret-key=$(python3 -c 'import secrets;
print(secrets.token_hex(16))')"
+The webserver key is also used to authorize requests to Celery workers when
logs are retrieved. The token
+generated using the secret key has a short expiry time though - make sure that
time on ALL the machines
+that you run airflow components on is synchronized (for example using ntpd)
otherwise you might get
+"forbidden" errors when the logs are accessed.
+
Extending and customizing Airflow Image
---------------------------------------
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index e9adb37..f0fc45b 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -1060,6 +1060,7 @@ nosasl
notificationChannels
npm
ntlm
+ntpd
nullable
num
oauth
