Hello Airflow team,
Please advise if you can. In our environment, we have noticed that dynamic
tasks place quite of stress on scheduler, webserver and increase MySQL DB
connections.
We are run about 1000 Dynamic Tasks every 30 min and parsing time increases
from 5 to 65 sec with Runtime from 2s
Hey Daniel,
Thank you so much for alternate logos!
@everyone please take a look at the link provided by Daniel.
I really like the last option, which is Alternate 2. Maybe we could really
consider a middle point b/w old and new ones?
Thanks,
Aizhamal
On Tue, Aug 13, 2019 at 23:33 Daniel Gruno w
Hello Airflow developers,
I just merged the initial version of pre-commit hooks to master of Airflow
with support for pylint, mypy, flake8, dockerfile linting (all the static
checks we had so far). This slightly changes the way how we run static
checks in Travis (one job for most static checks exc
If your dag generation process makes external calls over network, one
avenue of optimization is to cache your config (i.e. the thing that drives
your dag generation process) in local file (perhaps sqlite) and update that
file only periodically. Or, just convert to fixed config object (e.g. a
dicti
Hello there,
Please advise if you can. In our environment, we have noticed that dynamic
tasks place a lot of stress on scheduler, webserver and DB (connections wise)
We are running about 1000 Dynamic Tasks every 30 min and parsing time increases
from 5 to 65 sec with Runtime from 2sec to 350+ .
The documentation outlines 5 attributes "doc, doc_json, doc_yaml, doc_md,
doc_rst", but as best as I can tell only "doc_md" does anything and it the only
1 listed in the DAG class:
https://github.com/apache/airflow/blob/4e03d2390fc77e6a911fb97d8585fad482c589a6/airflow/models/dag.py#L204
I tri
