Hi Jarek,
Sorry for the late reply. It is very interesting! I like this idea. A very
simple optimization can have huge improvements.
One thing I am thinking Airflow core can expose certain env vars to the
dag_file, something like, AIRFLOW_PARSING_ENV=[scheduler|worker], and/or
AIRFLOW_DAG_ID= ins
I really like that - in my opinion definitely worth a note in the docs. Let's
just make sure we warn users of side effects which you already mentioned here.
From what I can say about dynamic dag generation at Trade Republic is that it
was a huge pain for us to get it done right. So that it does
I've added a PR to our docs (howto/dynamic-dags) to guide our users into
this possibility.
https://github.com/apache/airflow/pull/25121
On Tue, Jul 12, 2022 at 11:25 AM Abhishek Bhakat
wrote:
> Can vote for making it as an optional approach for fine-tuning (only for
> advance users).
>
> On 12-
I would love to hear what others think about the "in/out" approach - mine
is just the line of thoughts I've been exploring during the last few months
where I prepared my own line of thought about providers, maintenance,
incentive of entities maintaining open-source projects, and especially -
expect
I had some thoughts about it - this also connected with recent discussions
about mixed governance for providers, and I think it's worth using this
discussion to set some rules and "boundaries" on when and how and
especially why we want to accept some contributions, while for some other
contribution
