FYI. I've just updated my "robust" implementation
https://github.com/apache/airflow/pull/25161 to be such "future-possible".
J.
On Tue, Jul 19, 2022 at 5:32 PM Jarek Potiuk wrote:
> Actually this is even more complex as I found another case:
>
>
>
> I thought about it too, but this will be onl
Actually this is even more complex as I found another case:
I thought about it too, but this will be only for 2.4+ and a ton of people
could benefit from that even now.
However I have a thought. I was waiting for something like that to appear
to have a very good reason to implement the "apache-
I think we should only document the "DAG author friendly" approach (env var
or otherwise) and hold the existing PR until it is ready. My 2c.
Yes. Great minds think alike.
Fully agree - I have just found the right code to determine whether you are
in "task execution" or not and it is ... well... convoluted ... in the
current version.
So yeah - that was exactly my thought to add "API guaranteed" env vars that
will indicate what context
Hi Jarek,
Sorry for the late reply. It is very interesting! I like this idea. A very
simple optimization can have huge improvements.
One thing I am thinking Airflow core can expose certain env vars to the
dag_file, something like, AIRFLOW_PARSING_ENV=[scheduler|worker], and/or
AIRFLOW_DAG_ID= ins
I really like that - in my opinion definitely worth a note in the docs. Let's
just make sure we warn users of side effects which you already mentioned here.
From what I can say about dynamic dag generation at Trade Republic is that it
was a huge pain for us to get it done right. So that it does
I've added a PR to our docs (howto/dynamic-dags) to guide our users into
this possibility.
https://github.com/apache/airflow/pull/25121
On Tue, Jul 12, 2022 at 11:25 AM Abhishek Bhakat
wrote:
> Can vote for making it as an optional approach for fine-tuning (only for
> advance users).
>
> On 12-
Can vote for making it as an optional approach for fine-tuning (only for
advance users).
On 12-Jul-2022 at 7:44:35 AM, Jarek Potiuk wrote:
> Not interesting :) ?
>
> On Thu, Jul 7, 2022 at 10:41 AM Jarek Potiuk wrote:
>
>
> Hello everyone,
>
>
> We have just published a blog on our medium -
> h
Not interesting :) ?
On Thu, Jul 7, 2022 at 10:41 AM Jarek Potiuk wrote:
>
> Hello everyone,
>
> We have just published a blog on our medium -
> https://medium.com/apache-airflow/airflows-magic-loop-ec424b05b629 - that is
> a blog of one of our users Itay Bittan (thanks!) who had been inspired
Hello everyone,
We have just published a blog on our medium -
https://medium.com/apache-airflow/airflows-magic-loop-ec424b05b629 - that
is a blog of one of our users Itay Bittan (thanks!) who had been inspired
by our discussion on Slack on how they struggle with delays of loading
dynamic dags in t
10 matches
Mail list logo
