Thanks Kevin for the input.
I am working on this full-time as well with help from Ash.
Persistence to DB is what we want to achieve while making Webserver
stateless and hence mitigating the issues you described.
My opinion is that we should work towards that goal and aim to include it
in Airflow
For sure! I'll try my best to squeeze some time for it during the weekend
and see how I can help facilitate the effort( don't get the hopes up too
much tho, got ~2 million things piled in my TODO list :P ). Will bring this
up to the team and see if my team can help too.
You guys have showed pretty
Hi Kevin, it makes sense. Thanks for the explanation! Hope we can get DB
persistence move faster.
Zhou
On Mon, Jul 29, 2019, 5:46 PM Kevin Yang wrote:
> oops, s/consistent file/consistent file order/
>
> On Mon, Jul 29, 2019 at 5:42 PM Kevin Yang wrote:
>
>> Hi Zhou,
>>
>> Totally understood,
oops, s/consistent file/consistent file order/
On Mon, Jul 29, 2019 at 5:42 PM Kevin Yang wrote:
> Hi Zhou,
>
> Totally understood, thank you for that. Streaming logic does cover most
> cases, tho we still have the worst cases where os.walk doesn't give us
> consistent file and file/dir addition
Hi Zhou,
Totally understood, thank you for that. Streaming logic does cover most
cases, tho we still have the worst cases where os.walk doesn't give us
consistent file and file/dir additions/renames causing a different result
order of list_py_file_paths( e.g. right after we parsed the first dir, i
Hi Kevin,
Yes. DAG persistence in DB is definitely the way to go. I referred to the
aysnc dag loader because it may alleviate your current problem (since it is
code ready).
It actually reduces the time to 15min, because DAGs are refreshed by the
background process in a streaming way and you don't
Current process uses sphinx to generate documentation out of the .rst files
and documentation, and uses some examples from the code.
Sphinx has some nice i18n tools that can extract translatable "messages"
already (for example here:
https://jareddillard.com/blog/documentation-internationalization-u
Hi Zhou,
Thank you for the pointer. This solves the issue gunicorn restart rate
throttles webserver refresh rate but not the long DAG parsing time issue,
right? Worst case scenario we still wait 30 mins for the change to show up,
comparing to the previous 35 mins( I was wrong on the number, it sho
Hi Kevin,
The problem that DAG parsing takes a long time can be solved by
Asynchronous DAG loading: https://github.com/apache/airflow/pull/5594
The idea is the a background process parses DAG files, and sends DAGs to
webserver process every [webserver] dagbag_sync_interval = 10s.
We have launche
Thanks everyone for the discussion. The comments are very helpful.
AIP-24 that we proposed here is really a short-term one to minimize the
change for fast launch and compatibility. I agree with the benefits of the
long-term proposals. It would be great if AIP-24 can be a first step (if we
can agre
Had same in mind :-)
Should have some time late this week to dig into specifics of Airflow
process. And, need to explore how docs (source) organized, esp. whether
the parts to be translated are separate in some specific way.
On Sun, Jul 28, 2019 at 12:02 PM Jarek Potiuk
wrote:
> Such translati
Thanks all for the input and thanks Zhou too for the detailed AIP.
The WIP PR can be a good first step to overall optimization.
Let's sync-up on the progress you have already made & what we want to
target.
@Jarek Potiuk & @Fokko - If we manage to make
it entirely backward-compatible with an en
Thanks Jarek, adding my vote.
On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor wrote:
> Thanks Jarek, much clearer. I have voted there too.
>
> -ash
>
>
> > On 29 Jul 2019, at 08:13, Driesprong, Fokko
> wrote:
> >
> > Thanks Jarek, the Wiki works much better for me. I've added my vote there
>
Thanks Jarek, much clearer. I have voted there too.
-ash
> On 29 Jul 2019, at 08:13, Driesprong, Fokko wrote:
>
> Thanks Jarek, the Wiki works much better for me. I've added my vote there
> as well.
>
> You've convinced me on the second case. I've changed my vote on Case 2 from
> A to B :-)
>
Actually I am also doing a lot of v1-10-test merges during the last few
months (probably several tens of them already). Rarely the conflicts are
difficult to solve in fact. We have usually small, localised changes and
until we go for full Black file re-formatting, we should be ok (and the
change fr
I would be hesitant to merge it into 1.10.5. When I try to backport
anything into the 1.x branch, I get a whole bunch on merge conflicts, even
on the trivial tickets. For me, the only one who can really comment on this
would be Ash, since he's doing the bulk of the conflict resolving. Apart
from th
Thanks Jarek, the Wiki works much better for me. I've added my vote there
as well.
You've convinced me on the second case. I've changed my vote on Case 2 from
A to B :-)
Maybe we should leave the vote open a bit longer since it involves quite
some reading, and I would like to get some proper feed
17 matches
Mail list logo
