Sure, yeah I will add this to today's agenda and send a reminder email for today's meeting in a bit.
On Mon, Sep 28, 2020 at 3:43 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > I would like to add one more topic for today. It's been raised when I > spoke to my friend from GitLab (they solved similar problems) about the > Locking architecture of the HA scheduler. > It would be great to understand what kind of Database HA it will be OK > with. I think this is quite important to add to the documentation when we > merge it, but it might be good to talk about it today if we find the time. > I think this is a really important aspect before we ask the users to test > it. > Just wanted to give the heads up before - in case we need some exploration > before this we might talk about it at the next meeting as well. > > J. > > > On Mon, Sep 28, 2020 at 12:56 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > >> Hi Jarek, >> >> Thanks agree with MySQL 8 part, I will try to tackle that in a couple of >> days and help out with the other PRs / issues too. The Selective >> Optimisation one is a very interesting one too and will take a look at the >> WIP PR you have already created. >> >> Regards, >> Kaxil >> >> >> >> On Sat, Sep 26, 2020 at 10:13 AM Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> >>> Thanks, Kaxil, Good summary! >>> >>> Just a comment on progress here from my side. I have just updated the >>> status of three issues that are relevant for our Monday discussion. >>> >>> * Enable MySQL 8 CI jobs #11164 >>> https://github.com/apache/airflow/issues/11164 >>> >>> This one is a prerequisite IMHO to get HA Scheduler merged because we >>> have a dependency on MySQL 8 for it but we do not test MySQL 8 on CI at all >>> currently. I think this one should be implemented as part of the HA >>> Scheduler PR merge preparation. I'd love to take that on, but I have no >>> time to try it out even, but likely it is only adding the version and it >>> should work, however, there is quite a risk that we will need some fixes at >>> least in the tests and we for sure need to adapt Dockerfile to use MySQL 8 >>> client. >>> >>> It is also (not very strongly) depends on those three that are >>> closely related. It would be best if those three are all completed because >>> this will give us a chance to test the separation of packages and in case >>> we are going Semver (which we all agree is a better approach), there is >>> quite some work to implement SEMVER versioning for separate packages. This >>> separation is also needed to massively speed up our CI builds and it will >>> help us to tackle increased CI pressure when we add MySQL 8. >>> >>> * [OPTIMISATION] Selective builds for different types of tests #10507 >>> https://github.com/apache/airflow/issues/10507 >>> * Fully separate provider packages from the Airflow core (AIP-8) >>> https://github.com/apache/airflow/issues/11163 >>> * Release a 2nd wave of Backport packages #10014 >>> https://github.com/apache/airflow/issues/10014 >>> >>> Unfortunately, due to other obligations, I likely won't be able to >>> complete any of those this week and I will have very little time this week >>> in general, so any help on those is appreciated - if anyone would like to >>> take any of those - I linked all the WIP PRs for those - so if there is >>> anyone who would like to take it over - feel free. >>> >>> I will do my best to be able to take part in the meeting on Monday. >>> >>> J. >>> >>> >>> On Fri, Sep 25, 2020 at 9:09 PM Kaxil Naik <kaxiln...@apache.org> wrote: >>> >>>> Hi all, >>>> >>>> I have created a document to summarize the discussion from our dev call for >>>> Airflow 2.0 this Monday. Apologies for the delay in publishing the Meeting >>>> Notes. >>>> >>>> Thank you all who joined the call. >>>> >>>> *Doc Link*: >>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#5:21Sep2020 >>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%235:21Sep2020> >>>> >>>> To all those who attended, can you please double-check and add if I >>>> have missed anything? >>>> >>>> To all those who didn't join, if you disagree to anything in >>>> the Summary please voice your opinion. >>>> >>>> Also please let me know if someone wants to include an item in Next >>>> call's Agenda. >>>> >>>> Including the Summary here too (might potentially break formatting): >>>> >>>> *Key Decisions* >>>> >>>> - *API* >>>> - Progress: >>>> - Project Board: https://github.com/apache/airflow/projects/1 >>>> - The issues labelled with "Enhancement" are not a >>>> requirement for 2.0 >>>> - Endpoints: >>>> - Task Instance Endpoint >>>> <https://github.com/apache/airflow/pull/9597> is WIP, all >>>> the other endpoints have been implemented. >>>> - Permissions Model: >>>> - PR <https://github.com/apache/airflow/pull/10594> has >>>> been merged. >>>> - The next piece of work to be done is migrating existing >>>> Views to use resource-based permissions. (Github issue >>>> <https://github.com/apache/airflow/issues/10469>). This is >>>> mainly for standardizing the permissions model across API and >>>> UI. >>>> - *Providers* >>>> - Vote on AIP-8 took place on the mailing list >>>> >>>> <https://lists.apache.org/thread.html/rcd63bbe62a618c4547bd00b1c1d14dc329cfe1c09e4795571be28cb3%40%3Cdev.airflow.apache.org%3E> >>>> . >>>> - There is an ongoing discussion on the same thread about SemVer >>>> vs CalVer for the Providers package. >>>> - The people involved on the call were *leaning towards >>>> SemVer* to make a clear distinction about a breaking release. >>>> This will potentially increase the work on release managers but >>>> some >>>> automation around releasing (similar to backport providers) and >>>> automation >>>> around the generation of the changelog for the providers would >>>> make the >>>> effort less painful. >>>> - *Version Per Provide: *Each Providers package would have a >>>> separate versioning i.e. we might release "google-providers 3.1" and >>>> "amazon-providers" 3.7 at the same time but the versioning for a >>>> particular >>>> provider will be independent of other providers. >>>> - *DEV* >>>> - Would be good to have a release policy on when we can deprecate a >>>> feature, our release cadence. A good example is >>>> >>>> https://docs.djangoproject.com/en/3.1/internals/release-process/#release-cadence >>>> - *SubDag Deprecation* >>>> - There is a mailing list thread >>>> >>>> <https://lists.apache.org/thread.html/ra52746f9c8274469d343b5f0251199de776e75ab75ded6830886fb6a%40%3Cdev.airflow.apache.org%3E> >>>> on >>>> whether or not we want to deprecate SubDags in favor of Taskgroups, >>>> the >>>> majority on the call agreed that we *should not deprecate the >>>> Subdags yet* and wait till people have used TaskGroups and it >>>> has feature parity with SubDags. >>>> - However, we should *clearly recommend using TaskGroups >>>> compared to SubDags* in our docs and state limitations of the >>>> SubDags. >>>> - *Helm Chart Release* >>>> - Deferred until 2.0 is out >>>> - Will be available to use from the source code of Airflow on >>>> Github but the first official release of the Helm chart will only >>>> happen >>>> after Airflow 2.0 >>>> - *Docs* >>>> - Mailing list thread >>>> >>>> <https://lists.apache.org/thread.html/rc6331d0bf97d91dc88853c992513f4e886f113c1cff030876996859e%40%3Cdev.airflow.apache.org%3E> >>>> to >>>> get some feedback has been created and cross-posted across Slack and >>>> Twitter. Once we have enough feedback, Kaxil will create Github >>>> issues for >>>> them so that anyone willing to help on it can start working on it. >>>> - A separate section for Upgrading to 2.0 would be ideal, can be >>>> a duplicate of Updating.md but with a better structure and more >>>> organized. >>>> - *UI Changes* >>>> - *Github Issue: *https://github.com/apache/airflow/issues/10953 >>>> - There are some proposals from Ryan for the UI changes for >>>> which he has created some PRs (links below) and in the process of >>>> creating >>>> few more. >>>> - Task Instance Modal UX Enhancements · Issue #10944 · >>>> apache/airflow <https://github.com/apache/airflow/pull/10944> >>>> - Replace JS package toggle w/ pure CSS solution #11035 >>>> <https://github.com/apache/airflow/pull/11035> >>>> - Task Instance header/navigation pattern UX cleanup >>>> <https://github.com/apache/airflow/pull/11089> – Suggestions >>>> / VOTE needed here if anyone has strong opinions >>>> - *Scheduler HA* >>>> - *Reminder*: A draft PR for Scheduler HA >>>> <https://github.com/apache/airflow/pull/10956> is available for >>>> review. It would be good to get some more feedback from the wider >>>> community >>>> with their own DEV setup if possible. >>>> - *Process* >>>> - Any new PRs would continue to be merged until we complete the >>>> items for 2.0 and release alphas. >>>> - *NOTE: *The Timeline shown on the Planning page >>>> >>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning> >>>> will >>>> be revisited every week on the Dev Call and updated if needed based on >>>> the >>>> progress towards the major features of Airflow 2.0 >>>> >>>> >>>> Regards, >>>> Kaxil >>>> >>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > >
