Hi Jarek, Thanks agree with MySQL 8 part, I will try to tackle that in a couple of days and help out with the other PRs / issues too. The Selective Optimisation one is a very interesting one too and will take a look at the WIP PR you have already created.
Regards, Kaxil On Sat, Sep 26, 2020 at 10:13 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Thanks, Kaxil, Good summary! > > Just a comment on progress here from my side. I have just updated the > status of three issues that are relevant for our Monday discussion. > > * Enable MySQL 8 CI jobs #11164 > https://github.com/apache/airflow/issues/11164 > > This one is a prerequisite IMHO to get HA Scheduler merged because we have > a dependency on MySQL 8 for it but we do not test MySQL 8 on CI at all > currently. I think this one should be implemented as part of the HA > Scheduler PR merge preparation. I'd love to take that on, but I have no > time to try it out even, but likely it is only adding the version and it > should work, however, there is quite a risk that we will need some fixes at > least in the tests and we for sure need to adapt Dockerfile to use MySQL 8 > client. > > It is also (not very strongly) depends on those three that are > closely related. It would be best if those three are all completed because > this will give us a chance to test the separation of packages and in case > we are going Semver (which we all agree is a better approach), there is > quite some work to implement SEMVER versioning for separate packages. This > separation is also needed to massively speed up our CI builds and it will > help us to tackle increased CI pressure when we add MySQL 8. > > * [OPTIMISATION] Selective builds for different types of tests #10507 > https://github.com/apache/airflow/issues/10507 > * Fully separate provider packages from the Airflow core (AIP-8) > https://github.com/apache/airflow/issues/11163 > * Release a 2nd wave of Backport packages #10014 > https://github.com/apache/airflow/issues/10014 > > Unfortunately, due to other obligations, I likely won't be able to > complete any of those this week and I will have very little time this week > in general, so any help on those is appreciated - if anyone would like to > take any of those - I linked all the WIP PRs for those - so if there is > anyone who would like to take it over - feel free. > > I will do my best to be able to take part in the meeting on Monday. > > J. > > > On Fri, Sep 25, 2020 at 9:09 PM Kaxil Naik <kaxiln...@apache.org> wrote: > >> Hi all, >> >> I have created a document to summarize the discussion from our dev call for >> Airflow 2.0 this Monday. Apologies for the delay in publishing the Meeting >> Notes. >> >> Thank you all who joined the call. >> >> *Doc Link*: >> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#5:21Sep2020 >> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%235:21Sep2020> >> >> To all those who attended, can you please double-check and add if I have >> missed anything? >> >> To all those who didn't join, if you disagree to anything in >> the Summary please voice your opinion. >> >> Also please let me know if someone wants to include an item in Next call's >> Agenda. >> >> Including the Summary here too (might potentially break formatting): >> >> *Key Decisions* >> >> - *API* >> - Progress: >> - Project Board: https://github.com/apache/airflow/projects/1 >> - The issues labelled with "Enhancement" are not a >> requirement for 2.0 >> - Endpoints: >> - Task Instance Endpoint >> <https://github.com/apache/airflow/pull/9597> is WIP, all >> the other endpoints have been implemented. >> - Permissions Model: >> - PR <https://github.com/apache/airflow/pull/10594> has been >> merged. >> - The next piece of work to be done is migrating existing >> Views to use resource-based permissions. (Github issue >> <https://github.com/apache/airflow/issues/10469>). This is >> mainly for standardizing the permissions model across API and UI. >> - *Providers* >> - Vote on AIP-8 took place on the mailing list >> >> <https://lists.apache.org/thread.html/rcd63bbe62a618c4547bd00b1c1d14dc329cfe1c09e4795571be28cb3%40%3Cdev.airflow.apache.org%3E> >> . >> - There is an ongoing discussion on the same thread about SemVer >> vs CalVer for the Providers package. >> - The people involved on the call were *leaning towards SemVer* to >> make a clear distinction about a breaking release. This will >> potentially >> increase the work on release managers but some automation around >> releasing >> (similar to backport providers) and automation around the >> generation of the >> changelog for the providers would make the effort less painful. >> - *Version Per Provide: *Each Providers package would have a >> separate versioning i.e. we might release "google-providers 3.1" and >> "amazon-providers" 3.7 at the same time but the versioning for a >> particular >> provider will be independent of other providers. >> - *DEV* >> - Would be good to have a release policy on when we can deprecate a >> feature, our release cadence. A good example is >> >> https://docs.djangoproject.com/en/3.1/internals/release-process/#release-cadence >> - *SubDag Deprecation* >> - There is a mailing list thread >> >> <https://lists.apache.org/thread.html/ra52746f9c8274469d343b5f0251199de776e75ab75ded6830886fb6a%40%3Cdev.airflow.apache.org%3E> >> on >> whether or not we want to deprecate SubDags in favor of Taskgroups, the >> majority on the call agreed that we *should not deprecate the >> Subdags yet* and wait till people have used TaskGroups and it has >> feature parity with SubDags. >> - However, we should *clearly recommend using TaskGroups compared >> to SubDags* in our docs and state limitations of the SubDags. >> - *Helm Chart Release* >> - Deferred until 2.0 is out >> - Will be available to use from the source code of Airflow on >> Github but the first official release of the Helm chart will only >> happen >> after Airflow 2.0 >> - *Docs* >> - Mailing list thread >> >> <https://lists.apache.org/thread.html/rc6331d0bf97d91dc88853c992513f4e886f113c1cff030876996859e%40%3Cdev.airflow.apache.org%3E> >> to >> get some feedback has been created and cross-posted across Slack and >> Twitter. Once we have enough feedback, Kaxil will create Github issues >> for >> them so that anyone willing to help on it can start working on it. >> - A separate section for Upgrading to 2.0 would be ideal, can be a >> duplicate of Updating.md but with a better structure and more >> organized. >> - *UI Changes* >> - *Github Issue: *https://github.com/apache/airflow/issues/10953 >> - There are some proposals from Ryan for the UI changes for which >> he has created some PRs (links below) and in the process of creating >> few >> more. >> - Task Instance Modal UX Enhancements · Issue #10944 · >> apache/airflow <https://github.com/apache/airflow/pull/10944> >> - Replace JS package toggle w/ pure CSS solution #11035 >> <https://github.com/apache/airflow/pull/11035> >> - Task Instance header/navigation pattern UX cleanup >> <https://github.com/apache/airflow/pull/11089> – Suggestions / >> VOTE needed here if anyone has strong opinions >> - *Scheduler HA* >> - *Reminder*: A draft PR for Scheduler HA >> <https://github.com/apache/airflow/pull/10956> is available for >> review. It would be good to get some more feedback from the wider >> community >> with their own DEV setup if possible. >> - *Process* >> - Any new PRs would continue to be merged until we complete the >> items for 2.0 and release alphas. >> - *NOTE: *The Timeline shown on the Planning page >> >> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning> >> will >> be revisited every week on the Dev Call and updated if needed based on the >> progress towards the major features of Airflow 2.0 >> >> >> Regards, >> Kaxil >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > >
