On Sep 14 2020, at 11:01 am, Jarek Potiuk <[email protected]> wrote:
>>
>>
>> > We have to make sure that we have no dependencies core -> providers
>>
>> How do we handle writing logs to S3/GCS/Azure Blob storage, which
>> depends on the hook from the provider to work?
>>
>
> Good point. I will make sure that I address it in the PR as well. I think
> those should only work when
> the corresponding package his installed. And raise a warning otherwise.
>
An error at startup, I think would be better than an error.
>>
>> > Versioning proposal is CALVER with MAJOR AIRFLOW prefix (2.YYYY.MM.DD)
>> > for all 2.* Airflow releases. In the future 3.YYYY.MM.DD for all 3.*
>> > releases. Backwards incompatibility is only allowed between MAJOR
>> > Airflow Versions
>>
>> I'd personally prefer use to use SemVar to allow us to make the freedom
>> to make backwards-incompatible changes to a provider (within the same
>> major Airflow release). This is what HashiCorp have done with their
>> terraform providers modules.
>>
>
> This is detail that we can still decide on independently on voting.
> But in
> my opinion this
> one is super difficult (procedurally and regarding involvement of release
> management time
> and effort needed) if we want to release each provider separately.
>
> But I am open to it as well if you help to define how SEMVER implementation
> should look like and
> if we come up that does not require a heavy overhead of release management
> that cannot
> be automated.
>
> What is your proposal for the implementation of SEMVER in this case?
>
> I can see the following options (but if you have other proposal - I am
> happy to hear them):
>
> a) 2.0.N for all providers released in 2.0. But then according to semver
> those
> would be just bugfixes and no new features added) so technically it's not
> really SEMVER -
> it's basically the same as 2 + CALVER but with incrementing number instead
> of date.
>
> b) 2.0.X.Y incrementing X always when new features are added and Y's when
> there are bugfixes?
>
> c) 2.X.Y for all 2.* providers with X increasing when there are new
> features and Ys when there are bugfixes.
>
> I think, if we go for the b) c) who, when (and based on what criteria) will
> be deciding
> whether X or Y should be increased for those 60+operators? Would that be
> contributors committing the code or release manager releasing those
> packages?
> Note that the latter literally requires to review those all packages and
> decide on a
> case-by-case basis. I would never want to do it personally on regular basis.
>
> What do you think Ash? Which option do you propose?
I should preface this with my feeling isn't a strong desire, nor
blocking on the AIP being accepted.
I was thinking just a more strict SemVer (but starting at 2.0.0) -- i.e.
the providers version is not tied to the Airflow major version, so we
could have a provider at version 5.2.0 and it still works with Airflow
-- my thinking here was:
1. We start at 2.x just for the sake of it -- though it could just as
easily start every airflow-provider-* at 1.0.0 with the next release
2. We use the existing setup.py/python dep process to show limit what
version of apache-airflow a provider works with.
My main reason for thinking 2) is that I think it's more likely that the
providers will work with a hypotehtical Airflow 3 unchanged than not,
and so this avoids having to update/release a new provider version.
After all -- one of the goals of provider is to ease upgrades, so by
only having to upgrade core, and not upgrade the providers at the same
time makes it easier on users.
> Note that the latter literally requires to review those all packages and
> decide on a
> case-by-case basis.
Yes, this is a bit more work on the release manager, I accept that, but
we can use some automated process where we build up the changelog
programatically/from files.
For example, we could either use reno -- for example:
https://docs.openstack.org/magnum/pike/contributor/reno.html which lets
each PR specify the type of the change.
Or we could do the same thing but lighter weight -- a bunch of
Markdown/RST files in airflow/provider/x/changes/ that marks the type,
and then scripting we write ourselves to build up a CHANGELOG for each provider.
As I said above, I would _like_ us (the Airflow project) to do the work
of deciding SemVer/breaking changes, rather than each of our users
having to do this. But In this instance I am willing to be persauded otherwise.
-ash
>
> J.
>
>
>
>> -ash
>>
>> On Sep 13 2020, at 9:28 pm, Jarek Potiuk <[email protected]> wrote:
>>
>> > Hello Everyone,
>> >
>> > Last week, at the Airflow 2.0 meeting the people involved made a decision
>> > that we are releasing Airlow 2.0 as a set of separate "core" and
>> > "providers" packages - similarly to the 1.10 "backport providers"
>> packages.
>> >
>> > This decision effectively implements the "spirit" of the AIP-8
>> > proposed by
>> > Tim Swast in January 2019. It's not an exact implementation - we've
>> learned
>> > a lot since the original proposal and the final implementation is
>> slightly
>> > different - rather than "multiple" repositories we stay with the
>> mono-repo
>> > approach, but we aim to achieve many of the goals targeted by the AIP-8.
>> >
>> > I updated the original AIP-8
>> > <
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303
>> >
>> > to reflect those changes.
>> >
>> > This is the formal Vote for this proposal. It follows the "vote on code
>> > modification" policy of the ASF:
>> > https://www.apache.org/foundation/voting.html#votes-on-code-modification
>> >
>> > We need three +1 votes for the proposal to proceed. All committers
>> > have a
>> > binding vote.
>> >
>> > The vote will last 72 hours, which means that it ends on Wed 16th Sep
>> 2020,
>> > 10:30 PM CEST
>> >
>> > Consider this my binding +1.
>> >
>> > J
>> >
>> > --
>> > Jarek Potiuk
>> > Polidea | Principal Software Engineer
>> >
>> > M: +48 660 796 129
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
