> Do we want strict testing when we release a provider for the first time?
I believe we should ask contributors to confirm that they tested the provider E2E (or at least some parts of it). But it should not be required (resources are expensive, OSS is free). As long as there's no E2E (aka system) test for every operator, running from time to time we won't be able to assure that all works as expected. But that's ok. If something was broken, that's good because we know users use the code. We can fix it and the ad-hoc approach for releases seems to cover this case. My personal belief is that we won't be able to test the operators/hooks as well as our users will do. So in my opinion the best thing to do is to encourage users to help with testing. Cheers, Tomek On Sat, 3 Apr 2021 at 02:07, Elad Kalif <elad...@apache.org> wrote: > > I agree with Kaxil about testing. > I'm wondering if we should have a special case for testing a new provider. > Do we want strict testing when we release a provider for the first time? > > > > On Sat, Apr 3, 2021 at 2:24 AM Kaxil Naik <kaxiln...@gmail.com> wrote: >> >> Thanks all, looks like we have an agreement on 3 points, I have created a PR >> to add those in Project Guidelines: >> https://github.com/apache/airflow/pull/15168 >> >> For testing, I have not received enough feedback yet. Only Tomek and I have >> said something about it yet. I will wait for a week to hear any other >> thoughts on it before a create another PR to say "Not all changes need >> strict testing, "best-effort" and judgement is fine enough for providers >> because of the low-risk nature of them." >> >> Regards, >> Kaxil >> >> On Thu, Mar 11, 2021 at 10:37 AM Ash Berlin-Taylor <a...@apache.org> wrote: >>> >>> Not releasing doc only changes sounds like a better solution, and fair >>> point about SemVer, I was just thinking about what is possible with python >>> versions. >>> >>> On Tue, 9 Mar, 2021 at 22:42, Vikram Koka <vik...@astronomer.io.INVALID> >>> wrote: >>> >>> I agree with Batch vs Ad-hoc and Frequency. >>> >>> For doc-only changes, I would prefer NOT to change the version. Primarily >>> because of the end user perspective, as was articulated earlier in the >>> thread. >>> >>> Best regards, >>> Vikram >>> >>> >>> On Tue, Mar 9, 2021 at 6:05 PM Jarek Potiuk <ja...@potiuk.com> wrote: >>>> >>>> Fully agree batch and ad-hoc approach as well as with the frequency. >>>> >>>> For docs-only changes I think -post release in PyPI is not a good idea. It >>>> does not follow semver (https://semver.org/). The only way you can extend >>>> the number in SEMVER is for pre-releases: >>>> >>>> > A pre-release version MAY be denoted by appending a hyphen and a series >>>> > of dot separated identifiers immediately following the patch version. >>>> > Identifiers MUST comprise only ASCII alphanumerics and hyphens >>>> > [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST >>>> > NOT include leading zeroes. Pre-release versions have a lower precedence >>>> > than the associated normal version. A pre-release version indicates that >>>> > the version is unstable and might not satisfy the intended compatibility >>>> > requirements as denoted by its associated normal version. Examples: >>>> > 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–. >>>> >>>> Not only violating the semver but this might break a number of tools. >>>> >>>> However, I think we can do a different thing in this case. I do not think >>>> we strictly need to release the packages when we see doc-only change. What >>>> we can do - we can simply tag it with *-doc1, *-doc2 tags in the repo and >>>> add some logic to "skip" such doc-only commits next time when we prepare >>>> packages. >>>> Then we can simply skip (automatically) building/releasing/voting doc-only >>>> packages at all, However we will continue with documentation and only >>>> release the documentation (using the existing version). >>>> >>>> Unlike packages, our documentation is mutable and we can override it. >>>> >>>> J. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Mar 9, 2021 at 7:52 PM Tomasz Urbaszek <turbas...@apache.org> >>>> wrote: >>>>> >>>>> I'm ok with batch and ad-hoc approach as well as with the frequency. >>>>> >>>>> In case of docs-only changes I second what Ash proposed - let's not alter >>>>> the version. 100% of functionality is the same so users are not affected >>>>> and there's no need to update. >>>>> >>>>> As per testing I agree that E2E testing of all providers should not be >>>>> necessary to cast +1 vote. The point about ad-hoc releases allows us to >>>>> release a fix to a provider if users find something that is broken beyond >>>>> acceptance. >>>>> >>>>> Tomek >>>>> >>>>> >>>>> On Tue, 9 Mar 2021 at 11:15, Ash Berlin-Taylor <a...@apache.org> wrote: >>>>>> >>>>>> For doc-only changes there is one extra thing to decide: >>>>>> >>>>>> Should we do it as a patch version (1.0.2 etc) or as a "post" release? >>>>>> Python packages have this concept: >>>>>> https://www.python.org/dev/peps/pep-0440/#post-releases >>>>>> >>>>>> > Some projects use post-releases to address minor errors in a final >>>>>> > release that do not affect the distributed software (for example, >>>>>> > correcting an error in the release notes). >>>>>> >>>>>> So we could update our tooling to support this, or we could just bump >>>>>> the patch version and release re-release it. >>>>>> >>>>>> I have an ever so slight preference for doc only changes to be done this >>>>>> way, as I think is clearer to users that they don't have to update. >>>>>> >>>>>> What does everyone else think? >>>>>> >>>>>> -ash >>>>>> >>>>>> On Mon, 8 Mar, 2021 at 22:38, Kaxil Naik <kaxiln...@apache.org> wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I would like to open the release process of providers up for discussion. >>>>>> Testing and Voting needs more discussion, the other points are mostly >>>>>> straight-forward and had an agreement on the last dev call. >>>>>> >>>>>> (Backport Providers won't be released after the end of this month - link >>>>>> so let's ignore them in conversation) >>>>>> >>>>>> Batch vs Ad-hoc: >>>>>> >>>>>> Release Manager would default to releasing Providers in Batch >>>>>> ad-hoc releases are OK (i.e. if there is a critical bug that needs >>>>>> fixing in a single provider) >>>>>> >>>>>> Frequency: >>>>>> >>>>>> For Batch release, we will release every month (starting of the month - >>>>>> 1 to 7 most likely) >>>>>> Just a note that it generally takes around a week for the vote to pass >>>>>> even though we have 72 hours minimum period >>>>>> >>>>>> Doc-only changes >>>>>> >>>>>> When we have doc-only changes for Providers (during batch-release), we >>>>>> should still release a new version. The majority on the Dev call had >>>>>> agreed that releasing docs asap is good instead of waiting for the next >>>>>> release with a code-change. >>>>>> >>>>>> Testing >>>>>> >>>>>> License and Signature Checks are mandatory (following the ASF rules) >>>>>> For Providers, not all changes require strict testing -- you make a >>>>>> judgement based on the changes for a particular provider >>>>>> Wherever possible community can help test those but not strictly >>>>>> necessary. This is where more discussion is needed. >>>>>> >>>>>> Voting >>>>>> >>>>>> Automate and create separate voting threads to avoid confusions vs a >>>>>> single vote where we exclude the providers if someone finds a bug, let's >>>>>> keep the discussion for this on >>>>>> https://lists.apache.org/thread.html/r9dcc03840f478669e9bd0dc61f4088b725097da2b48ea274b7f0593e%40%3Cdev.airflow.apache.org%3E >>>>>> >>>>>> Regards, >>>>>> Kaxil >>>>>> >>>> >>>> >>>> -- >>>> +48 660 796 129
