Hi Jarek, I think all of this sounds fine, but I think that we should start a new vote with this understanding. I wouldn't feel comfortable assuming that any of the previous +1's are still applicable as we have changed what people are +1’ing. At the minimum I think we could have people re-affirm their votes on this thread based on the new proposal Once we figure that out then +1 from me :) On Wed, Mar 3, 2021 at 1:44 PM, Jarek Potiuk <ja...@potiuk.com> wrote: Hello Everyone,
We need one more PMC member vote in order to be able to release the packages. Just to describe what the current status of the batch is: Based on discussion here: https://github.com/apache/airflow/issues/14511 [https://github.com/apache/airflow/issues/14511] - I am planning to follow the process we had previously documented in our release procedures so far - that release manager tries to batch a number of providers in single "voting process" and when there are problems discovered with certain providers they might be excluded. I plan to skip the following providers from release now and release them together on a ad-hoc basis whenever all the relevant issues are merged: * apache.druid, microsoft.azure, apache.beam I also noticed that snowflake python connector has been released this morning as promised https://pypi.org/project/snowflake-connector-python/2.4.0/ [https://pypi.org/project/snowflake-connector-python/2.4.0/] - it fixes the last problem with the dependencies that were plaguing us, so I also plan to remove the snowflake provider from this batch. --------- I just wanted to use the opportunity to describe the current process for deciding providers, because apparently not everyone is aware that we already have an established and documented process for provider's releases that was discussed on the devlist and it is documented in our release process description. Possibly we will extract it into a separate policy and maybe discuss some aspects of it (the discussion was raised today at the dev call of ours but I wanted to make sure that we are all referring to the same "starting point" and the "process" I based my recent actions on. 1) Batching the providers as the default The decision on when to release and particularly preference for releasing providers in batches is described in the https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release [https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#decide-when-to-release] > Decide when to release > You can release provider packages separately from the main Airflow on an > ad-hoc basis, whenever we find that a given provider needs to be released - due to new features or due to bug fixes. You can release each provider package separately, but due to voting and release overhead we try to group releases of provider packages together. 2) Possibility of excluding certain packages from the release. The possibility of excluding certain packages which (for whatever reason) we decide to remove at the discretion of release manager is described here: https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate [https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#prepare-voting-email-for-providers-release-candidate] Prepare voting email for Providers release candidate .... > Due to the nature of packages, not all packages have to be released as > convenience packages in the final release. During the voting process the > voting PMCs might decide to exclude certain packages from the release if some > critical problems have been found in some packages. And the process of removing it is part of the described release process: > In case you decided to remove some of the packages. remove them from dist > folder now: > ls dist/*<provider>* > rm dist/*<provider>* The issue of excluding certain packages have been discussed in this thread in the mailing list : https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E [https://lists.apache.org/thread.html/rc620a8a503cc7b14850c0a2a1fca1f6051d08a7e3e6d2cbdeb691dda%40%3Cdev.airflow.apache.org%3E] - where we had a -1 veto from a PMC member on the whole batch of providers where we found a cncf.kubrenetes and google providers had critical problems. We discussed it then and the two PMC members proposed a solution that was not objected to by anyone in the VOTE thread - to remove the packages from the batch. I continued this in the continuation of the voting thread https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E [https://lists.apache.org/thread.html/r752c5d5171de4ff626663d30e1d50c4b0d2994f66bf8918d816dabd8%40%3Cdev.airflow.apache.org%3E] with the following message which specifically pointed out to my proposal where I specifically linked to the message above and asked for comments. > As discussed before: -1 on a single provider does not invalidate the whole vote (from https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate [https://github.com/apache/airflow/tree/master/dev#vote-and-verify-the-backport-providers-release-candidate] ): > "Due to the nature of backport packages, not all packages have to be released as convenience packages in the final release. During the voting process the voting PMCs might decide to exclude certain packages from the release if some critical problems have been found in some packages." > We will merge the fix and most likely release a new google package right after this one. Looking at the super-localized problem here my current decision will be to release 2020.10.29 'google package" - together with other packages and release 2020.11.01 (or smth) - but only the google one - right after we merge the fix. > Any comments to that? J. On Wed, Mar 3, 2021 at 1:23 AM Kaxil Naik < kaxiln...@gmail.com [kaxiln...@gmail.com] > wrote: +1 (binding). Verified Signature and SHA12. Based on the changes (and Changelog) I can verify that the following providers should work fine: * spark * kubernetes * jenkins * microsoft.azure * mysql * telegram * and all the ones that just have doc changes Regards, Kaxil On Tue, Mar 2, 2021 at 9:01 PM Ryan Hatter < ryannhat...@gmail.com [ryannhat...@gmail.com] > wrote: There were some changes to the operator after my PR was merged: https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py [https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/transfers/gdrive_to_gcs.py] Pak Andrey (Scuall1992 on GitHub) might be able to confirm the operator is functional. On Mar 2, 2021, at 13:16, Jarek Potiuk < ja...@potiuk.com [ja...@potiuk.com] > wrote: Hello everyone - just a reminder that we have voting (hopefully) finishing tomorrow. I'd love to get some votes for that. Just to clarify what the PMC votes mean, because I believe there were some question raised about the release process which we are going to discuss it tomorrow at the dev call but let me just express my interpretation of https://infra.apache.org/release-publishing.html [https://infra.apache.org/release-publishing.html] PMC member vote (as I understand it) does not mean that this PMC member tested the release functionality (neither Release Manager). This merely means that the PMC member agrees that the software was released according to the requirements and process described in https://infra.apache.org/release-publishing.html [https://infra.apache.org/release-publishing.html] and that the signatures, hash-sums and software packages are as expected by the process. This is how I interpret this part of the release process "Release managers do the mechanical work; but the PMC in general, and the PMC chair in particular (as an officer of the Foundation), are responsible for compliance with ASF requirements." My understanding is that it is not feasible (neither for Airflow nor Providers) that the PMC members (nor release manager) tests the software and all features/bugfixes. We've never done that and I believe we will never do. We are reaching out to the community to test and we make a best effort to test whatever we release automatically (unit tests, integration tests, testing if providers are installable/importable with Airflow 2.0 and latest source code of Airflow). And we hardly can do more than that. Happy to discuss it tomorrow, but in the meantime If some of the PMCs could do the review of the process and check the compliance, to be ready to cast your votes - I'd love that. J. On Tue, Mar 2, 2021 at 8:44 PM Jarek Potiuk < ja...@potiuk.com [ja...@potiuk.com] > wrote: Hey Ryan, There is no **must** in re-testing it. Providing that you tested it before with real GSuite account is for me enough of a confirmation ;). J. On Sun, Feb 28, 2021 at 10:00 PM Abdur-Rahmaan Janhangeer < arj.pyt...@gmail.com [arj.pyt...@gmail.com] > wrote: Salutes for having a GSuite account just for the functionality 👍👍👍 On Mon, 1 Mar 2021, 00:05 Ryan Hatter, < ryannhat...@gmail.com [ryannhat...@gmail.com] > wrote: I canceled my GSuite account when my PR for the gdrive to gcs operator was approved & merged. Could anyone maybe help me ensure correct functionality? On Feb 27, 2021, at 08:48, Jarek Potiuk < ja...@potiuk.com [ja...@potiuk.com] > wrote: I created issue, where we will track the status of tests for the providers (again - it is experiment - but I'd really love to get feedback on the new providers from those who contributed): https://github.com/apache/airflow/issues/14511 [https://github.com/apache/airflow/issues/14511] On Sat, Feb 27, 2021 at 4:28 PM Jarek Potiuk < ja...@potiuk.com [ja...@potiuk.com] > wrote: Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 72 hours + day for the weekend - which means that it will end on Wed 3 Mar 15:59:34 CET 2021. Consider this my (binding) +1. KIND REQUEST There was a recent discussion about test quality of the providers and I would like to try to address it, still keeping the batch release process every 3 weeks. We need a bit of help from the community. I have a kind request to the authors of fixes and new features. I group the providers into those that likely need more testing, and those that do not. I also added names of those who submitted the changes and are most likely to be able to verify if the RC packages are solving the problems/adding features. This is a bit of experiment (apologies for calling out) - but if we find that it works, we can automate that. I will create a separate Issue in Github where you will be able to "tick" the boxes for those providers which they are added to. It would not be a blocker if not tested, but it will be a great help if you could test the new RC provider and see if it works as expected according to your changes. Providers with new features and fixes - likely needs some testing.: * amazon : Cristòfol Torrens, Ruben Laguna, Arati Nagmal, Ivica Kolenkaš, JavierLopezT * apache.druid : Xinbin Huang * apache.spark : Igor Khrol * cncf.kubernetes : jpyen, Ash Berlin-Taylor, Daniel Imberman * google : Vivek Bhojawala, Xinbin Huang, Pak Andrey, uma66, Ryan Yuan, morrme, Sam Wheating, YingyingPeng22, Ryan Hatter,Tobiasz Kędzierski * jenkins : Maxim Lisovsky * microsift.azure : flvndh, yyu * mysql : Constantino Schillebeeckx * qubole : Xinbin Huang * salesforce : Jyoti Dhiman * slack : Igor Khrol * tableau : Jyoti Dhiman * telegram : Shekhar Sing, Adil Khashtamov Providers with doc only changes (no need to test): * apache-beam * apache-hive * dingding * docker * elasticsearch * exasol * http * neo4j * openfaas * papermill * presto * sendgrid * sftp * snowflake * sqlite * ssh * Airflow Providers are available at: https://dist.apache.org/repos/dist/dev/airflow/providers/ [https://dist.apache.org/repos/dist/dev/airflow/providers/] *apache-airflow-providers-<PROVIDER>-*-bin.tar.gz* are the binary Python "sdist" release - they are also official "sources" for the provider packages. *apache_airflow_providers_<PROVIDER>-*.whl are the binary Python "wheel" release. The test procedure for PMC members who would like to test the RC candidates are described in https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members [https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-by-pmc-members] and for Contributors: https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors [https://github.com/apache/airflow/blob/master/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors] Public keys are available at: https://dist.apache.org/repos/dist/release/airflow/KEYS [https://dist.apache.org/repos/dist/release/airflow/KEYS] Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason Only votes from PMC members are binding, but members of the community are encouraged to test the release and vote with "(non-binding)". Please note that the version number excludes the 'rcX' string. This will allow us to rename the artifact without modifying the artifact checksums when we actually release. Each of the packages contains a link to the detailed changelog. The changelogs are moved to the official airflow documentation: https://github.com/apache/airflow-site/ [https://github.com/apache/airflow-site/] <TODO COPY LINK TO BRANCH> <PASTE ANY HIGH-LEVEL DESCRIPTION OF THE CHANGES HERE!> Note the links to documentation from PyPI packages are not working until we merge the changes to airflow site after releasing the packages officially. https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/ [https://pypi.org/project/apache-airflow-providers-amazon/1.2.0rc1/] https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/ [https://pypi.org/project/apache-airflow-providers-apache-beam/1.0.1rc1/] https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/ [https://pypi.org/project/apache-airflow-providers-apache-druid/1.1.0rc1/] https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-apache-hive/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-apache-spark/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-dingding/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-docker/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-elasticsearch/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-exasol/1.1.1rc1/] https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/ [https://pypi.org/project/apache-airflow-providers-google/2.1.0rc1/] https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-http/1.1.1rc1/] https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/ [https://pypi.org/project/apache-airflow-providers-jenkins/1.1.0rc1/] https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/ [https://pypi.org/project/apache-airflow-providers-microsoft-azure/1.2.0rc1/] https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-mysql/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/ [https://pypi.org/project/apache-airflow-providers-neo4j/1.0.1rc1/] https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-openfaas/1.1.1rc1/] https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-papermill/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-presto/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-qubole/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/ [https://pypi.org/project/apache-airflow-providers-salesforce/2.0.0rc1/] https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-sendgrid/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-sftp/1.1.1rc1/] https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/ [https://pypi.org/project/apache-airflow-providers-slack/3.0.0rc1/] https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/ [https://pypi.org/project/apache-airflow-providers-snowflake/1.1.1rc1/] https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-sqlite/1.0.2rc1/] https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/ [https://pypi.org/project/apache-airflow-providers-ssh/1.2.0rc1/] https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/ [https://pypi.org/project/apache-airflow-providers-tableau/1.0.0rc1/] https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/ [https://pypi.org/project/apache-airflow-providers-telegram/1.0.2rc1/] Cheers, J. -- +48 660 796 129 -- +48 660 796 129 -- +48 660 796 129 -- +48 660 796 129 -- +48 660 796 129
