HAH.... Again canceling... seems like a routine now :). Seems that enough people (everyone who spoke) care for consistency and since (as discussed in https://lists.apache.org/thread.html/r456565d372dcd7199d5ca34881d43911bf2c6ce623e515252a3aec73%40%3Cdev.airflow.apache.org%3E ) we found an inconsistency in the names of transfer operators, the RC3 for backport operators is cancelled.
It's likely we will have another RC on Tuesday, It seems that we are moving away from the calver date 2020.5.20 so maybe we should consider abandoning 2020.5.20 and releasing a new RC1 candidate with a new date - 2020.06.05rc1 to be precise? Hopefully future releases will be much much faster :) WDYT? J. On Fri, May 29, 2020 at 1:53 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > Thanks Jarek :) > > On Fri, May 29, 2020 at 12:46 PM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > Absolutely. This discussion made me think we want to work on RC4 :). I > > think that's better to clarify it now since this is something that stay > > with us for quite some time. I will send a separate vote shortly. > > > > J. > > > > On Fri, May 29, 2020 at 1:11 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > We might want to create a separate DISCUSS / VOTING Thread for > > > https://apache-airflow.slack.com/archives/CCPRP7943/p1590742848402600 > > > > > > i.e. Naming convention for the AToB Operators or Transfer Operators > > > > > > I think we should wait for the results of that before we release the > > > Backport Packages i.e. create RC4 :( > > > > > > What do you think? > > > > > > Regards, > > > Kaxil > > > > > > > > > > > > On Fri, May 29, 2020 at 11:34 AM Jarek Potiuk < > jarek.pot...@polidea.com> > > > wrote: > > > > > > > Fixed :). Shift + Refresh > > > > > > > > On Fri, May 29, 2020 at 12:26 PM Jarek Potiuk < > > jarek.pot...@polidea.com> > > > > wrote: > > > > > > > > > Sha of sha .... Wow.. That's something like master level of > security > > > :). > > > > > > > > > > On Fri, May 29, 2020 at 10:48 AM Kaxil Naik <kaxiln...@gmail.com> > > > wrote: > > > > > > > > > >> I think we have some extra files at > > > > >> > > > > >> > > > > > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/ > > > > >> (maybe > > > > >> erroneously generated by the Bash scripts) > > > > >> > > > > >> Examples: > > > > >> > > > > >> - > > > > >> > > > > >> > > > > > > > > > > apache-airflow-backport-providers-amazon-2020.5.20rc3-bin.tar.gz.asc.sha512.sha512 > > > > >> < > > > > >> > > > > > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/apache-airflow-backport-providers-amazon-2020.5.20rc3-bin.tar.gz.asc.sha512.sha512 > > > > >> > > > > > >> - > > > > >> > > > > >> > > > > > > > > > > apache-airflow-backport-providers-amazon-2020.5.20rc3-bin.tar.gz.sha512.sha512.sha512 > > > > >> < > > > > >> > > > > > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/apache-airflow-backport-providers-amazon-2020.5.20rc3-bin.tar.gz.sha512.sha512.sha512 > > > > >> > > > > > >> - > > > > >> > > > > >> > > > > > > > > > > apache-airflow-backport-providers-apache-cassandra-2020.5.20rc3-bin.tar.gz.sha512.sha512.sha512 > > > > >> < > > > > >> > > > > > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/apache-airflow-backport-providers-apache-cassandra-2020.5.20rc3-bin.tar.gz.sha512.sha512.sha512 > > > > >> > > > > > >> > > > > >> > > > > >> Can we remove those first, please, they are not a blocker to the > > > release > > > > >> but makes it difficult to view the actual packages? > > > > >> > > > > >> Regards, > > > > >> Kaxil > > > > >> > > > > >> On Thu, May 28, 2020 at 6:22 PM Jarek Potiuk < > > > jarek.pot...@polidea.com> > > > > >> wrote: > > > > >> > > > > >> > Looking forward to some PMC votes :) But happy to hear more > > feedback > > > > >> from > > > > >> > others! > > > > >> > > > > > >> > Just a kind reminder (not sure if visible to everyone but let me > > > try): > > > > >> > > > > > >> > [image: majestic_rabbit.jpg] > > > > >> > > > > > >> > Fri May 29 22:42:38 CEST 2020. > > > > >> > > > > > >> > J. > > > > >> > > > > > >> > > > > > >> > On Tue, May 26, 2020 at 11:47 PM Jarek Potiuk < > > > > jarek.pot...@polidea.com > > > > >> > > > > > >> > wrote: > > > > >> > > > > > >> >> Indeed What an oversight :). Thanks for +1 - even non-binding > is > > > > super > > > > >> >> important :) Anyone else trying the backport packages is great > - > > I > > > > >> think > > > > >> >> it's super important for the Airflow community and great step > to > > > ease > > > > >> >> migration to 2.0 :D > > > > >> >> > > > > >> >> J. > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> On Tue, May 26, 2020 at 11:12 PM Bas Harenslak > > > > >> >> <basharens...@godatadriven.com.invalid> wrote: > > > > >> >> > > > > >> >>> Thanks for the compliment but I’m no PMC so here’s my > > non-binding > > > +1 > > > > >> :-) > > > > >> >>> > > > > >> >>> Bas > > > > >> >>> > > > > >> >>> On 26 May 2020, at 22:43, Jarek Potiuk < > > jarek.pot...@polidea.com > > > > >> <mailto: > > > > >> >>> jarek.pot...@polidea.com>> wrote: > > > > >> >>> > > > > >> >>> Hey all, > > > > >> >>> > > > > >> >>> I have cut Airflow Backport Providers 2020.5.20rc3. This email > > is > > > > >> >>> calling a > > > > >> >>> vote on the release, > > > > >> >>> which will last for 72 hours - which means that it will end on > > Fri > > > > >> May 29 > > > > >> >>> 22:42:38 CEST 2020. > > > > >> >>> > > > > >> >>> Consider this my (binding) +1. > > > > >> >>> > > > > >> >>> The changes vs. rc2 are explained at the end of the email. > > > > >> >>> > > > > >> >>> Airflow Backport Providers 2020.5.20rc3 are available at: > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> *apache-airflow-backport-providers-2020.5.20rc3-source.tar.gz* > > is > > > a > > > > >> >>> source > > > > >> >>> release that comes > > > > >> >>> with INSTALL instructions. > > > > >> >>> > > > > >> >>> > > > > > *apache-airflow-backport-providers-<PROVIDER>-2020.5.20rc3-bin.tar.gz* > > > > >> >>> are > > > > >> >>> the binary > > > > >> >>> Python "sdist" release. > > > > >> >>> > > > > >> >>> Public keys are available at: > > > > >> >>> https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > >> >>> > > > > >> >>> Please vote accordingly: > > > > >> >>> > > > > >> >>> [ ] +1 approve > > > > >> >>> [ ] +0 no opinion > > > > >> >>> [ ] -1 disapprove with the reason > > > > >> >>> > > > > >> >>> > > > > >> >>> Only votes from PMC members are binding, but members of the > > > > community > > > > >> are > > > > >> >>> encouraged to test the release and vote with "(non-binding)". > > > > >> >>> > > > > >> >>> Please note that the version number excludes the 'rcX' string, > > so > > > > it's > > > > >> >>> now > > > > >> >>> simply 2020.5.20. This will allow us to rename the artifact > > > without > > > > >> >>> modifying > > > > >> >>> the artifact checksums when we actually release it. > > > > >> >>> > > > > >> >>> Each of the packages contains a detailed changelog. Here is > the > > > list > > > > >> of > > > > >> >>> links to > > > > >> >>> the released packages and changelogs: > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-celery/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-discord/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-docker/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-email/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-google/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-http/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-imap/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-jira/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-presto/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-redis/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-samba/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-segment/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-slack/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> > > > > >> > > > > > > > > > > https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.5.20rc3/ > > > > >> >>> > > > > >> >>> J. > > > > >> >>> > > > > >> >>> > > > > >> >>> Appendix: The changes vs. RC2: > > > > >> >>> > > > > >> >>> * During testing we found a few (20 out of 537) > inconsistencies > > in > > > > the > > > > >> >>> names of the classes vs. the agreed AIP-21 standard. We fixed > > them > > > > all > > > > >> >>> and > > > > >> >>> also introduced automated tests that will fail PR in case > > someone > > > > >> adds an > > > > >> >>> inconsistently named class. > > > > >> >>> > > > > >> >>> * Several packages failed to import cleanly in 1.10. That was > a > > > > great > > > > >> >>> idea > > > > >> >>> from one of the PMCs (Bas Hanrenslak) to automatically import > > all > > > > the > > > > >> >>> classes from all the providers and see if they are importable > in > > > > >> 1.10. We > > > > >> >>> identified a few operators in a few providers (papermill, > odbc, > > > > hive, > > > > >> >>> and a > > > > >> >>> few others) that would not import properly on 1.10. This is > all > > > now > > > > >> fixed > > > > >> >>> (except papermill - see below). The good thing is that we also > > > added > > > > >> >>> automated tests for this: so we will be testing the > > importability > > > of > > > > >> >>> backported packages with every PR. This means that the > backport > > > > >> packages > > > > >> >>> will continue to be importable in the future and we check it > > with > > > > >> every > > > > >> >>> single PR! > > > > >> >>> > > > > >> >>> Of course, importability does not guarantee that the operators > > > will > > > > >> work > > > > >> >>> with !00% certainty - this can only be assured by automated > > system > > > > >> tests > > > > >> >>> - > > > > >> >>> this is something we will be working on the coming months. > > > > >> >>> > > > > >> >>> * Papermill operator has proven to be problematic. It uses the > > > > Airflow > > > > >> >>> 2.0 > > > > >> >>> lineage functionality that is not compatible with 1.10. That's > > why > > > > we > > > > >> >>> decided to remove papermill from the released backport > packages > > > and > > > > >> >>> remove > > > > >> >>> the previous RCs for it. > > > > >> >>> > > > > >> >>> * The most important reason why we needed RC3. We noticed that > > > while > > > > >> the > > > > >> >>> README package instructions were all fine, the CHANGELOG and > > > INSTALL > > > > >> in > > > > >> >>> source packages contained changelog/Install instructions for > > > Airflow > > > > >> 2.0. > > > > >> >>> We fixed that and the INSTALL and CHANGELOG in the packages > > > contain > > > > >> >>> instructions/changelog for the backport packages only, not for > > the > > > > >> whole > > > > >> >>> Airflow. > > > > >> >>> > > > > >> >>> Cheers, > > > > >> >>> > > > > >> >>> Jarek > > > > >> >>> > > > > >> >>> -- > > > > >> >>> > > > > >> >>> Jarek Potiuk > > > > >> >>> Polidea <https://www.polidea.com/> | Principal Software > > Engineer > > > > >> >>> > > > > >> >>> M: +48 660 796 129 <+48660796129> > > > > >> >>> [image: Polidea] <https://www.polidea.com/> > > > > >> >>> > > > > >> >>> > > > > >> >> > > > > >> >> -- > > > > >> >> > > > > >> >> Jarek Potiuk > > > > >> >> Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > > >> >> > > > > >> >> M: +48 660 796 129 <+48660796129> > > > > >> >> [image: Polidea] <https://www.polidea.com/> > > > > >> >> > > > > >> >> > > > > >> > > > > > >> > -- > > > > >> > > > > > >> > Jarek Potiuk > > > > >> > Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > > >> > > > > > >> > M: +48 660 796 129 <+48660796129> > > > > >> > [image: Polidea] <https://www.polidea.com/> > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > -- > > > > > > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
