Thanks Jarek :) On Fri, May 29, 2020 at 12:46 PM Jarek Potiuk <[email protected]> wrote:
> Absolutely. This discussion made me think we want to work on RC4 :). I > think that's better to clarify it now since this is something that stay > with us for quite some time. I will send a separate vote shortly. > > J. > > On Fri, May 29, 2020 at 1:11 PM Kaxil Naik <[email protected]> wrote: > > > We might want to create a separate DISCUSS / VOTING Thread for > > https://apache-airflow.slack.com/archives/CCPRP7943/p1590742848402600 > > > > i.e. Naming convention for the AToB Operators or Transfer Operators > > > > I think we should wait for the results of that before we release the > > Backport Packages i.e. create RC4 :( > > > > What do you think? > > > > Regards, > > Kaxil > > > > > > > > On Fri, May 29, 2020 at 11:34 AM Jarek Potiuk <[email protected]> > > wrote: > > > > > Fixed :). Shift + Refresh > > > > > > On Fri, May 29, 2020 at 12:26 PM Jarek Potiuk < > [email protected]> > > > wrote: > > > > > > > Sha of sha .... Wow.. That's something like master level of security > > :). > > > > > > > > On Fri, May 29, 2020 at 10:48 AM Kaxil Naik <[email protected]> > > wrote: > > > > > > > >> I think we have some extra files at > > > >> > > > >> > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/ > > > >> (maybe > > > >> erroneously generated by the Bash scripts) > > > >> > > > >> Examples: > > > >> > > > >> - > > > >> > > > >> > > > > > > apache-airflow-backport-providers-amazon-2020.5.20rc3-bin.tar.gz.asc.sha512.sha512 > > > >> < > > > >> > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/apache-airflow-backport-providers-amazon-2020.5.20rc3-bin.tar.gz.asc.sha512.sha512 > > > >> > > > > >> - > > > >> > > > >> > > > > > > apache-airflow-backport-providers-amazon-2020.5.20rc3-bin.tar.gz.sha512.sha512.sha512 > > > >> < > > > >> > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/apache-airflow-backport-providers-amazon-2020.5.20rc3-bin.tar.gz.sha512.sha512.sha512 > > > >> > > > > >> - > > > >> > > > >> > > > > > > apache-airflow-backport-providers-apache-cassandra-2020.5.20rc3-bin.tar.gz.sha512.sha512.sha512 > > > >> < > > > >> > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/apache-airflow-backport-providers-apache-cassandra-2020.5.20rc3-bin.tar.gz.sha512.sha512.sha512 > > > >> > > > > >> > > > >> > > > >> Can we remove those first, please, they are not a blocker to the > > release > > > >> but makes it difficult to view the actual packages? > > > >> > > > >> Regards, > > > >> Kaxil > > > >> > > > >> On Thu, May 28, 2020 at 6:22 PM Jarek Potiuk < > > [email protected]> > > > >> wrote: > > > >> > > > >> > Looking forward to some PMC votes :) But happy to hear more > feedback > > > >> from > > > >> > others! > > > >> > > > > >> > Just a kind reminder (not sure if visible to everyone but let me > > try): > > > >> > > > > >> > [image: majestic_rabbit.jpg] > > > >> > > > > >> > Fri May 29 22:42:38 CEST 2020. > > > >> > > > > >> > J. > > > >> > > > > >> > > > > >> > On Tue, May 26, 2020 at 11:47 PM Jarek Potiuk < > > > [email protected] > > > >> > > > > >> > wrote: > > > >> > > > > >> >> Indeed What an oversight :). Thanks for +1 - even non-binding is > > > super > > > >> >> important :) Anyone else trying the backport packages is great - > I > > > >> think > > > >> >> it's super important for the Airflow community and great step to > > ease > > > >> >> migration to 2.0 :D > > > >> >> > > > >> >> J. > > > >> >> > > > >> >> > > > >> >> > > > >> >> On Tue, May 26, 2020 at 11:12 PM Bas Harenslak > > > >> >> <[email protected]> wrote: > > > >> >> > > > >> >>> Thanks for the compliment but I’m no PMC so here’s my > non-binding > > +1 > > > >> :-) > > > >> >>> > > > >> >>> Bas > > > >> >>> > > > >> >>> On 26 May 2020, at 22:43, Jarek Potiuk < > [email protected] > > > >> <mailto: > > > >> >>> [email protected]>> wrote: > > > >> >>> > > > >> >>> Hey all, > > > >> >>> > > > >> >>> I have cut Airflow Backport Providers 2020.5.20rc3. This email > is > > > >> >>> calling a > > > >> >>> vote on the release, > > > >> >>> which will last for 72 hours - which means that it will end on > Fri > > > >> May 29 > > > >> >>> 22:42:38 CEST 2020. > > > >> >>> > > > >> >>> Consider this my (binding) +1. > > > >> >>> > > > >> >>> The changes vs. rc2 are explained at the end of the email. > > > >> >>> > > > >> >>> Airflow Backport Providers 2020.5.20rc3 are available at: > > > >> >>> > > > >> >>> > > > >> > > > > > > https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc3/ > > > >> >>> > > > >> >>> *apache-airflow-backport-providers-2020.5.20rc3-source.tar.gz* > is > > a > > > >> >>> source > > > >> >>> release that comes > > > >> >>> with INSTALL instructions. > > > >> >>> > > > >> >>> > > > *apache-airflow-backport-providers-<PROVIDER>-2020.5.20rc3-bin.tar.gz* > > > >> >>> are > > > >> >>> the binary > > > >> >>> Python "sdist" release. > > > >> >>> > > > >> >>> Public keys are available at: > > > >> >>> https://dist.apache.org/repos/dist/release/airflow/KEYS > > > >> >>> > > > >> >>> Please vote accordingly: > > > >> >>> > > > >> >>> [ ] +1 approve > > > >> >>> [ ] +0 no opinion > > > >> >>> [ ] -1 disapprove with the reason > > > >> >>> > > > >> >>> > > > >> >>> Only votes from PMC members are binding, but members of the > > > community > > > >> are > > > >> >>> encouraged to test the release and vote with "(non-binding)". > > > >> >>> > > > >> >>> Please note that the version number excludes the 'rcX' string, > so > > > it's > > > >> >>> now > > > >> >>> simply 2020.5.20. This will allow us to rename the artifact > > without > > > >> >>> modifying > > > >> >>> the artifact checksums when we actually release it. > > > >> >>> > > > >> >>> Each of the packages contains a detailed changelog. Here is the > > list > > > >> of > > > >> >>> links to > > > >> >>> the released packages and changelogs: > > > >> >>> > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-celery/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-discord/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-docker/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-email/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-google/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-http/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-imap/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-jira/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-presto/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-redis/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-samba/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-segment/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-slack/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.5.20rc3/ > > > >> >>> > > > >> >>> > > > >> > > > > > > https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.5.20rc3/ > > > >> >>> > > > >> >>> J. > > > >> >>> > > > >> >>> > > > >> >>> Appendix: The changes vs. RC2: > > > >> >>> > > > >> >>> * During testing we found a few (20 out of 537) inconsistencies > in > > > the > > > >> >>> names of the classes vs. the agreed AIP-21 standard. We fixed > them > > > all > > > >> >>> and > > > >> >>> also introduced automated tests that will fail PR in case > someone > > > >> adds an > > > >> >>> inconsistently named class. > > > >> >>> > > > >> >>> * Several packages failed to import cleanly in 1.10. That was a > > > great > > > >> >>> idea > > > >> >>> from one of the PMCs (Bas Hanrenslak) to automatically import > all > > > the > > > >> >>> classes from all the providers and see if they are importable in > > > >> 1.10. We > > > >> >>> identified a few operators in a few providers (papermill, odbc, > > > hive, > > > >> >>> and a > > > >> >>> few others) that would not import properly on 1.10. This is all > > now > > > >> fixed > > > >> >>> (except papermill - see below). The good thing is that we also > > added > > > >> >>> automated tests for this: so we will be testing the > importability > > of > > > >> >>> backported packages with every PR. This means that the backport > > > >> packages > > > >> >>> will continue to be importable in the future and we check it > with > > > >> every > > > >> >>> single PR! > > > >> >>> > > > >> >>> Of course, importability does not guarantee that the operators > > will > > > >> work > > > >> >>> with !00% certainty - this can only be assured by automated > system > > > >> tests > > > >> >>> - > > > >> >>> this is something we will be working on the coming months. > > > >> >>> > > > >> >>> * Papermill operator has proven to be problematic. It uses the > > > Airflow > > > >> >>> 2.0 > > > >> >>> lineage functionality that is not compatible with 1.10. That's > why > > > we > > > >> >>> decided to remove papermill from the released backport packages > > and > > > >> >>> remove > > > >> >>> the previous RCs for it. > > > >> >>> > > > >> >>> * The most important reason why we needed RC3. We noticed that > > while > > > >> the > > > >> >>> README package instructions were all fine, the CHANGELOG and > > INSTALL > > > >> in > > > >> >>> source packages contained changelog/Install instructions for > > Airflow > > > >> 2.0. > > > >> >>> We fixed that and the INSTALL and CHANGELOG in the packages > > contain > > > >> >>> instructions/changelog for the backport packages only, not for > the > > > >> whole > > > >> >>> Airflow. > > > >> >>> > > > >> >>> Cheers, > > > >> >>> > > > >> >>> Jarek > > > >> >>> > > > >> >>> -- > > > >> >>> > > > >> >>> Jarek Potiuk > > > >> >>> Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > >> >>> > > > >> >>> M: +48 660 796 129 <+48660796129> > > > >> >>> [image: Polidea] <https://www.polidea.com/> > > > >> >>> > > > >> >>> > > > >> >> > > > >> >> -- > > > >> >> > > > >> >> Jarek Potiuk > > > >> >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >> >> > > > >> >> M: +48 660 796 129 <+48660796129> > > > >> >> [image: Polidea] <https://www.polidea.com/> > > > >> >> > > > >> >> > > > >> > > > > >> > -- > > > >> > > > > >> > Jarek Potiuk > > > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >> > > > > >> > M: +48 660 796 129 <+48660796129> > > > >> > [image: Polidea] <https://www.polidea.com/> > > > >> > > > > >> > > > > >> > > > > > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
