That was exactly what I was proposing :). So we are in perfect sync here. On Tue, Oct 6, 2020 at 10:34 AM Kamil Breguła <[email protected]> wrote:
> Hi. > > I discussed this topic with Kaxil and we have a solution that could be > good for everyone. > > What do you think to release everything except Kubernetes now, and next > week two packages - Google and Kubernetes? This way, users will be able to > use all operators quickly and we will not have packages that have known > defects. > > Best regards, > Kamil Breguła > > On Tue, Oct 6, 2020 at 9:33 AM Jarek Potiuk <[email protected]> > wrote: > >> Kamil - I understand your concerns but we discussed it yesterday (at the >> 2.0 meeting - notes will be posted soon) that the whole point of the >> backport packages and separate providers is that we CAN release a subset of >> those separately. >> >> And the process that seems to be converging for releases is that we can >> exclude particular providers from the release without impacting >> the releasing of all the other providers. >> >> The only reason why we are releasing all the packages together is >> efficiency in voting (and we only release all of them together because we >> black-reformatted all of them and they have modifications that change >> source line numbers etc, so it makes sense to release all of them). >> >> I believe the -1 is not justified here. Can you please reconsider? I'd >> love we all agree on the process before I make the decision as release >> manager. >> >> Also just to remind the rules on package releases ( >> https://www.apache.org/foundation/voting.html) - voting on package >> releases cannot be vetoed: >> >> VOTES ON PACKAGE RELEASES >> >> Votes on whether a package is ready to be released use majority approval >> -- i.e. at least three PMC members must vote affirmatively for release, and >> there must be more positive than negative votes. Releases *may not be >> vetoed*. Generally the community will cancel the release vote if anyone >> identifies serious problems, but in most cases the ultimate decision, lies >> with the individual serving as release manager. The specifics of the >> process may vary from project to project, but the 'minimum quorum of three >> +1 votes' rule is universal. >> >> J. >> >> >> On Tue, Oct 6, 2020 at 12:14 AM Kamil Breguła <[email protected]> >> wrote: >> >>> Is there any time pressure or any other pressure for us to release this >>> package now? I think Kubernetes is highly anticipated by users and it is >>> worth considering the arguments that motivate us to make a given decision. >>> >>> -1 to release without cncf.kubernetes and an incomplete google package >>> (without GKEPodOperator). Sorry. >>> >>> On Mon, Oct 5, 2020 at 10:29 PM Kaxil Naik <[email protected]> wrote: >>> >>>> +1 to release without cncf.kubernetes >>>> >>>> On Mon, Oct 5, 2020 at 6:27 PM Jarek Potiuk <[email protected]> >>>> wrote: >>>> >>>>> (though of course releasing it without Kubernetes might be a little >>>>> less exciting Daniel ;) >>>>> >>>>> On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <[email protected]> >>>>> wrote: >>>>> >>>>>> We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet >>>>>> working. >>>>>> >>>>>> My proposal: >>>>>> >>>>>> * we continue with the voting (hopefully one more +1) and release all >>>>>> the sources + backport but do nto publish the cncf.kubernetes one. >>>>>> * we fix (who can take a look?) the cncf.kubernetes one and release >>>>>> and vote cncf.kubernetes separately. >>>>>> >>>>>> I think it will also happen in the future where we will be releasing >>>>>> a batch of providers and we detect problems with one of them. I think we >>>>>> need some agreed process for that and this might be what I explained. >>>>>> >>>>>> J, >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I see another problem. The kubernetes_engine.GKEStartPodOperator >>>>>>> operator is not included in the Google package and the kubernetes_engine >>>>>>> module contains references to the >>>>>>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator >>>>>>> operator instead of providers.cncf.kubernetes package. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I started testing and encountered three problems - Two minor and >>>>>>>> one serious. I used official Docker images for >>>>>>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE >>>>>>>> authorization >>>>>>>> >>>>>>>> 1. The configuration for StackdriverTaskHandlle is not obvious and >>>>>>>> is not documented anywhere. >>>>>>>> I managed to do this when I created the >>>>>>>> /opt/airflow/config/log_config.py file and set an environment variable >>>>>>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG >>>>>>>> File log_config.py: >>>>>>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5 >>>>>>>> >>>>>>>> However, I had more problems when I tried to use >>>>>>>> KubernetesPodOperator. >>>>>>>> >>>>>>>> 2. The following files are not included in the backport package >>>>>>>> cnd.kubernetes: >>>>>>>> - >>>>>>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py, >>>>>>>> >>>>>>>> - >>>>>>>> airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml >>>>>>>> >>>>>>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our >>>>>>>> repository and example DAG works >>>>>>>> >>>>>>>> 3. I haven't had such success with another example >>>>>>>> - example_kubernetes_operator. I performed the following steps which >>>>>>>> ended >>>>>>>> with an error. >>>>>>>> $ curl -LO >>>>>>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py >>>>>>>> $ cat example_kubernetes.py | sed >>>>>>>> 's/operators.bash/operators.bash_operator/g' | sed >>>>>>>> 's/in_cluster=True/in_cluster=False/g' > a.py >>>>>>>> $ mv a.py example_kubernetes.py >>>>>>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01 >>>>>>>> ..... >>>>>>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type >>>>>>>> object 'PodGenerator' has no attribute 'add_xcom_sidecar' >>>>>>>> Traceback (most recent call last): >>>>>>>> File >>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", >>>>>>>> line 984, in _run_raw_task >>>>>>>> result = task_copy.execute(context=context) >>>>>>>> File >>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", >>>>>>>> line 269, in execute >>>>>>>> self.pod = self.create_pod_request_obj() >>>>>>>> File >>>>>>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", >>>>>>>> line 405, in create_pod_request_obj >>>>>>>> pod = PodGenerator.add_xcom_sidecar(pod) >>>>>>>> AttributeError: type object 'PodGenerator' has no attribute >>>>>>>> 'add_xcom_sidecar' >>>>>>>> >>>>>>>> Have I missed any extra step I should do to run this example? >>>>>>>> >>>>>>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> +1 excited to finally encourage the Kubernetes providers! >>>>>>>>> >>>>>>>>> via Newton Mail >>>>>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2> >>>>>>>>> >>>>>>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>> Hey all, >>>>>>>>> >>>>>>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is >>>>>>>>> calling a vote on the release, >>>>>>>>> which will last for 72 hours - which means that it will end on >>>>>>>>> Mon, 5th of October 2020, 19:59:56 CEST. >>>>>>>>> >>>>>>>>> This is the first time we release "cncf.kubernetes" package after >>>>>>>>> big backporting of kubernetes changes >>>>>>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users >>>>>>>>> looking at the devlist test test this provider >>>>>>>>> - especially if you are waiting for it. >>>>>>>>> >>>>>>>>> Consider this my (binding) +1. >>>>>>>>> >>>>>>>>> Airflow Backport Providers 2020.10.5rc1 are available at: >>>>>>>>> >>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is >>>>>>>>> a source release that comes >>>>>>>>> with INSTALL instructions. >>>>>>>>> >>>>>>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz* >>>>>>>>> are the binary >>>>>>>>> Python "sdist" release. >>>>>>>>> >>>>>>>>> Public keys are available at: >>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS >>>>>>>>> >>>>>>>>> Please vote accordingly: >>>>>>>>> >>>>>>>>> [ ] +1 approve >>>>>>>>> [ ] +0 no opinion >>>>>>>>> [ ] -1 disapprove with the reason >>>>>>>>> >>>>>>>>> >>>>>>>>> Only votes from PMC members are binding, but members of the >>>>>>>>> community are >>>>>>>>> encouraged to test the release and vote with "(non-binding)". >>>>>>>>> >>>>>>>>> Please note that the version number excludes the 'rcX' string, so >>>>>>>>> it's now >>>>>>>>> simply 2020.10.5. This will allow us to rename the artifact >>>>>>>>> without modifying >>>>>>>>> the artifact checksums when we actually release. >>>>>>>>> >>>>>>>>> Each of the packages contains a detailed changelog. Here is the >>>>>>>>> list of links to >>>>>>>>> the released packages and changelogs: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/ >>>>>>>>> >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Jarek >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Jarek Potiuk >>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>>>>> >>>>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>>> >>>>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Jarek Potiuk >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>> >>>>>> M: +48 660 796 129 <+48660796129> >>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> Jarek Potiuk >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>> >>>>> M: +48 660 796 129 <+48660796129> >>>>> [image: Polidea] <https://www.polidea.com/> >>>>> >>>>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
