+1 to release without cncf.kubernetes On Mon, Oct 5, 2020 at 6:27 PM Jarek Potiuk <[email protected]> wrote:
> (though of course releasing it without Kubernetes might be a little less > exciting Daniel ;) > > On Mon, Oct 5, 2020 at 7:25 PM Jarek Potiuk <[email protected]> > wrote: > >> We have 2 +1s now and the comment about the 'cncf.kubernetes" not yet >> working. >> >> My proposal: >> >> * we continue with the voting (hopefully one more +1) and release all the >> sources + backport but do nto publish the cncf.kubernetes one. >> * we fix (who can take a look?) the cncf.kubernetes one and release and >> vote cncf.kubernetes separately. >> >> I think it will also happen in the future where we will be releasing a >> batch of providers and we detect problems with one of them. I think we need >> some agreed process for that and this might be what I explained. >> >> J, >> >> >> >> On Mon, Oct 5, 2020 at 1:56 AM Kamil Breguła <[email protected]> >> wrote: >> >>> I see another problem. The kubernetes_engine.GKEStartPodOperator >>> operator is not included in the Google package and the kubernetes_engine >>> module contains references to the >>> airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator >>> operator instead of providers.cncf.kubernetes package. >>> >>> >>> >>> On Mon, Oct 5, 2020 at 1:47 AM Kamil Breguła <[email protected]> >>> wrote: >>> >>>> I started testing and encountered three problems - Two minor and one >>>> serious. I used official Docker images for >>>> testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization >>>> >>>> 1. The configuration for StackdriverTaskHandlle is not obvious and is >>>> not documented anywhere. >>>> I managed to do this when I created the >>>> /opt/airflow/config/log_config.py file and set an environment variable >>>> - AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=log_config.LOGGING_CONFIG >>>> File log_config.py: >>>> https://gist.github.com/mik-laj/f94e9940bfd3f08cf69287ac62652bc5 >>>> >>>> However, I had more problems when I tried to use KubernetesPodOperator. >>>> >>>> 2. The following files are not included in the backport package >>>> cnd.kubernetes: >>>> - >>>> airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py, >>>> >>>> - >>>> airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_spark_pi.yaml >>>> >>>> I downloaded file /example_spark_kubernetes_spark_pi.yaml from our >>>> repository and example DAG works >>>> >>>> 3. I haven't had such success with another example >>>> - example_kubernetes_operator. I performed the following steps which ended >>>> with an error. >>>> $ curl -LO >>>> https://raw.githubusercontent.com/apache/airflow/master/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py >>>> $ cat example_kubernetes.py | sed >>>> 's/operators.bash/operators.bash_operator/g' | sed >>>> 's/in_cluster=True/in_cluster=False/g' > a.py >>>> $ mv a.py example_kubernetes.py >>>> $ airflow test example_kubernetes_operator write-xcom 2010-01-01 >>>> ..... >>>> [2020-10-04 23:29:06,623] {taskinstance.py:1150} ERROR - type object >>>> 'PodGenerator' has no attribute 'add_xcom_sidecar' >>>> Traceback (most recent call last): >>>> File >>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", >>>> line 984, in _run_raw_task >>>> result = task_copy.execute(context=context) >>>> File >>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", >>>> line 269, in execute >>>> self.pod = self.create_pod_request_obj() >>>> File >>>> "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", >>>> line 405, in create_pod_request_obj >>>> pod = PodGenerator.add_xcom_sidecar(pod) >>>> AttributeError: type object 'PodGenerator' has no attribute >>>> 'add_xcom_sidecar' >>>> >>>> Have I missed any extra step I should do to run this example? >>>> >>>> On Sun, Oct 4, 2020 at 7:11 PM Daniel Imberman < >>>> [email protected]> wrote: >>>> >>>>> +1 excited to finally encourage the Kubernetes providers! >>>>> >>>>> via Newton Mail >>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2> >>>>> >>>>> On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk < >>>>> [email protected]> wrote: >>>>> >>>>> Hey all, >>>>> >>>>> I have cut Airflow Backport Providers 2020.10.5rc1. This email is >>>>> calling a vote on the release, >>>>> which will last for 72 hours - which means that it will end on Mon, >>>>> 5th of October 2020, 19:59:56 CEST. >>>>> >>>>> This is the first time we release "cncf.kubernetes" package after big >>>>> backporting of kubernetes changes >>>>> to 1.10.11 and 1.10.12, so I'd appreciate, if some of our users >>>>> looking at the devlist test test this provider >>>>> - especially if you are waiting for it. >>>>> >>>>> Consider this my (binding) +1. >>>>> >>>>> Airflow Backport Providers 2020.10.5rc1 are available at: >>>>> >>>>> https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.10.5rc1/ >>>>> >>>>> *apache-airflow-backport-providers-2020.10.5rc1-source.tar.gz* is a >>>>> source release that comes >>>>> with INSTALL instructions. >>>>> >>>>> *apache-airflow-backport-providers-<PROVIDER>-2020.10.5rc1-bin.tar.gz* >>>>> are the binary >>>>> Python "sdist" release. >>>>> >>>>> Public keys are available at: >>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS >>>>> >>>>> Please vote accordingly: >>>>> >>>>> [ ] +1 approve >>>>> [ ] +0 no opinion >>>>> [ ] -1 disapprove with the reason >>>>> >>>>> >>>>> Only votes from PMC members are binding, but members of the community >>>>> are >>>>> encouraged to test the release and vote with "(non-binding)". >>>>> >>>>> Please note that the version number excludes the 'rcX' string, so it's >>>>> now >>>>> simply 2020.10.5. This will allow us to rename the artifact without >>>>> modifying >>>>> the artifact checksums when we actually release. >>>>> >>>>> Each of the packages contains a detailed changelog. Here is the list >>>>> of links to >>>>> the released packages and changelogs: >>>>> >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-kylin/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-celery/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-cncf-kubernetes/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-discord/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-docker/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-google/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-http/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-imap/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-jira/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-plexus/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-presto/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-redis/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-samba/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-segment/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-slack/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.10.5rc1/ >>>>> >>>>> https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.10.5rc1/ >>>>> >>>>> >>>>> Cheers, >>>>> Jarek >>>>> >>>>> -- >>>>> >>>>> Jarek Potiuk >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>> >>>>> M: +48 660 796 129 <+48660796129> >>>>> [image: Polidea] <https://www.polidea.com/> >>>>> >>>>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > >
