Re: [VOTE] Airflow Backport Providers 2020.10.5rc1

2020-10-04 Thread Kamil Breguła
I see another problem. The kubernetes_engine.GKEStartPodOperator operator is not included in the Google package and the kubernetes_engine module contains references to the airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator operator instead of providers.cncf.kubernetes package.

Re: [VOTE] Airflow Backport Providers 2020.10.5rc1

2020-10-04 Thread Kamil Breguła
I started testing and encountered three problems - Two minor and one serious. I used official Docker images for testing: apache/airflow:1.10.12 with gcloud installed for GKE authorization 1. The configuration for StackdriverTaskHandlle is not obvious and is not documented anywhere. I managed to do

Re: [VOTE] Airflow Backport Providers 2020.10.5rc1

2020-10-04 Thread Daniel Imberman
+1 excited to finally encourage the Kubernetes providers! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.14.6&source=email_footer_2] On Fri, Oct 2, 2020 at 11:00 AM, Jarek Potiuk wrote: Hey all, I have cut Airflow Backport Providers 2020.10.5rc1. This email is cal

Re: Question regarding data usage in the DAG itself

2020-10-04 Thread Tomasz Urbaszek
Chen take a look at `processor_poll_interval` and `min_file_process_interval` options in airflow configuration. But still, I would strongly recommend removing from your DAGs any top-level code that is executed. Cheers, Tomek On Sun, Oct 4, 2020 at 6:45 PM Chen Michaeli wrote: > > Hi, a quick fo

Re: Question regarding data usage in the DAG itself

2020-10-04 Thread Chen Michaeli
Hi, a quick follow-up. Is there a parameter I can configure to alter that behavior? Say I want a specific DAG/all DAGs to be parsed every X minutes instead of the default few seconds? Thanks again :) בתאריך יום ג׳, 22 בספט׳ 2020, 20:15, מאת Tomasz Urbaszek ‏< turbas...@apache.org>: > The DAG i

Re: [DISCUSS] Documentation Improvements for Airflow 2.0

2020-10-04 Thread Fabien BEURET
Hi, Definitely some examples on how to use operators in context would be helpful It's sometimes kind of hard to figure out how to use them So, just examples... (not to copy/paste, but to get inspired) Thx, Fabien On 2020/09/18 12:44:03, Kaxil Naik wrote: > Hi all, > > One of the thing that w