Current state of airflow on kubernetes

Hi Airflow devs, I am to new airflow and trying to figure out some details which will help me choose the right deployment for my company. I want to create a distributed airflow deployment either using Kube or Celery executor, and had a few questions. 1. I see that airflow on kube is a recent featu

Re: Deployment guide for airflow on kubernetes

Thanks, Maxime and Barni! On Tue, Apr 9, 2019 at 12:19 AM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Also looks like there's a Helm chart here. > https://github.com/helm/charts/tree/master/stable/airflow > > I haven't used it personally, but looks good at first glance > > On Mon, Ap

Re: Difference between Kubernetes Executor vs PodOperator

Thanks, James and Kamil! Please let me know if you have any examples of setting up Kubernetes Executor and Operator. On Tue, Apr 9, 2019 at 8:03 AM James Meickle wrote: > Yes, that summary is correct - the Executor is using Kubernetes to execute > all Airflow tasks (each wrapped by a temporary A

Re: [PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs

- The structure of the documentation requires rethinking. A lot of information is contained on one subpage - "concepts". Some information should not be on this subpage e.x. .airflowignore Other information should be extended, but then they will occupy too much space on this page e. x. creating rela

Re: [PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs

- Deployment in {AWS, Azure, GCP} - Monitoring, health checks, etc - Best practices for {dags, subdags, dag scheduling} Gerard Toonstra has some amazing docs here for some of these topics as well: https://gtoonstra.github.io/etl-with-airflow/ On 10

Re: [VOTE] Airflow 1.10.3rc2

+1 (binding) - Everything looks good, ran few sample dags, created/modified connections & variables. Found just 2 minor UI issues - TI Duration not visible in Graph View - PR raised for it - Markdown not correctly rendered. - Already resolved

Re: [PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs

- how to configure different types of connections - how to rerun a failed task. - tips for designing dags - writing custom operators (or tweaking the behaviour of a built-in one) without needing plug in. Ash On 9 April 2019 20:52:02 BST, Aizhamal Nurmamat kyzy wrote: >Hello all, > >Let's put t

[PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs

Hello all, Let's put together ideas for the Season of Docs. Can you please share any documentation pain points that you are aware of as a user or as a contributor? Anything where you think the Airflow docs could be improved significantly? Some ideas that members of the community have shared with

Re: Jira Administer permission

Hi all, thank you for granting me the permissions. After some research, it seems that sharing dashboards requires a JIRA global permission. I will see if someone from Apache can help me by simply making my dashboard public. + san...@apache.org is this something you could possibly help me with? The

Re: Difference between Kubernetes Executor vs PodOperator

Yes, that summary is correct - the Executor is using Kubernetes to execute all Airflow tasks (each wrapped by a temporary Airflow process), while the PodOperator is using Kubernetes only for that task, to execute one Pod (which likely won't run any Airflow code at all). On Tue, Apr 9, 2019 at 3:17

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

I agree with Ash here. The naming of "execution_date" is incredibly confusing to people who are new to Airflow, who think it has something to do with... execution. However, I think that there's still room for improvement with "period_start" and "period_end". Think about manually triggered tasks -

Re: [DISCUSS] period_start/period_end instead of execution_date/next_execution_date

Yeah, that's the other thing that has been talked about from time-to-time, which is a mode to change from "run at end of period, I need all the data available for this period" (the current) to "run at _this_ time on the schedule_interval, don't wait for the period to end". (No such flag exists

Re: FAB - New REST API in the works

Hi everyone, Thank you for your time. Just a heads up, I'll probably merge and release this feature today. Hope you had the time to look it up, since this could probably be a natural candidate for the foundation of Airflow's REST API. On 2019/04/04 14:28:20, Ash Berlin-Taylor wrote: > https:/

RE: [DISCUSS] period_start/period_end instead of execution_date/next_execution_date

Hi all, I'm new to this Airflow Dev mailing list so I wasn't expecting to reply to anything but I feel I am the target audience for this question. I am quite new to airflow and have been setting up an airflow environment for my business this last month. I find the current "execution_date" a sm

[DISCUSS] period_start/period_end instead of execution_date/next_execution_date

(trying to break this out in to another thread) The ML doesn't allow images, but I can guess that it is the deps section of a task instance details screen? I'm not saying it's not clear once you know to look there, but I'm trying remove/reduce the confusion in the first place. And I think we a

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

Ash, Personally, I think this is quite clear, there is a list of reasons why the job isn't being scheduled: [image: image.png] Coming back to the question of Bas, I believe that yesterday_ds does not make sense since we cannot assume that the schedule is daily. I don't see any usage of this varia

Re: Jira Administer permission

I have given her temporary Jira admin permission. @Aizhamal - Can you please let me know once you create the dashboard and no longer need this perms? Regards, Kaxil On 09/04/2019, 00:40, "Jakob Homan" wrote: Looks like Sid is the only admin right on the Airflow project in JIRA. We s

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

To (slightly) hijack this thread: On the subject of execuction_date: as I'm sure we're all aware the concept of execution_date is confusing to new-commers to Airflow (there are many questions about "why hasn't my DAG run yet"? "Why is my dag a day behind?" etc.) and although we mention this in

Re: Deployment guide for airflow on kubernetes

Also looks like there's a Helm chart here. https://github.com/helm/charts/tree/master/stable/airflow I haven't used it personally, but looks good at first glance On Mon, Apr 8, 2019 at 9:43 PM Barni Seetharaman wrote: > Please checkout > https://github.com/GoogleCloudPlatform/airflow-operator >

Re: Difference between Kubernetes Executor vs PodOperator

Hi Ashwin, I had exactly same question couple of days ago. Let me try to explain. If I'm wrong please someone correct me. Kubernetes Executor is used to execute TaskInstance, which means that Pod is created of that TaskInstance that for ex. could be BashOperator or SlackAPIOperator and after exec