Re: [DISCUSS] how to name classes with abbreviations?

I think we are at this point because AIP-21 handles more file paths rather than class names. Case 6 in AIP-21 says to make individual decisions and it says "consistency trumps compatibility." I'm in favor of: Gcs over GCS, Emr over EMR, Sql over SQL. To mention we also have operators like SqlToS3O

Re: [VOTE] deprecate days_ago helper function

+1 (binding) On Fri, Feb 4, 2022 at 11:10 PM Collin McNulty wrote: > +1 (non-binding) > > On Fri, Feb 4, 2022 at 2:59 PM Jed Cunningham > wrote: > >> +1 (binding) >> >

[VOTE] Airflow Providers prepared on February 05, 2022

Hey all, I have just cut the new wave Airflow Providers packages. This email is calling a vote on the release, which will last for 72 hours (+1 day for weekend) - which means that it will end on Wed Feb 8 22:01:06 CET 2022. Consider this my (binding) +1. Airflow Providers are available at: http

Re: [DISCUSSION] Specify tasks to skip when triggering DAG

I thought a bit about it and there is one more thing that I do not really like about the approach when you specify the task to skip in the dag run conf. Currently the "logic" of the dag run conf is exclusively "dag specific" and only used at "task execution" time (mind - it is not used in "dag sch

Re: [DISCUSSION] Specify tasks to skip when triggering DAG

> > [Cons] > 1) Not scalable / Inconvenient > To make a task skippable, one needs to modify existing DAG (to set > `pre_execute`). It seems not difficult, but when your Airflow host a > thousand DAGs own by different teams/users, it can be challenging. You can use task_policy to apply (or chain,

Re: [DISCUSSION] Specify tasks to skip when triggering DAG

Hey Jurek, Just a question about the future development, is the XComm backend replaceable now? The real power of Airflow is the 'defacto' the glue between different ways of mangling data, such as Python is the glue when you need to implement things a lower level, ie. Altair Simulation software is

Re: [DISCUSSION] Specify tasks to skip when triggering DAG

Hi everyone, [DISCLAIMER first: I'm working in the same team as Howie] Many thanks for the discussion. Very insightful! I have two points to share and would like to hear how you think of them: *# 1. Potential cons of `pre_execute` method* I do agree it's a nice idea and is almost perfect for

Re: [DISCUSSION] Specify tasks to skip when triggering DAG

Airflow is really a "pythonic" framework. The basic premise of Airflow is that everything "DAG" is not declarative but Imperative. DAG definition, Task code, decorated python callables as a "new" way of writing tasks, heck even triggers and more recently timetables are "python imperative code". Sim