Re: [AIP-34] Rewrite SubDagOperator

How would that look using the `>>`? Would have your tasks (task a, task b) in your main dag (parent_dag), then a dag factory that kicks out a dag (child_dag) with some property set denoting it as a child of `parent_dag`. Then your dependencies would look something like `task_a >> child_dag >> task_

Re: [AIP-34] Rewrite SubDagOperator

> I hadn’t thought about using the `>>` operator to tie dags together but I think that sounds pretty great! I think `>>` operator is great too! The question is how can we decide on which dag gets parsed into the dagbag or not? Maybe with a flag on the DAG object? Under the hood, I think we still

Re: [AIP-34] Rewrite SubDagOperator

Thanks Ash for reminding me the problem of image attachments :) Here are the image links for my previous message *Runtime DAG*: https://airflow.apache.org/docs/stable/_images/subdag_before.png *UI Visual*: https://airflow.apache.org/docs/stable/_images/subdag_after.png On Fri, Jun 12, 2020 at 11:

Re: [AIP-34] Rewrite SubDagOperator

> > - *Unpack SubDags during dag parsing*: This rewrites the *DagBag.bag_dag* > > method to unpack subdag while parsing, and it will give a flat > > structure at > > the task level > > The serialized_dag representation already does this I think. At least if > I've understood your idea here co

Re: [AIP-34] Rewrite SubDagOperator

I hadn’t thought about using the `>>` operator to tie dags together but I think that sounds pretty great! I wonder if we could essentially write in the ability to set dependencies to all starter-tasks for that DAG. I’m personally ok with SubDag being a mostly UI concept. It doesn’t need to exec

Re: [AIP-34] Rewrite SubDagOperator

I agree that SubDAGs are an overly complex abstraction. I think what is needed/useful is a TaskGroup concept. On a high level I think you want this functionality: - Tasks can be added to a TaskGroup - You *can* have dependencies between Tasks in the same TaskGroup, but *cannot* have depen

Re: [AIP-34] Rewrite SubDagOperator

Agree with James (and think it's actually the more important issue to fix), but I am still convinced Ash' idea is the right way forward (just it might require a bit more work to deprecate than adding visual grouping in the UI). There was a previous thread about this FYI with more context on why su

Re: [DISCUSS] Parametrized DAGs

I think this is a great idea! One thing that I think we should figure out before implementing is how to do so alongside DAG serialization, i.e. letting these params modify DAG topology might make it hard to store serialized representations for the Airflow services to consume and render, though that

Re: [AIP-34] Rewrite SubDagOperator

I'm not sure I totally agree it's just a UI concept. I use the subdag operator to simplify dependencies too. If you have a group of tasks that need to finish before another group of tasks start, using a subdag is a pretty quick way to set those dependencies and I think also make it easier to follow

Re: [AIP-34] Rewrite SubDagOperator

I second Ash’s grouping concept. On Fri, Jun 12, 2020 at 5:10 AM Ash Berlin-Taylor wrote: > Question: > > Do we even need the SubDagOperator anymore? > > Would removing it entirely and just replacing it with a UI grouping > concept be conceptually simpler, less to get wrong, and closer to what >

Re: dagrun_operator in Airflow Version 1.10.10 ERROR - No row was found for one() _run_raw_task result = task_copy.execute(context=context)

This issue has been resolved. We need to avoid using execution_date parameter while creating the `dagrun_operator` On 2020/06/09 16:33:32, SHANIT wrote: > On Migrating Airflow from V1.10.2 to V1.10.10 One of our DAG have a task > which is of dagrun_operator type. > The DAG runs all fine. In fa

Re: [AIP-34] Rewrite SubDagOperator

Question: Do we even need the SubDagOperator anymore? Would removing it entirely and just replacing it with a UI grouping concept be conceptually simpler, less to get wrong, and closer to what users actually want to achieve with subdags? With your proposed change, tasks in subdags could start ru

Re: [AIP-34] Rewrite SubDagOperator

Workon sub-dags is much needed, I'm excited to see how this progresses. > - *Unpack SubDags during dag parsing*: This rewrites the *DagBag.bag_dag* > method to unpack subdag while parsing, and it will give a flat > structure at > the task level The serialized_dag representation already doe

[AIP-34] Rewrite SubDagOperator

Hi everyone, Sending a message to everyone and collect feedback on the AIP-34 on rewriting SubDagOperator. This was previously briefly mentioned in the discussion about what needs to be done for Airflow 2.0, and one of the ideas is to make SubDagOperator attach tasks back to the root DAG. This AI