Re: [2.0 spring cleaning] Deprecate subdags

Hi all, Just want to bump this thread up again as I have a PR (https://github.com/apache/airflow/pull/5498 ) that makes SubDagOperator use normal scheduler instead of backfill scheduler to schedule tasks in subdags and would love to have some feedba

Re: [2.0 spring cleaning] Deprecate subdags

Hi all, I have a WIP PR that aims to make SubDagOperator to use normal scheduler instead of the backfill scheduler and would love to have some feedbacks from you with regard to the implementation. The basic idea is that we create a DagRun when e

Re: [2.0 spring cleaning] Deprecate subdags

I don't think fixing subdags to run in the scheduler is enough, although it's a huge improvement over the current implementation (especially the part that lets Subdags specify custom executors). From my experience with Subdags, I think what makes more sense is adding various operators to allow comb

Re: [2.0 spring cleaning] Deprecate subdags

-1 on deprecating subdags, because of the extra level of abstraction some of you already mentioned. We also use subdags in production. For example in cases where we get json data from an API but since we mostly need it to be in csv format we have a subdag like /specific_ap//i_specific_endpoint/_

Re: [2.0 spring cleaning] Deprecate subdags

+1 on keeping it. I think we should keep the SubDags as it provides a good abstraction layer. It just need some love from us to fix the underlying performance/reliability issues. On Fri, Apr 12, 2019 at 12:06 PM Ash Berlin-Taylor wrote: > This is what I was thinking - the dag collector in the s

Re: [2.0 spring cleaning] Deprecate subdags

This is what I was thinking - the dag collector in the scheduler should "just" be able to collect the tasks for subdags up to the parent dag. I'd possibly go as far as saying no DagRun object for subdags too. (Yes, "just" will never be that simple). -a On 12 April 2019 18:37:24 BST, Bolke de B

Re: [2.0 spring cleaning] Deprecate subdags

+1 on fixing the subdags. I think one issue I recall is that subdag uses backfill scheduler which doesn't have the on same functionality as normal scheduler. E.g the tasks within subdag are not resource bound by the pool. But with Chao-han's fix on https://github.com/apache/airflow/pull/4949, I th

Re: [2.0 spring cleaning] Deprecate subdags

+1 Sub dags should be fixed within the scheduler and run normally. On 12 April 2019 at 19:36:27, Feng Lu (fen...@google.com.invalid) wrote: Agree with others who think SubDag should stay, we should fix the SubDag implementation but not remove the abstraction itself. On Fri, Apr 12, 2019 at 8

Re: [2.0 spring cleaning] Deprecate subdags

Agree with others who think SubDag should stay, we should fix the SubDag implementation but not remove the abstraction itself. On Fri, Apr 12, 2019 at 8:42 AM Chen Tong wrote: > Is it possible to re-implement it in the view-level, not in operator level? > And this operator is just define a diffe

Re: [2.0 spring cleaning] Deprecate subdags

Is it possible to re-implement it in the view-level, not in operator level? And this operator is just define a different view in GUI, that these tasks will be collapsed into another view. On Fri, Apr 12, 2019 at 11:31 AM James Meickle wrote: > I have avoided using them because of outstanding iss

Re: [2.0 spring cleaning] Deprecate subdags

I have avoided using them because of outstanding issues like the open JIRA issues I linked above, or similar issues that I've read about on blog posts. If it were just GUI or UX issues I'd use them, but many people have reported issues which affect concurrency/stability, consistency, or correctness

Re: [2.0 spring cleaning] Deprecate subdags

I have been using SubDags in production and haven't had much problem with it. Can you list the issues you had? Regards, Kaxil On Fri, Apr 12, 2019, 16:16 James Meickle wrote: > Given their bad reputation, would it be appropriate to deprecate subDAGs > now to advertise that they're no longer c

Re: [2.0 spring cleaning] Deprecate subdags

Given their bad reputation, would it be appropriate to deprecate subDAGs now to advertise that they're no longer considered a suitable implementation? If a new and better implementation is created, would it even be similar enough to subDAGs that we'd want to continue to call the feature that? They

Re: [2.0 spring cleaning] Deprecate subdags

SubDAG is really useful in my use case, otherwise the graph will be hard to be managed. I guess it's better come out a better design rather than deprecate this feature. On Fri, Apr 12, 2019 at 10:51 AM Ash Berlin-Taylor wrote: > I'd like to find time to fix subdags as they do provide a useful

Re: [2.0 spring cleaning] Deprecate subdags

I'd like to find time to fix subdags as they do provide a useful abstraction - but I agree right now they aren't great (I avoid them because of this) I have half thoughts of how to it should work, I just need to look at the code in depth to see if that makes sense. Now 1.10.3 is out I might have

[2.0 spring cleaning] Deprecate subdags

I think we should deprecate SubDAGs given the complexity they add and the limited usage and use cases. Or, we should invest effort in redesigning their API and implementation. I think that having to account for subdag-introduced complexity makes Airflow's code much harder to maintain and buggier, l