Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Hi, +1 (binding) Beginning from Cloud Composer 1.7.7 users have the possibility to use DAG Serialization feature to improve reliability of their Airflow Webservers - we backported it to Airflow 1.10.2 container images that Composers can use while creating their Airflow environments. By default, w

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

I agree with what Jarek is saying. And if just for testing a user wants to see what his template field would look like they can used airflow render from the cli On Thu, Oct 24, 2019 at 3:19 PM Jarek Potiuk wrote: > Great work! > > I also think it's a pragmatic and reasonable approach to add it t

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Great work! I also think it's a pragmatic and reasonable approach to add it to 1.10 and work on improved version for 2.0 (and have it as the only option). This way we do not really add unnecessary complexity - we will just improve the serialized implementation in our 2.0 effort, we close all the g

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Thank you, I fine with adding it to 1.10 as the current PR where is it an option, and for now, the current behavior would be the old one, until smooth out all the issues. For 2.0 I'd love to see it as the only option so we can remove the logic and code of the SimpleDag to keep the codebase nice a

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Well at least we would love to start using it on our 1.10.* cluster before having to upgrade to 2.0. It is a huge change yes because it provides huge value so I would image there are more people like us want to start using it ASAP :D I agree that if the XCOM part of the rendered template is not as

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Speaking with my Astronomer hat on, we are going to include this in our next 1.10.x image as this improves the experience when deploying changes. We could not backport it to the v-1-10-test for inclusion in next release, but the work to backport it is already done. Yes, making this the default

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Awesome work all, I would be more in the opposite direction. Maybe we should not backport such a huge change to 1.10 and work on it for 2.0. Also, I'd like to see it as being default in 2.0, having it as an option will introduce another permutation in the configuration and this will make the whole

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Just saw the latest email from Kaxil and want to clarify. I was replying the previous posts and making general comments instead of trying to block the PR from being merged. I'm fine with merging it and improve incrementally. It is quite polished and already huge now, I can totally relate to the reb

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Oops, I guess I'm too slow reading the PRs to let me vote in time 😛 Congrats Zhou, Kaxil and Ash for getting it accepted 🎉 I'm pretty excited about this long-anticipated change and thank you guys for the great work. Though I do have a few comments. I agree with Dan's concern about maintaining two

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Hi guys, I have addressed all comments on the PR: https://github.com/apache/airflow/pull/5743 Can we merge this PR please if everything looks good and approved by the committers? It is becoming increasingly difficult to rebase on master and resolve conflicts. I also have the backport PR (https:

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

This vote passed (although not unanimous) and I'll mark this AIP as accepted. *Result*: +1 votes: 7 (6 binding and 1 non-binding vote) -1 votes: 2 (2 binding and 0 non-binding votes) *+1 (binding)*: Kaxil Naik Ash-Berlin Taylor Jarek Potiuk Kamil Breguła Fokko Driesprong Sumit Maheshwari *+1 (no

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Not sure I'm convinced, I think my core concern of maintaining two representations and not having a thought-out future plan of how the schema will evolve/how easy this migration will be still stands. It feels like we are trying to chase some short-term wins here. But hey it's a democracy, don't let

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Hi Dan, I understand your concern. Your +1 and suggestions are very important for us so let me try to explain in more details and see if I can convince you. Please check replies in-line I think this is the kind of core change we should make from the get-go, my > feeling is that we will be paying

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Responses inline. On Wed, Oct 16, 2019 at 6:07 AM Ash Berlin-Taylor wrote: > Thanks for the feedback and discussion everyone - it's nice to know people > feel strongly about this. Let's make sure we build this right! > > As a reminder: the main goals of this AIP: > > - To speed up bootup/worker

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Thanks for the feedback and discussion everyone - it's nice to know people feel strongly about this. Let's make sure we build this right! As a reminder: the main goals of this AIP: - To speed up bootup/worker recycle time of the webserver If you have a large number of DAGs or highly dynamic D

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Hello, I think it's worth to mark this feature as experimental, which means that its behavior may change. In this way, we will not have to ensure backward compatibility, but we will be able to test this feature in a real environment. This is how feature are introduced in the Kubernetes project, wh

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

On this, I'm more aligned with the +1 community here. While I agree that this is not the complete solution and work doesn't stop here, but I see it as a stepping stone in the right direction. It's a very big challenge to develop and deliver a change of this magnitude, so I believe that should be do

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Big +1 from my side, looking forward to make this happen. Two sides that aren't completely clear to me: - Are we going to extend the existing data model, to allow the RDBMS to optimize queries on fields that we use a lot? - How are we going to do state evolution when we extend the JSON m

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

I have been following it from the beginning as well. I understand there would be short-term wins for some users (I don't think a huge amount of users?), but I still feel like we are being a bit short-sighted here and that we are creating more work for ourselves and potentially our users in the futu

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Hello Dan, Alex, I believe all the points you make are super-valid ones. But maybe you are missing the full context a bit. I followed the original discussion from the very

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

-1 (binding) Good points made by Dan. We don't need to have the future plan implemented completely but it would be nice to see more detailed notes about how this will play out in the future. We shouldn't walk into a system that causes more pain in the future. (I can't say for sure that it does, but

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

-1 (binding), this may sound a bit FUD-y but I don't feel this has been thought through enough... Having both a SimpleDagBag representation and the JSON representation doesn't make sense to me at the moment: *"**Quoting from Airflow code, it is “a simplified representation of a DAG that contains a

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

+1 (binding) On Tue, Oct 15, 2019 at 2:57 AM Kaxil Naik wrote: > Hello, Airflow community, > > This email calls for a vote to add the DAG Serialization feature at > https://github.com/apache/airflow/pull/5743. > > *AIP*: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistenc

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

+1 non-binding On Mon, Oct 14, 2019 at 8:57 PM Kaxil Naik wrote: > Hello, Airflow community, > > This email calls for a vote to add the DAG Serialization feature at > https://github.com/apache/airflow/pull/5743. > > *AIP*: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persiste

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

(binding) On Tue, Oct 15, 2019 at 9:43 AM Jarek Potiuk wrote: > > I looked over your shoulders while you were implementing it. Big +1 from > me. > > On Tue, Oct 15, 2019 at 2:57 AM Kaxil Naik wrote: > >> Hello, Airflow community, >> >> This email calls for a vote to add the DAG Serialization fe

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

I looked over your shoulders while you were implementing it. Big +1 from me. On Tue, Oct 15, 2019 at 2:57 AM Kaxil Naik wrote: > Hello, Airflow community, > > This email calls for a vote to add the DAG Serialization feature at > https://github.com/apache/airflow/pull/5743. > > *AIP*: > https://c

[VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

Hello, Airflow community, This email calls for a vote to add the DAG Serialization feature at https://github.com/apache/airflow/pull/5743. *AIP*: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler *Previo