Re: [VOTE] AIP-48 Data Driven Scheduling

The way I've described it, it won't need any extra locking of rows and will tie in to the existing lock off there (source) dag run row. -Ash On 8 June 2022 21:45:26 BST, Ping Zhang wrote: >Thanks Ash. It is nice that this design will have next-to-no impact on >scheduling performance. > >One

Re: [VOTE] AIP-48 Data Driven Scheduling

Thanks Ash. It is nice that this design will have next-to-no impact on scheduling performance. One quick question when you say " DB as a queue ", will it leverage skip lock feature from the db? Thanks, Ping Thanks, Ping On Wed, Jun 8, 2022 at 9:59 AM Ash Berlin-Taylor wrote: > And the

Re: [VOTE] AIP-48 Data Driven Scheduling

And the reason for including dag_id/task_id in that new table is to avoid a new "top level query", but instead that we can add in to the existing "look at dagrun X" scheduling. (Strictly speaking we could just have a table of "dataset X has just been published", but for that to work we would

Re: [VOTE] AIP-48 Data Driven Scheduling

Hi Ping, Good idea. Very roughly: We will have a create a use a new database table to store a queue of datasets publishes to be actioned, with columns (dag_id, task_id, run_id, dataset_uri). Rows in that table are created by TaskInstance in the same transaction where that TI is marked to

Re: [VOTE] AIP-48 Data Driven Scheduling

Hi Ash, Thanks for introducing the data-driven scheduling and thoughtful future work section in the AIP. Could you please add a section to talk about the high level/early tech design in the AIP? This is a new scheduling pattern that can have many implications, thus we (our team) would love to

Re: [VOTE] AIP-48 Data Driven Scheduling

No momento estarei na instalação! Franklin Ferreira de Lima Em Dom, 5 de jun de 2022 22:38, escreveu: > +1 (non-binding) > > On Jun 4, 2022, at 7:45 PM, Igor Kholopov > wrote: > >  > +1 (non-binding) > Left some comments with my thoughts on the AIP wiki page. > > On Fri, Jun 3, 2022 at 3:11

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (non-binding) > On Jun 4, 2022, at 7:45 PM, Igor Kholopov > wrote: > >  > +1 (non-binding) > Left some comments with my thoughts on the AIP wiki page. > >> On Fri, Jun 3, 2022 at 3:11 PM Ankit Chaurasia wrote: >> +1 (non-binding) >> >> Ankit Chaurasia >> HomePage | LinkedIn |

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (non-binding) *Ankit Chaurasia* HomePage | LinkedIn | +91-9987351649 On Fri, Jun 3, 2022 at 1:04 PM Tomasz Urbaszek wrote: > +1 (binding) > > On Thu, 2 Jun 2022 at 15:49, Josh Fell > wrote: > >> +1 (binding) >>

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (binding) On Thu, 2 Jun 2022 at 15:49, Josh Fell wrote: > +1 (binding) > > On Thu, Jun 2, 2022 at 9:31 AM Hila Sofer Elyashiv > wrote: > >> +1 non-binding >> >> On 2022/06/01 16:34:13 Ash Berlin-Taylor wrote: >> > Hi All, >> > >> > Now that Summit is over (well done all the speakers! The

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (binding) On Thu, Jun 2, 2022 at 9:31 AM Hila Sofer Elyashiv wrote: > +1 non-binding > > On 2022/06/01 16:34:13 Ash Berlin-Taylor wrote: > > Hi All, > > > > Now that Summit is over (well done all the speakers! The talks I've > > caught so far have been great) I'm ready to push forward with

RE: [VOTE] AIP-48 Data Driven Scheduling

+1 non-binding On 2022/06/01 16:34:13 Ash Berlin-Taylor wrote: > Hi All, > > Now that Summit is over (well done all the speakers! The talks I've > caught so far have been great) I'm ready to push forward with Data > Driven Scheduling, and I would like to call for a vote on >

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (binding) On Wed, Jun 1, 2022 at 12:34 PM Ash Berlin-Taylor wrote: > Hi All, > > Now that Summit is over (well done all the speakers! The talks I've caught > so far have been great) I'm ready to push forward with Data Driven > Scheduling, and I would like to call for a vote on >

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (non binding) Looking forward to this as well, this feels like a great improvement! On Thu 2 Jun 2022 at 10:48, Phani Kumar wrote: > +1 non-binding > > On Thu, Jun 2, 2022 at 1:21 PM Dennis Akpenyi > wrote: > >> +1 non-binding >> >> On Thu 2. Jun 2022 at 09:45, Pankaj Koti wrote: >> >>>

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 non-binding On Thu, Jun 2, 2022 at 1:21 PM Dennis Akpenyi wrote: > +1 non-binding > > On Thu 2. Jun 2022 at 09:45, Pankaj Koti wrote: > >> +1 non-binding >> >> On Thu, 2 Jun 2022 at 01:50, Kaxil Naik wrote: >> >>> +1 binding >>> >>> On Wed, 1 Jun 2022 at 19:37, Drew Hubl >>> wrote: >>>

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 non-binding On Thu 2. Jun 2022 at 09:45, Pankaj Koti wrote: > +1 non-binding > > On Thu, 2 Jun 2022 at 01:50, Kaxil Naik wrote: > >> +1 binding >> >> On Wed, 1 Jun 2022 at 19:37, Drew Hubl >> wrote: >> >>> +1 (non-binding) >>> >>> On Jun 1, 2022, at 12:33 PM, Ephraim Anierobi < >>>

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 non-binding On Thu, 2 Jun 2022 at 01:50, Kaxil Naik wrote: > +1 binding > > On Wed, 1 Jun 2022 at 19:37, Drew Hubl > wrote: > >> +1 (non-binding) >> >> On Jun 1, 2022, at 12:33 PM, Ephraim Anierobi < >> ephr...@astronomer.io.INVALID> wrote: >> >> +1 (binding) >> >> On Wed, 1 Jun 2022 at

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (binding) On Wed, Jun 1, 2022 at 11:20 PM Kaxil Naik wrote: > +1 binding > > On Wed, 1 Jun 2022 at 19:37, Drew Hubl > wrote: > >> +1 (non-binding) >> >> On Jun 1, 2022, at 12:33 PM, Ephraim Anierobi < >> ephr...@astronomer.io.INVALID> wrote: >> >> +1 (binding) >> >> On Wed, 1 Jun 2022 at

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 binding On Wed, 1 Jun 2022 at 19:37, Drew Hubl wrote: > +1 (non-binding) > > On Jun 1, 2022, at 12:33 PM, Ephraim Anierobi < > ephr...@astronomer.io.INVALID> wrote: > > +1 (binding) > > On Wed, 1 Jun 2022 at 19:28, Vikram Koka > wrote: > >> +1 (binding) >> >> >> On Wed, Jun 1, 2022 at 10:49

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (non-binding) > On Jun 1, 2022, at 12:33 PM, Ephraim Anierobi > wrote: > > +1 (binding) > > On Wed, 1 Jun 2022 at 19:28, Vikram Koka wrote: > +1 (binding) > > > On Wed, Jun 1, 2022 at 10:49 AM Jarek Potiuk > wrote: > +1 (binding) > > On Wed, Jun 1, 2022 at

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (binding) On Wed, 1 Jun 2022 at 19:28, Vikram Koka wrote: > +1 (binding) > > > On Wed, Jun 1, 2022 at 10:49 AM Jarek Potiuk wrote: > >> +1 (binding) >> >> On Wed, Jun 1, 2022 at 7:47 PM Jed Cunningham >> wrote: >> >>> +1 (binding) >>> >>> I'm looking forward to this, thanks Ash. >>> >>

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (binding) On Wed, Jun 1, 2022 at 10:49 AM Jarek Potiuk wrote: > +1 (binding) > > On Wed, Jun 1, 2022 at 7:47 PM Jed Cunningham > wrote: > >> +1 (binding) >> >> I'm looking forward to this, thanks Ash. >> >

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (binding) On Wed, Jun 1, 2022 at 7:47 PM Jed Cunningham wrote: > +1 (binding) > > I'm looking forward to this, thanks Ash. >

Re: [VOTE] AIP-48 Data Driven Scheduling

+1 (binding) I'm looking forward to this, thanks Ash.

[VOTE] AIP-48 Data Driven Scheduling

Hi All, Now that Summit is over (well done all the speakers! The talks I've caught so far have been great) I'm ready to push forward with Data Driven Scheduling, and I would like to call for a vote on