It sure does! Thank you. On Thu, Jan 18, 2024 at 3:17 PM Oliveira, Niko <oniko...@amazon.com.invalid> wrote:
> Hey Ryan, > > Thanks for the reply! I'll make that note more clear in the AIP. It's a > nuanced point, and the wording is a bit vague at best right now or > borderline misleading >< > > What I'm trying to say here is that if you don't configure a specific > executor for a task, it will run on the default/environment level executor > and for each attempt that will be the case. But I was just pointing out > that it's also true that the default executor can be reconfigured to a > different executor at any time (with a scheduler restart of course) so that > env/default executor can be _any_ executor the user has currently > configured. And if that's done in the middle of a DAG run, or between > retries, etc the default level could be different than it was before. And > note that this is how Airflow behaves today already. > > Does that clear things up? Let me know if it doesn't and I'll have a third > go at it :) > > Cheers, > Niko > > ________________________________ > From: Ryan Hatter <ryan.hat...@astronomer.io.INVALID> > Sent: Thursday, January 18, 2024 1:27:00 PM > To: dev@airflow.apache.org > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] AIP-61 - Hybrid Executors > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > *IMPORTANT NOTE*: task instances that run on the default/environment > > executor (i.e. with no specific override provided) will not persist the > > executor in the same way so that they can be re-run/retried on any > executor. > > > Does this mean that any task that doesn't have the `executor` parameter > specified will run on the default executor for its first attempt, but any > executor on subsequent attempts? If so, how will users specify the default > executor? Also, I think this will provide some debugging challenges when a > task runs on one executor on one attempt, but a different executor on a > different attempt. It might be nice if users could set a > `use_default_executor_on_retry` kind of parameter. > > On Mon, Jan 15, 2024 at 1:05 PM Oliveira, Niko <oniko...@amazon.com.invalid > > > wrote: > > > Hey folks! > > > > I'd like to announce a new proposal for Airflow. I've teased this before > > in my talk on executors at this year's summit and it was also mentioned > in > > the townhall last week. > > > > It's a proposal to allow using multiple executors concurrently within a > > single Airflow environment. > > > > > > Let me know what you think! > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution > > > > Cheers, > > Niko > > >
