Hey Ryan, Thanks for the reply! I'll make that note more clear in the AIP. It's a nuanced point, and the wording is a bit vague at best right now or borderline misleading ><
What I'm trying to say here is that if you don't configure a specific executor for a task, it will run on the default/environment level executor and for each attempt that will be the case. But I was just pointing out that it's also true that the default executor can be reconfigured to a different executor at any time (with a scheduler restart of course) so that env/default executor can be _any_ executor the user has currently configured. And if that's done in the middle of a DAG run, or between retries, etc the default level could be different than it was before. And note that this is how Airflow behaves today already. Does that clear things up? Let me know if it doesn't and I'll have a third go at it :) Cheers, Niko ________________________________ From: Ryan Hatter <ryan.hat...@astronomer.io.INVALID> Sent: Thursday, January 18, 2024 1:27:00 PM To: dev@airflow.apache.org Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] AIP-61 - Hybrid Executors CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. > > *IMPORTANT NOTE*: task instances that run on the default/environment > executor (i.e. with no specific override provided) will not persist the > executor in the same way so that they can be re-run/retried on any executor. Does this mean that any task that doesn't have the `executor` parameter specified will run on the default executor for its first attempt, but any executor on subsequent attempts? If so, how will users specify the default executor? Also, I think this will provide some debugging challenges when a task runs on one executor on one attempt, but a different executor on a different attempt. It might be nice if users could set a `use_default_executor_on_retry` kind of parameter. On Mon, Jan 15, 2024 at 1:05 PM Oliveira, Niko <oniko...@amazon.com.invalid> wrote: > Hey folks! > > I'd like to announce a new proposal for Airflow. I've teased this before > in my talk on executors at this year's summit and it was also mentioned in > the townhall last week. > > It's a proposal to allow using multiple executors concurrently within a > single Airflow environment. > > > Let me know what you think! > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution > > Cheers, > Niko >
