Maybe file what i'm about to say under superfluous commentary but... I don't really think of this as "breaking with the API", or think there is much of a conflict. The API is just there for you to use. It has its own behavior. You understand how it works in order to do what you need to do. But *you* decide what you need to do, not the API. Sometimes that means you handle errors it throws at you.
Operators are a different layer of abstraction and governed by user needs and not design choices of the API it is using. So just because the API throws an error in a certain operation, perhaps even an operation that has a name similar to your operator, doesn't mean your code should explode when you are trying to use that operation. On Thu, Jan 15, 2026 at 1:21 PM Shahar Epstein <[email protected]> wrote: > As I reviewed a PR that fixed related behavior, I didn't know what the > appropriate approach is actually. I think that this confusion is justified > due to the conflict between achieving idempotency and breaking with the > original API (unlike other kinds of operators that typically follow it). > Enforcing might be hard, but I think that it should be documented as part > of guidelines for contributors and maintainers. Also, if we agree that this > is the desired behavior for most cases - deprecating the flags where > applicable, and aligning the relevant operators sounds like a reasonable > thing to do. > > > Shahar > > > On Tue, Jan 13, 2026 at 11:15 PM Oliveira, Niko <[email protected]> > wrote: > > > I think I agree with the crowd here, we shouldn't make a fuss if the > > resource is already deleted. > > But I would also strongly +1 Jarek's second point. Is this consistency we > > really need to do a lot of work to enact/enforce? Are users actually > > struggling even with this? Or have we so far settled on what makes the > most > > sense for each operator and not many folks are perfectly happy with the > > current state? > > > > > > Cheers, > > Niko > > > > ________________________________ > > From: Shahar Epstein <[email protected]> > > Sent: Tuesday, January 13, 2026 1:03:11 PM > > To: [email protected] > > Subject: RE: [EXT] [DISCUSS] Idempotency of "Delete" operators > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > le contenu ne présente aucun risque. > > > > > > > > Your wish is Gemini's command - I tried to scan the entire repo for > Delete > > operators with such flags. Luckily there aren't many + we are not that > > creative in naming :) > > https://github.com/apache/airflow/pull/60483 > > > > I've read the statements in the thread until now, and I'm not strongly > > against making all operators idempotent by default without a flag, as > > customizations could be made by subclassing + extra "if exist" statement. > > It's more important for me that the behavior will be mostly consistent in > > such operators (with granted exemptions, where applicable). This would > make > > it clearer for contributors and maintainers as it breaks with the API, > > which is not trivial IMO. > > > > I'll give this discussion a couple of more days, and if there are no > strong > > objections against the apparent consensus, I'll start a lazy consensus > > thread + create a GitHub issue to align all relevant operators. > > > > > > Shahar > > > > On Tue, Jan 13, 2026 at 9:07 PM Jarek Potiuk <[email protected]> wrote: > > > > > I think yes - we should strive to have the delete "kinds" of operations > > to > > > silently ignore already deleted content. We can agree on it as a > > "default" > > > behaviour - and probably have it written somewhere in best practices > for > > > writing operators. > > > > > > But - we also should be pragmatic in the sense that a) there might be > > > (justified) exceptions b) it's not very likely we will be able to > > > consistently and automatically enforce it, and it almost guarantees > that > > at > > > any point in time some of those will be following different semantics - > > > because we forget about it over time or won't realise this is > happening. > > > > > > I wish we had some way of enforcing or at least detecting it :) > > > > > > On Tue, Jan 13, 2026 at 7:17 PM Daniel Standish via dev < > > > [email protected]> wrote: > > > > > > > Yeah just suppress the exception. > > > > > > > > Task should not fail cus object already deleted. > > > > > > > > The intent is that the thing is not there, not to verify that it was > > and > > > > that you deleted it. > > > > > > > > If users want diff behavior they can subclass. > > > > > > > > Simpler is better than complex, re the flag idea. Does not seem like > a > > > case > > > > for a global param. > > > > > > > > > > > > > > > > On Tue, Jan 13, 2026 at 9:49 AM Ash Berlin-Taylor <[email protected]> > > > wrote: > > > > > > > > > Given then general rule of idempontency of Airflow operators, I > think > > > we > > > > > should break with the API. The intent of a “Delete” operator is to > > > ensure > > > > > the resource no longer exists. So not found is not an error in that > > > > model. > > > > > This makes such operators safe to retry. If it threw an error when > it > > > was > > > > > deleted then they are not safe to retry, and this is not the model > > > > Airflow > > > > > operators should have. > > > > > > > > > > No flag should be needed — if someone doesn’t want this behavoiur > > than > > > I > > > > > would say they should use the hook or API directly. The point of > the > > > > > pre-built operators is to package up behaviour like this in a > > > consistent > > > > > manner. > > > > > > > > > > My 2c: we should standardised on remove the “ignore_if_missing” > flags > > > on > > > > > operators and always catching not exists style errors in operators > > and > > > > > treating those as success. > > > > > > > > > > -ash > > > > > > > > > > > On 12 Jan 2026, at 22:06, Ferruzzi, Dennis <[email protected]> > > > > wrote: > > > > > > > > > > > > Yeah, generally when I add operators to the Amazon provider > > package, > > > my > > > > > intent is to preserve the behavior of the API,. I'm not claiming > > > that's > > > > > necessarily the right answer, but that's what i do and I like that > > > idea. > > > > > > > > > > > > I wouldn't be against the idea of a flag, but I think as a > general > > > rule > > > > > we should assume the default behavior should be familiar to someone > > who > > > > is > > > > > used to using the API before moving to Airflow without surprises. > Of > > > > > course, his may be an XKCD1172 issue [1] and maybe we SHOULD insist > > on > > > a > > > > > standard expectation across providers.... > > > > > > > > > > > > [1] https://xkcd.com/1172/ > > > > > > > > > > > > > > > > > > - ferruzzi > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > From: Shahar Epstein <[email protected]> > > > > > > Sent: Monday, January 12, 2026 12:51 PM > > > > > > To: [email protected] > > > > > > Subject: [EXT] [DISCUSS] Idempotency of "Delete" operators > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. > Do > > > not > > > > > click links or open attachments unless you can confirm the sender > and > > > > know > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > si > > > > vous > > > > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous > n’êtes > > > pas > > > > > certain que le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > Hey all, > > > > > > > > > > > > For the sake of this discussion, the idempotency of Delete > > operators > > > > > means > > > > > > that when an operator is applied to a non-existing resource, it > > > > catches a > > > > > > “not found” exception and returns success. By default, the > > operators > > > I > > > > > > refer to return an exception if the resource is not found. > > > > > > > > > > > > Following this PR #60083 < > > > https://github.com/apache/airflow/pull/60083 > > > > >, > > > > > > I've realized that we don't handle idempotency consistently in > > > > different > > > > > > "Delete" operators across different providers. For example, in > some > > > GCP > > > > > > operators there's a tendency to catch "NotFound" exception > without > > > any > > > > > flag > > > > > > (DeletePipelineJobOperator > > > > > > < > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/bd133c0ebb1219b72f4cb7998b2f5f55c8aff200/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py#L522 > > > > > >), > > > > > > while in some Microsoft's operators we handle it by flag (like > > > > > > "ignore_if_missing" in WASBDeleteBlob > > > > > > < > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/_api/airflow/providers/microsoft/azure/operators/wasb_delete_blob/index.html#module-contents > > > > > > > > > > > > ). > > > > > > > > > > > > My questions for discussion are as follows: > > > > > > 1. Do we want to require that idempotency for operators (that > > > normally > > > > > > return an error) will be handled by a flag? > > > > > > 2. If we agree on having a flag, what should be the default value > > of > > > > this > > > > > > flag? > > > > > > 3. Given the answer to the last question, should we break the > > > interface > > > > > of > > > > > > existing operators where applicable? > > > > > > > > > > > > My answers: > > > > > > 1. If the original API usually returns an error, catching > NotFound > > > > error > > > > > > and returning success "defies" the intent of the original API, > and > > I > > > > > think > > > > > > that it might be confusing and even limiting in some cases. > > > Therefore, > > > > if > > > > > > originally a "Delete" operator fails and we want to catch this > > > > exception > > > > > - > > > > > > I think that we should at least define a flag for that > (preferably > > > > with a > > > > > > consistent name across all operators). > > > > > > 2. Default behavior should reflect the original intent of the > API. > > > > > > 3. Slowly, but carefully - yes: changing both the default value, > > and > > > > the > > > > > > name of the flag to be consistent. > > > > > > > > > > > > I'll be happy to hear more opinions! > > > > > > > > > > > > > > > > > > Shahar > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > >
