Your wish is Gemini's command - I tried to scan the entire repo for Delete operators with such flags. Luckily there aren't many + we are not that creative in naming :) https://github.com/apache/airflow/pull/60483
I've read the statements in the thread until now, and I'm not strongly against making all operators idempotent by default without a flag, as customizations could be made by subclassing + extra "if exist" statement. It's more important for me that the behavior will be mostly consistent in such operators (with granted exemptions, where applicable). This would make it clearer for contributors and maintainers as it breaks with the API, which is not trivial IMO. I'll give this discussion a couple of more days, and if there are no strong objections against the apparent consensus, I'll start a lazy consensus thread + create a GitHub issue to align all relevant operators. Shahar On Tue, Jan 13, 2026 at 9:07 PM Jarek Potiuk <[email protected]> wrote: > I think yes - we should strive to have the delete "kinds" of operations to > silently ignore already deleted content. We can agree on it as a "default" > behaviour - and probably have it written somewhere in best practices for > writing operators. > > But - we also should be pragmatic in the sense that a) there might be > (justified) exceptions b) it's not very likely we will be able to > consistently and automatically enforce it, and it almost guarantees that at > any point in time some of those will be following different semantics - > because we forget about it over time or won't realise this is happening. > > I wish we had some way of enforcing or at least detecting it :) > > On Tue, Jan 13, 2026 at 7:17 PM Daniel Standish via dev < > [email protected]> wrote: > > > Yeah just suppress the exception. > > > > Task should not fail cus object already deleted. > > > > The intent is that the thing is not there, not to verify that it was and > > that you deleted it. > > > > If users want diff behavior they can subclass. > > > > Simpler is better than complex, re the flag idea. Does not seem like a > case > > for a global param. > > > > > > > > On Tue, Jan 13, 2026 at 9:49 AM Ash Berlin-Taylor <[email protected]> > wrote: > > > > > Given then general rule of idempontency of Airflow operators, I think > we > > > should break with the API. The intent of a “Delete” operator is to > ensure > > > the resource no longer exists. So not found is not an error in that > > model. > > > This makes such operators safe to retry. If it threw an error when it > was > > > deleted then they are not safe to retry, and this is not the model > > Airflow > > > operators should have. > > > > > > No flag should be needed — if someone doesn’t want this behavoiur than > I > > > would say they should use the hook or API directly. The point of the > > > pre-built operators is to package up behaviour like this in a > consistent > > > manner. > > > > > > My 2c: we should standardised on remove the “ignore_if_missing” flags > on > > > operators and always catching not exists style errors in operators and > > > treating those as success. > > > > > > -ash > > > > > > > On 12 Jan 2026, at 22:06, Ferruzzi, Dennis <[email protected]> > > wrote: > > > > > > > > Yeah, generally when I add operators to the Amazon provider package, > my > > > intent is to preserve the behavior of the API,. I'm not claiming > that's > > > necessarily the right answer, but that's what i do and I like that > idea. > > > > > > > > I wouldn't be against the idea of a flag, but I think as a general > rule > > > we should assume the default behavior should be familiar to someone who > > is > > > used to using the API before moving to Airflow without surprises. Of > > > course, his may be an XKCD1172 issue [1] and maybe we SHOULD insist on > a > > > standard expectation across providers.... > > > > > > > > [1] https://xkcd.com/1172/ > > > > > > > > > > > > - ferruzzi > > > > > > > > > > > > ________________________________ > > > > From: Shahar Epstein <[email protected]> > > > > Sent: Monday, January 12, 2026 12:51 PM > > > > To: [email protected] > > > > Subject: [EXT] [DISCUSS] Idempotency of "Delete" operators > > > > > > > > CAUTION: This email originated from outside of the organization. Do > not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > > certain que le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > Hey all, > > > > > > > > For the sake of this discussion, the idempotency of Delete operators > > > means > > > > that when an operator is applied to a non-existing resource, it > > catches a > > > > “not found” exception and returns success. By default, the operators > I > > > > refer to return an exception if the resource is not found. > > > > > > > > Following this PR #60083 < > https://github.com/apache/airflow/pull/60083 > > >, > > > > I've realized that we don't handle idempotency consistently in > > different > > > > "Delete" operators across different providers. For example, in some > GCP > > > > operators there's a tendency to catch "NotFound" exception without > any > > > flag > > > > (DeletePipelineJobOperator > > > > < > > > > > > https://github.com/apache/airflow/blob/bd133c0ebb1219b72f4cb7998b2f5f55c8aff200/providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py#L522 > > > >), > > > > while in some Microsoft's operators we handle it by flag (like > > > > "ignore_if_missing" in WASBDeleteBlob > > > > < > > > > > > https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/_api/airflow/providers/microsoft/azure/operators/wasb_delete_blob/index.html#module-contents > > > > > > > > ). > > > > > > > > My questions for discussion are as follows: > > > > 1. Do we want to require that idempotency for operators (that > normally > > > > return an error) will be handled by a flag? > > > > 2. If we agree on having a flag, what should be the default value of > > this > > > > flag? > > > > 3. Given the answer to the last question, should we break the > interface > > > of > > > > existing operators where applicable? > > > > > > > > My answers: > > > > 1. If the original API usually returns an error, catching NotFound > > error > > > > and returning success "defies" the intent of the original API, and I > > > think > > > > that it might be confusing and even limiting in some cases. > Therefore, > > if > > > > originally a "Delete" operator fails and we want to catch this > > exception > > > - > > > > I think that we should at least define a flag for that (preferably > > with a > > > > consistent name across all operators). > > > > 2. Default behavior should reflect the original intent of the API. > > > > 3. Slowly, but carefully - yes: changing both the default value, and > > the > > > > name of the flag to be consistent. > > > > > > > > I'll be happy to hear more opinions! > > > > > > > > > > > > Shahar > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
