GitHub user AffixIO created a discussion: Task retries after approval expiry: reconciling a committed export without authorising a new one
I'm testing a task where permission to perform a data export expires, but a retry may still need to confirm an export that already completed. I'd appreciate a sanity check on the Airflow lifecycle assumptions. The local case is deliberately small: a TaskFlow task validates a signed approval for fixed fictional inputs, writes the export and its operation record in one SQLite transaction, then raises an exception after commit. On retry, it checks for that operation first. An identical completed operation returns `RECONCILED_EXISTING`; if nothing was committed, the approval must still be valid before a new write. The operation ID stays fixed across attempts. On Airflow 3.1.0, `dag.test()` ran the first attempt, marked it up for retry, then completed on the second attempt with one export row. Separate tests cover expiry before a new write and reconciliation after expiry. This only demonstrates a local transactional sink, not exactly-once delivery to an external system or a distributed executor test. Is keeping that distinction entirely inside the task callable the appropriate pattern, particularly when a failed task is manually cleared? My intended behaviour is that clearing the task does not itself grant permission for a new export: the retained operation record still governs reconciliation, and a new logical export needs a newly approved operation ID. I'm not using `try_number` as the idempotency key; it is only used to inject the first-attempt failure in the demo. The runnable example and test results are at https://github.com/AffixIO/airflow-approved-export. I maintain AffixIO, whose local SDK supplies the signature check in this example. There's no request to add it to Airflow, and no real data or external export endpoint is involved. If there's an existing example addressing expiring authorisation versus completed-side-effect reconciliation, I'd rather build on that than duplicate it. GitHub link: https://github.com/apache/airflow/discussions/76421 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
