Hi Jarek, Thanks for reviewing the AIP and sharing the feedback here: https://lists.apache.org/thread.html/nzd1mvg5d7cgqbtqv0sj3h276kohw2s9
Bringing the conversation back to this discussion thread. I've worked through the points raised and updated the page accordingly. 1. Rollback only works for versions that were already pinned Fair point. The AIP now adds an opt-in mechanism to address this gap. When enabled, a code-only change creates a new DagVersion instead of refreshing one that task instances reference, so every version that has run stays a rollback target. It's disabled by default because it can add up to one version per bundle advance in a large shared bundle, and the AIP states what rollback covers with it disabled: https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-DagVersionImmutability Christos and I discussed the commit-based pin alternative thoroughly offline and are now aligned on the proposed DagVersion pin. Christos' comments on the page confirm this alignment. The AIP has also been revised according to the latest comments. 2. Relation to "trigger with bundle_version" The overlap is limited to one-off runs of healthy Dags. An explicit bundle_version doesn't change the version used by scheduled runs or other callers, and the trigger API rejects stale Dags and Dags with import errors, whereas the proposed pin keeps working. On the other hand, the adhoc trigger behavior can also be supported through the proposed mechanism. A pin update followed by a trigger can also mimic the behavior, but the AIP doesn't propose deprecating the other mechanism. The version lookup problem is also fixed at the source through the option proposed above, so the existing lookup resolves for both features. Details: https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-Relationtoexplicitbundle_versiontriggers 3. Divergence detection needs a home I agree with your analysis. Worker-side detection needs a new Execution API call, and it would need its own warning type. Upon thinking about it in detail, I'd like to move divergence detection out of AIP-109 into a follow-up discussion thread/AIP. I'd like your view before I change the page. Reasons for the proposal: - It isn't specific to pinning. An impure Dag can already diverge from its recorded version today when a task is rerun, or upon external state changes while a run is executing. - The serialized Dag may not capture every Dag or operator argument, so a hash comparison can't be exhaustive. Whether that is good enough, or a finer signal is needed, is a discussion of its own. - The choice between a purpose-specific (hash reconciliation) Execution API endpoint and a more general channel for warnings observed on workers is an Execution API and security model decision rather than a version pinning one. - Hashing on the worker needs the same serializer as on Dag processor, and comparisons are only valid when both sides use the same serializer version. If that works, I'll replace the Divergence Detection section with a pointer to the follow-up and drop it from the completion criteria. The limitation itself stays documented under "Execution-Time Divergence for Impure Dags" and Out of Scope > "Task Logic Divergence for Impure Dags". https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-Execution-TimeDivergenceforImpureDags https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-OutofScope 4. Open review threads - As mentioned above, Christos has replied to both threads: after we discussed it offline, he withdrew the commit-SHA suggestion. He's also aligned that any limitations inherited from Dag versioning shouldn't block the feature as long as they're clearly documented. They're covered in the Applicability, Rollback availability and Out of Scope sections. - I had already replied to both of Ash's comments on the AIP earlier, but I followed up again end of Sept whether those comments could be considered addressed. Ash said they'd take a look at the AIP again. Since then, they've suggested a more user-facing name for the AIP, which I've replied to, but I'm waiting to hear back. 5. Two more from checking the code - Permissions: pin changes now require a new access entity, DagAccessEntity.ACTIVE_VERSION, separate from Dag edit. It's proposed here: https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-AuthorizationforVersionPin/Unpin A related gap: triggering with bundle_version today only needs the trigger permission (POST on the RUN access entity). Changing that would affect existing callers of #61550, so I've left it out of this AIP, but it's essentially a loophole around the new access entity. - db clean: cleanup will skip pinned Dag rows and pinned versions, and the text is also updated accordingly. https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning#AIP109DAGVersionPinning-VersionRetention 6. Target release Added at the top of the page: "Target release: 3.5.0". 7. Other changes since the last voting round For anyone re-reviewing, these are the other notable edits: - Motivation and Use cases: rephrased to make the problem and the goals clearer. The motivation now focuses on the release risk of shared code (a change that passes its own CI can still break Dags owned by other teams or fail against the runtime on workers) and on parse timing deciding when code goes live. The use cases are now owner-controlled cutovers, quick selective recovery and tiered rollout. - Operating Models: split out rollback availability, which now lives with the DagVersion lifecycle changes - Added a high-level diagram covering the involved components and interaction edges. - API changes: added "Resolving the current Dag definition". Endpoints that load a Dag's current definition not in the context of a dag run or an explicit version (like Dag details, task list, graph, grid and calendar views, backfill dry-run) resolve the active version instead of the latest. "Synchronous Resolution on Pin/Unpin" moved under API changes, and the trigger API now admits pinned Dags that are stale or have import errors, matching the scheduler. - Dag Version Immutability: restructured as: existing lifecycle, the pin guard, a new opt-in option, rollback availability and the note on hash-identical versions. - Completion criteria: updated to match all of the above. Thanks again for the concrete list of comments, Jarek. It's helped make the document better. Please take some time to review when possible; I'll continue addressing any comments. Thanks, Piyush On Wed, Sep 23, 2026 at 3:07 PM Piyush Maheshwari < [email protected]> wrote: > Hi all, > This is a gentle nudge on the AIP-109 review. > > For context, please check the latest vote thread and my response here: > https://lists.apache.org/thread/78nbhmmnxxz4omdwx02v42jc87snn3zt > > If you have any open concerns, please share what changes or decisions we > should make before proceeding. > Aligning on the acceptance criteria would be really helpful so I can > address them and call another vote. > > If anyone needs more time to review the proposal, please let me know. I’m > also happy to follow up over Slack or another convenient communication > channel to clarify outstanding points and help bring the discussion to > closure. I can summarize and post any conclusions back on this thread. > > Thanks, > Piyush > > On Mon, Sep 7, 2026 at 12:03 PM Piyush Maheshwari < > [email protected]> wrote: > >> Hi everyone, >> >> As the Airflow summit ended last week, I've restarted a fresh vote on the >> AIP today. >> AIP: >> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning >> Vote thread: >> https://lists.apache.org/thread/78nbhmmnxxz4omdwx02v42jc87snn3zt >> >> Thanks again for the reviews. I've addressed all comments. >> >> Regards, >> Piyush >> >> On Sat, Aug 22, 2026 at 9:34 PM Przemysław Mirowski <[email protected]> >> wrote: >> >>> Hi Piyush, >>> >>> I checked the AIP again, and it looks much better! Thanks for iterating >>> over this proposal. I left a few minor editorial comments, and I also saw >>> that Christos left comments we should address before voting. >>> >>> Please also keep in mind that most of the maintainers are currently >>> occupied by the upcoming Airflow Summit. Personally, I would wait with >>> voting for a week or two after the conference to give maintainers time to >>> catch up. >>> >>> Regards, >>> Przemek >>> >>> On 2026/08/19 06:45:30 Piyush Maheshwari wrote: >>> > Hi everyone >>> > Gentle reminder to review the AIP >>> > >>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning >>> > . >>> > >>> > Thank you. >>> > >>> > On Thu, 13 Aug 2026 at 4:05 PM, Piyush Maheshwari < >>> > [email protected]> wrote: >>> > >>> > > Hi everyone, >>> > > I'd like to request the community's review of the AIP >>> > > >>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/421958312/AIP-109+DAG+Version+Pinning >>> > > . >>> > > If there are no differing opinions I'd like to take it to a vote >>> soon. >>> > > >>> > > Also, thank you Ephraim for the re-review. I've addressed all >>> comments >>> > > from the latest review and updated the AIP accordingly. >>> > > >>> > > Thanks and regards, >>> > > Piyush >>> > > >>> > > On Fri, Aug 7, 2026 at 4:10 PM Ephraim Anierobi < >>> > > [email protected]> wrote: >>> > > >>> > >> Hi Piyush, >>> > >> >>> > >> Thank you for this summary. It is accurate, and it matches where we >>> > >> landed. >>> > >> Thank you also for the many rounds of changes. The document is much >>> > >> clearer >>> > >> now than when we started. >>> > >> >>> > >> I want to state my position plainly for the thread. My main concern >>> was >>> > >> that a pin on a DagVersion could not hold, because the row could >>> change >>> > >> under it and the bundle pointer could move. The pin guard fixes >>> both. With >>> > >> that, plus the retention rule, the stale DAG change, the purity >>> scope, and >>> > >> the divergence warning, I no longer object to the pin target. The >>> AIP now >>> > >> says what it delivers and what it does not. That was my real goal. >>> > >> >>> > >> One thing I want to be clear about. My review was about whether the >>> design >>> > >> holds, not about whether the feature is worth building. Those are >>> two >>> > >> different questions, and the second one belongs to the community. >>> > >> >>> > >> The feature adds pin awareness in several core places: the version >>> write >>> > >> path, db clean, the scheduler's eligibility query for import errors >>> and >>> > >> stale DAGs, DagModel and asset field resolution in the DAG >>> processor, and >>> > >> a >>> > >> synchronous update when a pin changes. Each change is small on its >>> own. >>> > >> Together they mean these paths must consider pins from now on. >>> > >> >>> > >> What it gives back is per-DAG control over when a version takes >>> effect, >>> > >> for >>> > >> pure DAGs that come from versioned bundles. That is real and useful >>> for >>> > >> the >>> > >> teams who need it. Whether it is worth the added complexity in core >>> paths >>> > >> is a judgment call, and voters should make it with both sides >>> stated. I am >>> > >> not voting against it. I only do not want my resolved concerns to >>> be read >>> > >> as a view on that question. >>> > >> >>> > >> I have left my remaining comments on the document itself. They are >>> small >>> > >> and none of them changes the design, but I think they should be >>> settled >>> > >> before the vote. >>> > >> >>> > >> On the vote, I agree with starting it again instead of continuing >>> the old >>> > >> thread. The document has changed a lot. A fresh call, with a short >>> note on >>> > >> what changed, would be fair to everyone who already voted. >>> > >> >>> > >> Thanks again for your patience through all of this. The proposal is >>> in >>> > >> much >>> > >> better shape. >>> > >> >>> > >> Regards, >>> > >> Ephraim >>> > >> >>> > >> On Wed, 5 Aug 2026 at 10:55, Piyush Maheshwari < >>> > >> [email protected]> >>> > >> wrote: >>> > >> >>> > >> > Hi everyone, >>> > >> > >>> > >> > I'd like to thank Ephraim for the depth of this review. Following >>> the >>> > >> > previous email, we had a series of detailed follow-up >>> discussions, and I >>> > >> > believe we have converged. This email summarizes where we landed, >>> > >> answers >>> > >> > the concerns from the previous email for the benefit of the >>> thread, and >>> > >> > describes the updates I have now made to the AIP. I would like the >>> > >> > community's review of the updated AIP ( >>> > >> > >>> > >> > >>> > >> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >>> > >> > ) >>> > >> > >>> > >> > == Precise scope of the proposal == >>> > >> > >>> > >> > The discussion sharpened what the AIP promises, and it now states >>> this >>> > >> > explicitly: >>> > >> > >>> > >> > - The goal is deterministic, per-DAG control over when a new >>> version of >>> > >> a >>> > >> > DAG takes effect, plus an instant, API-driven rollback to a prior >>> > >> version, >>> > >> > without affecting other DAGs sharing the same bundle. >>> > >> > >>> > >> > - A pin fixes the serialized DAG the scheduler enforces (task set, >>> > >> edges, >>> > >> > schedule, display fields) and the bundle (name and version) the >>> worker >>> > >> > checks out. It provides graph and code-level determinism, not >>> > >> > behavior-level determinism, since a task that reads live external >>> state >>> > >> at >>> > >> > execution time still does so under a pin. >>> > >> > >>> > >> > - Pinning guarantees are stated for DAGs that are "pure", meaning >>> a >>> > >> given >>> > >> > bundle (name and version) always produces the same serialized >>> DAG. The >>> > >> AIP >>> > >> > now defines this purity condition up front. >>> > >> > >>> > >> > - For impure DAGs, different pinning guarantees are now >>> documented for >>> > >> > impure DAGs. A divergence between the pinned serialized DAG's >>> snapshot >>> > >> and >>> > >> > the worker's fresh parse for execution-time fields is silent. The >>> AIP >>> > >> now >>> > >> > explicitly documents this as "silent divergence," a limitation, >>> and adds >>> > >> > detection for it. >>> > >> > >>> > >> > == Answers to the five concerns from the previous email == >>> > >> > >>> > >> > 1. "The latest DagVersion is mutable, so a pin does not freeze >>> content." >>> > >> > Addressed by introducing a pin guard in the DAG Processor's >>> > >> > version-recording path. A pinned DagVersion is treated as >>> referenced, so >>> > >> > its serialized content is never overwritten in place; a content >>> change >>> > >> > creates a new DagVersion instead (the same path that already runs >>> when >>> > >> task >>> > >> > instances reference the row). Note that the DAG processor only >>> mutates >>> > >> the >>> > >> > latest DagVersion row, so a pin on an older version is already >>> immutable >>> > >> > today; the guard changes behavior only when the pinned version is >>> also >>> > >> the >>> > >> > latest. >>> > >> > >>> > >> > 2. "A pin would not pin the source code, because bundle_version is >>> > >> > refreshed in place on hash-identical parses." >>> > >> > Addressed by the same guard, which stops the in-place >>> > >> > bundle_version/version_data refresh from applying to a pinned >>> row. When >>> > >> the >>> > >> > bundle advances without a serialized-DAG change while the pinned >>> > >> version is >>> > >> > the latest, the processor instead creates one new DagVersion >>> carrying >>> > >> the >>> > >> > new bundle_version, which becomes the unpinned latest; subsequent >>> > >> no-change >>> > >> > bundle advances keep updating that new latest in place exactly as >>> today. >>> > >> > This preserves the intent of the in-place refresh (the DAG's >>> latest >>> > >> version >>> > >> > keeps tracking the bundle) while the pinned version's bundle >>> reference >>> > >> > stays frozen. Since new DagRuns will also carry the pinned >>> bundle, and >>> > >> > workers resolve their bundle checkout from the DagRun, the >>> scheduler's >>> > >> view >>> > >> > and the worker's code stay on the same version. Note that this >>> version >>> > >> > creation, without a hash change, occurs only per pin advance, not >>> once >>> > >> per >>> > >> > commit. >>> > >> > >>> > >> > 3. "Version identity is often accidental (impure DAGs)." >>> > >> > Addressed by scoping (guarantees stated for pure DAGs, with the >>> purity >>> > >> > condition defined in the AIP) and by divergence detection, where a >>> > >> > DagWarning is raised when a pinned DAG's fresh parse hash no >>> longer >>> > >> matches >>> > >> > the pinned DagVersion's hash, naming the pinned version. The >>> check is a >>> > >> > no-op for pure DAGs. >>> > >> > >>> > >> > 4. "Cleanup can delete a pinned version and silently unpin via ON >>> DELETE >>> > >> > SET NULL." >>> > >> > Addressed: airflow db clean will retain any DagVersion referenced >>> by an >>> > >> > active pin, the same way it retains task-instance-referenced >>> versions. >>> > >> > >>> > >> > 5. "The deletion rule works against the main use case." >>> > >> > Revised: a dag_id that disappears from the latest parse would >>> still be >>> > >> > marked stale on DagModel as today, but the scheduler's >>> eligibility query >>> > >> > will admit a stale DAG while it has an active pin, so it keeps >>> running >>> > >> on >>> > >> > its pinned version (parallel to the import-error exception >>> already in >>> > >> the >>> > >> > AIP). Actual removal requires an explicit unpin first, after >>> which the >>> > >> > normal stale-to-deleted path applies unchanged. This turns the >>> defect >>> > >> > scenario from a silent removal into an explicit act. >>> > >> > >>> > >> > == On pinning the bundle instead of the DagVersion == >>> > >> > >>> > >> > We discussed this alternative at length, in two variants, and the >>> AIP's >>> > >> > "Alternatives" section now covers both: >>> > >> > >>> > >> > - A global bundle pin (keep a history of bundle versions and >>> freeze the >>> > >> > whole bundle on one) is simple and atomic, but it pins every DAG >>> in the >>> > >> > bundle at once. It cannot keep a subset of DAGs on an older >>> version >>> > >> while >>> > >> > the rest advance, and per-DAG control is the primary use case, >>> since a >>> > >> > breaking change in a shared bundle typically affects only some of >>> the >>> > >> DAGs >>> > >> > it emits. We see it as a potentially complementary, >>> coarser-grained >>> > >> control >>> > >> > rather than a replacement. >>> > >> > >>> > >> > - A per-DAG pin keyed by the bundle (name and version) is not >>> uniquely >>> > >> > resolvable to a serialized DAG for impure DAGs, since the pair >>> can map >>> > >> to >>> > >> > several dag_version rows. It therefore cannot be expressed as a >>> foreign >>> > >> key >>> > >> > to a specific version, and the latest matching row can resolve >>> > >> differently >>> > >> > across invocations, which is behavior drift while supposedly >>> pinned. It >>> > >> is >>> > >> > also unstable over time, because a dag_version's bundle_version is >>> > >> > refreshed in place on hash-identical parses, so a previously valid >>> > >> bundle >>> > >> > version can stop resolving exactly when a rollback needs it. >>> Freezing >>> > >> that >>> > >> > would require an immutable bundle-to-dag_version mapping, which >>> amounts >>> > >> to >>> > >> > pinning the dag_version too and converges towards this proposal. >>> > >> > >>> > >> > == Operating models and rollback availability == >>> > >> > >>> > >> > Ephraim raised an important point about rollback availability. An >>> > >> on-demand >>> > >> > pin can only roll back to versions Airflow already recorded, so a >>> change >>> > >> > that never altered the serialized DAG (e.g. one confined to an >>> imported >>> > >> > helper module) has no distinct version to return to unless it was >>> > >> pinned at >>> > >> > the time. The AIP now states this honestly under a new "Usage >>> Models and >>> > >> > Rollback Availability" section, which describes two models the >>> primitive >>> > >> > serves: >>> > >> > >>> > >> > 1. On-demand pinning (incident response, planned cutovers): >>> rollback >>> > >> > targets are the recorded DAG versions. Changes that altered the >>> > >> serialized >>> > >> > DAG (and had task instances) are reachable; changes that did not >>> are a >>> > >> > stated limitation. >>> > >> > 2. Standing pins driven by an external release orchestrator: >>> every DAG >>> > >> > always carries a pin and the orchestrator advances it per >>> deployment. >>> > >> Every >>> > >> > pinned state remains an immutable rollback target, closing the gap >>> > >> above, >>> > >> > at the cost of operating the orchestrator. The consequences of >>> the same >>> > >> are >>> > >> > also covered in the AIP. >>> > >> > >>> > >> > Airflow itself only ships the pin/unpin primitive; neither model >>> is >>> > >> > mandatory to use this feature. >>> > >> > >>> > >> > == Updated AIP and proposed timeline == >>> > >> > >>> > >> > I have updated AIP-109 with all of the above: >>> > >> > >>> > >> > >>> > >> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >>> > >> > >>> > >> > I would appreciate the community's review of the updated >>> proposal. If >>> > >> > multiple maintainers review it and if no major open questions or >>> > >> comments >>> > >> > remain by Wednesday, August 12, I plan to restart the vote phase. >>> > >> > >>> > >> > Thanks again to Ephraim for the multiple rounds of thorough >>> review that >>> > >> got >>> > >> > the proposal to this shape. >>> > >> > >>> > >> > Regards, >>> > >> > Piyush >>> > >> > >>> > >> > On Tue, Jul 28, 2026 at 3:47 PM Ephraim Anierobi < >>> > >> > [email protected]> >>> > >> > wrote: >>> > >> > >>> > >> > > Thank you for this proposal, and for the careful work behind >>> it. I >>> > >> took a >>> > >> > > second, deeper look at the document, and I want to share my >>> concerns, >>> > >> > > because I believe they change the direction rather than the >>> details. >>> > >> > > >>> > >> > > First, I want to recognize the real problem the AIP identifies. >>> The >>> > >> parse >>> > >> > > loop controls when new code becomes active, and that timing is >>> not >>> > >> > > deterministic. Teams need a way to control activation time, and >>> they >>> > >> > need a >>> > >> > > fast rollback path. This is a genuine gap, and I am glad the >>> AIP puts >>> > >> a >>> > >> > > spotlight on it. >>> > >> > > >>> > >> > > After the second review, however, I believe the proposal rests >>> on a >>> > >> > > misunderstanding of what a DagVersion is. A DagVersion is a >>> historical >>> > >> > > record of what Airflow observed after a parse. It is an output >>> of the >>> > >> > > system, not an input to it. Dag versioning exists so that >>> running and >>> > >> > past >>> > >> > > task instances keep a faithful record of the code they ran >>> against. >>> > >> > *Bundle >>> > >> > > versioning* is the mechanism that decides which source code a >>> run >>> > >> uses. >>> > >> > The >>> > >> > > AIP reads DagVersion as a release artifact that users can >>> select and >>> > >> > > activate, and the current implementation does not support that >>> > >> reading: >>> > >> > > >>> > >> > > 1. *The latest DagVersion is mutable*. When no task instance >>> > >> references >>> > >> > it, >>> > >> > > write_dag() replaces its serialized content, hash, and Dag code >>> in >>> > >> place, >>> > >> > > on the same version ID. A pin does not create a task instance >>> > >> reference. >>> > >> > So >>> > >> > > a user could pin the current version before a risky merge, and >>> the >>> > >> next >>> > >> > > parse could replace the content behind that pin. The pin would >>> not >>> > >> give >>> > >> > the >>> > >> > > determinism it promises. >>> > >> > > >>> > >> > > 2. *A pin on the current DagVersion would not pin the source >>> code*. >>> > >> When >>> > >> > a >>> > >> > > new commit does not change the serialized content, `write_dag()` >>> > >> updates >>> > >> > > bundle_version on the latest DagVersion in place. A change to a >>> helper >>> > >> > > module or an SQL file does not change the serialized hash, so >>> workers >>> > >> > would >>> > >> > > run the new commit under the pinned version. >>> > >> > > >>> > >> > > 3. *Version identity is often accidental.* Parse-time >>> variability and >>> > >> > > serializer defects can create versions that no one intended. >>> Airflow >>> > >> > ships >>> > >> > > an inflation checker and a dedicated Dag warning type because >>> this >>> > >> > problem >>> > >> > > is real and common. Dag factories at large scale, the AIP's own >>> key >>> > >> > > example, are the most exposed. A pin would often select an >>> accidental >>> > >> > > snapshot rather than a deliberate release. >>> > >> > > >>> > >> > > 4. *The proposal does not address cleanup*. Today, `airflow db >>> clean` >>> > >> > keeps >>> > >> > > the latest version per Dag and skips only versions that task >>> instances >>> > >> > > reference. An older pinned version with no task instances is >>> eligible >>> > >> for >>> > >> > > deletion, and the proposed ON DELETE SET NULL behavior would >>> then >>> > >> > silently >>> > >> > > return the Dag to the latest version, exactly when the pin >>> matters >>> > >> most. >>> > >> > > The document does not cover this interaction. >>> > >> > > >>> > >> > > 5. *The deletion rule works against the main use case*. The AIP >>> states >>> > >> > that >>> > >> > > removal from a new bundle version deletes the Dag, even when it >>> is >>> > >> > pinned. >>> > >> > > A defect in a dag-factory file that changes or drops Dag IDs >>> removes >>> > >> Dags >>> > >> > > from the parse results. In that case, Airflow would delete the >>> Dag and >>> > >> > > discard the pin, so the protection would be absent in one of >>> the most >>> > >> > > common failure modes the AIP is motivated by. >>> > >> > > >>> > >> > > These are not implementation gaps that more safeguards can >>> close. They >>> > >> > all >>> > >> > > follow from the same source: the design asks a history table to >>> act >>> > >> as a >>> > >> > > deployment mechanism. Making that work would mean redefining >>> what >>> > >> > > DagVersion is, and I do not think we should. >>> > >> > > >>> > >> > > The goal itself is worth pursuing at the right layer. >>> Deterministic >>> > >> > cutover >>> > >> > > and instant rollback are operations on the *source artifact*. >>> Bundle >>> > >> > > versions already represent the source, and pinning or rolling >>> back at >>> > >> the >>> > >> > > bundle layer would be atomic for all Dags and all support files >>> at >>> > >> once. >>> > >> > I >>> > >> > > would be glad to help explore that direction. >>> > >> > > >>> > >> > > Thanks again for raising this discussion. The problem is real, >>> and I >>> > >> hope >>> > >> > > we can solve it together on a foundation that supports it. >>> > >> > > >>> > >> > > - Ephraim >>> > >> > > >>> > >> > > On Mon, 27 Jul 2026 at 06:53, Piyush Maheshwari < >>> > >> > > [email protected]> >>> > >> > > wrote: >>> > >> > > >>> > >> > > > Thank you, Sumit. >>> > >> > > > >>> > >> > > > I've started a voting thread as suggested: >>> > >> > > > >>> https://lists.apache.org/thread/bff758c5p68v1mtzhflytn7yzv0d8l1t >>> > >> > > > >>> > >> > > > Regards, >>> > >> > > > Piyush >>> > >> > > > >>> > >> > > > On Fri, Jul 24, 2026 at 12:08 PM Sumit Maheshwari < >>> > >> > > [email protected]> >>> > >> > > > wrote: >>> > >> > > > >>> > >> > > > > I've reviewed it again today and left some comments. >>> Overall, it >>> > >> > looks >>> > >> > > > > great and would be a great feature add in Airflow. I think >>> you can >>> > >> > > start >>> > >> > > > an >>> > >> > > > > official voting thread on this AIP. >>> > >> > > > > >>> > >> > > > > >>> > >> > > > > On Sun, Jul 19, 2026 at 7:32 PM Piyush Maheshwari < >>> > >> > > > > [email protected]> wrote: >>> > >> > > > > >>> > >> > > > > > Hi everyone, >>> > >> > > > > > Thanks for the reviews on AIP-109 ( >>> > >> > > > > > >>> > >> > > > > > >>> > >> > > > > >>> > >> > > > >>> > >> > > >>> > >> > >>> > >> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >>> > >> > > > > > ). >>> > >> > > > > > Over the past few weeks, I implemented the feature in our >>> fork >>> > >> > > (nearly >>> > >> > > > > > done). >>> > >> > > > > > Based on the insights and learnings from this process, I >>> have >>> > >> > further >>> > >> > > > > > refined the AIP while addressing all comments. >>> > >> > > > > > >>> > >> > > > > > Please review the updated version. I'll wait to collect >>> some >>> > >> > feedback >>> > >> > > > and >>> > >> > > > > > then take it to a vote. >>> > >> > > > > > I can start contributing PRs as soon as the AIP is >>> approved. >>> > >> > > > > > >>> > >> > > > > > Thanks and regards, >>> > >> > > > > > Piyush >>> > >> > > > > > >>> > >> > > > > > On Wed, Jun 3, 2026 at 2:01 PM Ephraim Anierobi < >>> > >> > > > > [email protected] >>> > >> > > > > > > >>> > >> > > > > > wrote: >>> > >> > > > > > >>> > >> > > > > > > Hi Piyush, >>> > >> > > > > > > >>> > >> > > > > > > Thanks for the AIP. I’ve added a few comments to the >>> doc. >>> > >> Please >>> > >> > > > take a >>> > >> > > > > > > look, as I think we should iron out those areas and >>> fully >>> > >> > > understand >>> > >> > > > > the >>> > >> > > > > > > implications before moving forward. >>> > >> > > > > > > >>> > >> > > > > > > Regards >>> > >> > > > > > > - Ephraim >>> > >> > > > > > > >>> > >> > > > > > > On Tue, 26 May 2026 at 18:24, Przemysław Mirowski < >>> > >> > > [email protected] >>> > >> > > > > >>> > >> > > > > > > wrote: >>> > >> > > > > > > >>> > >> > > > > > > > I checked the API - +1. Thanks for writing this up! >>> > >> > > > > > > > >>> > >> > > > > > > > Thanks Nathan for mentioning the #63884 PR. It is nice >>> > >> addition >>> > >> > > > > > (already >>> > >> > > > > > > > merged), and I think that it will be really useful >>> for users >>> > >> > who >>> > >> > > > got >>> > >> > > > > > used >>> > >> > > > > > > > to Airflow 2 retry mechanism. I think also that your >>> PR and >>> > >> > > API-109 >>> > >> > > > > are >>> > >> > > > > > > > complementary as one is focusing on the versioning >>> behaviour >>> > >> > for >>> > >> > > > > > retrying >>> > >> > > > > > > > task, the latter is focusing on versioning behaviour >>> for new >>> > >> > Dag >>> > >> > > > > Runs. >>> > >> > > > > > > > ________________________________ >>> > >> > > > > > > > From: Christos Bisias <[email protected]> >>> > >> > > > > > > > Sent: 25 May 2026 09:46 >>> > >> > > > > > > > To: [email protected] <[email protected]> >>> > >> > > > > > > > Subject: Re: [DISCUSS] DAG Version Pinning for >>> Deployment >>> > >> > Gating >>> > >> > > > > > > (Building >>> > >> > > > > > > > on AIP-63) >>> > >> > > > > > > > >>> > >> > > > > > > > Hello, >>> > >> > > > > > > > >>> > >> > > > > > > > I'm a little late on the discussion but I just came >>> across >>> > >> the >>> > >> > > AIP >>> > >> > > > > and >>> > >> > > > > > I >>> > >> > > > > > > > like this idea. I've actually been thinking of >>> working on >>> > >> > > something >>> > >> > > > > > > > similar, to allow people to handle a bad rollout by >>> > >> reverting >>> > >> > to >>> > >> > > > old >>> > >> > > > > > code >>> > >> > > > > > > > for that run without a full slow release pipeline. >>> And this >>> > >> > > covers >>> > >> > > > > it. >>> > >> > > > > > > > >>> > >> > > > > > > > In my opinion, this seems more like a natural step >>> towards >>> > >> what >>> > >> > > dag >>> > >> > > > > > > > versioning is supposed to do, than a new feature. >>> > >> > > > > > > > >>> > >> > > > > > > > Thank you, >>> > >> > > > > > > > Christos >>> > >> > > > > > > > >>> > >> > > > > > > > >>> > >> > > > > > > > On Mon, May 25, 2026 at 8:15 AM Piyush Maheshwari < >>> > >> > > > > > > > [email protected]> wrote: >>> > >> > > > > > > > >>> > >> > > > > > > > > Hi everyone, >>> > >> > > > > > > > > I wanted to send a gentle nudge to review AIP-109: >>> DAG >>> > >> > Version >>> > >> > > > > > Pinning >>> > >> > > > > > > ( >>> > >> > > > > > > > > >>> > >> > > > > > > > > >>> > >> > > > > > > > >>> > >> > > > > > > >>> > >> > > > > > >>> > >> > > > > >>> > >> > > > >>> > >> > > >>> > >> > >>> > >> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >>> > >> > > > > > > > > ). >>> > >> > > > > > > > > If there are no major concerns, I would like to >>> take this >>> > >> to >>> > >> > a >>> > >> > > > vote >>> > >> > > > > > > soon. >>> > >> > > > > > > > > >>> > >> > > > > > > > > Thanks, >>> > >> > > > > > > > > Piyush >>> > >> > > > > > > > > >>> > >> > > > > > > > > On Fri, May 15, 2026 at 7:44 AM Piyush Maheshwari < >>> > >> > > > > > > > > [email protected]> wrote: >>> > >> > > > > > > > > >>> > >> > > > > > > > > > Thanks for the note, Sumit. Based on the >>> feedback, I've >>> > >> > > drafted >>> > >> > > > > an >>> > >> > > > > > > AIP >>> > >> > > > > > > > > > that is now up for review. >>> > >> > > > > > > > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > > > >>> > >> > > > > > > > >>> > >> > > > > > > >>> > >> > > > > > >>> > >> > > > > >>> > >> > > > >>> > >> > > >>> > >> > >>> > >> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-109+DAG+Version+Pinning >>> > >> > > > > > > > > > >>> > >> > > > > > > > > > Would like to get the community's feedback on the >>> same. >>> > >> > > > > > > > > > >>> > >> > > > > > > > > > Nathan, I remember seeing your work (the issue >>> and an >>> > >> older >>> > >> > > PR) >>> > >> > > > > > while >>> > >> > > > > > > > > > reviewing all ongoing work related to DAG >>> versions. I >>> > >> agree >>> > >> > > > with >>> > >> > > > > > the >>> > >> > > > > > > > PR's >>> > >> > > > > > > > > > intent, although I haven't reviewed it yet. >>> > >> > > > > > > > > > I understand your PR makes the version-pinned >>> execution >>> > >> > > > behavior >>> > >> > > > > of >>> > >> > > > > > > > > reruns >>> > >> > > > > > > > > > and backfills configurable. >>> > >> > > > > > > > > > However, this discussion revolves around the >>> behavior >>> > >> for >>> > >> > new >>> > >> > > > > runs >>> > >> > > > > > > > only. >>> > >> > > > > > > > > > We need the capability to pin a DAG to a specific >>> > >> version >>> > >> > for >>> > >> > > > > > future >>> > >> > > > > > > > runs >>> > >> > > > > > > > > > instead. >>> > >> > > > > > > > > > Hope that clarifies. >>> > >> > > > > > > > > > >>> > >> > > > > > > > > > Regards, >>> > >> > > > > > > > > > Piyush >>> > >> > > > > > > > > > >>> > >> > > > > > > > > > On Thu, May 14, 2026 at 5:11 PM Nathan Hadfield < >>> > >> > > > > > > > > [email protected]> >>> > >> > > > > > > > > > wrote: >>> > >> > > > > > > > > > >>> > >> > > > > > > > > >> Hello, >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> I saw AIP-109 that was created in relation to >>> this >>> > >> > > discussion >>> > >> > > > > and >>> > >> > > > > > > > > thought >>> > >> > > > > > > > > >> I’d better mention this PR that I’ve been >>> working on >>> > >> for a >>> > >> > > > while >>> > >> > > > > > and >>> > >> > > > > > > > is >>> > >> > > > > > > > > >> close to being approved. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> https://github.com/apache/airflow/pull/63884 >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> It is very much related to the motivations >>> described >>> > >> and >>> > >> > > > > > implements >>> > >> > > > > > > > the >>> > >> > > > > > > > > >> desire for control over the behaviour when >>> > >> > > > clearing/backfilling >>> > >> > > > > > > runs. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> Happy to discuss best steps for this here or on >>> the PR. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> Cheers, >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> Nathan >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> From: Przemysław Mirowski <[email protected]> >>> > >> > > > > > > > > >> Date: Tuesday, 28 April 2026 at 21:54 >>> > >> > > > > > > > > >> To: [email protected] < >>> [email protected]> >>> > >> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for >>> > >> Deployment >>> > >> > > > Gating >>> > >> > > > > > > > > >> (Building on AIP-63) >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> This Message Is From an External Sender >>> > >> > > > > > > > > >> This message came from outside your organization. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> > P.S. In my opinion, what can be done in/around >>> git, >>> > >> > should >>> > >> > > > be >>> > >> > > > > > done >>> > >> > > > > > > > > >> there. Recreation of CI/CD in any form inside of >>> > >> Airflow >>> > >> > > > itself >>> > >> > > > > is >>> > >> > > > > > > > > >> something which should not be done. >>> > >> > > > > > > > > >> > I'm glad we agree on this :) I suppose we just >>> > >> disagree >>> > >> > on >>> > >> > > > > what >>> > >> > > > > > is >>> > >> > > > > > > > > >> possible outside of Airflow :p >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> I think that we just disagree on what the issue >>> is, >>> > >> not on >>> > >> > > > what >>> > >> > > > > is >>> > >> > > > > > > > > >> possible/should be outside of Airflow. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> > I think we are trying to duplicate what we >>> already >>> > >> have >>> > >> > in >>> > >> > > > > Git. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> Not really if we are only referring to version >>> > >> pinning. As >>> > >> > > far >>> > >> > > > > as >>> > >> > > > > > I >>> > >> > > > > > > am >>> > >> > > > > > > > > >> aware of how things are working, there is no >>> > >> possibility >>> > >> > to >>> > >> > > > > > > determine >>> > >> > > > > > > > > that >>> > >> > > > > > > > > >> Dag after e.g. deployment on 1:00:00 PM will be >>> exactly >>> > >> > > parsed >>> > >> > > > > and >>> > >> > > > > > > > used >>> > >> > > > > > > > > >> since 1:01:00 PM forward. Basically, what version >>> > >> pinning >>> > >> > > > would >>> > >> > > > > > > > provide >>> > >> > > > > > > > > is >>> > >> > > > > > > > > >> the full control of the time since the given >>> version >>> > >> will >>> > >> > be >>> > >> > > > > used >>> > >> > > > > > > > > >> (currently we can only have more-or-less timing >>> which >>> > >> in >>> > >> > > some >>> > >> > > > > > cases, >>> > >> > > > > > > > is >>> > >> > > > > > > > > not >>> > >> > > > > > > > > >> sufficient). The "quick revert" is the >>> consequence of >>> > >> > having >>> > >> > > > > above >>> > >> > > > > > > > > >> possibility. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> Looking at the general concerns, with having that >>> > >> feature >>> > >> > or >>> > >> > > > > not, >>> > >> > > > > > > > users >>> > >> > > > > > > > > >> can pretty easily test things on production, but >>> it >>> > >> just >>> > >> > > > > requires >>> > >> > > > > > > more >>> > >> > > > > > > > > time >>> > >> > > > > > > > > >> between iterations without it. IMHO it will not >>> change >>> > >> the >>> > >> > > > need >>> > >> > > > > > for >>> > >> > > > > > > > > >> Airflow-related platform teams which makes sure, >>> by >>> > >> > > > > > > > standards/policies, >>> > >> > > > > > > > > >> that things are properly tested before production >>> > >> > > deployment. >>> > >> > > > I >>> > >> > > > > > > think >>> > >> > > > > > > > > that >>> > >> > > > > > > > > >> assumption that some users will misuse this >>> feature is >>> > >> > true >>> > >> > > > > (like >>> > >> > > > > > > with >>> > >> > > > > > > > > most >>> > >> > > > > > > > > >> of the features really), but on the other hand >>> it would >>> > >> > > > provide >>> > >> > > > > > more >>> > >> > > > > > > > > >> control for other users. The other solution >>> possibly >>> > >> would >>> > >> > > be >>> > >> > > > to >>> > >> > > > > > > make >>> > >> > > > > > > > > Dag >>> > >> > > > > > > > > >> Processor work more on "events" instead of >>> "simple" >>> > >> > parsing >>> > >> > > > loop >>> > >> > > > > > (I >>> > >> > > > > > > > > recall >>> > >> > > > > > > > > >> that there was some PR couple years ago with PoC >>> of >>> > >> that, >>> > >> > > but >>> > >> > > > I >>> > >> > > > > > > > couldn't >>> > >> > > > > > > > > >> quickly find it). >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> ________________________________ >>> > >> > > > > > > > > >> From: Jarek Potiuk <[email protected]> >>> > >> > > > > > > > > >> Sent: 28 April 2026 17:02 >>> > >> > > > > > > > > >> To: [email protected] < >>> [email protected]> >>> > >> > > > > > > > > >> Subject: Re: [DISCUSS] DAG Version Pinning for >>> > >> Deployment >>> > >> > > > Gating >>> > >> > > > > > > > > >> (Building on AIP-63) >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> Same concerns. I think we are trying to >>> duplicate what >>> > >> we >>> > >> > > > > already >>> > >> > > > > > > have >>> > >> > > > > > > > > in >>> > >> > > > > > > > > >> Git—branches and reverts, for example—by moving >>> what >>> > >> > should >>> > >> > > be >>> > >> > > > > > > managed >>> > >> > > > > > > > > as >>> > >> > > > > > > > > >> part of the development process to Airflow UI. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> Almost everything you describe can be done with: >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> * having a dev/staging system configured >>> properly to >>> > >> use >>> > >> > > > > > dev/staging >>> > >> > > > > > > > > >> branches >>> > >> > > > > > > > > >> * Having a process of managing development and a >>> proper >>> > >> > > > > branching >>> > >> > > > > > > > > strategy >>> > >> > > > > > > > > >> * single git command (for example, `git revert >>> XXXX` >>> > >> > > followed >>> > >> > > > by >>> > >> > > > > > > push >>> > >> > > > > > > > to >>> > >> > > > > > > > > >> the right branch) >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> J. >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> On Tue, Apr 28, 2026 at 10:34 AM Pierre Jeambrun >>> < >>> > >> > > > > > > > [email protected] >>> > >> > > > > > > > > > >>> > >> > > > > > > > > >> wrote: >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> > At first glance I tend to agree with Jens and >>> Niko. >>> > >> > > > > > > > > >> > >>> > >> > > > > > > > > >> > I understand the request, but I agree that this >>> > >> resolves >>> > >> > > > CI/CD >>> > >> > > > > > and >>> > >> > > > > > > > > >> testing >>> > >> > > > > > > > > >> > issues that should probably be remain outside >>> > >> Airflow. >>> > >> > > > > > > > > >> > >>> > >> > > > > > > > > >> > On Mon, Apr 27, 2026 at 7:43 PM Oliveira, Niko >>> < >>> > >> > > > > > > [email protected] >>> > >> > > > > > > > > >>> > >> > > > > > > > > >> > wrote: >>> > >> > > > > > > > > >> > >>> > >> > > > > > > > > >> > > Hey folks! >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > > P.S. In my opinion, what can be done >>> in/around >>> > >> git, >>> > >> > > > should >>> > >> > > > > > be >>> > >> > > > > > > > done >>> > >> > > > > > > > > >> > > there. Recreation of CI/CD in any form >>> inside of >>> > >> > Airflow >>> > >> > > > > > itself >>> > >> > > > > > > is >>> > >> > > > > > > > > >> > > something which should not be done. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > I'm glad we agree on this :) I suppose we >>> just >>> > >> > disagree >>> > >> > > on >>> > >> > > > > > what >>> > >> > > > > > > is >>> > >> > > > > > > > > >> > > possible outside of Airflow :p >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > But at this point I will bow out of the >>> > >> conversation >>> > >> > and >>> > >> > > > let >>> > >> > > > > > > > others >>> > >> > > > > > > > > >> weigh >>> > >> > > > > > > > > >> > > in. I'm not fully convinced any of these >>> requested >>> > >> > > > > behaviours >>> > >> > > > > > > > > require >>> > >> > > > > > > > > >> > > changes to Airflow (I think that's just >>> masking >>> > >> some >>> > >> > dev >>> > >> > > > ops >>> > >> > > > > > > > work). >>> > >> > > > > > > > > >> But >>> > >> > > > > > > > > >> > > also I'm not completely opposed to the change >>> > >> either, >>> > >> > > I'm >>> > >> > > > > more >>> > >> > > > > > > on >>> > >> > > > > > > > > the >>> > >> > > > > > > > > >> > > fence, so if others love the feature by all >>> means >>> > >> > > > implement >>> > >> > > > > > it! >>> > >> > > > > > > :) >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > Cheers, >>> > >> > > > > > > > > >> > > Niko >>> > >> > > > > > > > > >> > > ________________________________ >>> > >> > > > > > > > > >> > > From: Przemysław Mirowski <[email protected] >>> > >>> > >> > > > > > > > > >> > > Sent: Thursday, April 23, 2026 3:06 PM >>> > >> > > > > > > > > >> > > To: [email protected] < >>> [email protected] >>> > >> > >>> > >> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version >>> Pinning >>> > >> for >>> > >> > > > > > Deployment >>> > >> > > > > > > > > Gating >>> > >> > > > > > > > > >> > > (Building on AIP-63) >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > CAUTION: This email originated from outside >>> of the >>> > >> > > > > > organization. >>> > >> > > > > > > > Do >>> > >> > > > > > > > > >> not >>> > >> > > > > > > > > >> > > click links or open attachments unless you >>> can >>> > >> confirm >>> > >> > > the >>> > >> > > > > > > sender >>> > >> > > > > > > > > and >>> > >> > > > > > > > > >> > know >>> > >> > > > > > > > > >> > > the content is safe. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique >>> provient >>> > >> d’un >>> > >> > > > > > expéditeur >>> > >> > > > > > > > > >> externe. >>> > >> > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune >>> pièce >>> > >> > > jointe >>> > >> > > > si >>> > >> > > > > > > vous >>> > >> > > > > > > > ne >>> > >> > > > > > > > > >> > pouvez >>> > >> > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et >>> si vous >>> > >> > > n’êtes >>> > >> > > > > pas >>> > >> > > > > > > > > certain >>> > >> > > > > > > > > >> > que >>> > >> > > > > > > > > >> > > le contenu ne présente aucun risque. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > Hi, >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > I think that CI/CD and version pining are a >>> little >>> > >> two >>> > >> > > > > > different >>> > >> > > > > > > > > >> things >>> > >> > > > > > > > > >> > > here. In a use cases with some critical >>> systems >>> > >> > > involved, >>> > >> > > > > the >>> > >> > > > > > > > > >> situation >>> > >> > > > > > > > > >> > > when the Dag changes the version to the >>> latest >>> > >> without >>> > >> > > > > > > possibility >>> > >> > > > > > > > > to >>> > >> > > > > > > > > >> > > determine when it will exactly happen (CI/CD >>> will >>> > >> have >>> > >> > > > some >>> > >> > > > > > > > > >> more-or-less >>> > >> > > > > > > > > >> > > time to deploy the change, the same goes for >>> Dag >>> > >> > > Processor >>> > >> > > > > > > parsing >>> > >> > > > > > > > > >> time) >>> > >> > > > > > > > > >> > is >>> > >> > > > > > > > > >> > > rather hard to do and in some systems it can >>> make >>> > >> > change >>> > >> > > > > > > > deployment >>> > >> > > > > > > > > >> > harder >>> > >> > > > > > > > > >> > > and less safe. Of course, the ideal solution >>> would >>> > >> be >>> > >> > to >>> > >> > > > > have >>> > >> > > > > > > > proper >>> > >> > > > > > > > > >> > > non-prod environment, which is fully >>> > >> representative in >>> > >> > > > > > > comparison >>> > >> > > > > > > > to >>> > >> > > > > > > > > >> > > production (in some cases exposing non-prod >>> to prod >>> > >> > > > > > > > > data/traffic/etc. >>> > >> > > > > > > > > >> is, >>> > >> > > > > > > > > >> > > just, not an option - e.g. security), but it >>> is not >>> > >> > > always >>> > >> > > > > > > > possible >>> > >> > > > > > > > > >> to do >>> > >> > > > > > > > > >> > > due to various reasons like costs, licenses, >>> space >>> > >> > > and/or >>> > >> > > > > > > vendors. >>> > >> > > > > > > > > I'm >>> > >> > > > > > > > > >> > > agreeing especially with point 5 of Piyush >>> latest >>> > >> > > message. >>> > >> > > > > > > Having >>> > >> > > > > > > > > >> above >>> > >> > > > > > > > > >> > in >>> > >> > > > > > > > > >> > > mind, I think that version pinning would be >>> a nice >>> > >> > > > addition >>> > >> > > > > to >>> > >> > > > > > > the >>> > >> > > > > > > > > Dag >>> > >> > > > > > > > > >> > > Versioning feature with an assumption that >>> it is >>> > >> for >>> > >> > > > > critical >>> > >> > > > > > > > > Airflow >>> > >> > > > > > > > > >> > Dags >>> > >> > > > > > > > > >> > > when full control of the Dags version change >>> time >>> > >> is >>> > >> > > > > required >>> > >> > > > > > > > (maybe >>> > >> > > > > > > > > >> > there >>> > >> > > > > > > > > >> > > is also another way to achieve that). >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > P.S. In my opinion, what can be done >>> in/around git, >>> > >> > > should >>> > >> > > > > be >>> > >> > > > > > > done >>> > >> > > > > > > > > >> there. >>> > >> > > > > > > > > >> > > Recreation of CI/CD in any form inside of >>> Airflow >>> > >> > itself >>> > >> > > > is >>> > >> > > > > > > > > something >>> > >> > > > > > > > > >> > which >>> > >> > > > > > > > > >> > > should not be done. >>> > >> > > > > > > > > >> > > ________________________________ >>> > >> > > > > > > > > >> > > From: Oliveira, Niko <[email protected]> >>> > >> > > > > > > > > >> > > Sent: 23 April 2026 01:50 >>> > >> > > > > > > > > >> > > To: [email protected] < >>> [email protected] >>> > >> > >>> > >> > > > > > > > > >> > > Subject: Re: [DISCUSS] DAG Version Pinning >>> for >>> > >> > > Deployment >>> > >> > > > > > Gating >>> > >> > > > > > > > > >> > (Building >>> > >> > > > > > > > > >> > > on AIP-63) >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > Hey Piyush, >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > Thanks for your reply, I do love how clearly >>> it is >>> > >> > > written >>> > >> > > > > > and I >>> > >> > > > > > > > see >>> > >> > > > > > > > > >> > > exactly the problem you're trying to solve! >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > I'm still just not convinced this needs to >>> be done >>> > >> in >>> > >> > > > > Airflow, >>> > >> > > > > > > at >>> > >> > > > > > > > > >> least >>> > >> > > > > > > > > >> > > not with a first class feature. As >>> interesting as I >>> > >> > > think >>> > >> > > > > your >>> > >> > > > > > > > > >> > microservice >>> > >> > > > > > > > > >> > > analogy is, Airflow is not a microservice >>> > >> component, >>> > >> > it >>> > >> > > > is a >>> > >> > > > > > > > (very, >>> > >> > > > > > > > > >> very) >>> > >> > > > > > > > > >> > > fancy cron scheduler. And I'm not sure the >>> > >> complexity >>> > >> > is >>> > >> > > > > worth >>> > >> > > > > > > the >>> > >> > > > > > > > > use >>> > >> > > > > > > > > >> > > case. Since any new code added to Airflow >>> must be >>> > >> > > > maintained >>> > >> > > > > > by >>> > >> > > > > > > > this >>> > >> > > > > > > > > >> > > community and we must be cautious that any >>> new >>> > >> pieces >>> > >> > > > serves >>> > >> > > > > > > > enough >>> > >> > > > > > > > > >> use >>> > >> > > > > > > > > >> > > cases/users to make it worth it. >>> > >> > > > > > > > > >> > > To me this should either be managed outside >>> of an >>> > >> > > > individual >>> > >> > > > > > > > Airflow >>> > >> > > > > > > > > >> > > environment e.g. you have an entirely >>> separate >>> > >> > > > > > staging/gamma/dev >>> > >> > > > > > > > > >> Airflow >>> > >> > > > > > > > > >> > > environment, which is exposed to some level >>> of >>> > >> > > production >>> > >> > > > > > > traffic >>> > >> > > > > > > > > (to >>> > >> > > > > > > > > >> > > borrow your microservice analogy) until it >>> can >>> > >> > graduate >>> > >> > > to >>> > >> > > > > the >>> > >> > > > > > > > > >> production >>> > >> > > > > > > > > >> > > environment. And if you really need on the >>> fly >>> > >> > toggling >>> > >> > > > of a >>> > >> > > > > > > > > version, >>> > >> > > > > > > > > >> as >>> > >> > > > > > > > > >> > > you say, Airflow does this quite >>> responsively, if >>> > >> you >>> > >> > > > > deploy a >>> > >> > > > > > > new >>> > >> > > > > > > > > >> > version >>> > >> > > > > > > > > >> > > of your dags it will parse and start using >>> that new >>> > >> > > > version >>> > >> > > > > > > > > >> immediately >>> > >> > > > > > > > > >> > > (the problem you're trying to solve can be a >>> > >> benefit >>> > >> > > > here). >>> > >> > > > > > You >>> > >> > > > > > > > can >>> > >> > > > > > > > > >> even >>> > >> > > > > > > > > >> > > have multiple versions of your dags deployed >>> at >>> > >> once >>> > >> > and >>> > >> > > > use >>> > >> > > > > > > > > >> > configuration >>> > >> > > > > > > > > >> > > to control which dag directory Airflow reads >>> from >>> > >> (or >>> > >> > > > > > > move/symlink >>> > >> > > > > > > > > >> Dags >>> > >> > > > > > > > > >> > in >>> > >> > > > > > > > > >> > > and out of the Dags directory as needed from >>> a >>> > >> known >>> > >> > > good >>> > >> > > > or >>> > >> > > > > > > > pinned >>> > >> > > > > > > > > >> > > source). Or use variables or some other >>> parameter >>> > >> > store >>> > >> > > to >>> > >> > > > > > > control >>> > >> > > > > > > > > >> other >>> > >> > > > > > > > > >> > > pieces of runtime behaviour inside the Dags >>> > >> > themselves. >>> > >> > > > > > Between >>> > >> > > > > > > > > CI/CD, >>> > >> > > > > > > > > >> > dev >>> > >> > > > > > > > > >> > > ops and making use of existing Airflow >>> primitives I >>> > >> > > think >>> > >> > > > > you >>> > >> > > > > > > can >>> > >> > > > > > > > > >> achieve >>> > >> > > > > > > > > >> > > what you're looking for. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > But as always, this is open and community >>> based >>> > >> > > software, >>> > >> > > > so >>> > >> > > > > > I'm >>> > >> > > > > > > > > >> happy to >>> > >> > > > > > > > > >> > > disagree and commit if the rest of the >>> community >>> > >> > thinks >>> > >> > > > this >>> > >> > > > > > is >>> > >> > > > > > > a >>> > >> > > > > > > > > >> > valuable >>> > >> > > > > > > > > >> > > feature :) >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > Cheers, >>> > >> > > > > > > > > >> > > Niko >>> > >> > > > > > > > > >> > > ________________________________ >>> > >> > > > > > > > > >> > > From: Piyush Maheshwari < >>> [email protected]> >>> > >> > > > > > > > > >> > > Sent: Tuesday, April 21, 2026 10:46 PM >>> > >> > > > > > > > > >> > > To: [email protected] < >>> [email protected] >>> > >> > >>> > >> > > > > > > > > >> > > Subject: RE: [EXT] [DISCUSS] DAG Version >>> Pinning >>> > >> for >>> > >> > > > > > Deployment >>> > >> > > > > > > > > Gating >>> > >> > > > > > > > > >> > > (Building on AIP-63) >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > CAUTION: This email originated from outside >>> of the >>> > >> > > > > > organization. >>> > >> > > > > > > > Do >>> > >> > > > > > > > > >> not >>> > >> > > > > > > > > >> > > click links or open attachments unless you >>> can >>> > >> confirm >>> > >> > > the >>> > >> > > > > > > sender >>> > >> > > > > > > > > and >>> > >> > > > > > > > > >> > know >>> > >> > > > > > > > > >> > > the content is safe. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > AVERTISSEMENT: Ce courrier électronique >>> provient >>> > >> d’un >>> > >> > > > > > expéditeur >>> > >> > > > > > > > > >> externe. >>> > >> > > > > > > > > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune >>> pièce >>> > >> > > jointe >>> > >> > > > si >>> > >> > > > > > > vous >>> > >> > > > > > > > ne >>> > >> > > > > > > > > >> > pouvez >>> > >> > > > > > > > > >> > > pas confirmer l’identité de l’expéditeur et >>> si vous >>> > >> > > n’êtes >>> > >> > > > > pas >>> > >> > > > > > > > > certain >>> > >> > > > > > > > > >> > que >>> > >> > > > > > > > > >> > > le contenu ne présente aucun risque. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > Hi Ephraim, Jarek, Jens, and Niko, >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > Thank you for the candid feedback. I want to >>> > >> clarify a >>> > >> > > few >>> > >> > > > > > > things, >>> > >> > > > > > > > > as >>> > >> > > > > > > > > >> I >>> > >> > > > > > > > > >> > > completely agree with Jens and Niko that >>> "testing >>> > >> in >>> > >> > > > > > production" >>> > >> > > > > > > > is >>> > >> > > > > > > > > an >>> > >> > > > > > > > > >> > > anti-pattern. That is absolutely not the >>> intention >>> > >> > here. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > 1. I view this as bringing standard >>> > >> microservice-like >>> > >> > > > > > deployment >>> > >> > > > > > > > > >> maturity >>> > >> > > > > > > > > >> > > to DAGs. >>> > >> > > > > > > > > >> > > Before service deployments in our org, code >>> is >>> > >> tested >>> > >> > > > > locally, >>> > >> > > > > > > in >>> > >> > > > > > > > a >>> > >> > > > > > > > > >> dev >>> > >> > > > > > > > > >> > > environment, and via strict unit/e2e >>> integration >>> > >> tests >>> > >> > > > > before >>> > >> > > > > > it >>> > >> > > > > > > > > ever >>> > >> > > > > > > > > >> > makes >>> > >> > > > > > > > > >> > > it to main. But even after merging and >>> passing >>> > >> those >>> > >> > CI >>> > >> > > > > > > pipelines, >>> > >> > > > > > > > > we >>> > >> > > > > > > > > >> > still >>> > >> > > > > > > > > >> > > use load tests, pre-prod soak times, shadow >>> > >> traffic, >>> > >> > and >>> > >> > > > > gated >>> > >> > > > > > > > > >> production >>> > >> > > > > > > > > >> > > rollouts with automated rollback triggers. >>> Having >>> > >> > > > deployment >>> > >> > > > > > > gates >>> > >> > > > > > > > > for >>> > >> > > > > > > > > >> > the >>> > >> > > > > > > > > >> > > production environment doesn't mean the >>> pre-merge >>> > >> > checks >>> > >> > > > > > weren't >>> > >> > > > > > > > > >> strict >>> > >> > > > > > > > > >> > or >>> > >> > > > > > > > > >> > > that the change wasn't tested beforehand -- >>> it just >>> > >> > > allows >>> > >> > > > > us >>> > >> > > > > > to >>> > >> > > > > > > > > place >>> > >> > > > > > > > > >> > > additional safety gates for the code to take >>> > >> effect, >>> > >> > > > exactly >>> > >> > > > > > > like >>> > >> > > > > > > > in >>> > >> > > > > > > > > >> the >>> > >> > > > > > > > > >> > > service world. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > 2. The core issue we are trying to solve is >>> that >>> > >> > Airflow >>> > >> > > > > > > currently >>> > >> > > > > > > > > >> > > inseparably links Code Distribution (a file >>> > >> arriving >>> > >> > on >>> > >> > > > the >>> > >> > > > > > > > > >> dag-processor >>> > >> > > > > > > > > >> > > and being parsed) with Release Activation >>> (the >>> > >> > scheduler >>> > >> > > > > > > executing >>> > >> > > > > > > > > >> that >>> > >> > > > > > > > > >> > > code). >>> > >> > > > > > > > > >> > > To extend the microservices analogy, I can >>> think of >>> > >> > the >>> > >> > > > DAG >>> > >> > > > > > > > > processor >>> > >> > > > > > > > > >> > > parsing all files as "building the >>> artifact(s)," >>> > >> while >>> > >> > > the >>> > >> > > > > > > > scheduler >>> > >> > > > > > > > > >> and >>> > >> > > > > > > > > >> > > executor acting on the DAG versions created >>> > >> thereafter >>> > >> > > as >>> > >> > > > > > > > > "deploying" >>> > >> > > > > > > > > >> or >>> > >> > > > > > > > > >> > > running the changed code. >>> > >> > > > > > > > > >> > > We simply want to decouple the build from the >>> > >> > > deployment. >>> > >> > > > > This >>> > >> > > > > > > > does >>> > >> > > > > > > > > >> not >>> > >> > > > > > > > > >> > > mean that the code arriving on the >>> dag-processor >>> > >> will >>> > >> > be >>> > >> > > > > > tested >>> > >> > > > > > > > for >>> > >> > > > > > > > > >> the >>> > >> > > > > > > > > >> > > first time straight in production. It >>> should've >>> > >> > already >>> > >> > > > > > passed a >>> > >> > > > > > > > set >>> > >> > > > > > > > > >> of >>> > >> > > > > > > > > >> > > checks in the CI pipeline. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > 3. It is also worth calling out that Airflow >>> > >> already >>> > >> > > > > supports >>> > >> > > > > > > this >>> > >> > > > > > > > > >> > > decoupled behavior at the run level for task >>> > >> re-runs >>> > >> > and >>> > >> > > > > > > > > mid-execution >>> > >> > > > > > > > > >> > DAG >>> > >> > > > > > > > > >> > > version bumps (by pinning the version for >>> the rest >>> > >> of >>> > >> > > the >>> > >> > > > > > > > execution >>> > >> > > > > > > > > or >>> > >> > > > > > > > > >> > the >>> > >> > > > > > > > > >> > > rerun). We are simply trying to expose this >>> > >> existing >>> > >> > > > > > capability >>> > >> > > > > > > at >>> > >> > > > > > > > > the >>> > >> > > > > > > > > >> > DAG >>> > >> > > > > > > > > >> > > level so users can govern which version new >>> > >> scheduled >>> > >> > > runs >>> > >> > > > > are >>> > >> > > > > > > > > created >>> > >> > > > > > > > > >> > > with. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > 4. I also agree that Airflow itself should >>> not be >>> > >> > aware >>> > >> > > of >>> > >> > > > > our >>> > >> > > > > > > > CI/CD >>> > >> > > > > > > > > >> > > pipeline, nor would it manage the deployment >>> > >> > > orchestration >>> > >> > > > > or >>> > >> > > > > > > > > testing. >>> > >> > > > > > > > > >> > > For our requirements, I just need Airflow to >>> expose >>> > >> > APIs >>> > >> > > > to >>> > >> > > > > > > deploy >>> > >> > > > > > > > > >> (pin) >>> > >> > > > > > > > > >> > a >>> > >> > > > > > > > > >> > > DAG version, and to remove the pin (to >>> > >> restore/enable >>> > >> > > the >>> > >> > > > > > > default >>> > >> > > > > > > > > >> > > "auto-deploy latest" behavior). >>> > >> > > > > > > > > >> > > Beyond that, we intend to use an external >>> release >>> > >> > > > > orchestrator >>> > >> > > > > > > > that >>> > >> > > > > > > > > >> can >>> > >> > > > > > > > > >> > > explicitly tell Airflow when a parsed >>> version is >>> > >> > > actually >>> > >> > > > > > > allowed >>> > >> > > > > > > > to >>> > >> > > > > > > > > >> run. >>> > >> > > > > > > > > >> > > Until that API call is made, the previously >>> pinned >>> > >> > > version >>> > >> > > > > > > remains >>> > >> > > > > > > > > >> > active. >>> > >> > > > > > > > > >> > > This ensures we don't introduce assumptions >>> or >>> > >> > awareness >>> > >> > > > of >>> > >> > > > > > the >>> > >> > > > > > > > > >> presence >>> > >> > > > > > > > > >> > of >>> > >> > > > > > > > > >> > > any external gating mechanisms to Airflow. >>> > >> > > > > > > > > >> > > Also note that the intention is to keep the >>> default >>> > >> > > > > > auto-deploy >>> > >> > > > > > > > > >> behavior >>> > >> > > > > > > > > >> > > unless a user (or a system on their behalf) >>> > >> explicitly >>> > >> > > > asks >>> > >> > > > > > > > Airflow >>> > >> > > > > > > > > to >>> > >> > > > > > > > > >> > pin >>> > >> > > > > > > > > >> > > a DAG to a specific version. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > 5. Most importantly, this feature provides an >>> > >> incident >>> > >> > > > > > response >>> > >> > > > > > > > > >> > "rollback" >>> > >> > > > > > > > > >> > > behavior. If a bad DAG version slips through >>> CI/CD >>> > >> > into >>> > >> > > > > > > > production, >>> > >> > > > > > > > > >> > either >>> > >> > > > > > > > > >> > > an on-call engineer or a rollback-trigger >>> > >> > > > (airflow-external) >>> > >> > > > > > can >>> > >> > > > > > > > > >> > instantly >>> > >> > > > > > > > > >> > > roll back to the previous pinned version via >>> the >>> > >> > API/UI >>> > >> > > to >>> > >> > > > > > > > mitigate. >>> > >> > > > > > > > > >> > > Without this, users have to revert the code >>> in Git >>> > >> and >>> > >> > > > wait >>> > >> > > > > > for >>> > >> > > > > > > > the >>> > >> > > > > > > > > >> > entire >>> > >> > > > > > > > > >> > > CI/CD pipeline and file-sync process to run, >>> which >>> > >> is >>> > >> > > > often >>> > >> > > > > > too >>> > >> > > > > > > > slow >>> > >> > > > > > > > > >> > during >>> > >> > > > > > > > > >> > > an outage. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > 6. Jarek - You are right, database schema >>> changes >>> > >> can >>> > >> > be >>> > >> > > > > > > discussed >>> > >> > > > > > > > > >> later. >>> > >> > > > > > > > > >> > > My intention was only to share a very brief >>> > >> summary of >>> > >> > > > how I >>> > >> > > > > > > > deemed >>> > >> > > > > > > > > >> it to >>> > >> > > > > > > > > >> > > be technically feasible for early feedback. >>> I did >>> > >> > > briefly >>> > >> > > > > > share >>> > >> > > > > > > > the >>> > >> > > > > > > > > >> > > high-level use cases ("Safe Deployment >>> Gating" and >>> > >> > > > "Instant >>> > >> > > > > > > > > >> Rollbacks") >>> > >> > > > > > > > > >> > in >>> > >> > > > > > > > > >> > > the original mail, but I completely agree >>> that >>> > >> > aligning >>> > >> > > on >>> > >> > > > > the >>> > >> > > > > > > UX >>> > >> > > > > > > > > >> first >>> > >> > > > > > > > > >> > > would be a good next step. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > If there are no major remaining concerns >>> after this >>> > >> > > > > response, >>> > >> > > > > > I >>> > >> > > > > > > > can >>> > >> > > > > > > > > >> draft >>> > >> > > > > > > > > >> > > and share an AIP to detail the UX, followed >>> by a >>> > >> > > > high-level >>> > >> > > > > > > > > proposal, >>> > >> > > > > > > > > >> > > caveats and next steps. >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > Thanks for your time. >>> > >> > > > > > > > > >> > > Regards, >>> > >> > > > > > > > > >> > > Piyush >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > On Tue, Apr 21, 2026 at 5:59 PM Oliveira, >>> Niko < >>> > >> > > > > > > > [email protected] >>> > >> > > > > > > > > > >>> > >> > > > > > > > > >> > > wrote: >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > > > I am with Jens on this one. I think we're >>> > >> > complicating >>> > >> > > > > > Airflow >>> > >> > > > > > > > to >>> > >> > > > > > > > > >> get >>> > >> > > > > > > > > >> > > > around a bad practice. If stability of >>> your Dags >>> > >> is >>> > >> > > > > critical >>> > >> > > > > > > and >>> > >> > > > > > > > > >> they >>> > >> > > > > > > > > >> > are >>> > >> > > > > > > > > >> > > > highly versioned then I think as Jens >>> suggested >>> > >> > > running >>> > >> > > > > them >>> > >> > > > > > > > > >> through a >>> > >> > > > > > > > > >> > > > pipeline that first deploys them to a dev >>> or >>> > >> gamma >>> > >> > > > > > environment >>> > >> > > > > > > > > which >>> > >> > > > > > > > > >> > > > verifies that quality of the Dags is what >>> you >>> > >> > expect. >>> > >> > > If >>> > >> > > > > > > > something >>> > >> > > > > > > > > >> > slips >>> > >> > > > > > > > > >> > > > through, then it's just normal software >>> > >> practices of >>> > >> > > > > either >>> > >> > > > > > > > > >> reverting >>> > >> > > > > > > > > >> > and >>> > >> > > > > > > > > >> > > > rolling back or rolling forward with a fix >>> pushed >>> > >> > > > through >>> > >> > > > > > the >>> > >> > > > > > > > > >> > pipeline. I >>> > >> > > > > > > > > >> > > > don't think Airflow should be aware of that >>> > >> process >>> > >> > or >>> > >> > > > > > > > opinionated >>> > >> > > > > > > > > >> > about >>> > >> > > > > > > > > >> > > it. >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > Cheers, >>> > >> > > > > > > > > >> > > > Niko >>> > >> > > > > > > > > >> > > > ------------------------------ >>> > >> > > > > > > > > >> > > > *From:* Jens Scheffler < >>> [email protected]> >>> > >> > > > > > > > > >> > > > *Sent:* Monday, April 20, 2026 11:17 AM >>> > >> > > > > > > > > >> > > > *To:* [email protected] < >>> > >> > [email protected]> >>> > >> > > > > > > > > >> > > > *Subject:* RE: [EXT] [DISCUSS] DAG Version >>> > >> Pinning >>> > >> > for >>> > >> > > > > > > > Deployment >>> > >> > > > > > > > > >> > Gating >>> > >> > > > > > > > > >> > > > (Building on AIP-63) >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > CAUTION: This email originated from >>> outside of >>> > >> the >>> > >> > > > > > > organization. >>> > >> > > > > > > > > Do >>> > >> > > > > > > > > >> not >>> > >> > > > > > > > > >> > > > click links or open attachments unless you >>> can >>> > >> > confirm >>> > >> > > > the >>> > >> > > > > > > > sender >>> > >> > > > > > > > > >> and >>> > >> > > > > > > > > >> > > know >>> > >> > > > > > > > > >> > > > the content is safe. >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > AVERTISSEMENT: Ce courrier électronique >>> provient >>> > >> > d’un >>> > >> > > > > > > expéditeur >>> > >> > > > > > > > > >> > externe. >>> > >> > > > > > > > > >> > > > Ne cliquez sur aucun lien et n’ouvrez >>> aucune >>> > >> pièce >>> > >> > > > jointe >>> > >> > > > > si >>> > >> > > > > > > > vous >>> > >> > > > > > > > > ne >>> > >> > > > > > > > > >> > > pouvez >>> > >> > > > > > > > > >> > > > pas confirmer l’identité de l’expéditeur >>> et si >>> > >> vous >>> > >> > > > n’êtes >>> > >> > > > > > pas >>> > >> > > > > > > > > >> certain >>> > >> > > > > > > > > >> > > que >>> > >> > > > > > > > > >> > > > le contenu ne présente aucun risque. >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > Hi, >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > I am still quite sceptical. Yes, if such >>> pinning >>> > >> is >>> > >> > > > made, >>> > >> > > > > > then >>> > >> > > > > > > > per >>> > >> > > > > > > > > >> Dag >>> > >> > > > > > > > > >> > a >>> > >> > > > > > > > > >> > > > change need to be possible via UI and API. >>> But I >>> > >> > still >>> > >> > > > see >>> > >> > > > > > it >>> > >> > > > > > > as >>> > >> > > > > > > > > >> > > > checken-and-egg - so you want to run a >>> pinned >>> > >> > version >>> > >> > > > but >>> > >> > > > > > then >>> > >> > > > > > > > how >>> > >> > > > > > > > > >> do >>> > >> > > > > > > > > >> > > > you test the changes (w/o moving a version >>> pin)? >>> > >> > Then >>> > >> > > > > again >>> > >> > > > > > > some >>> > >> > > > > > > > > >> test >>> > >> > > > > > > > > >> > > > mode is needed or per run you need to make >>> a >>> > >> "test >>> > >> > > run" >>> > >> > > > > with >>> > >> > > > > > > > > another >>> > >> > > > > > > > > >> > > > version. Smells a bit like mis-using a >>> production >>> > >> > > system >>> > >> > > > > for >>> > >> > > > > > > > > >> testing. >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > On the other hand, yes if all Dags share >>> the same >>> > >> > Git >>> > >> > > > repo >>> > >> > > > > > > then >>> > >> > > > > > > > > >> merging >>> > >> > > > > > > > > >> > > > a branch to some other will switch all >>> Dags at >>> > >> the >>> > >> > > same >>> > >> > > > > > time. >>> > >> > > > > > > > > Still >>> > >> > > > > > > > > >> you >>> > >> > > > > > > > > >> > > > could utilize standard Git tools and >>> cherry-pick >>> > >> > > > > individual >>> > >> > > > > > > > > changes >>> > >> > > > > > > > > >> and >>> > >> > > > > > > > > >> > > > no force to always make a full rollout. At >>> least >>> > >> 80% >>> > >> > > > > > possible >>> > >> > > > > > > > with >>> > >> > > > > > > > > >> > > > standard CI/CD tools and Git. >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > TLDR I see the danger that instead of a >>> proper >>> > >> CI/CD >>> > >> > > and >>> > >> > > > > > test >>> > >> > > > > > > > > system >>> > >> > > > > > > > > >> > > > such a feature might feel like you can >>> easily >>> > >> test >>> > >> > on >>> > >> > > a >>> > >> > > > > > > > production >>> > >> > > > > > > > > >> > > > system. Effectively it would be needed >>> allowing >>> > >> to >>> > >> > > > start a >>> > >> > > > > > Dag >>> > >> > > > > > > > > with >>> > >> > > > > > > > > >> any >>> > >> > > > > > > > > >> > > > version to also be able to jump back as a >>> > >> reversion. >>> > >> > > > Even >>> > >> > > > > > > > though, >>> > >> > > > > > > > > >> yes, >>> > >> > > > > > > > > >> > > > agree, all is technically possible. >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > Jens >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > On 20.04.26 16:40, Jarek Potiuk wrote: >>> > >> > > > > > > > > >> > > > > +1 to what Ephraim wrote. I think that >>> was a >>> > >> > natural >>> > >> > > > > next >>> > >> > > > > > > step >>> > >> > > > > > > > > we >>> > >> > > > > > > > > >> > > > > discussed, but it needs significant >>> refinement, >>> > >> > > > starting >>> > >> > > > > > > with >>> > >> > > > > > > > > the >>> > >> > > > > > > > > >> > > actual >>> > >> > > > > > > > > >> > > > > use cases it should serve and the UX for >>> user >>> > >> > > > > > interaction. I >>> > >> > > > > > > > > think >>> > >> > > > > > > > > >> > > > related >>> > >> > > > > > > > > >> > > > > database changes are pretty secondary. >>> Use >>> > >> cases >>> > >> > > cover >>> > >> > > > > > runs, >>> > >> > > > > > > > > >> re-runs, >>> > >> > > > > > > > > >> > > > > backfills, CI testing, rollbacks, etc. >>> > >> Following >>> > >> > the >>> > >> > > > > > > > > >> "documentation >>> > >> > > > > > > > > >> > > > first" >>> > >> > > > > > > > > >> > > > > approach discussed in separate thread, >>> > >> describing >>> > >> > > the >>> > >> > > > > > > context >>> > >> > > > > > > > > and >>> > >> > > > > > > > > >> > > > intention >>> > >> > > > > > > > > >> > > > > of what we want to achieve is much more >>> > >> important >>> > >> > > than >>> > >> > > > > DB >>> > >> > > > > > > > schema >>> > >> > > > > > > > > >> > > changes. >>> > >> > > > > > > > > >> > > > > Once we know which use cases we want to >>> serve, >>> > >> the >>> > >> > > DB >>> > >> > > > > > schema >>> > >> > > > > > > > > >> changes >>> > >> > > > > > > > > >> > > and >>> > >> > > > > > > > > >> > > > > other related items will emerge >>> naturally. >>> > >> > > > > > > > > >> > > > > >>> > >> > > > > > > > > >> > > > > On Mon, Apr 20, 2026 at 3:15 PM Ephraim >>> > >> Anierobi < >>> > >> > > > > > > > > >> > > > [email protected]> >>> > >> > > > > > > > > >> > > > > wrote: >>> > >> > > > > > > > > >> > > > > >>> > >> > > > > > > > > >> > > > >> Hi Piyush, thanks for starting this >>> > >> discussion. >>> > >> > > > > > > > > >> > > > >> >>> > >> > > > > > > > > >> > > > >> I like the proposal. We can introduce an >>> > >> active >>> > >> > > > > execution >>> > >> > > > > > > > > version >>> > >> > > > > > > > > >> > for >>> > >> > > > > > > > > >> > > > >> "versioned bundles" and make >>> scheduler/API >>> > >> > resolve >>> > >> > > > > > through >>> > >> > > > > > > > it. >>> > >> > > > > > > > > >> The >>> > >> > > > > > > > > >> > > hard >>> > >> > > > > > > > > >> > > > >> part of this is making airflow able to >>> > >> > distinguish >>> > >> > > > the >>> > >> > > > > > > latest >>> > >> > > > > > > > > >> parsed >>> > >> > > > > > > > > >> > > > >> dagmodel's metadata from active >>> scheduling >>> > >> > > metadata. >>> > >> > > > I >>> > >> > > > > > will >>> > >> > > > > > > > > >> suggest >>> > >> > > > > > > > > >> > > you >>> > >> > > > > > > > > >> > > > >> draft this in a google docs and share >>> for >>> > >> further >>> > >> > > > > > > > discussions. >>> > >> > > > > > > > > >> > > > >> >>> > >> > > > > > > > > >> > > > >> Regards >>> > >> > > > > > > > > >> > > > >> - Ephraim >>> > >> > > > > > > > > >> > > > >> >>> > >> > > > > > > > > >> > > > >> On Mon, 20 Apr 2026 at 01:31, Piyush >>> > >> Maheshwari < >>> > >> > > > > > > > > >> > > [email protected]> >>> > >> > > > > > > > > >> > > > >> wrote: >>> > >> > > > > > > > > >> > > > >> >>> > >> > > > > > > > > >> > > > >>> Thanks for sharing your thoughts Jens. >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>>> be able to test it? … a Q&A/Testing >>> > >> environment >>> > >> > > to >>> > >> > > > be >>> > >> > > > > > > able >>> > >> > > > > > > > to >>> > >> > > > > > > > > >> > > sign-off >>> > >> > > > > > > > > >> > > > >>> changes. >>> > >> > > > > > > > > >> > > > >>> Yes, we’ve have built an isolated >>> airflow >>> > >> > > > environment >>> > >> > > > > to >>> > >> > > > > > > run >>> > >> > > > > > > > > >> > > regression >>> > >> > > > > > > > > >> > > > >>> checks before promoting to production. >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>> As you suggested, we’re already >>> running both >>> > >> > > generic >>> > >> > > > > and >>> > >> > > > > > > > > >> DAG-custom >>> > >> > > > > > > > > >> > > > >> static >>> > >> > > > > > > > > >> > > > >>> checks in a CI job as a required step >>> to >>> > >> merge >>> > >> > to >>> > >> > > > the >>> > >> > > > > > main >>> > >> > > > > > > > > >> branch. >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>>> But then the "main" branch might be >>> best >>> > >> suited >>> > >> > > if >>> > >> > > > > > > > > >> > > > >>> implemented on the test system >>> > >> > > > > > > > > >> > > > >>> In this case, problematic commits on >>> “main” >>> > >> can >>> > >> > > > choke >>> > >> > > > > > > other >>> > >> > > > > > > > > >> > unrelated >>> > >> > > > > > > > > >> > > > >>> changes. >>> > >> > > > > > > > > >> > > > >>> So the other option would be to revert >>> the >>> > >> > > > problematic >>> > >> > > > > > > > commits >>> > >> > > > > > > > > >> and >>> > >> > > > > > > > > >> > > > deploy >>> > >> > > > > > > > > >> > > > >>> forward. >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>> However, a key limitation with this >>> approach >>> > >> > that >>> > >> > > > > > remains >>> > >> > > > > > > is >>> > >> > > > > > > > > >> that a >>> > >> > > > > > > > > >> > > > >> commit >>> > >> > > > > > > > > >> > > > >>> affecting multiple DAGs goes live for >>> either >>> > >> all >>> > >> > > > DAGs >>> > >> > > > > or >>> > >> > > > > > > > none. >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>> Second important feature we get with >>> this is >>> > >> > > instant >>> > >> > > > > > > > DAG-level >>> > >> > > > > > > > > >> > > rollback >>> > >> > > > > > > > > >> > > > >>> without waiting for a revert commit to >>> merge >>> > >> and >>> > >> > > be >>> > >> > > > > > picked >>> > >> > > > > > > > by >>> > >> > > > > > > > > >> > > airflow. >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>> I think DAG-level version pinning can >>> also >>> > >> > unlock >>> > >> > > a >>> > >> > > > > lot >>> > >> > > > > > of >>> > >> > > > > > > > > >> > > flexibility >>> > >> > > > > > > > > >> > > > >> for >>> > >> > > > > > > > > >> > > > >>> deployments including tiered rollouts, >>> > >> > > auto-rollback >>> > >> > > > > > > > triggers, >>> > >> > > > > > > > > >> > timed >>> > >> > > > > > > > > >> > > > >>> deployment windows and so on. >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>> Looking forward to hear your thoughts. >>> > >> > > > > > > > > >> > > > >>> Regards, >>> > >> > > > > > > > > >> > > > >>> Piyush >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>> On Sun, 19 Apr 2026 at 3:12 PM, Jens >>> > >> Scheffler < >>> > >> > > > > > > > > >> > [email protected]> >>> > >> > > > > > > > > >> > > > >>> wrote: >>> > >> > > > > > > > > >> > > > >>> >>> > >> > > > > > > > > >> > > > >>>> Thanks Piyush for dropping the >>> discussion! >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> I think in general QA processes are >>> > >> important >>> > >> > > and a >>> > >> > > > > > valid >>> > >> > > > > > > > use >>> > >> > > > > > > > > >> > case. >>> > >> > > > > > > > > >> > > So >>> > >> > > > > > > > > >> > > > >> a >>> > >> > > > > > > > > >> > > > >>>> kind of pinning Dag versions really is >>> > >> > important. >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> Thinking about it, if you pin the >>> version >>> > >> ... >>> > >> > how >>> > >> > > > > would >>> > >> > > > > > > you >>> > >> > > > > > > > > >> then >>> > >> > > > > > > > > >> > be >>> > >> > > > > > > > > >> > > > >> able >>> > >> > > > > > > > > >> > > > >>>> to test it? I assume you would need >>> (and >>> > >> should >>> > >> > > > have >>> > >> > > > > or >>> > >> > > > > > > > > invest >>> > >> > > > > > > > > >> > > into) a >>> > >> > > > > > > > > >> > > > >>>> Q&A/Testing environment to be able to >>> > >> sign-off >>> > >> > > > > changes. >>> > >> > > > > > > > Both >>> > >> > > > > > > > > in >>> > >> > > > > > > > > >> > > > >>>> infrastructure but also for Dag >>> changes. >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> If you are changing Dags first of all >>> static >>> > >> > > checks >>> > >> > > > > on >>> > >> > > > > > > Dag >>> > >> > > > > > > > > code >>> > >> > > > > > > > > >> > are >>> > >> > > > > > > > > >> > > > >> very >>> > >> > > > > > > > > >> > > > >>>> much proposed as well as you can have >>> tests >>> > >> > > > > implemented >>> > >> > > > > > > and >>> > >> > > > > > > > > >> test >>> > >> > > > > > > > > >> > > your >>> > >> > > > > > > > > >> > > > >>>> Dags and logic. Similar like software >>> a >>> > >> CI/CD >>> > >> > > > system >>> > >> > > > > > will >>> > >> > > > > > > > be >>> > >> > > > > > > > > a >>> > >> > > > > > > > > >> > good >>> > >> > > > > > > > > >> > > > >>>> setup. Alongside Dag changes also have >>> > >> logical >>> > >> > > > > changes >>> > >> > > > > > > that >>> > >> > > > > > > > > >> mostly >>> > >> > > > > > > > > >> > > can >>> > >> > > > > > > > > >> > > > >>>> only be tested in a live system and >>> not as >>> > >> > static >>> > >> > > > > > checks. >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> Have you considered using Git and a >>> set of >>> > >> > > branches >>> > >> > > > > for >>> > >> > > > > > > > > >> > implementing >>> > >> > > > > > > > > >> > > > >>>> such staging? E.g. you have a git >>> repo and >>> > >> you >>> > >> > > plan >>> > >> > > > > to >>> > >> > > > > > > make >>> > >> > > > > > > > > >> > changes. >>> > >> > > > > > > > > >> > > > >>>> Then you would open a PR for the >>> change and >>> > >> > merge >>> > >> > > > it >>> > >> > > > > to >>> > >> > > > > > > the >>> > >> > > > > > > > > >> "main" >>> > >> > > > > > > > > >> > > > >>>> branch - and there in your CI/CD you >>> can >>> > >> check >>> > >> > > all >>> > >> > > > > > sorts >>> > >> > > > > > > of >>> > >> > > > > > > > > >> static >>> > >> > > > > > > > > >> > > > >>>> checks and tests. But then the "main" >>> branch >>> > >> > > might >>> > >> > > > be >>> > >> > > > > > > best >>> > >> > > > > > > > > >> suited >>> > >> > > > > > > > > >> > if >>> > >> > > > > > > > > >> > > > >>>> implemented on the test system. Once >>> you >>> > >> > validate >>> > >> > > > the >>> > >> > > > > > > > changes >>> > >> > > > > > > > > >> > > > >> end-to-end >>> > >> > > > > > > > > >> > > > >>>> you could make another PR for example >>> to a >>> > >> > "prod" >>> > >> > > > > > branch. >>> > >> > > > > > > > And >>> > >> > > > > > > > > >> if >>> > >> > > > > > > > > >> > > your >>> > >> > > > > > > > > >> > > > >>>> production system is only pulling >>> Dags from >>> > >> the >>> > >> > > > > "prod" >>> > >> > > > > > > > branch >>> > >> > > > > > > > > >> then >>> > >> > > > > > > > > >> > > you >>> > >> > > > > > > > > >> > > > >>>> can have this merging strategy as a >>> staging >>> > >> > > setup. >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> Would this resolve your PING problem? >>> Or >>> > >> which >>> > >> > > > other >>> > >> > > > > > > detail >>> > >> > > > > > > > > in >>> > >> > > > > > > > > >> the >>> > >> > > > > > > > > >> > > use >>> > >> > > > > > > > > >> > > > >>>> case would require a PIN on top of a >>> staging >>> > >> > > > > strategy? >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> Jens >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> P.S.: Have enabled your confluence >>> account >>> > >> > after >>> > >> > > it >>> > >> > > > > was >>> > >> > > > > > > > > >> created in >>> > >> > > > > > > > > >> > > > >> order >>> > >> > > > > > > > > >> > > > >>>> to write to Confluence, sorry, typical >>> > >> pitfall >>> > >> > > > after >>> > >> > > > > > > > account >>> > >> > > > > > > > > >> > > creation >>> > >> > > > > > > > > >> > > > >>>> permissions were not set. Now it >>> should >>> > >> work. >>> > >> > Let >>> > >> > > > me >>> > >> > > > > > know >>> > >> > > > > > > > if >>> > >> > > > > > > > > >> not. >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> On 19.04.26 01:40, Piyush Maheshwari >>> wrote: >>> > >> > > > > > > > > >> > > > >>>>> Hi everyone, >>> > >> > > > > > > > > >> > > > >>>>> I'm a new contributor to Airflow. >>> I'd like >>> > >> to >>> > >> > > > > propose >>> > >> > > > > > a >>> > >> > > > > > > > new >>> > >> > > > > > > > > >> > feature >>> > >> > > > > > > > > >> > > > >> for >>> > >> > > > > > > > > >> > > > >>>> Airflow: DAG Version Pinning. >>> > >> > > > > > > > > >> > > > >>>>> Building on the foundation >>> introduced by >>> > >> > AIP-63: >>> > >> > > > DAG >>> > >> > > > > > > > > >> Versioning ( >>> > >> > > > > > > > > >> > > > >> >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >>> > >> > > > > > > > >>> > >> > > > > > > >>> > >> > > > > > >>> > >> > > > > >>> > >> > > > >>> > >> > > >>> > >> > >>> > >> >>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-63*3A*DAG*Versioning__;JSsr!!Ci6f514n9QsL8ck!l3ZKTOw996h9qu4NR0VT4ouUryUdk_HmXUAVPbwCHwPwn0N2CCptVdx95-V0BoRFjws9huE_1Vy-THL8jw$ >>> > >> > > > > > > > > >> > > > >>> ), >>> > >> > > > > > > > > >> > > > >>>> this proposal aims to extend Airflow's >>> > >> > > capabilities >>> > >> > > > > to >>> > >> > > > > > > > > support >>> > >> > > > > > > > > >> > true >>> > >> > > > > > > > > >> > > > >>>> continuous deployment (CD) gating and >>> safer >>> > >> > > release >>> > >> > > > > > > cycles. >>> > >> > > > > > > > > >> > > > >>>>> The Problem & Use Cases >>> > >> > > > > > > > > >> > > > >>>>> Currently, the scheduler always >>> creates >>> > >> > DagRuns >>> > >> > > > > using >>> > >> > > > > > > the >>> > >> > > > > > > > > >> latest >>> > >> > > > > > > > > >> > > > >> parsed >>> > >> > > > > > > > > >> > > > >>>> DagVersion. This means that the >>> updated DAG >>> > >> > code >>> > >> > > is >>> > >> > > > > > > > deployed >>> > >> > > > > > > > > >> > (takes >>> > >> > > > > > > > > >> > > > >>> effect) >>> > >> > > > > > > > > >> > > > >>>> right after the dag-processor >>> processes it. >>> > >> > While >>> > >> > > > > this >>> > >> > > > > > is >>> > >> > > > > > > > > great >>> > >> > > > > > > > > >> > for >>> > >> > > > > > > > > >> > > > >> rapid >>> > >> > > > > > > > > >> > > > >>>> development, teams running >>> business-critical >>> > >> > > > > pipelines >>> > >> > > > > > > > often >>> > >> > > > > > > > > >> need >>> > >> > > > > > > > > >> > > > >>> stricter >>> > >> > > > > > > > > >> > > > >>>> deployment mechanisms. Specifically: >>> > >> > > > > > > > > >> > > > >>>>> * >>> > >> > > > > > > > > >> > > > >>>>> Safe Deployment Gating: The ability >>> to pin >>> > >> a >>> > >> > DAG >>> > >> > > > to >>> > >> > > > > > its >>> > >> > > > > > > > last >>> > >> > > > > > > > > >> > known >>> > >> > > > > > > > > >> > > > >>>> stable version while new code is >>> parsed in >>> > >> the >>> > >> > > > > > > background. >>> > >> > > > > > > > > This >>> > >> > > > > > > > > >> > > allows >>> > >> > > > > > > > > >> > > > >>> the >>> > >> > > > > > > > > >> > > > >>>> new version to be held back until it >>> passes >>> > >> > > > automated >>> > >> > > > > > > > > >> regression >>> > >> > > > > > > > > >> > > tests >>> > >> > > > > > > > > >> > > > >> or >>> > >> > > > > > > > > >> > > > >>>> receives explicit manual approval. >>> > >> > > > > > > > > >> > > > >>>>> * >>> > >> > > > > > > > > >> > > > >>>>> Instant Rollbacks: If an issue is >>> detected >>> > >> in >>> > >> > a >>> > >> > > > > newly >>> > >> > > > > > > > > promoted >>> > >> > > > > > > > > >> > DAG >>> > >> > > > > > > > > >> > > > >>>> version, users need the capability to >>> > >> instantly >>> > >> > > > roll >>> > >> > > > > > back >>> > >> > > > > > > > to >>> > >> > > > > > > > > a >>> > >> > > > > > > > > >> > > > previous >>> > >> > > > > > > > > >> > > > >>>> version via the UI/API, without >>> having to >>> > >> > revert >>> > >> > > > the >>> > >> > > > > > > > > underlying >>> > >> > > > > > > > > >> > code >>> > >> > > > > > > > > >> > > > >> and >>> > >> > > > > > > > > >> > > > >>>> wait for the repository sync and DAG >>> > >> processing >>> > >> > > > > cycle. >>> > >> > > > > > > > > >> > > > >>>>> High-Level Proposed Solution >>> > >> > > > > > > > > >> > > > >>>>> Introduce an optional >>> > >> active_dag_version_id to >>> > >> > > the >>> > >> > > > > > > > DagModel. >>> > >> > > > > > > > > >> This >>> > >> > > > > > > > > >> > > > >> field >>> > >> > > > > > > > > >> > > > >>>> can be used to pin a DAG version for >>> > >> scheduling >>> > >> > > and >>> > >> > > > > > > > > execution, >>> > >> > > > > > > > > >> > while >>> > >> > > > > > > > > >> > > > >> the >>> > >> > > > > > > > > >> > > > >>>> dag-processor can continue to parse >>> and >>> > >> > register >>> > >> > > > > newer >>> > >> > > > > > > DAG >>> > >> > > > > > > > > >> > versions. >>> > >> > > > > > > > > >> > > > >>>>> * >>> > >> > > > > > > > > >> > > > >>>>> When this pin is set, the scheduler >>> and API >>> > >> > will >>> > >> > > > > > respect >>> > >> > > > > > > > the >>> > >> > > > > > > > > >> > pinned >>> > >> > > > > > > > > >> > > > >>>> version for creating runs and >>> executing >>> > >> tasks, >>> > >> > > > > > separating >>> > >> > > > > > > > the >>> > >> > > > > > > > > >> > > parsing >>> > >> > > > > > > > > >> > > > >> of >>> > >> > > > > > > > > >> > > > >>>> new code from the execution of new >>> code. >>> > >> > > > > > > > > >> > > > >>>>> * >>> > >> > > > > > > > > >> > > > >>>>> If the pin is NULL, the system >>> defaults to >>> > >> the >>> > >> > > > > current >>> > >> > > > > > > > > >> behavior >>> > >> > > > > > > > > >> > > > >> (always >>> > >> > > > > > > > > >> > > > >>>> executing the latest parsed version). >>> This >>> > >> way, >>> > >> > > we >>> > >> > > > > can >>> > >> > > > > > > > > maintain >>> > >> > > > > > > > > >> > > > >> complete >>> > >> > > > > > > > > >> > > > >>>> backwards compatibility. >>> > >> > > > > > > > > >> > > > >>>>> I have put together some detailed >>> notes >>> > >> > covering >>> > >> > > > the >>> > >> > > > > > > data >>> > >> > > > > > > > > >> model >>> > >> > > > > > > > > >> > > > >>> changes, >>> > >> > > > > > > > > >> > > > >>>> database migrations, and edge cases >>> with >>> > >> this >>> > >> > > > > approach. >>> > >> > > > > > > If >>> > >> > > > > > > > > >> there >>> > >> > > > > > > > > >> > is >>> > >> > > > > > > > > >> > > > >>> general >>> > >> > > > > > > > > >> > > > >>>> alignment that this fits the vision >>> for >>> > >> > Airflow, >>> > >> > > I >>> > >> > > > > > would >>> > >> > > > > > > > like >>> > >> > > > > > > > > >> to >>> > >> > > > > > > > > >> > > take >>> > >> > > > > > > > > >> > > > >>> this >>> > >> > > > > > > > > >> > > > >>>> proposal through the formal AIP review >>> > >> process. >>> > >> > > > > > > > > >> > > > >>>>> But I would love to get the >>> community's >>> > >> > feedback >>> > >> > > > on >>> > >> > > > > > the >>> > >> > > > > > > > > >> feature >>> > >> > > > > > > > > >> > and >>> > >> > > > > > > > > >> > > > >> the >>> > >> > > > > > > > > >> > > > >>>> high-level approach. >>> > >> > > > > > > > > >> > > > >>>>> I'll also need someone to grant me >>> access >>> > >> to >>> > >> > > > create >>> > >> > > > > > > > content >>> > >> > > > > > > > > on >>> > >> > > > > > > > > >> > the >>> > >> > > > > > > > > >> > > > >>>> Airflow Confluence wiki. >>> > >> > > > > > > > > >> > > > >>>>> Thanks for your time! >>> > >> > > > > > > > > >> > > > >>>>> Regards, >>> > >> > > > > > > > > >> > > > >>>>> Piyush >>> > >> > > > > > > > > >> > > > >>>>> >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >>> > >> > > > > >>> > >> >>> --------------------------------------------------------------------- >>> > >> > > > > > > > > >> > > > >>>> To unsubscribe, e-mail: >>> > >> > > > > > > [email protected] >>> > >> > > > > > > > > >> > > > >>>> For additional commands, e-mail: >>> > >> > > > > > > > [email protected] >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>>> >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> >>> > >> > > > > > > >>> > >> > > >>> --------------------------------------------------------------------- >>> > >> > > > > > > > > >> > > > To unsubscribe, e-mail: >>> > >> > > > > [email protected] >>> > >> > > > > > > > > >> > > > For additional commands, e-mail: >>> > >> > > > > > [email protected] >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > > >>> > >> > > > > > > > > >> > > >>> > >> > > > > > > > > >> > >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > >>> > >> > > > > > > > >>> > >> > > > > > > >>> > >> > > > > > >>> > >> > > > > >>> > >> > > > >>> > >> > > >>> > >> > >>> > >> >>> > > >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>>
