> In my opinion, we can simply add an optional `manifest` field (or another suitable name). I don’t think we need to introduce a new table via DbManager; an additional field for storing metadata about the external state (such as prefix and object versions for all dags in the bundle, in the case of S3DagBundle) should suffice. We could introduce a new parent subclass, such as `RemoteDagBundle` or `ObjectStoreDagBundle`, in the common provider to define the structure for serializing and deserializing the `manifest` field.
This is a good solution. It goes along the idea of a "generic" solution that does not need an "amazon specific" table and DB manager. If the manifest serialized field can be used for all other "bundles" (even if manifest format itself is specific to S3 bundle), I am very happy with that solution. One thing to consider (but this is entirely up to the S3 bundle implementation) is handling versioning of such manifest during serialization/deserialization to allow downgrading and upgrading the provider seamlessly. On Fri, Jul 18, 2025 at 5:56 AM Zhe You Liu <jason...@apache.org> wrote: > Sorry for the late response. > > Both approaches work for me; I just wanted to share my opinion as we > settle on a final decision. > > From my perspective, the DagBundle acts as a client that pulls external > state and stores only the version identifier in the Airflow metadata DB. > > For example, with GitDagBundle, the Git repository serves as the external > storage. The GitDagBundle pulls DAG files locally and stores the commit > hash as the `version` field in `DagBundleModel.version`. > > 1. If we choose to store the manifest in the Airflow metadata DB: > > In my opinion, we can simply add an optional `manifest` field (or another > suitable name). I don’t think we need to introduce a new table via > DbManager; an additional field for storing metadata about the external > state (such as prefix and object versions for all dags in the bundle, in > the case of S3DagBundle) should suffice. We could introduce a new parent > subclass, such as `RemoteDagBundle` or `ObjectStoreDagBundle`, in the > common provider to define the structure for serializing and deserializing > the `manifest` field. > > 2. If we decide to store the manifest outside the Airflow metadata DB: > > We will need to clarify: > > a) The required parameters for all DagBundles that pull DAGs from object > storage. Based on the discussion above, we would need the `conn_id`, > `bucket`, and `prefix` for the manifest file. > > b) The interface for calculating the bundle version based on the external > state or DAG content hash. > > Here is a concrete example of how the manifest could be stored: > https://github.com/apache/airflow/pull/46621#issuecomment-3078208467 > > Thank you all for the insightful discussion! > > Best, > Jason > > On 2025/07/10 21:56:31 "Oliveira, Niko" wrote: > > Thanks for the reply Jarek :) > > > > Indeed we have different philosophies about this so we will certainly > keep going in circles about where to draw the line on making things easy > and enjoyable to use, whether to intentionally add friction or not, etc, > etc. > > > > I think if we have optional paths to take and it's not immensely harder > we should err on the side of making OSS Airflow as good as it can be, > despite whatever managed services we have in the community. I'm not sure > where it has come from recently but this new push to make Airflow > intentionally hard to use so that managed services stay in business is a > bit unsettling. We're certainly not asking for that, and those around that > I've chatted to (since I'm now seeing this mentioned frequently) are also > not asking for this. I'm curious where this new pressure is coming from and > why you feel it recently. > > > > But regardless of the curiosity above, I'll return to the drawing board, > and see what else can be done for this particular problem. If there are > other Bundle types who need to solve the same problem perhaps we can find a > more acceptable implementation in Airflow core to support this. And if not, > I'll proceed with externalizing the storage of the S3 Bundle version > metadata outside of Airflow. > > > > Cheers, > > Niko > > > > ________________________________ > > From: Jarek Potiuk <ja...@potiuk.com> > > Sent: Wednesday, July 9, 2025 11:59:06 PM > > To: dev@airflow.apache.org > > Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > > > > > > To me, I'm always working from a user perspective. My goal is to make > > their lives easier, their deployments easier, the product the most > > enjoyable for them to use. To me, the best user experience is that they > > should enable bundle versioning and it should just work with as little or > > no extra steps and with as little infra as possible, and with the fewest > > possible pit falls for them to fall into. From a user perspective, > they've > > already provisioned a database for airflow metadata, why is this portion > of > > metadata leaking out to other forms of external storage? Now this is > > another resource they now need to be aware of and manage the lifecycle of > > (or allow us write access into their accounts to manage for them). > > > > > > *TL;DR; I think our goal in open-source is to have frictionless and "out > of > > the box" experience only for basic cases, but not for more complex > > deployments.* > > > > It's a long read if you want to read it .. so beware :). > > > > I think that is an important "optimization goal" for sure to provide > > frictionless and enjoyable experience - but I think it's one of many > goals > > that are sometimes contradicting with long term open-source project > > sustainability and it's very import to clarify which "user" we are > talking > > about. > > > > To be honest, I am not sure that our goal should be "airflow should work > > out of the box in case of integration with external services in > production' > > if it complicates our code and makes it service-dependent - and as Jens > > noticed, if we can come up with a "generic" thing that can be reusable > > across multiple services, we can invest more in making it works "out of > the > > box", but if you anyhow need to integrate and make work with external > > service, it adds very little "deployment complexity" to use another piece > > of the service - and this is basically the job of deployment manager > > anyway. > > > > The "just work" goal as I see it should only cover those individual users > > who want to try and use airflow in it's basic form and "standalone" > > configuration - not for "deployment managers". > > > > I think yes - our goal should be to make things extremely easy for users > > who want to use airflow in its basic form where things should **just > > work**. Like "docker run -it apache/airflow standalone" - this is what > > currently **just works**, 0 configuration, 0 work for external > > integrations, and we even had a discussion that we could make it "low > > production ready" (which I think we could - just implement automated > > backup/recovery of sqlite db and maybe document mounting a folder with > DAGs > > and db, better handling of logs rather than putting them as mixed output > on > > stdout and we are practically done). But when you add "S3" as the dag > > storage you already need to make a lot of decisions - mostly about > service > > accounts, security, access, versioning, backup of the s3 objects, etc. > etc. > > And that's not a "standalone user' case - that is a "deployment manager" > > work (where "deployment manager" is a role - not necessarily title of the > > job you have. > > > > I think - and that is a bit of philosophical - but I've been talking > about > > it to Maciek Obuchowski yesterday - that there is a pretty clear boundary > > of what open-source solutions delivers and it should match expectations > of > > people using it. Maintainers and community developing open-source should > > mostly deliver a working, generic solutions that are extendable with > > various deployment options and we should make it possible for those > > deployments to happen - and provide building blocks for them. But it's > > "deployment manager" work to make sure to put things together and make it > > works. And we should not do it "for them". It's their job to figure out > how > > to configure and set-up things, make backups, set security boundaries > etc. > > - we should make it possible, document the options, document security > model > > and make it "easy" to configure things - but there should not be an > > expectation from the deploiyment manager that it "just works". > > > > And I think your approach is perfectly fine - but only for "managed > > services" - there, indeed manage service user's expectations can be that > > things "just work" and they are willing to pay for it with real money, > > rather than their time and effort to make it so. And there I think, those > > who deliver such a service should have the "just work" as primary goal - > > also because users will have such expectations - because they actually > pay > > for it to "just work". Not so much for open-source product - where "just > > work" often involves complexity, additional maintenance overhead and > making > > opinionated decisions on "how it just works". For those "managed service" > > teams - "just work" is very much a primary goal. But for "open source > > community" - having such a goal is actually not good - it's dangerous > > because it might result in wrong expectations from the users. If we start > > making airflow "just works" in all kinds of deployment with zero work > from > > the users who want to deploy it in production and at scale, they will > > expect it to happen for everything - why don't we have automated log > > trimming, why don't we have automated backup of the Database, why don't > we > > auto vacuum the db, why don't we provide one-click deployment option on > > AWS. GCS. Azure, why don't we provide DDOS protection in our webserver, > why > > don't we ..... you name it. > > > > That's a bit of philosophy - those are the same assumptions and goals > that > > I had in mind when designing multi-team - and there it's also why we had > > different views - I just feel that some level of friction is a "property" > > of open-source product. > > > > Also a bit of "business" side - this is also "good" for those who provide > > managed services and airflow to keep sustainable open-source business > model > > working - because what people are paying them is precisely to "remove the > > friction". If take the "frictionless user experience" goal case to > extreme > > - Airflow would essentially be killed IMHO. Imagine if Airflow would be > > frictioness for all kinds of deployments and had "everything" working out > > of the box. There would be no business for any of the managed services > > (because users would not need to pay for it). Then we would only have > users > > who expect thigns to "just work" and most of them would not even think > > about contributing back. And there would be no managed services people > > (like you) whose job is paid by the services - or people like me who > work > > with and get money from several of those - which would basically slow > down > > active development and maintenance for Airflow to a halt - because even > if > > we had a lot of people willing to contribute, maintainers would have very > > little - own - time to keep things running. There is a fine balance that > we > > keep now between the open-source and stakeholders, and open-source > product > > "friction" is an important property that the balance is built on. > > > > J. > > > > > > On Wed, Jul 9, 2025 at 9:21 PM Oliveira, Niko > <oniko...@amazon.com.invalid> > > wrote: > > > > > To me, I'm always working from a user perspective. My goal is to make > > > their lives easier, their deployments easier, the product the most > > > enjoyable for them to use. To me, the best user experience is that they > > > should enable bundle versioning and it should just work with as little > or > > > no extra steps and with as little infra as possible, and with the > fewest > > > possible pit falls for them to fall into. From a user perspective, > they've > > > already provisioned a database for airflow metadata, why is this > portion of > > > metadata leaking out to other forms of external storage? Now this is > > > another resource they now need to be aware of and manage the lifecycle > of > > > (or allow us write access into their accounts to manage for them). > > > > > > Ultimately, we should not be afraid of doing sometimes difficult work > to > > > make a good product for our users, it's for them in the end :) > > > > > > However, I see your perspectives as well, making our code and DB > > > management more complex is more work and complication for us. And from > the > > > feedback so far I'm out voted, so I'm happy as always to disagree and > > > commit, and do as you wish :) > > > > > > Thanks for the feedback everyone! > > > > > > Cheers, > > > Niko > > > > > > ________________________________ > > > From: Jens Scheffler <j_scheff...@gmx.de.INVALID> > > > Sent: Wednesday, July 9, 2025 12:07:08 PM > > > To: dev@airflow.apache.org > > > Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > know > > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > My 2ct on the discussions are similar like the opinions before. > > > > > > From my Edge3 experience migrating DB from provider - even if > > > technically enabled - is a bit of a pain. Adding a lot of boilerplate, > > > you need to consider your provider should also still be compatible with > > > AF2 (I assume) and once a user wants to downgrade it is a bit of manual > > > effort to downgrade DB as well. > > > > > > As long as we are not adding a generic Key/Value store to core (similar > > > liek Variables but for general purpose internal use not exposed to > users > > > - but then in case of trougbleshooting how to "manage/admin it?) I > would > > > also see it like Terraform - a secondary bucked for state os cheap and > > > convenient. Yes write access would be needed but only for Airflow. And > > > as it is separated from other should not be a general security harm... > > > just a small deployment complexity. And I assume versining is optional. > > > So no requirement to have it on per default and if a user wants to move > > > to/enable versioing then just the state bucket would need to be added > to > > > Bundle-config? > > > > > > TLDR I would favor a bucket, else if DB is the choice then a common > > > solution in core might be easier than a DB handling in provider. But > > > would also not block any other, just from point of complexity I'd not > > > favor provider specifc DB tables. > > > > > > Jens > > > > > > On 09.07.25 19:57, Jarek Potiuk wrote: > > > > What about the DynamoDB idea ? What you are trying to trade-off is > > > "writing > > > > to airflow metadata DB" with "writing to another DB" really. So yes > it > > > is - > > > > another thing you will need to have access to write to - other than > > > Airflow > > > > DB, but it's really the question should the boundaries be on > "Everything > > > > writable should be in Airflow" vs. "Everything writable should be in > the > > > > "cloud" that the integration is about. > > > > > > > > Yes - it makes the management using S3 versioning a bit more > "write-y" - > > > > but on the other hand it does allow to confine complexity to a pure > > > > "amazon" provider - with practically 0 impact on Airflow core and > > > airflow > > > > DB. Which I really like to be honest. > > > > > > > > And yes "co-location" is also my goal. And I think this is a perfect > way > > > to > > > > explain it as well why it is better to keep "S3 versioning" close to > "S3" > > > > and not to Airflow - especially that there will be a lot of > "S3-specific" > > > > things in the state that are not easy to abstract and have "common" > for > > > > other Airflow versioning implementations. > > > > > > > > You can think about it this way: > > > > > > > > Airflow has already done its job with abstractions - versioning > changes > > > and > > > > metadata DB is implemented in Airflow DB. If there are any missing > pieces > > > > in the abstraction that will be usable across multiple > implementations of > > > > versioning, we should - of course - add it to Airflow metadata DB - > in > > > the > > > > way that they can be used by those different implementations. But the > > > code > > > > to manage and use it should be in airflow-core. > > > > If there is anything specific for the implementation of S3 / Amazon > > > > integration -> it should be implemented independently from Airflow > > > Metadata > > > > DB. There are many complexities in managing and upgrading core DB > and we > > > > should not use the db to make provider-specific things. The > discussion > > > > about shared code and isolation is interesting in this context. > Because I > > > > think we might get to the point when we go deeper and deeper in this > > > > direction that we will have (and we already do it more or less) NO > > > > (regular) providers needed with whatever CLI or tooling we will need > to > > > > manage the Metadata DB. FAB and Edge are currently exceptions - but > they > > > > are by no means "regular" providers. > > > > > > > > So I'd say - if while designing/ implementing S3 versioning you will > see > > > > that part of the implementation can be abstracted away and added to > the > > > > core and used by other implementations - 100% - let's add it to the > core. > > > > But only then. If it is something that only Amazon provider needs > and S3 > > > > needs - let's make it use Amazon **whatever** as backing storage. > > > > > > > > I would even say - talk to the Google team and try to come up with an > > > > abstraction that can be used for versioning in both S3 and GCS, > agree on > > > > it, and let's see if this abstraction should find its way to the > core. > > > That > > > > would be my proposal. > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > On Wed, Jul 9, 2025 at 7:37 PM Oliveira, Niko > > > <oniko...@amazon.com.invalid> > > > > wrote: > > > > > > > >> Thanks for engaging folks! > > > >> > > > >> I don’t love the idea of using another bucket. For one, this means > > > Airflow > > > >> needs write access to S3 which is not ideal; some users/customers > are > > > very > > > >> sensitive about ever allowing write access to things. And two, you > will > > > >> commonly get issues with a design that leaks state into customer > managed > > > >> accounts/resources, they may delete the bucket not knowing what it > is, > > > they > > > >> may not migrate it to a new account or region if they ever move. I > think > > > >> it’s best for the data to be stored transparently to the user and > > > >> co-located with the data it strongly relates to (i.e. the dag runs > that > > > are > > > >> associated with those bundle versions). > > > >> > > > >> Is using DB Manager completely unacceptable these days? What are > folks' > > > >> thoughts on that? > > > >> > > > >> Cheers, > > > >> Niko > > > >> > > > >> ________________________________ > > > >> From: Jarek Potiuk <ja...@potiuk.com> > > > >> Sent: Wednesday, July 9, 2025 6:23:54 AM > > > >> To: dev@airflow.apache.org > > > >> Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager > > > >> > > > >> CAUTION: This email originated from outside of the organization. Do > not > > > >> click links or open attachments unless you can confirm the sender > and > > > know > > > >> the content is safe. > > > >> > > > >> > > > >> > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > externe. > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > > pouvez > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain > > > que > > > >> le contenu ne présente aucun risque. > > > >> > > > >> > > > >> > > > >>> Another option also would be Using dynamodb table? that also > supports > > > >> snapshots and i feel it works very well with state management. > > > >> > > > >> Yep that would also work. > > > >> > > > >> Anything "Amazon" to keep state would do. I think that it should be > our > > > >> "default" approach that if we have to keep state and the state is > > > connected > > > >> with specific "provider's" implementation, it's best to not keep > state > > > in > > > >> Airflow, but in the "integration" that the provider works with if > > > possible. > > > >> We cannot do it in "generic" case because we do not know what > > > >> "integrations" the user has - but since this is "provider's" > > > functionality, > > > >> using anything else that the given integration provides makes > perfect > > > >> sense. > > > >> > > > >> J. > > > >> > > > >> > > > >> On Wed, Jul 9, 2025 at 3:12 PM Pavankumar Gopidesu < > > > >> gopidesupa...@gmail.com> > > > >> wrote: > > > >> > > > >>> Agree another s3 bucket also works here > > > >>> > > > >>> Another option also would be Using dynamodb table? that also > supports > > > >>> snapshots and i feel it works very well with state management. > > > >>> > > > >>> > > > >>> Pavan > > > >>> > > > >>> On Wed, Jul 9, 2025 at 2:06 PM Jarek Potiuk <ja...@potiuk.com> > wrote: > > > >>> > > > >>>> One of the options would be to use a similar approach as terraform > > > >> uses - > > > >>>> i.e. use dedicated "metadata" state storage in a DIFFERENT s3 > bucket > > > >> than > > > >>>> DAG files. Since we know there must be an S3 available > (obviously) - > > > it > > > >>>> seems not too excessive to assume that there might be another > bucket, > > > >>>> independent of the DAG bucket where the state is stored - same > bucket > > > >>> (and > > > >>>> dedicated connection id) could even be used to store state for > > > multiple > > > >>> S3 > > > >>>> dag bundles - each Dag bundle could have a dedicated object > storing > > > the > > > >>>> state. The metadata is not huge, so continuously reading and > replacing > > > >> it > > > >>>> should not be an issue. > > > >>>> > > > >>>> What's nice about it - this single object could even > **actually** > > > use > > > >> S3 > > > >>>> versioning to keep historical state - to optimize things and > keep a > > > >> log > > > >>> of > > > >>>> changes potentially. > > > >>>> > > > >>>> J. > > > >>>> > > > >>>> On Wed, Jul 9, 2025 at 3:01 AM Oliveira, Niko > > > >>> <oniko...@amazon.com.invalid > > > >>>> wrote: > > > >>>> > > > >>>>> Hey folks, > > > >>>>> > > > >>>>> tl;dr I’d like to get some thoughts on a proposal to use DB > Manager > > > >> for > > > >>>> S3 > > > >>>>> Dag Bundle versioning. > > > >>>>> > > > >>>>> The initial commit for S3 Dag Bundles was recently merged [1] > but it > > > >>>> lacks > > > >>>>> Bundle versioning (since this isn’t trivial with something like > S3, > > > >>> like > > > >>>> it > > > >>>>> is with Git). The proposed solution involves building a snapshot > of > > > >> the > > > >>>> S3 > > > >>>>> bucket at the time each Bundle version is created, noting the > version > > > >>> of > > > >>>>> all the objects in the bucket (using S3’s native bucket > versioning > > > >>>> feature) > > > >>>>> and creating a manifest to store those versions and then giving > that > > > >>>> whole > > > >>>>> manifest itself some unique id/version/uuid. These manifests now > need > > > >>> to > > > >>>> be > > > >>>>> stored somewhere for future use/retrieval. The proposal is to > use the > > > >>>>> Airflow database using the DB Manager feature. Other options > include > > > >>>> using > > > >>>>> the local filesystem to store them (but this obviously wont work > in > > > >>>>> Airflow’s distributed architecture) or the S3 bucket itself (but > this > > > >>>>> requires write access to the bucket and we will always be at the > > > >> mercy > > > >>> of > > > >>>>> the user accidentally deleting/modifying the manifests as they > try to > > > >>>>> manage the lifecycle of their bucket, they should not need to be > > > >> aware > > > >>> of > > > >>>>> or need to account for this metadata). So the Airflow DB works > nicely > > > >>> as > > > >>>> a > > > >>>>> persistent and internally accessible location for this data. > > > >>>>> > > > >>>>> But I’m aware of the complexities of using the DB Manager and the > > > >>>>> discussion we had during the last dev call about providers > vending DB > > > >>>>> tables (concerning migrations and ensuring smooth upgrades or > > > >>> downgrades > > > >>>> of > > > >>>>> the schema). So I wanted to reach out to see what folks thought. > I > > > >> have > > > >>>>> talked to Jed, the Bundle Master (tm), and we haven’t come up > with > > > >>>> anything > > > >>>>> else that solves the problem as cleanly, so the DB Manager is > still > > > >> my > > > >>>> top > > > >>>>> choice. I think what we go with will pave the way for other > Bundle > > > >>>>> providers of a similar type as well, so it's worth thinking > deeply > > > >>> about > > > >>>>> this decision. > > > >>>>> > > > >>>>> Let me know what you think and thanks for your time! > > > >>>>> > > > >>>>> Cheers, > > > >>>>> Niko > > > >>>>> > > > >>>>> [1] https://github.com/apache/airflow/pull/46621 > > > >>>>> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
