o-nikolas commented on PR #46621: URL: https://github.com/apache/airflow/pull/46621#issuecomment-3079664938
> > I was talking to Jed about something very similar last week. I don't even think the interface needs to enforce the "how" for the versioning, that can be delegated to each Bundle provider. > > Cool! Glad to hear we’re on the same page. I don’t have a strong opinion on this, my main concern is making sure we align on the overall changes before moving forward. Cool, I'm glad as well! Let's definitely wait for @jedcunningham to weigh in as well. > > I agree, this approach might be simpler. > > > IMHO the interface should expect a string payload back from the bundle provider which is the representation of the dag bundle version (which can be whatever mechanism the provider wants/can use to serialize the state of the dags at that time). > > I’ve considered this scenario, and I agree it should be a mandatory requirement if any provider supports versioning. > > However, it seems challenging to get a specific snapshot version at the bucket + subpath level in most object stores. As far as I know, current cloud object stores only support versioning at the object level, not for a combination of bucket + subpath. > > One possible solution is: The user could provide a special path for storing a metadata file (maybe in JSON or YAML format), which would track the current bundle version ID along with the version IDs of the relevant objects (Dag files) and their prefixes. I’m not sure if this is the best approach, but I’d love to hear your thoughts. > > The solution will similar to metadata file in Open Table Format (e.g. Apache Iceberg, Delta Lake) and metadata file for supporting bundle versioning might be like: > > ```yaml > - bundle_version: <bundle version> > dag_files: > - object_version_id: <object version id from object store> > prefix: /path/dag/file > ``` > Yupp, this is also basically what I already proposed in my email on the dev list for the S3 Bundle :smiley: I'll quote this sentence for a gist: > The proposed solution involves building a snapshot of the S3 bucket at the time each Bundle version is created, noting the version of all the objects in the bucket (using S3’s native bucket versioning feature) and creating a manifest to store those versions and then giving that whole manifest itself some unique id/version/uuid. And I think that payload, and ones for other providers (whatever they decide/need to do for their storage), are exactly what can just be serialized into a string payload and returned through the Bundle interface. > > I'd also vote that the interface creates the version id for that, otherwise each provider will create their own semantics and shape of the id which is going to create inconsistency. > > Thanks for agreeing that the interface should create the version ID. I’ll continue to gather more ideas and suggestions on this. Sounds good! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
