o-nikolas commented on PR #46621: URL: https://github.com/apache/airflow/pull/46621#issuecomment-3074540257
> Hi all, > > Would it be a good idea to introduce a `RemoteDagBundle` interface (or perhaps extend `BaseDagBundle` or add a common utility in the `bundles.base` module) to support **versioning** for remote Dag bundles like `S3DagBundle`? > > We could support bundle versioning using one of the following approaches (potentially controlled by an `enable_versioning` parameter in `RemoteDagBundle`): > > 1. **Content-based hash**: Calculate a hash of the bundle by hashing each Dags file and incorporating the directory structure, then return a final hash value as the version. > > 2. **Timestamp-based version**: Use the last modified timestamp of the bundle. Most cloud providers support a `LastModified` or `updated` metadata field that we can leverage. > > > If this sounds reasonable, I will open a new issue to track this feature and raise PR for it. CC @jedcunningham I was talking to Jed about something very similar last week. I don't even think the interface needs to enforce the "how" for the versioning, that can be delegated to each Bundle provider. But I agree that the interface should support this, especially from the persistence side of things. IMHO the interface should expect a string payload back from the bundle provider which is the representation of the dag bundle version (which can be whatever mechanism the provider wants/can use to serialize the state of the dags at that time). I'd also vote that the interface creates the version id for that, otherwise each provider will create their own semantics and shape of the id which is going to create inconsistency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
