jason810496 commented on PR #46621:
URL: https://github.com/apache/airflow/pull/46621#issuecomment-3078208467
> I was talking to Jed about something very similar last week. I don't even
think the interface needs to enforce the "how" for the versioning, that can be
delegated to each Bundle provider.
Cool! Glad to hear we’re on the same page. I don’t have a strong opinion on
this, my main concern is making sure we align on the overall changes before
moving forward.
I agree, this approach might be simpler.
> IMHO the interface should expect a string payload back from the bundle
provider which is the representation of the dag bundle version (which can be
whatever mechanism the provider wants/can use to serialize the state of the
dags at that time).
I’ve considered this scenario, and I agree it should be a mandatory
requirement if any provider supports versioning.
However, it seems challenging to get a specific snapshot version at the
bucket + subpath level in most object stores. As far as I know, current cloud
object stores only support versioning at the object level, not for a
combination of bucket + subpath.
One possible solution is: The user could provide a special path for storing
a metadata file (maybe in JSON or YAML format), which would track the current
bundle version ID along with the version IDs of the relevant objects (Dag
files) and their prefixes. I’m not sure if this is the best approach, but I’d
love to hear your thoughts.
The solution will similar to metadata file in Open Table Format (e.g. Apache
Iceberg, Delta Lake) and metadata file for supporting bundle versioning might
be like:
```yaml
- bundle_version: <bundle version>
dag_files:
- object_version_id: <object version id from object store>
prefix: /path/dag/file
```
> I'd also vote that the interface creates the version id for that,
otherwise each provider will create their own semantics and shape of the id
which is going to create inconsistency.
Thanks for agreeing that the interface should create the version ID. I’ll
continue to gather more ideas and suggestions on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
