ap-- opened a new issue, #37067: URL: https://github.com/apache/airflow/issues/37067
### Apache Airflow version main (development) ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? I'm opening this issue to track a needed refactor of `airflow.io.path.ObjectStoragePath` to support the next release of `universal_pathlib` without having to have a special implementation for Python-3.12 ### What you think should happen instead? Tagging @bolkedebruin, @uranusjr, @potiuk It would be great if I could have some insight into the intended behaviour for the custom `UPath` subclass `airflow.io.path.ObjectStoragePath` Looking at the current implementation it seems that these are the intended changes: - (1) extracting a connection ID from the URI https://github.com/apache/airflow/blob/07fd3646fa765859a152731bf135ea944d0753d7/airflow/io/path.py#L148-L151 - (2) delegating fsspec filesystem creation to `airflow.io.store.ObjectStore` https://github.com/apache/airflow/blob/07fd3646fa765859a152731bf135ea944d0753d7/airflow/io/path.py#L49-L65 - (3) customization and extension of the `UPath` interface with several custom methods. I am working on supporting (1) and (2) via custom methods that can be overwritten in UPath subclasses, and allow to have a single implementation for python versions prior to 3.12 and 3.12+, see: fsspec/universal_pathlib#172 Once that is integrated in UPath it should simplify the `ObjectStoragePath` implementation a lot. The custom accessor will be replaced by a factory method for creating fsspec filesystems, and the custom `__new__` implementation by a method for parsing additional storage_options. Let me know if that sounds good, and what's the best way to collaborate on this issue and move forward. Cheers, Andreas 😃 ### How to reproduce N/A ### Operating System N/A ### Versions of Apache Airflow Providers --- ### Deployment Other ### Deployment details N/A ### Anything else? N/A ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
