ap-- commented on PR #37524:
URL: https://github.com/apache/airflow/pull/37524#issuecomment-1955041434
> Great! Even better because I think there is a regression now @ap-- . The
modules are eagerly loaded now ie:
>
> ```
> o = ObjectStoragePath("s3://xxx")
> ```
>
> Will now load `s3fs` right away, which it shouldn't do as this code will
be often placed at the top of a DAG and thus loaded every single time it is
parsed.
This design was intentional, because fsspec filesystems can implement their
own storage_option parsing and protocol strip methods, and correct parsing
behaviour can only be ensured if the filesystem class is importable.
I will see if I can make changes to universal_pathlib to delay that as long
as possible, or offer another way of handling these cases. What operations on
an `ObjectStoragePath` would you expect to not import the underlying
filesystem_spec AbstractFileSystem class? And is `ObjectStoragePath` supposed
to handle all `UPath` filesystems, or only a subset?
I'll open an issue in universal_pathlib for further discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
