bolkedebruin commented on PR #34729: URL: https://github.com/apache/airflow/pull/34729#issuecomment-1748698174
> What I’m envisioning is something like > > ```python > warehouse_mnt = afs.mount("s3://warehouse") # Can have conn_id too, it’s orthogontal. > > @dag > def my_dag: > > @task > def load_file(src): > with afs.open(src) as f: > f.read() > > load_file(warehouse_mnt / "my_data.csv") > ``` > > instead of exposing the mount to the user, we encapsulate the data inside the Mount object and expose a Path-like interface to let the user operate on it directly. You can work with the mount directly as well, either by passing a mount point explicitly to `mount` or by accessing `mnt.mount_location` (or whatever, returns the location as as string) and work with that. > I think I like that. If we can use both patterns that would be pretty cool. > The Dataset part I’m thinking now is pretty simple, just make the Mount object inherit from Dataset (or _is_ Dataset?) so that object can be used for both purposes without duplicating the URL if you need that. Not that useful but the two are really the same idea (a reference to some resource) that I feel shouldn’t be two things. Not entirely sure about this. To me, for now, both are quite different. A dataset points to data and a mount provides an interface that allows you to manipulate file like objects. So not really a reference to a resource imho. But maybe I am seeing that wrongly. If you have an example how you think that would work on the user side it would help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org