Joris Van den Bossche created ARROW-8766: --------------------------------------------
Summary: [Python] A FileSystem implementation based on Python callbacks Key: ARROW-8766 URL: https://issues.apache.org/jira/browse/ARROW-8766 Project: Apache Arrow Issue Type: Improvement Components: Python Reporter: Joris Van den Bossche The new {{pyarrow.fs}} filesystems are now actual C++ objects, and no longer "just" a python interface. So they can't easily be expanded from the Python side, and the existing integration with {{fsspec}} filesystems is therefore also not working anymore. One possible solution is to have a C++ filesystem that calls back into a python object for each of its methods (possibly similar to how you can implement a flight server in Python, I suppose). Such a FileSystem implementation would allow to make a {{pyarrow.fs}} wrapper for {{fsspec}} filesystems, and thus allow such filesystems to be used in pyarrow where new filesystems are expected. -- This message was sent by Atlassian Jira (v8.3.4#803005)