umangbudhwar opened a new issue, #6249: URL: https://github.com/apache/opendal/issues/6249
### Feature Description This feature proposes implementing the fsspec.AsyncFileSystem interface directly within the opendal Python bindings. This would allow OpenDAL to serve as a backend-compatible filesystem for widely-used Python libraries such as **DuckDB, Polars, PyArrow**, and others that rely on fsspec for abstracted filesystem access. An optional implementation of pyarrow.fs.FSSpecHandler could also be provided for better integration with PyArrow’s native filesystem API. ### Problem and Solution **Problem:** Many Python-based data tools (DuckDB, Polars, PyArrow, etc.) rely on fsspec for interacting with storage backends. While OpenDAL offers a unified interface for accessing remote and local storage, it currently does not expose an fsspec-compatible interface, making it non-trivial to use OpenDAL directly with these tools. There exists an external experimental project ([opendalfs](https://github.com/fsspec/opendalfs)) to bridge this gap via Rust, but it duplicates functionality already available in OpenDAL’s Python bindings and adds overhead in terms of maintenance and packaging. **Solution:** By implementing fsspec.AsyncFileSystem inside OpenDAL’s Python bindings (possibly as an optional extra), OpenDAL can natively power fsspec-based workflows without reinventing the wheel. This solution: - Allows immediate compatibility with popular Python data tools - Avoids redundant Rust+Python implementations (as in opendalfs) - Keeps maintenance, versioning, and testing centralised in the OpenDAL repo - Reduces friction for users who want to use OpenDAL as a unified data access layer in Python - Boosts real-world adoption of OpenDAL through seamless integration with existing ecosystems ### Additional Context This proposal was discussed in detail on a GitHub conversation in the OpenDAL repository: 📍 Reference Thread: [(rust) How to use opendal to load data into DuckDB or Polars LazyFrame? #5972](https://github.com/apache/opendal/discussions/5972) - Explored how current limitations in the Rust APIs of Polars and DuckDB make direct integration difficult. - Highlighted that fsspec is already widely supported in Python bindings for these tools. - Suggested that instead of building a standalone solution like opendalfs, integrating it directly into OpenDAL’s Python bindings would lead to better long-term maintainability and adoption. - Aligns with the direction acknowledged by maintainers like @Xuanwo. ### Are you willing to contribute to the development of this feature? - [x] Yes, I am willing to contribute to the development of this feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
