I would vote for a Python FileIO implementation using OpenDAL. FileIO tends to be a simpler interface and we already have support for multiple options (PyArrow and FSSpec). This would give users the choice.
On Fri, Aug 2, 2024 at 7:53 AM Renjie Liu <liurenjie2...@gmail.com> wrote: > Hi: > > Thanks Xuanwo for raising this. > > As mentioned in another thread, I think using iceberg-rust in pyiceberg is > a good idea. > > About this idea, would you have a more detailed design? For example, > where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What > kind of interface should we provide to pyiceberg, FileIO or OpenDAL? > > I think this is a good first step moving forward to make pyiceberg backed > iceberg-rust. In the future we can replace components gradually. > > On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xua...@apache.org> wrote: > >> > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL >> implementations via pyo3 / fsspec bindings >> <https://github.com/apache/opendal/issues/4511>? >> >> Hi, Raschkowski, good question! >> >> It's possible. There is an ongoing project developing fsspec bindings for >> opendal at https://github.com/fsspec/opendalfs. Once complete, we can >> directly use opendal through fsspec. >> >> This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should >> be able to use opendalfs as an alternative implementation of the fsspec >> AbstractFileSystem class. >> >> On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote: >> >> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL >> implementations via pyo3 / fsspec bindings >> <https://github.com/apache/opendal/issues/4511>? >> >> >> ------------------------------ >> >> *From:* Joe Stein <crypt...@gmail.com> >> *Sent:* Thursday, August 1, 2024 3:37 AM >> *To:* dev@iceberg.apache.org <dev@iceberg.apache.org> >> *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io >> >> *CAUTION:* This email originates from an external party (outside of >> Palantir). If you believe this message is suspicious in nature, please use >> the "Report Message" button built into Outlook. >> >> Kafka did this with librdkafka and was wildly successful. The underlying >> bindings being in rust are great with a layer for access in Python +1 >> >> >> ~ Joe Stein >> >> >> On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xua...@apache.org> wrote: >> >> Hello everyone >> >> I start this thread to discuss the idea about using iceberg-rust as >> pyiceberg file io. >> >> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io >> [hackmd.io] >> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$> >> >> In summary, we can leverage the work from iceberg-rust to help pyiceberg >> in developing a fast and compact file IO system that benefits users with >> specific constraints. >> >> Welcome to join in the discussion. >> >> Xuanwo >> >> https://xuanwo.io/ [xuanwo.io] >> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$> >> >> Xuanwo >> >> https://xuanwo.io/ >> >> -- Ryan Blue Databricks
