While I understand Kyle's concerns, I'm all for a C++ or Rust implementation.
We know that this is going to help a lot of people that want to integrate Iceberg in engines that are outside the JVM ecosystem. I think it would be great to work with anyone that is interested and build up the community in this area! Ryan On Wed, Jun 8, 2022 at 3:16 AM OpenInx <open...@gmail.com> wrote: > As a cloud-native table format standard for the big-data ecosystem, I > believe supporting multiple languages is the correct direction so that > different languages can connect to the apache iceberg table format. > > But I can also get Kyle's point about lacking enough resources(developers > and reviewers ) to accomplish this goal. In my mind, Python, Golang, C++, > Rust , all of them can be regarded as the native language support. we may > just need to support the Rust SDK and then all of the other languages can > just wrap the Rust SDK to access the table format. > > Anyway, we will need to wait for the REST catalog finished before we > introduce another languages support , because we can not access the iceberg > table by invoking the JVM catalog interfaces. > > On Tue, Jun 7, 2022 at 4:41 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > >> There’s also the question of how useful this would be in practice given >>> the complexity of using C++ (or Rust etc) within some of the major >>> frameworks. >>> >> >> One place this would be useful is for the Arrow's DataSet API [1]. An >> option the Arrow community might be open to is hosting parts of the code >> there (this is what is done for Apache Parquet C++). This helps shape some >> of the answers to other questions posed (ORC and Parquet are already in the >> Repo, it provides a Filesystem interface, etc). The project doesn't >> currently consume Avro, and I think the preferred approach is to make a >> clean room Avro parser. But I agree this is a non-trivial effort to get >> underway. >> >> Another area to consider is compatibility testing. I think before a >> third officially supported community library is introduced it would be good >> to have a compatibility framework in place to make sure implementations are >> all interpreting the specification correctly. If there isn't already an >> effort here, I'd like to start contributing something (probably will have >> bandwidth sometime place in Q3). >> >> Thanks, >> -Micah >> >> >> [1] https://arrow.apache.org/docs/cpp/dataset.html >> >> On Sun, Jun 5, 2022 at 11:07 PM Kyle Bendickson <k...@tabular.io> wrote: >> >>> Hi caneGuy, >>> >>> I personally don’t dislike this idea. I understand the performance >>> benefits. >>> >>> But this would be a huge undertaking for the community. We’d need to >>> ensure we had sufficient developer support for reviews (likely one of the >>> biggest issues), as well as a number of other things. Particularly >>> dependencies, package management, etc. We’d also need to scope support down >>> to specific OS / compilers etc. >>> >>> We’d also need to be sure we had adequate developer support from a wide >>> enough range of the community to support the project long term. One issue >>> in open source is that developers will work on something tangential to >>> their project in another repository, but nobody is available to maintain it. >>> >>> There’s also the question of how useful this would be in practice given >>> the complexity of using C++ (or Rust etc) within some of the major >>> frameworks. >>> >>> Again, I’m not opposed to the idea but just trying to be realistic about >>> the realities of such an undertaking. It would need full community support >>> (or at least support from enough community members to be sustainable). >>> >>> If you wanted to make a design doc, the milestones tab in the Iceberg >>> project has some that you might use as reference. >>> >>> *I highly suggest you come to the next community sync and bring this up >>> to the community then.* >>> >>> If you’re not already on the invite list for the monthly community sync, >>> you can get on it by joining the Google group. You’ll receive incites when >>> they go out: >>> https://groups.google.com/g/iceberg-sync >>> >>> Looking forward to seeing you at the next community sync. >>> >>> A design document and/or any prior art would be very helpful as the >>> community sync does discuss many topics (possibly there is existing C++ >>> support in StarRocks for Iceberg V1?). >>> >>> Thank you, >>> Kyle Bendickson >>> GitHub: kbendick >>> >>> On Sun, Jun 5, 2022 at 10:44 PM Sam Redai <s...@tabular.io> wrote: >>> >>>> Currently there is no existing effort to develop a C++ package. That >>>> being said I think it would be awesome to have one! If anyone is willing to >>>> start that development effort, I can help with some of the ground work to >>>> kickstart it. >>>> >>>> I would say the first step would be for someone to prepare a high-level >>>> proposal. >>>> >>>> -Sam >>>> >>>> On Sun, Jun 5, 2022 at 11:02 PM 周康 <zhoukang199...@gmail.com> wrote: >>>> >>>>> Hi team >>>>> I am a dev from StarRocks community, and we have supported iceberg v1 >>>>> format. >>>>> We are also planning to support v2 format. If there is a C++ package, >>>>> it will be very convenient for our implementation. >>>>> At the same time, other c++ computing engines support v2 format will >>>>> also be faster. >>>>> >>>>> Do we have plans to support c++ version sdk? >>>>> -- >>>>> caneGuy >>>>> >>>> -- >>>> >>>> Sam Redai <s...@tabular.io> >>>> >>>> Developer Advocate | Tabular <https://tabular.io/> >>>> >>>> c (267) 226-8606 >>>> >>> -- Ryan Blue Tabular
