You might be able to find relevant examples in the Delta lake Rust implementation[1] which I believe features DataFusion catalog integration
You might also find the "Catalogs, Schemas, and Tables" section of the library developers guide[2] helpful. [1] https://github.com/delta-io/delta-rs [2] https://arrow.apache.org/datafusion/library-user-guide/catalogs.html On Wed, Oct 4, 2023 at 1:52 PM Kothapalli, Vamsi <vamsi_kothapa...@intuit.com.invalid> wrote: > Thanks @chao ,@raphael for the suggestions and getting the discussion > started 🙂 > > I have a question regarding the client side implementation how would it > look like or > what libraries are already available to leveraged to build this feature! > Any suggestions are much > appreciated! > > A client code talking to hive metastore to get list of tables and then > registering them in runtime env > of data fusion? is there a lib for that kind of functionality in rust that > you know of? > > > I see that to build this functionality, I should get started with a repo > in datafusion-contrib, > or start working on it in my repo and then if everything looks good then > we can move it to > datafusion-contrib, github org? > > Thanks, > Vamsi > ________________________________ > From: Chao Sun <sunc...@apache.org> > Sent: Wednesday, October 4, 2023 8:35 AM > To: dev@arrow.apache.org <dev@arrow.apache.org> > Subject: Re: [DISCUSS][Rust][DataFusion][HiveMetaStore] Possible Metastore > integration with Data Fusion > > This email is from an external sender. > > > I think for DataFusion to Hive metastore we just need to implement the HMS > thrift API and create a client implementation. Agree this is best suited > under datafusion-contrib. > > On Wed, Oct 4, 2023 at 4:06 AM Raphael Taylor-Davies > <r.taylordav...@googlemail.com.invalid> wrote: > > > Hi, > > > > I think [1] might be a good place to start and handle coordination for > > this undertaking. I suspect it would probably want to live under the > > datafusion-contrib organisation, similar to HDFS [2] > > > > Kind Regards, > > > > Raphael > > > > [1]: https://github.com/apache/arrow-datafusion/issues/2209 > > [2]: https://github.com/datafusion-contrib/datafusion-objectstore-hdfs > > > > On 04/10/2023 00:18, Kothapalli, Vamsi wrote: > > > Hi devs, > > > > > > I would like to start discussion with possible integration of Hive > > Metastore with Data fusion, > > > > > > > > > Simmilarly along the lines of this Glue catalog itegration > > > https://github.com/apache/arrow-datafusion/issues/2206, Can anyone > > suggest me how the code should look like if > > > I or some would like to work on building HiveMetastore as catalog > > provider feature in data fusion > > > > > > Thanks, > > > Vamsi > > > > > > > > > [ > > > https://opengraph.githubassets.com/e562cb13412873d4ec10975083b49f02937926cfb878b1c07f4daf147c1f7494/apache/arrow-datafusion/issues/2206 > > ]<https://github.com/apache/arrow-datafusion/issues/2206> > > > [datafusion-contrib] AWS Glue Integration · Issue #2206 · > > apache/arrow-datafusion< > > https://github.com/apache/arrow-datafusion/issues/2206> > > > Is your feature request related to a problem or challenge? Please > > describe what you are trying to do. This has been discussed in various > > places, #907 and datafusion-contrib/datafusion-objectstore-s... > > > github.com > > > > > > > > >
