It's nice to hear gRPC support in Hive Metastore again. Thanks for contributing back to the community.
It seems that there is an old design doc about gRPC support in Hive Metastore https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158869886, could we continue on this doc? Best Regards, Rascal Wu Stamatis Zampetakis <zabe...@gmail.com> 于2022年8月24日周三 21:24写道： > Hi Rohan and team, > > The work sounds exciting thanks for considering contributing back to the > community. > > The design document didn't arrive cause attachements are not allowed in > many Apache lists. > > Maybe as a first step it would be nice to share a link to a Google doc > where people can add comments and possibly provide some feedback on it. > Then, I guess it makes sense to put in the wiki > under the respective section (design documents) and/or upload it in the > JIRA case. > > I am not very familiar with the area so not sure if I can help much pushing > this forward but I am definitely interested to learn more about this work. > > Best, > Stamatis > > On Tue, Aug 23, 2022, 2:05 AM Cameron Moberg <cjmob...@google.com.invalid> > wrote: > > > > > *Sending on behalf of Rohan where policies don't allow sending outside of > > our domain for interns:* > > Hello - > > > > During my internship I’ve been working on gRPC native support in the > > standalone hive metastore as it comes with a variety of benefits. As a > > proof of concept, my team, Dataproc Metastore on GCP currently uses a > > client side proxy to translate Thrift requests to gRPC coupled with a > > server side proxy to translate the gRPC requests back to Thrift. The > > process is repeated in reverse to deliver the server response to the > > client. While this approach has been successful, native gRPC support has > > several cloud-centric advantages over the current configuration: > > > > - enables streaming support > > - allows for native integrations in Hive ecosystem for various query > > engines like Impala, Spark SQL, and Trino to take advantage of > streaming > > (eventually) > > - has support for custom interceptors for more fine-grained control > > over the server action > > - built on HTTP/2 protocol > > > > I’ve opened a PR here <https://github.com/apache/hive/pull/3534> (just > > fyi, no rush), some background – this proto3 definition has been > refactored > > to take a MethodNameRequest and MethodNameResponse to stop any future > > backwards incompatibilities. Unfortunately, the other metastore.proto > which > > has SplitInfo uses a `required` field setting, which makes upgrading it > not > > feasible since moving away from `required` will change the SerDe of > proto, > > potentially a breaking change depending on clients. > > While this is the last week of my internship my hosts > cjmob...@google.com > > and hchinch...@google.com will continue to develop in this area with > > further implementation building on the proto. > > > > Attached is the full design doc, I’m not sure how I’m supposed to share > > documents like this, so I can reupload this somewhere or convert to the > > wiki. > > > > Comments are of course appreciated! > > > > Thank you, > > Rohan Sonecha > > >
