Thanks for showing interest and for raising valid questions.
I do not currently have a forked branch to show how it is done. However, I
can certainly make it happen. It will take me a couple of weeks or so to do
it. I will need to get myself familiarized with the contribution process.
To answer other questions:
1. The current implementation is in Java. However, it can also be easily
ported to other languages. Happy to help in that direction as well.
2. The solution does not modify the definition of any structure (eg,
Payload => SlimPayload). The output instance has the same type regardless
of whether we use full deserialization or partial deserialization. In that
sense, it is 100% compatible in both directions (partial <=> full). The
solution enables selectively deserializing any arbitrary nested subset.
Kumar
On Mon, Jul 12, 2021 at 4:10 PM Duru Can Celasun <dcela...@apache.org>
wrote:
> This is definitely an interesting problem at scale and it'd be great to
> have a solution upstream.
>
> I second Yuxuan's questions. From the blog post it seems you have an
> implementation for Java, but it would be great to have at least one more.
>
> On Tue, 13 Jul 2021, at 00:00, Yuxuan Wang wrote:
> > Hi Bhalchandra,
> >
> > Do you have any open source code (e.g. forked thrift) to show how you
> > implemented it? The blog post stated what's the problem but really lack
> > information on the solution side:
> >
> > 1. How did you solve the problem?
> > 2. In which language(s) did you implement the solution?
> >
> > Without that information it's really not much we can do here.
> >
> > Also just based on the problem you described, a simple solution would be
> > just to duplicate the struct definition and remove the fields you don't
> > need, for example:
> >
> > // Original struct
> > struct Payload {
> > 1: optional Type1 field1,
> > 2: optional Type2 field2,
> > 3: optional Type3 field3,
> > ...
> > }
> >
> > // Slim struct
> > struct SlimPayload {
> > 1: optional Type1 field1,
> > // field2 removed because we don't care about it in this use case
> > 3: optional SlimType3 field3,
> > ...
> > }
> >
> > But of course it's hard to keep SlimPayload in sync with the original
> > Payload so I can see there are some values to have some helpers to help
> > that, but as long as you don't make breaking changes into Payload "keep
> > them in sync" is a false problem.
> >
> > On Mon, Jul 12, 2021 at 12:56 PM Bhalchandra Pandit
> > <kpan...@pinterest.com.invalid> wrote:
> >
> > > Hi All,
> > > I work for Pinterest. I developed a technique for partial
> deserialization
> > > of Thrift that has been very useful in significantly improving
> efficiency
> > > of the data processing at Pinterest. I would like to contribute that
> > > feature to Apache Thrift. More details on this technique are available
> in
> > > this blog I recently wrote:
> > >
> > >
> https://medium.com/pinterest-engineering/improving-data-processing-efficiency-using-partial-deserialization-of-thrift-16bc3a4a38b4
> > >
> > > I would like to know if any of you are interested in helping with
> > > contributing this work to the main branch.
> > >
> > > Kumar
> > >
> >
>
