On Tue, 13 Jul 2021, at 01:07, Bhalchandra Pandit wrote: > Thanks for showing interest and for raising valid questions. > > I do not currently have a forked branch to show how it is done. However, I > can certainly make it happen. It will take me a couple of weeks or so to do > it. I will need to get myself familiarized with the contribution process.
The short version is that you need to open a ticket on Jira [1] and then send a PR against the master branch on Github [2] with the title "THRIFT-NNNN: Your title". Everything else can be figured out later. [1] https://issues.apache.org/jira/projects/THRIFT/issues [2] https://github.com/apache/thrift/ > > To answer other questions: > > 1. The current implementation is in Java. However, it can also be easily > ported to other languages. Happy to help in that direction as well. > 2. The solution does not modify the definition of any structure (eg, > Payload => SlimPayload). The output instance has the same type regardless > of whether we use full deserialization or partial deserialization. In that > sense, it is 100% compatible in both directions (partial <=> full). The > solution enables selectively deserializing any arbitrary nested subset. > > Kumar > > On Mon, Jul 12, 2021 at 4:10 PM Duru Can Celasun <dcela...@apache.org> > wrote: > > > This is definitely an interesting problem at scale and it'd be great to > > have a solution upstream. > > > > I second Yuxuan's questions. From the blog post it seems you have an > > implementation for Java, but it would be great to have at least one more. > > > > On Tue, 13 Jul 2021, at 00:00, Yuxuan Wang wrote: > > > Hi Bhalchandra, > > > > > > Do you have any open source code (e.g. forked thrift) to show how you > > > implemented it? The blog post stated what's the problem but really lack > > > information on the solution side: > > > > > > 1. How did you solve the problem? > > > 2. In which language(s) did you implement the solution? > > > > > > Without that information it's really not much we can do here. > > > > > > Also just based on the problem you described, a simple solution would be > > > just to duplicate the struct definition and remove the fields you don't > > > need, for example: > > > > > > // Original struct > > > struct Payload { > > > 1: optional Type1 field1, > > > 2: optional Type2 field2, > > > 3: optional Type3 field3, > > > ... > > > } > > > > > > // Slim struct > > > struct SlimPayload { > > > 1: optional Type1 field1, > > > // field2 removed because we don't care about it in this use case > > > 3: optional SlimType3 field3, > > > ... > > > } > > > > > > But of course it's hard to keep SlimPayload in sync with the original > > > Payload so I can see there are some values to have some helpers to help > > > that, but as long as you don't make breaking changes into Payload "keep > > > them in sync" is a false problem. > > > > > > On Mon, Jul 12, 2021 at 12:56 PM Bhalchandra Pandit > > > <kpan...@pinterest.com.invalid> wrote: > > > > > > > Hi All, > > > > I work for Pinterest. I developed a technique for partial > > deserialization > > > > of Thrift that has been very useful in significantly improving > > efficiency > > > > of the data processing at Pinterest. I would like to contribute that > > > > feature to Apache Thrift. More details on this technique are available > > in > > > > this blog I recently wrote: > > > > > > > > > > https://medium.com/pinterest-engineering/improving-data-processing-efficiency-using-partial-deserialization-of-thrift-16bc3a4a38b4 > > > > > > > > I would like to know if any of you are interested in helping with > > > > contributing this work to the main branch. > > > > > > > > Kumar > > > > > > > > > >
