The custom cell help us to save memory consumption. We don't have own serialization/deserialization mechanism, hence to transform data from client to server needs many conversion phase (user data -> Put/Cell -> pb object). The cost of conversion is large in transferring bulk data. In fact, we also have custom mutation to manage the memory usage of inner cell collection.
On 2017-09-30 02:43, Andrew Purtell <apurt...@apache.org> wrote: > What are the use cases for a custom cell? It seems a dangerously low level > thing to attempt and perhaps we should unwind support for it. But perhaps > there is a compelling justification. > > > On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <chia7...@apache.org> > wrote: > > > Thanks for all comment. > > > > The problem i want to resolve is the valid code should be exposed as > > IA.Public. Otherwise, end user have to access the IA.Private class to build > > the custom cell. > > > > For example, I have a use case which plays a streaming role in our > > appliaction. It > > applies the CellBuilder(HBASE-18519) to build custom cells. These cells > > have many same fields so they are put in shared-memory for avoiding GC > > pause. Everything is wonderful. However, we have to access the IA.Private > > class - KeyValue#Type - to get the valid code of Put. > > > > I believe there are many use cases of custom cell, and consequently it is > > worth adding a way to get the valid type via IA.Public class. Otherwise, it > > may imply that the custom cell is based on a unstable way, because the > > related code can be changed at any time. > > -- > > Chia-Ping > > > > On 2017-09-29 00:49, Andrew Purtell <apurt...@apache.org> wrote: > > > I agree with Stack. Was typing up a reply to Anoop but let me move it > > down > > > here. > > > > > > The type code exposes some low level details of how our current stores > > are > > > architected. But what if in the future you could swap out HStore > > implements > > > Store with PStore implements Store, where HStore is backed by HFiles and > > > PStore is backed by Parquet? Just as a hypothetical example. I know there > > > would be larger issues if this were actually attempted. Bear with me. You > > > can imagine some different new Store implementation that has some > > > advantages but is not a design derived from the log structured merge tree > > > if you like. Most values from a new Cell.Type based on KeyValue.Type > > > wouldn't apply to cells from such a thing because they are particular to > > > how LSMs work. I'm sure such a project if attempted would make a number > > of > > > changes requiring a major version increment and low level details could > > be > > > unwound from Cell then, but if we could avoid doing it in the first > > place, > > > I think it would better for maintainability. > > > > > > > > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote: > > > > > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <chia7...@apache.org> > > > > wrote: > > > > > > > > > hi folks, > > > > > > > > > > User is allowed to create custom cell but the valid code of type - > > > > > KeyValue#Type - is declared as IA.Private. As i see it, we should > > expose > > > > > KeyValue#Type as Public Client. Three possible ways are shown below: > > > > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public > > > > > 2) Move KeyValue#Type into Cell. > > > > > 3) Move KeyValue#Type to upper level > > > > > > > > > > Any suggestions? > > > > > > > > > > > > > > What is the problem that we are trying to solve Chia-Ping? You want to > > make > > > > Cells of a new Type? > > > > > > > > My first reaction is that KV#Type is particular to the KV > > implementation. > > > > Any new Cell implementation should not have to adopt the KeyValue > > typing > > > > mechanism. > > > > > > > > S > > > > > > > > > > > > > > > > > > > > > -- > > > > > Chia-Ping > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Andrew > > > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > > decrepit hands > > > - A23, Crosstalk > > > > > > > > > -- > Best regards, > Andrew > > Words like orphans lost among the crosstalk, meaning torn from truth's > decrepit hands > - A23, Crosstalk >
