Guys, I want to be clear:
* "Per-field compression" design is the result of a research of the binary
infrastructure of Ignite and some other its places (querying, indexing,
etc.)
* Full-compression of object will be more effective, but in this case there
is no capability with querying and indexing (or there is large overhead by
way of decompressing of full object (or caches pages) on demand)
* "Per-field compression" is a one of ways to implement the compression
feature
I'm new to Ignite also I can be mistaken in some things.
Last 3-4 month I've tryed to start dicussion about a design, but nobody
answers nothing (except Dmitry and Valentin who was interested how it
works).
But I understand that this is community and nobody is obliged to anybody.
There are strong Ignite experts.
If they can help me and community with a design of the compression feature
it will be great.
At the moment I have a desire and time to be engaged in development of
compression feature in Ignite.
Let's use this opportunity :)
2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> Igniters,
>
> I have never seen a single Ignite user asking about compressing a single
> field. However, we have had requests to secure certain fields, e.g.
> passwords.
>
> I personally do not think per-field compression is needed, unless we can
> point out some concrete real life use cases.
>
> D.
>
> On Thu, Jun 8, 2017 at 3:42 AM, Vyacheslav Daradur <daradu...@gmail.com>
> wrote:
>
> > Anton,
> >
> > >> I thought that if there will storing compressed data in the memory,
> data
> > >> will transmit over wire in compression too. Is it right?
> >
> > In per-field compression case - yes.
> >
> > 2017-06-08 13:36 GMT+03:00 Антон Чураев <churaev...@gmail.com>:
> >
> > > Guys, could you please help me.
> > > I thought that if there will storing compressed data in the memory,
> data
> > > will transmit over wire in compression too. Is it right?
> > >
> > > 2017-06-08 13:30 GMT+03:00 Vyacheslav Daradur <daradu...@gmail.com>:
> > >
> > > > Vladimir,
> > > >
> > > > The main problem which I'am trying to solve is storing data in memory
> > in
> > > a
> > > > compression form via Ignite.
> > > > The main goal is using memory more effectivelly.
> > > >
> > > > >> here the much simpler step would be to full
> > > > compression on per-cache basis rather than dealing with per-fields
> > case.
> > > >
> > > > Please explain your idea. Compess data by memory-page?
> > > > Is it compatible with quering and indexing?
> > > >
> > > > >> In the end, if user would like to compress particular field, he
> can
> > > > always to it on his own
> > > > I think we mustn't think in this way, if user need something he
> trying
> > to
> > > > choose a tool which has this feature OOTB.
> > > >
> > > >
> > > >
> > > > 2017-06-08 12:53 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
> > > >
> > > > > Igniters,
> > > > >
> > > > > Honestly I still do not see how to apply it gracefully this feature
> > ti
> > > > > Ignite. And overall approach to compress only particular fields
> looks
> > > > > overcomplicated to me. Remember, that our main use case is an
> > > application
> > > > > without classes on the server. It means that any kind of
> annotations
> > > are
> > > > > inapplicable. To be more precise: proper API should be implemented
> to
> > > > > handle no-class case (e.g. how would build such an object through
> > > > > BinaryBuilder without a class?), and only then add annotations as
> > > > > convenient addition to more basic API.
> > > > >
> > > > > It seems to me that full implementation, which takes in count
> proper
> > > > > "classless" API, changes to binary metadata to reflect compressed
> > > fields,
> > > > > changes to SQL, changes to binary protocol, and porting to .NET and
> > > CPP,
> > > > > will yield very complex solution with little value to the product.
> > > > >
> > > > > Instead, as I proposed earlier, it seems that we'd better start
> with
> > > the
> > > > > problem we are trying to solve. Basically, compression could help
> in
> > > two
> > > > > cases:
> > > > > 1) Transmitting data over wire - it should be implemented on
> > > > communication
> > > > > layer and should not affect binary serialization component a lot.
> > > > > 2) Storing data in memory - here the much simpler step would be to
> > full
> > > > > compression on per-cache basis rather than dealing with per-fields
> > > case.
> > > > >
> > > > > In the end, if user would like to compress particular field, he can
> > > > always
> > > > > to it on his own, and set already compressed field to our
> > BinaryObject.
> > > > >
> > > > > Vladimir.
> > > > >
> > > > >
> > > > > On Thu, Jun 8, 2017 at 12:37 PM, Vyacheslav Daradur <
> > > daradu...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Valentin,
> > > > > >
> > > > > > Yes, I have the prototype[1][2]
> > > > > >
> > > > > > You can see an example of Java class[3] that I used in my
> > benchmark.
> > > > > > For example:
> > > > > > class Foo {
> > > > > > @BinaryCompression
> > > > > > String data;
> > > > > > }
> > > > > > If user make decision to store the object in compressed form, he
> > can
> > > > use
> > > > > > the annotation @BinaryCompression as shown above.
> > > > > > It means annotated field 'data' will be compressed at
> marshalling.
> > > > > >
> > > > > > [1] https://github.com/apache/ignite/pull/1951
> > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-5226
> > > > > > [3]
> > > > > > https://github.com/daradurvs/ignite-compression/blob/
> > > > > > master/src/main/java/ru/daradurvs/ignite/compression/
> > > > model/Audit1F.java
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2017-06-08 2:04 GMT+03:00 Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > Vyacheslav, Anton,
> > > > > > >
> > > > > > > Are there any ideas and/or prototypes for the API? Your design
> > > > > > suggestions
> > > > > > > seem to make sense, but I would like to see how it all this
> will
> > > like
> > > > > > from
> > > > > > > user's standpoint.
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Wed, Jun 7, 2017 at 1:06 AM, Антон Чураев <
> > churaev...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Vyacheslav, correct me if something wrong
> > > > > > > >
> > > > > > > > We could provide opportunity of choose between CPU usage and
> > > > MEM/NET
> > > > > > > usage
> > > > > > > > for users by compression some attributes of stored objects.
> > > > > > > > You have learned design, and it is possible to localize
> changes
> > > in
> > > > > > > > marshalling without performance affect and current
> > functionality.
> > > > > > > >
> > > > > > > > I think, that it's usefull for our project and users.
> > > > > > > > Community, what do you think about this proposal?
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-06-06 17:29 GMT+03:00 Vyacheslav Daradur <
> > > daradu...@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > In short,
> > > > > > > > >
> > > > > > > > > During marshalling a fields is represented as
> > > BinaryFieldAccessor
> > > > > > which
> > > > > > > > > manages its marshalling. It checks if the field is marked
> by
> > > > > > annotation
> > > > > > > > > @BinaryCompression, in that case - binary representation
> of
> > > > field
> > > > > > > (bytes
> > > > > > > > > array) will be compressed. It will be marked as compressed
> by
> > > > types
> > > > > > > > > constant (GridBinaryMarshaller.COMPRESSED), after this the
> > > > > > compressed
> > > > > > > > > bytes
> > > > > > > > > array wiil be include in binary representation of whole
> > object.
> > > > > Note,
> > > > > > > > > header of marshalled object will not be compressed.
> > Compression
> > > > > > > affected
> > > > > > > > > only object's field representation.
> > > > > > > > >
> > > > > > > > > Objects in IgniteCache is represented as BinaryObject which
> > is
> > > > > > wrapper
> > > > > > > > over
> > > > > > > > > bytes array of marshalled object.
> > > > > > > > > BinaryObject provides some usefull methods, which are used
> by
> > > > > Ignite
> > > > > > > > > systems.
> > > > > > > > > For example, the Queries use BinaryObject#field method,
> which
> > > > > > > > deserializes
> > > > > > > > > only field of object, without deserializing of whole
> object.
> > > > > > > > > BinaryObject#field method during deserialization, if meets
> > the
> > > > > > constant
> > > > > > > > of
> > > > > > > > > compressed type, decompress this bytes array, then continue
> > > > > > > unmarshalling
> > > > > > > > > as usual.
> > > > > > > > >
> > > > > > > > > Now, I introduced the Compressor interface in
> > > > IgniteConfigurations,
> > > > > > it
> > > > > > > > > allows user to use own implementation of compressor - it is
> > the
> > > > > > > > requirement
> > > > > > > > > in the task[1].
> > > > > > > > >
> > > > > > > > > As far as I know, Vladimir Ozerov doesn't like the idea of
> > > > granting
> > > > > > > this
> > > > > > > > > opportunity to the user.
> > > > > > > > > In that case we can choose a compression algorithm which we
> > > will
> > > > > > > provide
> > > > > > > > by
> > > > > > > > > default and will move the interface to internals of binary
> > > > > > > > infractructure.
> > > > > > > > > For this case I've prepared benchmarked, which I've sent
> > > earlier.
> > > > > > > > >
> > > > > > > > > I vote for ZSTD algorithm[2], it provides good compression
> > > ratio
> > > > > and
> > > > > > > good
> > > > > > > > > throughput. It has implementation in Java, .NET and C++,
> and
> > > has
> > > > > > > > > ASF-friendly license, we can use it in the all Ignite
> > > platforms.
> > > > > > > > > You can look at an assessment of this algorithm in my
> > > benchmark's
> > > > > > > > >
> > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-3592
> > > > > > > > > [2]https://github.com/facebook/zstd
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2017-06-06 16:02 GMT+03:00 Антон Чураев <
> > churaev...@gmail.com
> > > >:
> > > > > > > > >
> > > > > > > > > > Looks good for me.
> > > > > > > > > >
> > > > > > > > > > Could You propose design of implementation in couple of
> > > > > sentences?
> > > > > > > > > > So that we can estimate the completeness and complexity
> of
> > > the
> > > > > > > > proposal.
> > > > > > > > > >
> > > > > > > > > > 2017-06-06 15:26 GMT+03:00 Vyacheslav Daradur <
> > > > > daradu...@gmail.com
> > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Anton,
> > > > > > > > > > >
> > > > > > > > > > > Of course, the solution does not affect on existing
> > > > > > > implementation. I
> > > > > > > > > > mean,
> > > > > > > > > > > there is no changes if user not use the annotation
> > > > > > > > @BinaryCompression.
> > > > > > > > > > (no
> > > > > > > > > > > performance changes)
> > > > > > > > > > > Only if user make decision to use compression on
> specific
> > > > field
> > > > > > or
> > > > > > > > > fields
> > > > > > > > > > > of a class - in that case compression will be used at
> > > > > marshalling
> > > > > > > in
> > > > > > > > > > > relation to annotated fields.
> > > > > > > > > > >
> > > > > > > > > > > 2017-06-06 15:10 GMT+03:00 Антон Чураев <
> > > > churaev...@gmail.com
> > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Vyacheslav,
> > > > > > > > > > > >
> > > > > > > > > > > > Is it possible to propose implementation that can be
> > > > switched
> > > > > > on
> > > > > > > > > > > on-demand?
> > > > > > > > > > > > In this case it should not affect performance of
> > current
> > > > > > > solution.
> > > > > > > > > > > >
> > > > > > > > > > > > I mean, that users should make decision what is more
> > > > > important
> > > > > > > for
> > > > > > > > > > them:
> > > > > > > > > > > > throutput or memory/net usage.
> > > > > > > > > > > > May be they will be choose not all objects, or only
> > some
> > > > > > > attributes
> > > > > > > > > of
> > > > > > > > > > > > objects for compress.
> > > > > > > > > > > >
> > > > > > > > > > > > 2017-06-06 14:48 GMT+03:00 Vyacheslav Daradur <
> > > > > > > daradu...@gmail.com
> > > > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > Conclusion:
> > > > > > > > > > > > > Provided solution allows reduce size of an object
> in
> > > > > > > IgniteCache
> > > > > > > > at
> > > > > > > > > > the
> > > > > > > > > > > > > cost of throughput reduction (small - in some
> cases),
> > > it
> > > > > > > depends
> > > > > > > > on
> > > > > > > > > > > part
> > > > > > > > > > > > of
> > > > > > > > > > > > > object which will be compressed and compression
> > > > algorithm.
> > > > > > > > > > > > > I mean, we can make more effective use of memory,
> and
> > > in
> > > > > some
> > > > > > > > cases
> > > > > > > > > > it
> > > > > > > > > > > > can
> > > > > > > > > > > > > reduce loading of the interconnect. (replication,
> > > > > > rebalancing)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Especially, it will be particularly useful for
> > object's
> > > > > > fields
> > > > > > > > > which
> > > > > > > > > > > are
> > > > > > > > > > > > > large text (>~ 250 bytes) and can be effectively
> > > > > compressed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2017-06-06 12:00 GMT+03:00 Антон Чураев <
> > > > > > churaev...@gmail.com
> > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Vyacheslav, thank you! But could you please
> > provide a
> > > > > > > > conclusions
> > > > > > > > > > or
> > > > > > > > > > > > > > proposals based on this benchmarks?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur <
> > > > > > > > > daradu...@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Dmitry,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Excel-pages:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1). "Compression ratio (2)" - shows object
> size,
> > > with
> > > > > > > > > compression
> > > > > > > > > > > and
> > > > > > > > > > > > > > > without compression. (Conditions: literal text)
> > > > > > > > > > > > > > > 1st graph shows compression ratios of using
> > > different
> > > > > > > > > compression
> > > > > > > > > > > > > > algrithms
> > > > > > > > > > > > > > > depending on size of compressed field.
> > > > > > > > > > > > > > > 2nd graph shows evaluation of size of objects
> > > > depending
> > > > > > on
> > > > > > > > > sizes
> > > > > > > > > > > and
> > > > > > > > > > > > > > > compression algorithms.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2). "Compression ratio (1)" - shows object
> size,
> > > with
> > > > > > > > > compression
> > > > > > > > > > > and
> > > > > > > > > > > > > > > without compression. (Conditions: badly
> > compressed
> > > > > > > character
> > > > > > > > > > > > sequence)
> > > > > > > > > > > > > > > 1st graph shows compression ratios of using
> > > different
> > > > > > > > > compression
> > > > > > > > > > > > > > > algrithms depending on size of compressed
> field.
> > > > > > > > > > > > > > > 2nd graph shows evaluation of size of objects
> > > > depending
> > > > > > on
> > > > > > > > > sizes
> > > > > > > > > > > and
> > > > > > > > > > > > > > > compression algorithms.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3) 'put-avg" - shows average time of the "put"
> > > > > operation
> > > > > > > > > > depending
> > > > > > > > > > > on
> > > > > > > > > > > > > > size
> > > > > > > > > > > > > > > and compression algorithms.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 4) 'put-thrpt" - shows throughput of the "put"
> > > > > operation
> > > > > > > > > > depending
> > > > > > > > > > > on
> > > > > > > > > > > > > > size
> > > > > > > > > > > > > > > and compression algorithms.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 5) 'get-avg" - shows average time of the "get"
> > > > > operation
> > > > > > > > > > depending
> > > > > > > > > > > on
> > > > > > > > > > > > > > size
> > > > > > > > > > > > > > > and compression algorithms.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 6) 'get-thrpt" - shows throughput of the "get"
> > > > > operation
> > > > > > > > > > depending
> > > > > > > > > > > on
> > > > > > > > > > > > > > size
> > > > > > > > > > > > > > > and compression algorithms.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan <
> > > > > > > > > > > dsetrak...@apache.org
> > > > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Vladimir, I am not sure how to interpret the
> > > > graphs?
> > > > > > What
> > > > > > > > are
> > > > > > > > > > we
> > > > > > > > > > > > > > looking
> > > > > > > > > > > > > > > > at?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav
> > > > Daradur <
> > > > > > > > > > > > > > daradu...@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi, Igniters.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I've prepared some benchmarking. Results
> [1].
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > And I've prepared the evaluation in the
> form
> > of
> > > > > > > diagrams
> > > > > > > > > [2].
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I hope that helps to interest the community
> > and
> > > > > > > > > accelerates a
> > > > > > > > > > > > > > reaction
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > this improvment :)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > https://github.com/daradurvs/
> > > > > > ignite-compression/tree/
> > > > > > > > > > > > > > > > > master/src/main/resources/result
> > > > > > > > > > > > > > > > > [2] https://drive.google.com/file/d/
> > > > > > > > > > > > 0B2CeUAOgrHkoMklyZ25YTEdKcEk/
> > > > > > > > > > > > > > view
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2017-05-24 9:49 GMT+03:00 Vyacheslav
> Daradur
> > <
> > > > > > > > > > > > daradu...@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Guys, any thoughts?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2017-05-16 13:40 GMT+03:00 Vyacheslav
> > > Daradur <
> > > > > > > > > > > > > daradu...@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> Hi guys,
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> I've prepared the PR to show my idea.
> > > > > > > > > > > > > > > > > >> https://github.com/apache/
> > > > > ignite/pull/1951/files
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> About querying - I've just copied
> existing
> > > > tests
> > > > > > and
> > > > > > > > > have
> > > > > > > > > > > > > > annotated
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >> testing data.
> > > > > > > > > > > > > > > > > >> https://github.com/apache/
> > > > > > > > ignite/pull/1951/files#diff-
> > > > > > > > > > > c19a9d
> > > > > > > > > > > > > > > > > >> f4058141d059bb577e75244764
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> It means fields which will be marked by
> > > > > > > > > @BinaryCompression
> > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > >> compressed at marshalling via
> > > > BinaryMarshaller.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> This solution has no effect on existing
> > data
> > > > or
> > > > > > > > project
> > > > > > > > > > > > > > > architecture.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> I'll be glad to see your thougths.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> 2017-05-15 19:18 GMT+03:00 Vyacheslav
> > > Daradur
> > > > <
> > > > > > > > > > > > > > daradu...@gmail.com
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>> Dmitriy,
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> I have ready prototype. I want to show
> > it.
> > > > > > > > > > > > > > > > > >>> It is always easier to discuss on
> > example.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> 2017-05-15 19:02 GMT+03:00 Dmitriy
> > > Setrakyan
> > > > <
> > > > > > > > > > > > > > > dsetrak...@apache.org
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>>> Vyacheslav,
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> I think it is a bit premature to
> > provide a
> > > > PR
> > > > > > > > without
> > > > > > > > > > > > getting
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > >>>> community
> > > > > > > > > > > > > > > > > >>>> consensus on the dev list. Please
> allow
> > > some
> > > > > > time
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > >>>> respond.
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> D.
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> On Mon, May 15, 2017 at 6:36 AM,
> > > Vyacheslav
> > > > > > > Daradur
> > > > > > > > <
> > > > > > > > > > > > > > > > > >>>> daradu...@gmail.com>
> > > > > > > > > > > > > > > > > >>>> wrote:
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> > I created the ticket:
> > > > > > > > > https://issues.apache.org/jira
> > > > > > > > > > > > > > > > > >>>> /browse/IGNITE-5226
> > > > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > > > > > > >>>> > I'll prepare a PR with described
> > > solution
> > > > in
> > > > > > > > couple
> > > > > > > > > of
> > > > > > > > > > > > days.
> > > > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > > > > > > >>>> > 2017-05-15 15:05 GMT+03:00
> Vyacheslav
> > > > > Daradur
> > > > > > <
> > > > > > > > > > > > > > > > daradu...@gmail.com
> > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > > > > > > >>>> > > Hi, Igniters!
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > Apache 2.0 is released.
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > Let's continue the discussion
> about
> > a
> > > > > > > > compression
> > > > > > > > > > > > design.
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > At the moment, I found only one
> > > solution
> > > > > > which
> > > > > > > > is
> > > > > > > > > > > > > compatible
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > >>>> > querying
> > > > > > > > > > > > > > > > > >>>> > > and indexing, this is
> > > per-objects-field
> > > > > > > > > compression.
> > > > > > > > > > > > > > > > > >>>> > > Per-fields compression means that
> > > > metadata
> > > > > > (a
> > > > > > > > > > header)
> > > > > > > > > > > of
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > >>>> won't
> > > > > > > > > > > > > > > > > >>>> > > be compressed, only serialized
> > values
> > > of
> > > > > an
> > > > > > > > object
> > > > > > > > > > > > fields
> > > > > > > > > > > > > > (in
> > > > > > > > > > > > > > > > > bytes
> > > > > > > > > > > > > > > > > >>>> array
> > > > > > > > > > > > > > > > > >>>> > > form) will be compressed.
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > This solution have some
> contentious
> > > > > issues:
> > > > > > > > > > > > > > > > > >>>> > > - small values, like primitives
> and
> > > > short
> > > > > > > > arrays -
> > > > > > > > > > > there
> > > > > > > > > > > > > > isn't
> > > > > > > > > > > > > > > > > >>>> sense to
> > > > > > > > > > > > > > > > > >>>> > > compress them;
> > > > > > > > > > > > > > > > > >>>> > > - there is no possible to use
> > > > compression
> > > > > > with
> > > > > > > > > > > > > > java-predefined
> > > > > > > > > > > > > > > > > >>>> types;
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > We can provide an annotation,
> > > > > > > > @IgniteCompression -
> > > > > > > > > > for
> > > > > > > > > > > > > > > example,
> > > > > > > > > > > > > > > > > >>>> which can
> > > > > > > > > > > > > > > > > >>>> > > be used by users for marking
> fields
> > to
> > > > > > > compress.
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > Any thoughts?
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > Maybe someone already have ready
> > > design?
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > 2017-04-10 11:06 GMT+03:00
> > Vyacheslav
> > > > > > Daradur
> > > > > > > <
> > > > > > > > > > > > > > > > > daradu...@gmail.com
> > > > > > > > > > > > > > > > > >>>> >:
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > >> Alexey,
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >> Yes, I've read it.
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >> Ok, let's discuss about public
> API
> > > > > design.
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >> I think we need to add some a
> > > configure
> > > > > > > entity
> > > > > > > > to
> > > > > > > > > > > > > > > > > >>>> CacheConfiguration,
> > > > > > > > > > > > > > > > > >>>> > >> which will contain the Compressor
> > > > > interface
> > > > > > > > > > > > > implementation
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > >>>> > usefull
> > > > > > > > > > > > > > > > > >>>> > >> parameters.
> > > > > > > > > > > > > > > > > >>>> > >> Or maybe to provide a
> > > BinaryMarshaller
> > > > > > > > decorator,
> > > > > > > > > > > which
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > >>>> compress
> > > > > > > > > > > > > > > > > >>>> > >> data after marshalling.
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >> 2017-04-10 10:40 GMT+03:00 Alexey
> > > > > > Kuznetsov <
> > > > > > > > > > > > > > > > > akuznet...@apache.org
> > > > > > > > > > > > > > > > > >>>> >:
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >>> Vyacheslav,
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>> Did you read initial discussion
> > [1]
> > > > > about
> > > > > > > > > > > compression?
> > > > > > > > > > > > > > > > > >>>> > >>> As far as I remember we agreed
> to
> > > add
> > > > > only
> > > > > > > > some
> > > > > > > > > > > > > > "top-level"
> > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > >>>> > order
> > > > > > > > > > > > > > > > > >>>> > >>> to
> > > > > > > > > > > > > > > > > >>>> > >>> provide a way for
> > > > > > > > > > > > > > > > > >>>> > >>> Ignite users to inject some sort
> > of
> > > > > custom
> > > > > > > > > > > > compression.
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>> [1]
> > > > > > > > > > > > > > > > > >>>> > >>> http://apache-ignite-developer
> > > > > > > > > s.2346864.n4.nabble
> > > > > > > > > > .
> > > > > > > > > > > > > > > com/Data-c
> > > > > > > > > > > > > > > > > >>>> > >>> ompression-in-Ignite-2-0-
> > > td10099.html
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>> On Mon, Apr 10, 2017 at 2:19 PM,
> > > > > > daradurvs <
> > > > > > > > > > > > > > > > daradu...@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >>>> > wrote:
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>> > Hi Igniters!
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> > I am interested in this task.
> > > > > > > > > > > > > > > > > >>>> > >>> > Provide some kind of pluggable
> > > > > > compression
> > > > > > > > SPI
> > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > >>>> > >>> > <https://issues.apache.org/
> > > > > > > > > > > jira/browse/IGNITE-3592>
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> > I developed a solution on
> > > > > > > > > > BinaryMarshaller-level,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > reviewer
> > > > > > > > > > > > > > > > > >>>> has
> > > > > > > > > > > > > > > > > >>>> > >>> rejected
> > > > > > > > > > > > > > > > > >>>> > >>> > it.
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> > Let's continue discussion of
> > task
> > > > > goals
> > > > > > > and
> > > > > > > > > > > solution
> > > > > > > > > > > > > > > design.
> > > > > > > > > > > > > > > > > >>>> > >>> > As I understood that, the main
> > > goal
> > > > of
> > > > > > > this
> > > > > > > > > task
> > > > > > > > > > > is
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > >>>> data in
> > > > > > > > > > > > > > > > > >>>> > >>> > compressed form.
> > > > > > > > > > > > > > > > > >>>> > >>> > This is what I need from
> Ignite
> > as
> > > > its
> > > > > > > user.
> > > > > > > > > > > > > Compression
> > > > > > > > > > > > > > > > > >>>> provides
> > > > > > > > > > > > > > > > > >>>> > >>> economy
> > > > > > > > > > > > > > > > > >>>> > >>> > on
> > > > > > > > > > > > > > > > > >>>> > >>> > servers.
> > > > > > > > > > > > > > > > > >>>> > >>> > We can store more data on same
> > > > servers
> > > > > > at
> > > > > > > > the
> > > > > > > > > > cost
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > >>>> increasing CPU
> > > > > > > > > > > > > > > > > >>>> > >>> > utilization.
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> > I'm researching a possibility
> of
> > > > > > > > > implementation
> > > > > > > > > > of
> > > > > > > > > > > > > > > > compression
> > > > > > > > > > > > > > > > > >>>> at the
> > > > > > > > > > > > > > > > > >>>> > >>> > cache-level.
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> > Any thoughts?
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> > --
> > > > > > > > > > > > > > > > > >>>> > >>> > Best regards,
> > > > > > > > > > > > > > > > > >>>> > >>> > Vyacheslav
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>> > --
> > > > > > > > > > > > > > > > > >>>> > >>> > View this message in context:
> > > > > > > > > > > http://apache-ignite-
> > > > > > > > > > > > > > > > > >>>> > >>> > developers.2346864.n4.nabble.
> > > > > > > > > > > > com/Data-compression-in-
> > > > > > > > > > > > > > > > > >>>> > >>> > Ignite-2-0-tp10099p16317.html
> > > > > > > > > > > > > > > > > >>>> > >>> > Sent from the Apache Ignite
> > > > Developers
> > > > > > > > mailing
> > > > > > > > > > > list
> > > > > > > > > > > > > > > archive
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > >>>> > >>> Nabble.com.
> > > > > > > > > > > > > > > > > >>>> > >>> >
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>> --
> > > > > > > > > > > > > > > > > >>>> > >>> Alexey Kuznetsov
> > > > > > > > > > > > > > > > > >>>> > >>>
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >> --
> > > > > > > > > > > > > > > > > >>>> > >> Best Regards, Vyacheslav
> > > > > > > > > > > > > > > > > >>>> > >>
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> > > --
> > > > > > > > > > > > > > > > > >>>> > > Best Regards, Vyacheslav
> > > > > > > > > > > > > > > > > >>>> > >
> > > > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > > > > > > >>>> > --
> > > > > > > > > > > > > > > > > >>>> > Best Regards, Vyacheslav
> > > > > > > > > > > > > > > > > >>>> >
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> --
> > > > > > > > > > > > > > > > > >>> Best Regards, Vyacheslav
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> --
> > > > > > > > > > > > > > > > > >> Best Regards, Vyacheslav
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > Best Regards, Vyacheslav
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > Best Regards, Vyacheslav
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Best Regards, Vyacheslav
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best Regards, Anton Churaev
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Best Regards, Vyacheslav
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > Best Regards, Anton Churaev
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best Regards, Vyacheslav
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Best Regards, Anton Churaev
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best Regards, Vyacheslav
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Best Regards, Anton Churaev
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards, Vyacheslav
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards, Vyacheslav
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best Regards, Anton Churaev
> > >
> >
> >
> >
> > --
> > Best Regards, Vyacheslav
> >
>
--
Best Regards, Vyacheslav
