Hello!
I want to share with the developer community my compression prototype.
Long story short, it compresses BinaryObject's byte[] as they are written
to Durable Memory page, operating on a pre-built dictionary. Typical
compression ratio is 0.4 (meaning 2.5x compression) using custo
gt; configuration,
> > > where
> > > > > > you
> > > > > > > can enable compression, enable key compression (which has
> heavier
> > > > > > > performance implications), adjust dictionary gathering
> settings,
> > &
> >
>> > > > > > On Thu, Aug 23, 2018 at 5:17 AM Ilya Kasnacheev <
>> > > > > ilya.kasnach...@gmail.com
>> > > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hel
Hello again!
I've been running various compression parameters through cod dataset.
It looks like the best compression level in terms of speed is either 1 or 2.
The default for Zstd seems to be 3 which would almost always perform worse.
For best performance a dictionary of 1024 is optimal, for bet
Hi Ilya,
This is very useful. Is the compression going to be per-page, in which case
the dictionary is going to be kept inside of a page? Or do you have some
other design in mind?
D.
On Mon, Sep 3, 2018 at 10:36 AM, Ilya Kasnacheev
wrote:
> Hello again!
>
> I've been running various compressio
Hello!
The compression is per-binary-object, but dictionary is external, shared
between multiple (millions of) entries and stored alongside compressed data.
Regards,
--
Ilya Kasnacheev
вт, 4 сент. 2018 г. в 2:40, Dmitriy Setrakyan :
> Hi Ilya,
>
> This is very useful. Is the compression going
On Tue, Sep 4, 2018 at 1:16 AM, Ilya Kasnacheev
wrote:
> Hello!
>
> The compression is per-binary-object, but dictionary is external, shared
> between multiple (millions of) entries and stored alongside compressed
> data.
>
I was under a different impression. If the dictionary is for the whole d
Hello!
Each node has a local dictionary (per node currently, per cache planned).
Dictionary is never shared between nodes. As data patterns shift,
dictionary rotation is also planned.
With Zstd, the best dictionary size seems to be 1024 bytes. I imagine It is
enough to store common BinaryObject b
On Tue, Sep 4, 2018 at 2:55 AM, Ilya Kasnacheev
wrote:
> Hello!
>
> Each node has a local dictionary (per node currently, per cache planned).
> Dictionary is never shared between nodes. As data patterns shift,
> dictionary rotation is also planned.
>
> With Zstd, the best dictionary size seems to
Hello!
In case of Apache Ignite, most of savings is due to BinaryObject format,
which encodes types and fields with byte sequences. Any enum/string flags
will also get in dictionary. And then as it processes a record it fills up
its individual dictionary.
But, in one cache, most if not all entrie
In my view, dictionary of 1024 bytes is not going to be nearly enough.
On Tue, Sep 4, 2018 at 8:06 AM, Ilya Kasnacheev
wrote:
> Hello!
>
> In case of Apache Ignite, most of savings is due to BinaryObject format,
> which encodes types and fields with byte sequences. Any enum/string flags
> will a
Hello!
Of course, this setting will be configurable.
Regards,
--
Ilya Kasnacheev
ср, 5 сент. 2018 г. в 3:21, Dmitriy Setrakyan :
> In my view, dictionary of 1024 bytes is not going to be nearly enough.
>
> On Tue, Sep 4, 2018 at 8:06 AM, Ilya Kasnacheev >
> wrote:
>
> > Hello!
> >
> > In cas
de restart?
Sincerely,
Dmitriy Pavlov
чт, 23 авг. 2018 г. в 13:09, Ilya Kasnacheev :
> Hello!
>
> I want to share with the developer community my compression prototype.
>
> Long story short, it compresses BinaryObject's byte[] as they are written
> to Durable Memory page, op
rt?
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 23 авг. 2018 г. в 13:09, Ilya Kasnacheev :
>
> > Hello!
> >
> > I want to share with the developer community my compression prototype.
> >
> > Long story short, it compresses BinaryObject's byte[] as th
hare with the developer community my compression prototype.
>
> Long story short, it compresses BinaryObject's byte[] as they are written
> to Durable Memory page, operating on a pre-built dictionary. Typical
> compression ratio is 0.4 (meaning 2.5x compression) using custom
> L
lov
> >
> > чт, 23 авг. 2018 г. в 13:09, Ilya Kasnacheev >:
> >
> > > Hello!
> > >
> > > I want to share with the developer community my compression prototype.
> > >
> > > Long story short, it compresses BinaryObject's byte[] as
ve the
> ability to extend compressions algorithms in future
>
> On Thu, Aug 23, 2018 at 1:09 PM, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> wrote:
>
> > Hello!
> >
> > I want to share with the developer community my compression prototype.
> >
>
at instead the boolean type I suggest to use the enum and reserve the
> > ability to extend compressions algorithms in future
> >
> > On Thu, Aug 23, 2018 at 1:09 PM, Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com>
> > wrote:
> >
> > > Hello!
>
to introduce it as an option of Ignite configuration?
> In
> > > that instead the boolean type I suggest to use the enum and reserve the
> > > ability to extend compressions algorithms in future
> > >
> > > On Thu, Aug 23, 2018 at 1:09 PM, Ilya Kasnacheev <
; > >
> > > 2018-08-23 15:13 GMT+03:00 Sergey Kozlov :
> > >
> > > > Hi Ilya
> > > >
> > > > Is there a plan to introduce it as an option of Ignite configuration?
> > In
> > > > that instead the boolean type I s
to data format, of course there will be room for further
> > > > extension.
> > > >
> > > > Regards,
> > > >
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > > 2018-08-23 15:13 GMT+03:00 Sergey Kozlov :
&
between all caches
> vs.
> > > > > having separate dictionary for every cache.
> > > > >
> > > > > With regards to data format, of course there will be room for
> further
> > > > > extension.
> > > > >
> > > > > Re
; > > assumption is that you can always just use latest&greatest, but
> > maybe
> > > > we
> > > > > > can have e.g. very fast and not very strong vs. slower but stronger
> > > > one.
> > > > > >
> > > > > > I'm n
a compression section to cache configuration,
> > > where
> > > > > > you
> > > > > > > can enable compression, enable key compression (which has heavier
> > > > > > > performance implications), adjust dictionary gath
gt; wrote:
> > > > > > >
> > > > > > > > Hello!
> > > > > > > >
> > > > > > > > My plan was to add a compression section to cache
> configuration,
> > > > where
> > > > > > > you
> > >
25 matches
Mail list logo
