Hi,
I guess page level compression make sense on page loading / eviction.
In this case we can decrease I/O operation and performance boost can be
reached.
What is goal for in-memory compression? Holds about 2-5x data in memory
with performance drop?
Also please clarify the case with compression/decompression for hot and
cold pages.
Is it right for your approach:
1. Hot pages are always decompressed in memory because many read/write
operations touch ones.
2. So we can compress only cold pages.
So the way is suitable when the hot data size << available RAM size.
Thoughts?
On 05.03.2018 20:18, Vyacheslav Daradur wrote:
Hi Igniters!
I’d like to do next step in our data compression discussion [1].
Most Igniters vote for per-data-page compression.
I’d like to accumulate main theses to start implementation:
- page will be compressed with the dictionary-based approach (e.g.LZV)
- page will be compressed in batch mode (not on every change)
- page compression should been initiated by an event, for example, a
page’s free space drops below 20%
- compression process will be under page write lock
Vladimir Ozerov has written:
What we do not understand yet:
1) Granularity of compression algorithm.
1.1) It could be per-entry - i.e. we compress the whole entry content, but
respect boundaries between entries. E.g.: before - [ENTRY_1][ENTRY_2],
after - [COMPRESSED_ENTRY_1][COMPRESSED_ENTRY_2] (as opposed to [COMPRESSED
ENTRY_1 and ENTRY_2]).
v1.2) Or it could be per-field - i.e. we compress fields, but respect binary
object layout. First approach is simple, straightforward, and will give
acceptable compression rate, but we will have to compress the whole binary
object on every field access, what may ruin our SQL performance. Second
approach is more complex, we are not sure about it's compression rate, but
as BinaryObject structure is preserved, we will still have fast
constant-time per-field access.
I think there are advantages in both approaches and we will be able to
compare different approaches and algorithms after prototype
implementation.
Main approach in brief:
1) When page’s free space drops below 20% will be triggered compression event
2) Page will be locked by write lock
3) Page will be passed to page’s compressor implementation
4) Page will be replaced by compressed page
Whole object or a field reading:
1) If page marked as compressed then the page will be handled by
page’s compressor implementation, otherwise, it will be handled as
usual.
Thoughts?
Should we create new IEP and register tickets to start implementation?
This will allow us to watch for the feature progress and related
tasks.
[1]
http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-tc20679.html
--
Taras Ledkov
Mail-To: tled...@gridgain.com
