[
https://issues.apache.org/jira/browse/CASSANDRA-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714182#comment-16714182
]
Joseph Lynch commented on CASSANDRA-14482:
------------------------------------------
{quote}We're in a freeze, but I can't imagine this would break anyone's testing
{quote}
Just want to +1 merging this if we can as zstd will be a significant
performance improvement over deflate and imo should be the default "high
compression" option. Even if we don't want to merge it in 4.0 it could be
shipped first as a standalone jar (there's been
[cassandra-zstd|https://github.com/MatejTymes/cassandra-zstd] for a while now).
I do think that if we merge it for 4.0 having some "before and after" blog
posts of zstd vs deflate for both performance and ratio would be really great
(zstd should beat deflate both in ratio and performance).
[~aweisberg]
{quote}This doesn't seem to use the killer feature of ZSTD which is the
possibility of using a dictionary?
How good is zstd if you don't take advantage of that?
{quote}
Sushma can confirm but in our experiments zstd without dictionaries strictly
dominates deflate as in it is significantly faster and still get's better
compression ratios on most datasets. Zstd is actually on the same order of
magnitude speed wise as LZ4 which is very impressive given the additional ratio
but I guess it makes sense since it's written by the same person. Basically,
even without dictionaries, zstd is a really good option for use cases that need
to get high ratio.
That being said, I've been playing around with integrating a dictionary per
SSTable approach for zstd and it works _really_ well. You can run with like 4kb
block size with a fixed 10kb of dictionary stored after the offsets in the
{{CompressionInfo.db}} file and get similar ratios for many types of data as
LZ4 did with a 64kb block size. Also, because the dictionaries are part of the
{{CompressionInfo.db}} files it's also compatible with most backup/restore and
offline processing systems as well. I haven't posted a patch yet because this
strategy requires non trivial changes to the {{ICompressor}} contract which I
think would violate the freeze and to be frank I haven't been able to get it
fully working in Cassandra itself I believe due to early re-open (it looks like
there is a code path that can re-open the {{SSTableReader}} and in my case it
causes reads to use the wrong dictionary and everything blows up).
All my prototypes have lived outside Cassandra and use dictionaries stored
outside the database, but I think it's safe to say that the "hard" part of
adding dictionary support is going to be how to generate and manage them. I
think the best path forwards will be to hook into training the dictionaries at
SSTable write time, store them in {{CompressionInfo.db}} and then have an
instance of the compressor per sstable as the dictionaries can be held off heap
by the zstd compressor so there would be little overhead. If we went with the
dictionary per sstable approach you'd have a 10kb dictionary per sstable and
2TB of data / 160MB / sstable = 12500 stables * 10kb offheap memory / sstable =
125 megabytes of offheap memory for the dictionaries. This seems pretty worth
it to me since you'll probably be getting another 20-30% ratio.
Anyways, I believe dictionaries are going to be a game changer, but I also
think it should be a follow up item to this ticket. Adding _just_ a zstd
compressor without dictionaries should be a big win, and then for 5.0 we can
work on integrating dictionaries and your in memory offset representation for a
_huge_ win.
> ZSTD Compressor support in Cassandra
> ------------------------------------
>
> Key: CASSANDRA-14482
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14482
> Project: Cassandra
> Issue Type: New Feature
> Components: Compression, Libraries
> Reporter: Sushma A Devendrappa
> Assignee: Sushma A Devendrappa
> Priority: Major
> Labels: performance, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> ZStandard has a great speed and compression ratio tradeoff.
> ZStandard is open source compression from Facebook.
> More about ZSTD
> [https://github.com/facebook/zstd]
> https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
