Hints replay incompatible between 2.x and 3.x
In evaluating 3.x, we found that hints are unable to be replayed between 2.x and 3.x nodes. This introduces a risk during the upgrade path for some of our write-heavy clusters - nodes will accumulate upwards of 1TB of hints if a node goes/remains down for <1hr. Any suggestions to mitigate this issue?
Re: CASSANDRA-9472 Reintroduce off heap memtables - patch to 3.0
Hi Jay, Here's the backport to 3.0.14 - https://github.com/whangsf/cassandra/commit/8db2e3ed412e42fed1da2d85ee7d086edcc8ae4c. This should pass all unit tests, but please let me know if you have any issues. Thanks, Andrew On Mon, Aug 28, 2017 at 7:35 PM, Jay Zhuang wrote: > Hi Andrew, > > Do you mind sharing the backport patch? We're very interested in that, > 20-30% improvement sounds great to us. > > Thanks, > Jay > > On 7/27/17 11:52 PM, Andrew Whang wrote: > > Yes, seeing latency improvement after backporting 9472 to 3.0.13. We are > > measuring p99 latency, thus moving objects off heap improved gc stalls, > > which directly affects our read/write p99 latency. > > > > On Thu, Jul 27, 2017 at 10:54 PM, Jeff Jirsa wrote: > > > >> This is after you backported 9472 to 3.0? > >> > >> -- > >> Jeff Jirsa > >> > >> > >>> On Jul 27, 2017, at 10:33 PM, Andrew Whang > >> wrote: > >>> > >>> Jay, > >>> > >>> We see ~20% write latency improvement on 3.0.13 in a write-heavy > >> workload, > >>> using offheap_objects. offheap_buffers only offered minimal > improvement. > >>> > >>> On Thu, Jul 27, 2017 at 10:06 PM, Jay Zhuang > >> > >>> wrote: > >>> > >>>> Hi Andrew, > >>>> > >>>> Do you see performance gain from reintroducing off-heap memtables for > >>>> 3.0.x? When we were on 2.2.x we saw big improvements from enabling > >>>> off-heap memtables. > >>>> > >>>> Thanks, > >>>> Jay > >>>> > >>>>> On 7/27/17 9:37 PM, Andrew Whang wrote: > >>>>> I'm wondering if anyone has been able to patch CASSANDRA-9472 to 3.0, > >>>>> without breaking unit tests. The patch was introduced in 3.4, but > 3.0.x > >>>>> contains unit tests and code from later 3.x releases, which makes > >>>> debugging > >>>>> unit test failures difficult - i.e. SSTableCorruptionDetectionTest, > >>>> which > >>>>> was introduced in 3.7 and is found in 3.0.14, but not in 3.4. > >>>>> > >>>> > >>>> - > >>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >>>> For additional commands, e-mail: dev-h...@cassandra.apache.org > >>>> > >>>> > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > >> > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: CASSANDRA-9472 Reintroduce off heap memtables - patch to 3.0
Yes, seeing latency improvement after backporting 9472 to 3.0.13. We are measuring p99 latency, thus moving objects off heap improved gc stalls, which directly affects our read/write p99 latency. On Thu, Jul 27, 2017 at 10:54 PM, Jeff Jirsa wrote: > This is after you backported 9472 to 3.0? > > -- > Jeff Jirsa > > > > On Jul 27, 2017, at 10:33 PM, Andrew Whang > wrote: > > > > Jay, > > > > We see ~20% write latency improvement on 3.0.13 in a write-heavy > workload, > > using offheap_objects. offheap_buffers only offered minimal improvement. > > > > On Thu, Jul 27, 2017 at 10:06 PM, Jay Zhuang > > > wrote: > > > >> Hi Andrew, > >> > >> Do you see performance gain from reintroducing off-heap memtables for > >> 3.0.x? When we were on 2.2.x we saw big improvements from enabling > >> off-heap memtables. > >> > >> Thanks, > >> Jay > >> > >>> On 7/27/17 9:37 PM, Andrew Whang wrote: > >>> I'm wondering if anyone has been able to patch CASSANDRA-9472 to 3.0, > >>> without breaking unit tests. The patch was introduced in 3.4, but 3.0.x > >>> contains unit tests and code from later 3.x releases, which makes > >> debugging > >>> unit test failures difficult - i.e. SSTableCorruptionDetectionTest, > >> which > >>> was introduced in 3.7 and is found in 3.0.14, but not in 3.4. > >>> > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: CASSANDRA-9472 Reintroduce off heap memtables - patch to 3.0
Jay, We see ~20% write latency improvement on 3.0.13 in a write-heavy workload, using offheap_objects. offheap_buffers only offered minimal improvement. On Thu, Jul 27, 2017 at 10:06 PM, Jay Zhuang wrote: > Hi Andrew, > > Do you see performance gain from reintroducing off-heap memtables for > 3.0.x? When we were on 2.2.x we saw big improvements from enabling > off-heap memtables. > > Thanks, > Jay > > On 7/27/17 9:37 PM, Andrew Whang wrote: > > I'm wondering if anyone has been able to patch CASSANDRA-9472 to 3.0, > > without breaking unit tests. The patch was introduced in 3.4, but 3.0.x > > contains unit tests and code from later 3.x releases, which makes > debugging > > unit test failures difficult - i.e. SSTableCorruptionDetectionTest, > which > > was introduced in 3.7 and is found in 3.0.14, but not in 3.4. > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
CASSANDRA-9472 Reintroduce off heap memtables - patch to 3.0
I'm wondering if anyone has been able to patch CASSANDRA-9472 to 3.0, without breaking unit tests. The patch was introduced in 3.4, but 3.0.x contains unit tests and code from later 3.x releases, which makes debugging unit test failures difficult - i.e. SSTableCorruptionDetectionTest, which was introduced in 3.7 and is found in 3.0.14, but not in 3.4.
Perf regression between 2.2.5 and 3.11
Hi, I'm seeing perf regressions (using cassandra-stress) between 2.2.5 and 3.11. I understand these versions are quite far apart, but just wondering if there are stress results publicly available that compare 2.x to 3.x? Thanks.
