Given your data model, there’s two ways you may read a tombstone:
You select an expired row, or you scan the whole table.
If you select an expired row, you’re going to scan one tombstone. With
sufficiently high read rate, that’ll look like you’re scanning a lot - each
read will add one to the h
Thanks Jeff. I'm trying to figure out why the tombstones scans are
happening if possible eliminate it.
On Sat, Feb 23, 2019, 10:50 PM Jeff Jirsa wrote:
> G1GC with an 8g heap may be slower than CMS. Also you don’t typically set
> new gen size on G1.
>
> Again though - what problem are you solvin
gt; *From:* Rahul Reddy [mailto:rahulreddy1...@gmail.com]
> *Sent:* Saturday, February 23, 2019 7:26 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Tombstones in memtable
>
>
>
> ```jvm setting
>
>
>
> -XX:+UseThreadPriorities
>
> -XX:ThreadPriorityPolicy=42
>
>
When the CPU utilization spikes from 5-10% to 50%, how many nodes does it
happen to at the same time?
From: Rahul Reddy [mailto:rahulreddy1...@gmail.com]
Sent: Saturday, February 23, 2019 7:26 PM
To: user@cassandra.apache.org
Subject: Re: Tombstones in memtable
```jvm setting
-XX
G1GC with an 8g heap may be slower than CMS. Also you don’t typically set new
gen size on G1.
Again though - what problem are you solving here? If you’re serving reads and
sitting under 50% cpu, it’s not clear to me what you’re trying to fix.
Tombstones scanned won’t matter for your table, so i
```jvm setting
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-XX:+HeapDumpOnOutOfMemoryError
-Xss256k
-XX:StringTableSize=103
-XX:+AlwaysPreTouch
-XX:-UseBiasedLocking
-XX:+UseTLAB
-XX:+ResizeTLAB
-XX:+UseNUMA
-XX:+PerfDisableSharedMem
-Djava.net.preferIPv4Stack=true
-XX:+UseG1GC
-XX:G1
Thanks Jeff,
Since low writes and high reads most of the time data in memtables only.
When I noticed intially issue no stables on disk everything in memtable
only.
On Sat, Feb 23, 2019, 10:01 PM Jeff Jirsa wrote:
> Also given your short ttl and low write rate, you may want to think about
> how
Also given your short ttl and low write rate, you may want to think about how
you can keep more in memory - this may mean larger memtable and high flush
thresholds (reading from the memtable), or perhaps the partition cache (if you
are likely to read the same key multiple times). You’ll also pro
You’ll only ever have one tombstone per read, so your load is based on normal
read rate not tombstones. The metric isn’t wrong, but it’s not indicative of a
problem here given your data model.
You’re using STCS do you may be reading from more than one sstable if you
update column2 for a given
...@gmail.com]
Sent: Saturday, February 23, 2019 5:56 PM
To: user@cassandra.apache.org
Subject: Re: Tombstones in memtable
Changing gcgs didn't help
CREATE KEYSPACE ksname WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '3&
Do you see anything wrong with this metric.
metric to scan tombstones
increase(cassandra_Table_TombstoneScannedHistogram{keyspace="mykeyspace",Table="tablename",function="Count"}[5m])
And sametime CPU Spike to 50% whenever I see high tombstone alert.
On Sat, Feb 23, 2019, 9:25 PM Jeff Jirsa wro
Your schema is such that you’ll never read more than one tombstone per select
(unless you’re also doing range reads / table scans that you didn’t mention) -
I’m not quite sure what you’re alerting on, but you’re not going to have
tombstone problems with that table / that select.
--
Jeff Jirsa
-patterns-queues-and-queue-like-datasets
Kenneth Brotman
From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Saturday, February 23, 2019 4:47 PM
To: user@cassandra.apache.org
Subject: Re: Tombstones in memtable
Would also be good to see your schema (anonymized if needed) and the select
Changing gcgs didn't help
CREATE KEYSPACE ksname WITH replication = {'class':
'NetworkTopologyStrategy', 'dc1': '3', 'dc2': '3'} AND durable_writes =
true;
```CREATE TABLE keyspace."table" (
"column1" text PRIMARY KEY,
"column2" text
) WITH bloom_filter_fp_chance = 0.01
AND caching
Would also be good to see your schema (anonymized if needed) and the select
queries you’re running
--
Jeff Jirsa
> On Feb 23, 2019, at 4:37 PM, Rahul Reddy wrote:
>
> Thanks Jeff,
>
> I'm having gcgs set to 10 mins and changed the table ttl also to 5 hours
> compared to insert ttl to 4 h
I’m not parsing this - did the lower gcgs help or not ? Seeing the table
histograms is the next step if this is still a problem
The table level TTL doesn’t matter if you set a TTL on each insert
--
Jeff Jirsa
> On Feb 23, 2019, at 4:37 PM, Rahul Reddy wrote:
>
> Thanks Jeff,
>
> I'm ha
Thanks Jeff,
I'm having gcgs set to 10 mins and changed the table ttl also to 5 hours
compared to insert ttl to 4 hours . Tracing on doesn't show any tombstone
scans for the reads. And also log doesn't show tombstone scan alerts. Has
the reads are happening 5-8k reads per node during the peak h
If all of your data is TTL’d and you never explicitly delete a cell without
using s TTL, you can probably drop your GCGS to 1 hour (or less).
Which compaction strategy are you using? You need a way to clear out those
tombstones. There exist tombstone compaction sub properties that can help
enco
Can we see the histogram? Why wouldn’t you at times have that many tombstones?
Makes sense.
Kenneth Brotman
From: Rahul Reddy [mailto:rahulreddy1...@gmail.com]
Sent: Thursday, February 21, 2019 7:06 AM
To: user@cassandra.apache.org
Subject: Tombstones in memtable
We have small table
19 matches
Mail list logo
