Re: error deleting messages
50100 inserts or deletes? also how much ram / cpu do you have on the server running this, and what's the ram / cpu usage at about the time it fails? On Tue, Mar 24, 2015 at 5:29 PM, joss Earl j...@rareformnewmedia.com wrote: on a stock install, it gets to about 50100 before grinding to a halt On 24 March 2015 at 12:19, Ali Akhtar ali.rac...@gmail.com wrote: What happens when you run it? How far does it get before stopping? On Tue, Mar 24, 2015 at 5:13 PM, joss Earl j...@rareformnewmedia.com wrote: sure: https://gist.github.com/joss75321/7d85e4c75c06530e9d80 On 24 March 2015 at 12:04, Ali Akhtar ali.rac...@gmail.com wrote: Can you put your code on gist.github.com or pastebin? On Tue, Mar 24, 2015 at 4:58 PM, joss Earl j...@rareformnewmedia.com wrote: I run into trouble after a while if I delete rows, this happens in both 2.1.3 and 2.0.13, and I encountered the same problem when using either the datastax java driver or the stock python driver. The problem is reproducible using the attached python program. Once the problem is encountered, the table becomes unusable.. cqlsh:test1 select id from msg limit 1; Request did not complete within rpc_timeout. So, questions are: am I doing something wrong ? is this expected behaviour ? is there some way to fix the table and make it usable again once this has happened ? if this is a bug, what is the best way of reporting it ? Many thanks Joss
Re: error deleting messages
Hi Joss We faced similar issue recently. The problem seems to be related to huge number of tombstones generated after deletion. I would suggest you to increase tombstone warning and failure threshold in cassandra.yaml. Once you do that and run your program make sure that you monitor Cassandra Heap usage using nodetool info command. If heap is near full Cassandra halts are obvious. So you need to increase heap. Due to increased tombstones your query is unable to complete within short time..I would suggest increasing read timeout in cassandra.yaml so that query may complete. Please look at your logs to make sure that there are no exception. Thanks Anuj Wadehra From:joss Earl j...@rareformnewmedia.com Date:Tue, 24 Mar, 2015 at 6:17 pm Subject:Re: error deleting messages It inserts 100,000 messages, I then start deleting the messages by grabbing chunks of 100 at a time and then individually deleting each message. So, the 100,000 messages get inserted without any trouble, I run into trouble when I have deleted about half of them. I've run this on machines with 4,8, and 16gig of ram and behaviour was consistent (I fail after 5 or so messages on that table, or maybe 30,000 messages on a table with more columns). On 24 March 2015 at 12:35, Ali Akhtar ali.rac...@gmail.com wrote: 50100 inserts or deletes? also how much ram / cpu do you have on the server running this, and what's the ram / cpu usage at about the time it fails? On Tue, Mar 24, 2015 at 5:29 PM, joss Earl j...@rareformnewmedia.com wrote: on a stock install, it gets to about 50100 before grinding to a halt On 24 March 2015 at 12:19, Ali Akhtar ali.rac...@gmail.com wrote: What happens when you run it? How far does it get before stopping? On Tue, Mar 24, 2015 at 5:13 PM, joss Earl j...@rareformnewmedia.com wrote: sure: https://gist.github.com/joss75321/7d85e4c75c06530e9d80 On 24 March 2015 at 12:04, Ali Akhtar ali.rac...@gmail.com wrote: Can you put your code on gist.github.com or pastebin? On Tue, Mar 24, 2015 at 4:58 PM, joss Earl j...@rareformnewmedia.com wrote: I run into trouble after a while if I delete rows, this happens in both 2.1.3 and 2.0.13, and I encountered the same problem when using either the datastax java driver or the stock python driver. The problem is reproducible using the attached python program. Once the problem is encountered, the table becomes unusable.. cqlsh:test1 select id from msg limit 1; Request did not complete within rpc_timeout. So, questions are: am I doing something wrong ? is this expected behaviour ? is there some way to fix the table and make it usable again once this has happened ? if this is a bug, what is the best way of reporting it ? Many thanks Joss
Re: error deleting messages
Hi Anuj
Yes, thanks.. looking at my log file I see:
ERROR [SharedPool-Worker-2] 2015-03-24 13:52:06,751
SliceQueryFilter.java:218 - Scanned over 10 tombstones in test1.msg;
query aborted (see tombstone_\
failure_threshold)
WARN [SharedPool-Worker-2] 2015-03-24 13:52:06,759
AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2\
,5,main]: {}
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
I'm reading up about how to deal with this now, thanks..
On 24 March 2015 at 13:16, Anuj Wadehra anujw_2...@yahoo.co.in wrote:
Hi Joss
We faced similar issue recently. The problem seems to be related to huge
number of tombstones generated after deletion. I would suggest you to
increase tombstone warning and failure threshold in cassandra.yaml.
Once you do that and run your program make sure that you monitor Cassandra
Heap usage using nodetool info command. If heap is near full Cassandra
halts are obvious. So you need to increase heap.
Due to increased tombstones your query is unable to complete within short
time..I would suggest increasing read timeout in cassandra.yaml so that
query may complete.
Please look at your logs to make sure that there are no exception.
Thanks
Anuj Wadehra
--
*From*:joss Earl j...@rareformnewmedia.com
*Date*:Tue, 24 Mar, 2015 at 6:17 pm
*Subject*:Re: error deleting messages
It inserts 100,000 messages, I then start deleting the messages by
grabbing chunks of 100 at a time and then individually deleting each
message.
So, the 100,000 messages get inserted without any trouble, I run into
trouble when I have deleted about half of them. I've run this on machines
with 4,8, and 16gig of ram and behaviour was consistent (I fail after 5
or so messages on that table, or maybe 30,000 messages on a table with more
columns).
On 24 March 2015 at 12:35, Ali Akhtar ali.rac...@gmail.com wrote:
50100 inserts or deletes? also how much ram / cpu do you have on the
server running this, and what's the ram / cpu usage at about the time it
fails?
On Tue, Mar 24, 2015 at 5:29 PM, joss Earl j...@rareformnewmedia.com
wrote:
on a stock install, it gets to about 50100 before grinding to a halt
On 24 March 2015 at 12:19, Ali Akhtar ali.rac...@gmail.com wrote:
What happens when you run it? How far does it get before stopping?
On Tue, Mar 24, 2015 at 5:13 PM, joss Earl j...@rareformnewmedia.com
wrote:
sure: https://gist.github.com/joss75321/7d85e4c75c06530e9d80
On 24 March 2015 at 12:04, Ali Akhtar ali.rac...@gmail.com wrote:
Can you put your code on gist.github.com or pastebin?
On Tue, Mar 24, 2015 at 4:58 PM, joss Earl j...@rareformnewmedia.com
wrote:
I run into trouble after a while if I delete rows, this happens in
both 2.1.3 and 2.0.13, and I encountered the same problem when using
either
the datastax java driver or the stock python driver.
The problem is reproducible using the attached python program.
Once the problem is encountered, the table becomes unusable..
cqlsh:test1 select id from msg limit 1;
Request did not complete within rpc_timeout.
So, questions are:
am I doing something wrong ?
is this expected behaviour ?
is there some way to fix the table and make it usable again once
this has happened ?
if this is a bug, what is the best way of reporting it ?
Many thanks
Joss
Re: error deleting messages
It inserts 100,000 messages, I then start deleting the messages by grabbing chunks of 100 at a time and then individually deleting each message. So, the 100,000 messages get inserted without any trouble, I run into trouble when I have deleted about half of them. I've run this on machines with 4,8, and 16gig of ram and behaviour was consistent (I fail after 5 or so messages on that table, or maybe 30,000 messages on a table with more columns). On 24 March 2015 at 12:35, Ali Akhtar ali.rac...@gmail.com wrote: 50100 inserts or deletes? also how much ram / cpu do you have on the server running this, and what's the ram / cpu usage at about the time it fails? On Tue, Mar 24, 2015 at 5:29 PM, joss Earl j...@rareformnewmedia.com wrote: on a stock install, it gets to about 50100 before grinding to a halt On 24 March 2015 at 12:19, Ali Akhtar ali.rac...@gmail.com wrote: What happens when you run it? How far does it get before stopping? On Tue, Mar 24, 2015 at 5:13 PM, joss Earl j...@rareformnewmedia.com wrote: sure: https://gist.github.com/joss75321/7d85e4c75c06530e9d80 On 24 March 2015 at 12:04, Ali Akhtar ali.rac...@gmail.com wrote: Can you put your code on gist.github.com or pastebin? On Tue, Mar 24, 2015 at 4:58 PM, joss Earl j...@rareformnewmedia.com wrote: I run into trouble after a while if I delete rows, this happens in both 2.1.3 and 2.0.13, and I encountered the same problem when using either the datastax java driver or the stock python driver. The problem is reproducible using the attached python program. Once the problem is encountered, the table becomes unusable.. cqlsh:test1 select id from msg limit 1; Request did not complete within rpc_timeout. So, questions are: am I doing something wrong ? is this expected behaviour ? is there some way to fix the table and make it usable again once this has happened ? if this is a bug, what is the best way of reporting it ? Many thanks Joss
Re: error deleting messages
This talk from DataStax was talking about deletes as an Anti-pattern. It may
be worth watching.
Thanks for your interest in the following webinar:
Avoiding anti-patterns: How to stay in love with Cassandra
Here are the links to the video recording and presentation slides.
Thanks,
The DataStax Team
On Mar 24, 2015, at 9:09 AM, joss Earl j...@rareformnewmedia.com wrote:
Hi Anuj
Yes, thanks.. looking at my log file I see:
ERROR [SharedPool-Worker-2] 2015-03-24 13:52:06,751 SliceQueryFilter.java:218
- Scanned over 10 tombstones in test1.msg; query aborted (see tombstone_\
failure_threshold)
WARN [SharedPool-Worker-2] 2015-03-24 13:52:06,759
AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2\
,5,main]: {}
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
I'm reading up about how to deal with this now, thanks..
On 24 March 2015 at 13:16, Anuj Wadehra anujw_2...@yahoo.co.in wrote:
Hi Joss
We faced similar issue recently. The problem seems to be related to huge
number of tombstones generated after deletion. I would suggest you to
increase tombstone warning and failure threshold in cassandra.yaml.
Once you do that and run your program make sure that you monitor Cassandra
Heap usage using nodetool info command. If heap is near full Cassandra halts
are obvious. So you need to increase heap.
Due to increased tombstones your query is unable to complete within short
time..I would suggest increasing read timeout in cassandra.yaml so that query
may complete.
Please look at your logs to make sure that there are no exception.
Thanks
Anuj Wadehra
From:joss Earl j...@rareformnewmedia.com
Date:Tue, 24 Mar, 2015 at 6:17 pm
Subject:Re: error deleting messages
It inserts 100,000 messages, I then start deleting the messages by grabbing
chunks of 100 at a time and then individually deleting each message.
So, the 100,000 messages get inserted without any trouble, I run into trouble
when I have deleted about half of them. I've run this on machines with 4,8,
and 16gig of ram and behaviour was consistent (I fail after 5 or so
messages on that table, or maybe 30,000 messages on a table with more
columns).
On 24 March 2015 at 12:35, Ali Akhtar ali.rac...@gmail.com wrote:
50100 inserts or deletes? also how much ram / cpu do you have on the server
running this, and what's the ram / cpu usage at about the time it fails?
On Tue, Mar 24, 2015 at 5:29 PM, joss Earl j...@rareformnewmedia.com wrote:
on a stock install, it gets to about 50100 before grinding to a halt
On 24 March 2015 at 12:19, Ali Akhtar ali.rac...@gmail.com wrote:
What happens when you run it? How far does it get before stopping?
On Tue, Mar 24, 2015 at 5:13 PM, joss Earl j...@rareformnewmedia.com wrote:
sure: https://gist.github.com/joss75321/7d85e4c75c06530e9d80
On 24 March 2015 at 12:04, Ali Akhtar ali.rac...@gmail.com wrote:
Can you put your code on gist.github.com or pastebin?
On Tue, Mar 24, 2015 at 4:58 PM, joss Earl j...@rareformnewmedia.com wrote:
I run into trouble after a while if I delete rows, this happens in both 2.1.3
and 2.0.13, and I encountered the same problem when using either the datastax
java driver or the stock python driver.
The problem is reproducible using the attached python program.
Once the problem is encountered, the table becomes unusable..
cqlsh:test1 select id from msg limit 1;
Request did not complete within rpc_timeout.
So, questions are:
am I doing something wrong ?
is this expected behaviour ?
is there some way to fix the table and make it usable again once this has
happened ?
if this is a bug, what is the best way of reporting it ?
Many thanks
Joss
Re: error deleting messages
Hi Joss, On 24/03/15 12:58, joss Earl wrote: I run into trouble after a while if I delete rows, this happens in both 2.1.3 and 2.0.13, and I encountered the same problem when using either the datastax java driver or the stock python driver. The problem is reproducible using the attached python program. Once the problem is encountered, the table becomes unusable.. cqlsh:test1 select id from msg limit 1; Request did not complete within rpc_timeout. my guess is that due to previous delete attempts your table is filled with tombstones, and this is why the read times out: it hits the tombstone limit. Check your Cassandra node logs to see if they mention tombstones. Ciao, Duncan. So, questions are: am I doing something wrong ? is this expected behaviour ? is there some way to fix the table and make it usable again once this has happened ? if this is a bug, what is the best way of reporting it ? Many thanks Joss
error deleting messages
I run into trouble after a while if I delete rows, this happens in both
2.1.3 and 2.0.13, and I encountered the same problem when using either the
datastax java driver or the stock python driver.
The problem is reproducible using the attached python program.
Once the problem is encountered, the table becomes unusable..
cqlsh:test1 select id from msg limit 1;
Request did not complete within rpc_timeout.
So, questions are:
am I doing something wrong ?
is this expected behaviour ?
is there some way to fix the table and make it usable again once this has
happened ?
if this is a bug, what is the best way of reporting it ?
Many thanks
Joss
from cassandra.cluster import Cluster
cluster = Cluster()
session = cluster.connect()
num_msgs = 10
# session.execute(DROP KEYSPACE test1 IF EXISTS test1;)
session.execute(CREATE KEYSPACE test1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 1};)
session.execute(CREATE TABLE test1.msg (
author text,
id timeuuid,
txttext,
PRIMARY KEY (author,id)
) WITH CLUSTERING ORDER BY (id DESC);)
fetch_st = session.prepare(SELECT id,author,txt FROM test1.msg WHERE author=? limit 100);
insert_st = session.prepare(INSERT INTO test1.msg(id,author,txt) VALUES(now(),?,?));
del_st = session.prepare(DELETE FROM test1.msg where author=? and id=?);
author = some author
for i in range(0,num_msgs):
session.execute(insert_st, [author,some message + str(i)])
msgs = session.execute(fetch_st,[author])
deleted = 0
while len(msgs) 0:
for m in msgs:
session.execute(del_st,[author,m[0]])
deleted += len(msgs)
print deleted + str(deleted)
msgs = session.execute(fetch_st,[author])
