Re: no change observed in read latency after switching from EBS to SSD storage

[Benedict Elliott Smith]

It is possible this is CPU bound. In 2.1 we have optimised the comparison
of clustering columns (CASSANDRA-5417
), but in 2.0 it is
quite expensive. So for a large row with several million comparisons to
perform (to merge, filter, etc.) it could be a significant proportion of
the cost. Note that these costs for a given query are all bound by a single
core, there is no parallelism, since the assumption is we are serving more
queries at once than there are cores (in general Cassandra is not designed
to serve workloads consisting of single large queries, at least not yet)

On Thu, Sep 18, 2014 at 7:29 AM, Mohammed Guller 
wrote:

>  Chris,
>
> I agree that reading 250k row is a bit excessive and that breaking up the
> partition would help reduce the query time. That part is well understood.
> The part that we can’t figure out is why read time did not change when we
> switched from a slow Network Attached Storage (AWS EBS) to local SSD.
>
>
>
> One possibility is that the read is not bound by disk i/o, but it is not
> cpu or memory bound either. So where is it spending all that time? Another
> possibility is that even though it is returning only 193311 cells, C* reads
> the entire partition, which may have a lot more cells. But even in that
> case reading from a local SSD should have been a lot faster than reading
> from non-provisioned EBS.
>
>
>
> Mohammed
>
>
>
> *From:* Chris Lohfink [mailto:clohf...@blackbirdit.com]
> *Sent:* Wednesday, September 17, 2014 7:17 PM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: no change observed in read latency after switching from
> EBS to SSD storage
>
>
>
> "Read 193311 live and 0 tombstoned cells "
>
>
>
> is your killer.  returning 250k rows is a bit excessive, you should really
> page this in smaller chunks, what client are you using to access the data?
>  This partition (a, b, c, d, e, f) may be too large as well (can check
> partition max size from output of *nodetool cfstats*), may be worth
> including g to break it up more - but I dont know enough about your data
> model.
>
>
>
> ---
>
> Chris Lohfink
>
>
>
> On Sep 17, 2014, at 4:53 PM, Mohammed Guller 
> wrote:
>
>
>
>   Thank you all for your responses.
>
>
>
> Alex –
>
>   Instance (ephemeral) SSD
>
>
>
> Ben –
>
> the query reads data from just one partition. If disk i/o is the
> bottleneck, then in theory, if reading from EBS takes 10 seconds, then it
> should take lot less when reading the same amount of data from local SSD.
> My question is not about why it is taking 10 seconds, but why is the read
> time same for both EBS (network attached storage) and local SSD?
>
>
>
> Tony –
>
> if the data was cached in memory, then a read should not take 10 seconds
> just for 20MB data
>
>
>
> Rob –
>
> Here is the schema, query, and trace. I masked the actual column names to
> protect the innocents J
>
>
>
> create table dummy(
>
>   a   varchar,
>
>   b   varchar,
>
>   c   varchar,
>
>   d   varchar,
>
>   e   varchar,
>
>   f   varchar,
>
>   g   varchar,
>
>   h   timestamp,
>
>   i   int,
>
>   non_key1   varchar,
>
>   ...
>
>   non_keyN   varchar,
>
>   PRIMARY KEY ((a, b, c, d, e, f), g, h, i)
>
> ) WITH CLUSTERING ORDER BY (g ASC, h DESC, i ASC)
>
>
>
> SELECT h, non_key100, non_key200 FROM dummy WHERE a='' AND b='bb'
> AND c='ccc' AND d='dd' AND e='' AND f='ff' AND
> g='g'AND h >='2014-09-10T00:00:00' AND h<='2014-09-10T23:40:41';
>
>
>
> The above query returns around 250,000 CQL rows.
>
>
>
> cqlsh trace:
>
>
>
> activity | timestamp| source  | source_elapsed
>
>
> -
>
> execute_cql3_query | 21:57:16,830 | 10.10.100.5 |  0
>
> Parsing query; | 21:57:16,830 | 10.10.100.5 |673
>
> Preparing statement | 21:57:16,831 | 10.10.100.5 |   1602
>
> Executing single-partition query on event | 21:57:16,845 | 10.10.100.5
> |  14871
>
> Acquiring sstable references | 21:57:16,845 | 10.10.100.5 |  14896
>
> Merging memtable tombstones | 21:57:16,845 | 10.10.100.5 |  14954
>
> Bloom filter allows skipping sstable 1049 | 21:57:16,845 | 10.10.100.5
> |  15090
>
> Bloom filter allows skipping sstable 989 | 21:57:16,845 | 10.10.100.5
> |  15146
>
> Partition index with 0 entries found for sstable 937 | 21:57:16,845 |
> 10.10.100.5 |  15565
>
> Seeking to partition indexed section in data file | 21:57:16,845 |
> 10.10.100.5 |  15581
>
> Partition index with 7158 entries found for sstable 884 | 21:57:16,898 |
> 10.10.100.5 |  68644
>
> Seeking to partition indexed section in data file | 21:57:16,899 |
> 10.10.100.5 |  69014
>
> Partition index with 20819 entries found for sstable 733 | 21:57:16,916 |
> 10.10.100.5 |  86121
>
> Seeking to partition indexed section in data file | 21:57:16,916 |
> 10.10.100.5 |  86412
>
> Sk

Re: no change observed in read latency after switching from EBS to SSD storage

[Tony Anecito]

Thanks for the detail.
1. So how do we know from the trace if the read was from memory or I/O? You 
might want to use a java profiler like VisualVM and a plugin for it to show 
I/O. But you will have to be a Cassandra developer to understand the output and 
tune your filters for the tool.

2.  Also, I have never seen such a big primary key (and mostly varchars) before 
but maybe that is common for the data you are using? I usually use keys that 
have int but then I use a STAR schema with combo keys where at most 2 type int 
for the key. My queries take less than 100 microsecnds and that is two queries 
one to get a list of keys and other to take that list for the second query. I 
also changed the default key to an int rather than a varchar that Cassandra 
uses. Also I may return only 40-100 records.
3. I have done performance testing where Oracle and SQL Server and I rarely see 
this long of a response time except maybe for ad-hoc queries against a main 
frame.

Again a different set or 
requirements and data that I have and I have not even optimized the design and 
architecture yet but hope for sub-microsecond overall time.  I  am very pleased 
with Cassandra for what I am doing.


Good Luck,
-Tony



On Thursday, September 18, 2014 3:15 AM, Benedict Elliott Smith 
 wrote:
 


It is possible this is CPU bound. In 2.1 we have optimised the comparison of 
clustering columns (CASSANDRA-5417), but in 2.0 it is quite expensive. So for a 
large row with several million comparisons to perform (to merge, filter, etc.) 
it could be a significant proportion of the cost. Note that these costs for a 
given query are all bound by a single core, there is no parallelism, since the 
assumption is we are serving more queries at once than there are cores (in 
general Cassandra is not designed to serve workloads consisting of single large 
queries, at least not yet)


On Thu, Sep 18, 2014 at 7:29 AM, Mohammed Guller  wrote:

Chris,
>I agree that reading 250k row is a bit excessive and that breaking up the 
>partition would help reduce the query time. That part is well understood. The 
>part that we can’t figure out is why read time did not change when we switched 
>from a slow Network Attached Storage (AWS EBS) to local SSD.
> 
>One possibility is that the read is not bound by disk i/o, but it is not cpu 
>or memory bound either. So where is it spending all that time? Another 
>possibility is that even though it is returning only 193311 cells, C* reads 
>the entire partition, which may have a lot more cells. But even in that case 
>reading from a local SSD should have been a lot faster than reading from 
>non-provisioned EBS.
> 
>Mohammed
> 
>From:Chris Lohfink [mailto:clohf...@blackbirdit.com] 
>Sent: Wednesday, September 17, 2014 7:17 PM
>
>To: user@cassandra.apache.org
>Subject: Re: no change observed in read latency after switching from EBS to 
>SSD storage
> 
>"Read 193311 live and 0 tombstoned cells " 
> 
>is your killer.  returning 250k rows is a bit excessive, you should really 
>page this in smaller chunks, what client are you using to access the data?  
>This partition (a, b, c, d, e, f) may be too large as well (can check 
>partition max size from output of nodetool cfstats), may be worth including g 
>to break it up more - but I dont know enough about your data model.
> 
>---
>Chris Lohfink
> 
>On Sep 17, 2014, at 4:53 PM, Mohammed Guller  wrote:
>
>
>
>Thank you all for your responses.
> 
>Alex –
>  Instance (ephemeral) SSD
> 
>Ben –
>the query reads data from just one partition. If disk i/o is the bottleneck, 
>then in theory, if reading from EBS takes 10 seconds, then it should take lot 
>less when reading the same amount of data from local SSD. My question is not 
>about why it is taking 10 seconds, but why is the read time same for both EBS 
>(network attached storage) and local SSD?
> 
>Tony –
>if the data was cached in memory, then a read should not take 10 seconds just 
>for 20MB data
> 
>Rob –
>Here is the schema, query, and trace. I masked the actual column names to 
>protect the innocents J
> 
>create table dummy(
>  a   varchar,
>  b   varchar,
>  c   varchar,
>  d   varchar,
>  e   varchar,
>  f   varchar,
>  g   varchar,
>  h   timestamp,
>  i   int,
>  non_key1   varchar,
>  ...
>  non_keyN   varchar,
>  PRIMARY KEY ((a, b, c, d, e, f), g, h, i)
>) WITH CLUSTERING ORDER BY (g ASC, h DESC, i ASC)
> 
>SELECT h, non_key100, non_key200 FROM dummy WHERE a='' AND b='bb' AND 
>c='ccc' AND d='dd' AND e='' AND f='ff' AND 
>g='g'AND h >='2014-09-10T00:00:00' AND h<='2014-09-10T23:40:41';
> 
>The above query returns around 250,000 CQL rows.
> 
>cqlsh trace:
> 
>activity | timestamp| source  | source_elapsed
>-
>execute_cql3_query | 21:57:16,830 | 10.10.100.5 |  0
>Parsing query; | 21:57:16,830 | 10.10.100.5 |673
>Preparing statement | 21:57:16,831 | 10.10.100

Re: cqlsh 2.1 to cassandra 2.0.7

[Adam Holmberg]

There is now a ticket open to look into this and produce a more informative
error message:
https://datastax-oss.atlassian.net/browse/PYTHON-157

Adam

On Wed, Sep 17, 2014 at 4:47 PM, Adam Holmberg 
wrote:

> This is not really supported. Presently cqlsh hard-codes CQL and protocol
> to versions only supported in 2.1:
>
> https://github.com/apache/cassandra/blob/cassandra-2.1.0/bin/cqlsh#L144-L147
>
> Your best bet is probably downloading a 2.0.x tarball and running out of
> there.
>
> Adam Holmberg
>
> On Wed, Sep 17, 2014 at 4:33 PM, Tyler Tolley 
> wrote:
>
>> Just upgraded cassandra to 2.1 on my own box and when I try to connect to
>> our server running 2.0.7, I get the following error.
>>
>> Connection error: ('Unable to connect to any servers', {'10.0.16.144':
>> ConnectionShutdown('Connection > 10.0.16.144:9042 (closed)> is already closed',)})
>>
>> With debug on I get the following output.
>>
>> Using CQL driver: > '/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.0.post.zip/cassandra-driver-2.1.0.post/cassandra/__init__.py'>
>> Connection error: ('Unable to connect to any servers', {'10.0.16.144':
>> ConnectionShutdown('Connection > 10.0.16.144:9042 (closed)> is already closed',)})
>>
>> If I try to connect using an earlier version of cqlsh, it connects fine.
>> Anyone seen this before or have any advice on how to dig into the problem
>> further? Thanks,
>>  -Tyler
>>
>>
>

Re: cqlsh 2.1 to cassandra 2.0.7

[groupie02]

did you try running cqlsh with --cqlversion?

On Thu, Sep 18, 2014 at 9:17 AM, Adam Holmberg 
wrote:

> There is now a ticket open to look into this and produce a more
> informative error message:
> https://datastax-oss.atlassian.net/browse/PYTHON-157
>
> Adam
>
>
> On Wed, Sep 17, 2014 at 4:47 PM, Adam Holmberg  > wrote:
>
>> This is not really supported. Presently cqlsh hard-codes CQL and protocol
>> to versions only supported in 2.1:
>>
>> https://github.com/apache/cassandra/blob/cassandra-2.1.0/bin/cqlsh#L144-L147
>>
>> Your best bet is probably downloading a 2.0.x tarball and running out of
>> there.
>>
>> Adam Holmberg
>>
>> On Wed, Sep 17, 2014 at 4:33 PM, Tyler Tolley 
>> wrote:
>>
>>> Just upgraded cassandra to 2.1 on my own box and when I try to connect
>>> to our server running 2.0.7, I get the following error.
>>>
>>> Connection error: ('Unable to connect to any servers', {'10.0.16.144':
>>> ConnectionShutdown('Connection >> 10.0.16.144:9042 (closed)> is already closed',)})
>>>
>>> With debug on I get the following output.
>>>
>>> Using CQL driver: >> '/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.0.post.zip/cassandra-driver-2.1.0.post/cassandra/__init__.py'>
>>> Connection error: ('Unable to connect to any servers', {'10.0.16.144':
>>> ConnectionShutdown('Connection >> 10.0.16.144:9042 (closed)> is already closed',)})
>>>
>>> If I try to connect using an earlier version of cqlsh, it connects fine.
>>> Anyone seen this before or have any advice on how to dig into the problem
>>> further? Thanks,
>>>  -Tyler
>>>
>>>
>>
>

index partition key

[Marcelo Elias Del Valle]

Hi,

This question is just for curiosity purposes, I don't need this in my
solution, but it's something I was asking myself...

Is there a way of indexing the partition key values in Cassandra? Does
anyone needed this and found a solution of any kind?

For example, I know the sample bellow doesn't work, but would it be
possible somehow?
I know select * from testcf where token(key1) > token('Lucas'); works, but
the question is ordering by the values, even using another CF or index of
any kind...

CREATE KEYSPACE IF NOT EXISTS testks
  WITH REPLICATION = { 'class' : 'SimpleStrategy',
  'replication_factor': '1' };


CREATE TABLE IF NOT EXISTS testcf (
  key1 varchar,
  value varchar,
  PRIMARY KEY ((key1)));

CREATE INDEX testidx on testks.testcf (key1);

insert into testcf (key1, value) values ('Lucas', 'value1');
insert into testcf (key1, value) values ('Luiz', 'value2');
insert into testcf (key1, value) values ('Edu', 'value3');
insert into testcf (key1, value) values ('Marcelo', 'value4');

select * from testcf order by key1;
select * from testcf where key1 > 'Lucas';


Best regards,
Marcelo.

New Debian/Ubuntu Repository GPG Signer

[Michael Shuler]

The Apache Cassandra apt repository welcomes "T Jake Luciani 
" as a new package signer.


This means that apt repository users will need to import a new GPG key 
(the wiki page is being updated).


$ gpg --keyserver pgp.mit.edu --recv-keys 749D6EEC0353B12C
$ gpg --export --armor 749D6EEC0353B12C | sudo apt-key add -

For context, I'm attaching the output you might see from `apt-get update`.

Kind regards,
Michael
mshuler@hana:~$ sudo apt-get update
<...>
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not 
updated and the previous index files will be used. GPG error: 
http://www.apache.org 21x InRelease: The following signatures couldn't be 
verified because the public key is not available: NO_PUBKEY 749D6EEC0353B12C

W: Failed to fetch 
http://www.apache.org/dist/cassandra/debian/dists/21x/InRelease  

W: Some index files failed to download. They have been ignored, or old ones 
used instead.
mshuler@hana:~$ apt-cache policy cassandra
cassandra:
  Installed: (none)
  Candidate: 2.1.0
  Version table:
 2.1.0 0
500 http://www.apache.org/dist/cassandra/debian/ 21x/main amd64 Packages

mshuler@hana:~$ gpg --keyserver pgp.mit.edu --recv-keys 749D6EEC0353B12C
gpg: requesting key 0353B12C from hkp server pgp.mit.edu
gpg: key 0353B12C: public key "T Jake Luciani " imported
gpg: Total number processed: 1
gpg:   imported: 1  (RSA: 1)
mshuler@hana:~$ gpg --export --armor 749D6EEC0353B12C | sudo apt-key add -  
  
OK

mshuler@hana:~$ sudo apt-get update
<...>
Reading package lists... Done
mshuler@hana:~$ apt-cache policy cassandra
cassandra:
  Installed: (none)
  Candidate: 2.1.0-2
  Version table:
 2.1.0-2 0
500 http://www.apache.org/dist/cassandra/debian/ 21x/main amd64 Packages

Re: cqlsh 2.1 to cassandra 2.0.7

[Tyler Tolley]

I did and got the same error.

On Thu, Sep 18, 2014 at 9:10 AM, groupie02  wrote:

> did you try running cqlsh with --cqlversion?
>
> On Thu, Sep 18, 2014 at 9:17 AM, Adam Holmberg  > wrote:
>
>> There is now a ticket open to look into this and produce a more
>> informative error message:
>> https://datastax-oss.atlassian.net/browse/PYTHON-157
>>
>> Adam
>>
>>
>> On Wed, Sep 17, 2014 at 4:47 PM, Adam Holmberg <
>> adam.holmb...@datastax.com> wrote:
>>
>>> This is not really supported. Presently cqlsh hard-codes CQL and
>>> protocol to versions only supported in 2.1:
>>>
>>> https://github.com/apache/cassandra/blob/cassandra-2.1.0/bin/cqlsh#L144-L147
>>>
>>> Your best bet is probably downloading a 2.0.x tarball and running out of
>>> there.
>>>
>>> Adam Holmberg
>>>
>>> On Wed, Sep 17, 2014 at 4:33 PM, Tyler Tolley 
>>> wrote:
>>>
 Just upgraded cassandra to 2.1 on my own box and when I try to connect
 to our server running 2.0.7, I get the following error.

 Connection error: ('Unable to connect to any servers', {'10.0.16.144':
 ConnectionShutdown('Connection >>> 10.0.16.144:9042 (closed)> is already closed',)})

 With debug on I get the following output.

 Using CQL driver: >>> '/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.0.post.zip/cassandra-driver-2.1.0.post/cassandra/__init__.py'>
 Connection error: ('Unable to connect to any servers', {'10.0.16.144':
 ConnectionShutdown('Connection >>> 10.0.16.144:9042 (closed)> is already closed',)})

 If I try to connect using an earlier version of cqlsh, it connects
 fine. Anyone seen this before or have any advice on how to dig into the
 problem further? Thanks,
  -Tyler


>>>
>>
>

purge gossip info

[S C]

I recently added nodes to existing cluster and removed some. "nodetool 
gossipinfo" doesn't show the non existing nodes but a thread dump on cassandra 
reveals it is trying to write to the non existing old node. I tried restarting 
the cluster using "-Dcassandra.load_ring_state=false" on each node but has no 
effect to the behavior. Is there a way to purge this through CQL?

Thanks,SC 

Re: index partition key

[DuyHai Doan]

Your query "select * from testcf where key1 > 'Lucas';" does work if you
choose the old OrderPartioner but since it's considered absolute evil (for
good reason, load distribution is horrendous), pratically it is not
possible to do your query.

And it's the same thing with secondary index. Query with inequality on
secondary index translates into full cluster scan...

Regards

 Duy Hai DOAN

On Thu, Sep 18, 2014 at 5:44 PM, Marcelo Elias Del Valle <
marc...@s1mbi0se.com.br> wrote:

> Hi,
>
> This question is just for curiosity purposes, I don't need this in my
> solution, but it's something I was asking myself...
>
> Is there a way of indexing the partition key values in Cassandra? Does
> anyone needed this and found a solution of any kind?
>
> For example, I know the sample bellow doesn't work, but would it be
> possible somehow?
> I know select * from testcf where token(key1) > token('Lucas'); works, but
> the question is ordering by the values, even using another CF or index of
> any kind...
>
> CREATE KEYSPACE IF NOT EXISTS testks
>   WITH REPLICATION = { 'class' : 'SimpleStrategy',
>   'replication_factor': '1' };
>
>
> CREATE TABLE IF NOT EXISTS testcf (
>   key1 varchar,
>   value varchar,
>   PRIMARY KEY ((key1)));
>
> CREATE INDEX testidx on testks.testcf (key1);
>
> insert into testcf (key1, value) values ('Lucas', 'value1');
> insert into testcf (key1, value) values ('Luiz', 'value2');
> insert into testcf (key1, value) values ('Edu', 'value3');
> insert into testcf (key1, value) values ('Marcelo', 'value4');
>
> select * from testcf order by key1;
> select * from testcf where key1 > 'Lucas';
>
>
> Best regards,
> Marcelo.
>

Re: How to get rid of stale info in gossip

[DuyHai Doan]

unsafeAssassinate remove gossip info and token range ownership for a node
WITHOUT streaming data.

On Tue, Aug 5, 2014 at 2:56 AM, Rahul Neelakantan  wrote:

> It no longer shows up, but I will definitely investigate that option. What
> exactly does it do?
>
> Rahul Neelakantan
>
> On Aug 4, 2014, at 7:49 PM, Patricia Gorla 
> wrote:
>
> Rahul,
>
> If the removed node is still showing up in gossip state, you can remove it
> with 'unsafeAssassinate' via a JMX tool such as jconsole or jmxterm.
>
>
> On Fri, Jul 25, 2014 at 6:10 AM, Rahul Neelakantan  wrote:
>
>> Yes, and this is a really old version of casandra 1.0.8.
>>
>>
>> Rahul Neelakantan
>> 678-451-4545
>>
>> On Jul 25, 2014, at 7:29 AM, Mark Reddy  wrote:
>>
>> After removing a node, it's information can persist in the Gossiper for
>> up to 3 days, after which time it should be removed.
>>
>> Are you having issues with a removed node state persisting for longer?
>>
>>
>> Mark
>>
>>
>> On Fri, Jul 25, 2014 at 11:33 AM, Rahul Neelakantan 
>> wrote:
>>
>>> Is there a way to get rid of stale information that shows up for
>>> removed/dead nodes in gossip, without a complete cluster bounce?
>>>
>>> Rahul Neelakantan
>>>
>>>
>>
>
>
> --
> Patricia Gorla
> @patriciagorla
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com 
>
>

[RELEASE] Apache Cassandra 1.2.19 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 1.2.19.

Cassandra is a highly scalable second-generation distributed database,
bringing together Dynamo's fully distributed design and Bigtable's
ColumnFamily-based data model. You can read more here:

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a maintenance/bug fix release[1] on the 1.2 series. As
always,
please pay attention to the release notes[2] and Let us know[3] if you were
to
encounter any problem. This will likely be the final release in the 1.2
series.

Enjoy!

[1]: http://goo.gl/F6szqv (CHANGES.txt)
[2]: http://goo.gl/9VsZ88 (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: index partition key

[Marcelo Elias Del Valle]

Yes, indeed using the evil ordered partitioner it would work, but it should
be avoided at all costs.
What I am trying to figure is: Can I tell Cassandra is not good for cases
where you need range queries over all the cluster?
It's just a kind of "architectural rule" I am trying to use to decide
whether when to use a kind of technology or another.
Solr, for instance, allows you to query the way you want over all the
cluster, so data stax enterprise could "solve" this problem, but I wonder
if it's the only way.

[]s

2014-09-18 13:35 GMT-03:00 DuyHai Doan :

> Your query "select * from testcf where key1 > 'Lucas';" does work if you
> choose the old OrderPartioner but since it's considered absolute evil (for
> good reason, load distribution is horrendous), pratically it is not
> possible to do your query.
>
> And it's the same thing with secondary index. Query with inequality on
> secondary index translates into full cluster scan...
>
> Regards
>
>  Duy Hai DOAN
>
> On Thu, Sep 18, 2014 at 5:44 PM, Marcelo Elias Del Valle <
> marc...@s1mbi0se.com.br> wrote:
>
>> Hi,
>>
>> This question is just for curiosity purposes, I don't need this in my
>> solution, but it's something I was asking myself...
>>
>> Is there a way of indexing the partition key values in Cassandra? Does
>> anyone needed this and found a solution of any kind?
>>
>> For example, I know the sample bellow doesn't work, but would it be
>> possible somehow?
>> I know select * from testcf where token(key1) > token('Lucas'); works,
>> but the question is ordering by the values, even using another CF or index
>> of any kind...
>>
>> CREATE KEYSPACE IF NOT EXISTS testks
>>   WITH REPLICATION = { 'class' : 'SimpleStrategy',
>>   'replication_factor': '1' };
>>
>>
>> CREATE TABLE IF NOT EXISTS testcf (
>>   key1 varchar,
>>   value varchar,
>>   PRIMARY KEY ((key1)));
>>
>> CREATE INDEX testidx on testks.testcf (key1);
>>
>> insert into testcf (key1, value) values ('Lucas', 'value1');
>> insert into testcf (key1, value) values ('Luiz', 'value2');
>> insert into testcf (key1, value) values ('Edu', 'value3');
>> insert into testcf (key1, value) values ('Marcelo', 'value4');
>>
>> select * from testcf order by key1;
>> select * from testcf where key1 > 'Lucas';
>>
>>
>> Best regards,
>> Marcelo.
>>
>
>

RE: no change observed in read latency after switching from EBS to SSD storage

[Mohammed Guller]

Benedict,
That makes perfect sense. Even though the node has multiple cores, I do see 
that only one core is pegged at 100%.

Interestingly, after I switched to 2.1, cqlsh trace now shows that the same 
query takes only 600ms. However, cqlsh still waits for almost 20-30 seconds 
before it starts showing the result. I noticed similar latency when I ran the 
query from our app, which uses the Astyanax driver. So I thought perhaps there 
is a bug in the cqlsh code that tracks the statistics and the reported numbers 
are incorrect. But, I guess the numbers shown by cqlsh trace is correct, but 
the bottleneck is somewhere else now. In other words, the read operation itself 
is much faster in 2.1, but something else delays the response back to the 
client.

Mohammed

From: Benedict Elliott Smith [mailto:belliottsm...@datastax.com]
Sent: Thursday, September 18, 2014 2:15 AM
To: user@cassandra.apache.org
Cc: Chris Lohfink
Subject: Re: no change observed in read latency after switching from EBS to SSD 
storage

It is possible this is CPU bound. In 2.1 we have optimised the comparison of 
clustering columns 
(CASSANDRA-5417), but in 
2.0 it is quite expensive. So for a large row with several million comparisons 
to perform (to merge, filter, etc.) it could be a significant proportion of the 
cost. Note that these costs for a given query are all bound by a single core, 
there is no parallelism, since the assumption is we are serving more queries at 
once than there are cores (in general Cassandra is not designed to serve 
workloads consisting of single large queries, at least not yet)

On Thu, Sep 18, 2014 at 7:29 AM, Mohammed Guller 
mailto:moham...@glassbeam.com>> wrote:
Chris,
I agree that reading 250k row is a bit excessive and that breaking up the 
partition would help reduce the query time. That part is well understood. The 
part that we can’t figure out is why read time did not change when we switched 
from a slow Network Attached Storage (AWS EBS) to local SSD.

One possibility is that the read is not bound by disk i/o, but it is not cpu or 
memory bound either. So where is it spending all that time? Another possibility 
is that even though it is returning only 193311 cells, C* reads the entire 
partition, which may have a lot more cells. But even in that case reading from 
a local SSD should have been a lot faster than reading from non-provisioned EBS.

Mohammed

From: Chris Lohfink 
[mailto:clohf...@blackbirdit.com]
Sent: Wednesday, September 17, 2014 7:17 PM

To: user@cassandra.apache.org
Subject: Re: no change observed in read latency after switching from EBS to SSD 
storage

"Read 193311 live and 0 tombstoned cells "

is your killer.  returning 250k rows is a bit excessive, you should really page 
this in smaller chunks, what client are you using to access the data?  This 
partition (a, b, c, d, e, f) may be too large as well (can check partition max 
size from output of nodetool cfstats), may be worth including g to break it up 
more - but I dont know enough about your data model.

---
Chris Lohfink

On Sep 17, 2014, at 4:53 PM, Mohammed Guller 
mailto:moham...@glassbeam.com>> wrote:

Thank you all for your responses.

Alex –
  Instance (ephemeral) SSD

Ben –
the query reads data from just one partition. If disk i/o is the bottleneck, 
then in theory, if reading from EBS takes 10 seconds, then it should take lot 
less when reading the same amount of data from local SSD. My question is not 
about why it is taking 10 seconds, but why is the read time same for both EBS 
(network attached storage) and local SSD?

Tony –
if the data was cached in memory, then a read should not take 10 seconds just 
for 20MB data

Rob –
Here is the schema, query, and trace. I masked the actual column names to 
protect the innocents ☺

create table dummy(
  a   varchar,
  b   varchar,
  c   varchar,
  d   varchar,
  e   varchar,
  f   varchar,
  g   varchar,
  h   timestamp,
  i   int,
  non_key1   varchar,
  ...
  non_keyN   varchar,
  PRIMARY KEY ((a, b, c, d, e, f), g, h, i)
) WITH CLUSTERING ORDER BY (g ASC, h DESC, i ASC)

SELECT h, non_key100, non_key200 FROM dummy WHERE a='' AND b='bb' AND 
c='ccc' AND d='dd' AND e='' AND f='ff' AND g='g'AND 
h >='2014-09-10T00:00:00' AND h<='2014-09-10T23:40:41';

The above query returns around 250,000 CQL rows.

cqlsh trace:

activity | timestamp| source  | source_elapsed
-
execute_cql3_query | 21:57:16,830 | 10.10.100.5 |  0
Parsing query; | 21:57:16,830 | 10.10.100.5 |673
Preparing statement | 21:57:16,831 | 10.10.100.5 |   1602
Executing single-partition query on event | 21:57:16,845 | 10.10.100.5 |
  14871
Acquiring sstable references | 21:57:16,845 | 10.10.100.5 |  14896
Merging memtable 

Why column with timestamp datatype come in different format?

[Check Peck]

I have a Cassandra cluster version as -

cqlsh:dataks> show version;
[cqlsh 2.3.0 | Cassandra 2.0.6 | CQL spec 3.0.0 | Thrift protocol
19.39.0]

And I have a table like this -

CREATE TABLE data_test (
  valid_id int,
  data_id text,
  client_name text,
  creation_date timestamp,
  last_modified_date timestamp,
  PRIMARY KEY (valid_id, data_id)
)

I am inserting the data like this in my above table -

insert into data_test (valid_id, data_id, client_name, creation_date,
last_modified_date) values (1, 'b4b61aa', 'TTLAP', dateOf(now()),
dateOf(now()));

After I do a select on my table, I am seeing creation_date and
last_modified_date coming in some other format, not sure why?

 valid_id | data_id | client_name | creation_date  |
last_modified_date

--+
  1   | b4b61aa | TTLAP   | \x00\x00\x01H\x89\xf0\xb6I |
\x00\x00\x01H\x89\xf0\xb6I

Does anyone know why creation_date and last_modified_date is coming like
this and how we can get actual timestamp in those columns?

Re: [RELEASE] Apache Cassandra 1.2.19 released

[Jake Luciani]

Apologies, the correct url for CHANGES.txt is http://goo.gl/eB973i

On Thu, Sep 18, 2014 at 12:58 PM, Jake Luciani  wrote:

> The Cassandra team is pleased to announce the release of Apache Cassandra
> version 1.2.19.
>
> Cassandra is a highly scalable second-generation distributed database,
> bringing together Dynamo's fully distributed design and Bigtable's
> ColumnFamily-based data model. You can read more here:
>
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions are listed in our download
> section:
>
>  http://cassandra.apache.org/download/
>
> This version is a maintenance/bug fix release[1] on the 1.2 series. As
> always,
> please pay attention to the release notes[2] and Let us know[3] if you
> were to
> encounter any problem. This will likely be the final release in the 1.2
> series.
>
> Enjoy!
>
> [1]: http://goo.gl/F6szqv (CHANGES.txt)
> [2]: http://goo.gl/9VsZ88 (NEWS.txt)
> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>

Re: hs_err_pid3013.log, out of memory?

[J. Ryan Earl]

On Wed, Sep 17, 2014 at 8:35 PM, Yatong Zhang  wrote:

> @Chris Lohfink I have 16G memory per node, all the other settings are
> default
>
> @J. Ryan Earl I am not sure. I am using the default settings.
>
> But I've found out it might be because some settings in
> '/etc/sysctl.conf'. I am still testing it
>


If JNA is installed, it will try to memlock all of the JVM process.  For
this to happen, you have to adjust the settings for the user you run
Cassandra as under /etc/security/limits.conf or limits.d/ and you have to
modifying kernel.shmmax in sysctl.conf accordingly.  If you do not, and JNA
is installed, the memlock will fail with the error you gave.

Re: ava.lang.OutOfMemoryError: unable to create new native thread

[J. Ryan Earl]

What's the 'ulimit -a' output of the user cassandra runs as?  From this and
your previous OOM thread, is sounds like you skipped the requisite OS
configuration.

On Wed, Sep 17, 2014 at 9:43 AM, Yatong Zhang  wrote:

> Hi there,
>
> I am using leveled compaction strategy and have many sstable files. The
> error was during the startup, so any idea about this?
>
>
>> ERROR [FlushWriter:4] 2014-09-17 22:36:59,383 CassandraDaemon.java (line
>> 199) Exception in thread Thread[FlushWriter:4,5,main]
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:693)
>> at
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>> at
>> java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> ERROR [FlushWriter:2] 2014-09-17 22:36:59,472 CassandraDaemon.java (line
>> 199) Exception in thread Thread[FlushWriter:2,5,main]
>> FSReadError in
>> /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-23-Index.db
>> at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
>> at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
>> at
>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:334)
>> at
>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:324)
>> at
>> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:394)
>> at
>> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342)
>> at
>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> Caused by: java.io.IOException: Map failed
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
>> at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
>> ... 10 more
>> Caused by: java.lang.OutOfMemoryError: Map failed
>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846)
>> ... 11 more
>>
>
>

Re: Why column with timestamp datatype come in different format?

[Jens Rantil]

Hi,


Have tried removing your dateOf calls and only inserting now()?




Cheers,

Jens

On Thu, Sep 18, 2014 at 8:13 PM, Check Peck 
wrote:

> I have a Cassandra cluster version as -
> cqlsh:dataks> show version;
> [cqlsh 2.3.0 | Cassandra 2.0.6 | CQL spec 3.0.0 | Thrift protocol
> 19.39.0]
> And I have a table like this -
> CREATE TABLE data_test (
>   valid_id int,
>   data_id text,
>   client_name text,
>   creation_date timestamp,
>   last_modified_date timestamp,
>   PRIMARY KEY (valid_id, data_id)
> )
> I am inserting the data like this in my above table -
> insert into data_test (valid_id, data_id, client_name, creation_date,
> last_modified_date) values (1, 'b4b61aa', 'TTLAP', dateOf(now()),
> dateOf(now()));
> After I do a select on my table, I am seeing creation_date and
> last_modified_date coming in some other format, not sure why?
>  valid_id | data_id | client_name | creation_date  |
> last_modified_date
> --+
>   1   | b4b61aa | TTLAP   | \x00\x00\x01H\x89\xf0\xb6I |
> \x00\x00\x01H\x89\xf0\xb6I
> Does anyone know why creation_date and last_modified_date is coming like
> this and how we can get actual timestamp in those columns?

Re: Slow down of secondary index query with VNODE (C* version 1.2.18, jre6).

[DuyHai Doan]

Hello Jay

Your query is : "select * from keyspaceuser.company_testusers where
lastname = ‘lau’ LIMIT 1"

 Why do you think that the slowness is due to vnodes and not your query
asking for 10 000 results ?

On Fri, Sep 19, 2014 at 3:33 AM, Jay Patel  wrote:

> Hi there,
>
> We are seeing extreme slow down (500ms to 1s) in query on secondary index
> with vnode. I'm seeing multiple secondary index scans on a given node in
> trace output when vnode is enabled. Without vnode, everything is good.
>
> Cluster size: 6 nodes
> Replication factor: 3
> Consistency level: local_quorum. Same behavior happens with consistency
> level of ONE.
>
> Snippet from the trace output. Pls see the attached output1.txt for the
> full log. Are we hitting any bug? Do not understand why coordinator sends
> requests multiple times to the same node (e.g. 192.168.51.22 in below
> output) for different token ranges.
>
> >>>
>
> Executing indexed scan for [min(-9223372036854775808),
> max(-9193352069377957523)] | 23:11:30,992 | 192.168.51.22 |
> Executing indexed scan for (max(-9193352069377957523),
> max(-9136021049555745100)] | 23:11:30,998 | 192.168.51.25 |
>  Executing indexed scan for (max(-9136021049555745100),
> max(-8959555493872108621)] | 23:11:30,999 | 192.168.51.22 |
>  Executing indexed scan for (max(-8959555493872108621),
> max(-8929774302283364912)] | 23:11:31,000 | 192.168.51.25 |
> Executing indexed scan for (max(-8929774302283364912),
> max(-8854653908608918942)] | 23:11:31,001 | 192.168.51.22 |
>  Executing indexed scan for (max(-8854653908608918942),
> max(-8762620856967633953)] | 23:11:31,002 | 192.168.51.25 |
>   Executing indexed scan for (max(-8762620856967633953),
> max(-8668275030769104047)] | 23:11:31,003 | 192.168.51.22 |
> Executing indexed scan for (max(-8668275030769104047),
> max(-8659066486210615614)] | 23:11:31,003 | 192.168.51.25 |
>  Executing indexed scan for (max(-8659066486210615614),
> max(-8419137646248370231)] | 23:11:31,004 | 192.168.51.22 |
>  Executing indexed scan for (max(-8419137646248370231),
> max(-8416786876632807845)] | 23:11:31,005 | 192.168.51.25 |
>  Executing indexed scan for (max(-8416786876632807845),
> max(-8315889933848495185)] | 23:11:31,006 | 192.168.51.22 |
> Executing indexed scan for (max(-8315889933848495185),
> max(-8270922890152952193)] | 23:11:31,006 | 192.168.51.25 |
> Executing indexed scan for (max(-8270922890152952193),
> max(-8260813759533312175)] | 23:11:31,007 | 192.168.51.22 |
>  Executing indexed scan for (max(-8260813759533312175),
> max(-8234845345932129353)] | 23:11:31,008 | 192.168.51.25 |
>  Executing indexed scan for (max(-8234845345932129353),
> max(-8216636461332030758)] | 23:11:31,008 | 192.168.51.22 |
>
> Thanks,
> Jay
>
>