date:20141216

In place vnode conversion possible?

2014-12-16 Thread Jonas Borgström

Hi,

I know that adding a new vnode enabled DC is the recommended method to
convert and existing cluster to vnode. And that the cassandra-shuffle
utility has been removed.

That said, I've done some testing and it appears to be possible to
perform an in place conversion as long as all nodes contain all data (3
nodes and replication factor 3 for example) like this:

for each node:
- nodetool -h localhost disablegossip (Not sure if this is needed)

- cqlsh localhost
  UPDATE system.local SET tokens=$NEWTOKENS WHERE key='local';

- nodetool -h localhost disablethrift (Not sure if this is needed)
- nodetool -h localhost drain
- service cassandra restart

And the following python snippet was used to generate $NEWTOKENS for
each node (RandomPartitioner):
"""
import random
print str([str(x) for x in sorted(random.randint(0,2**127-1) for x in
range(256))]).replace('[', '{').replace(']', '}')
"""

I've tested this in a test cluster and it seems to work just fine.

Has anyone else done anything similar?

Or if manually changing tokens is impossible and something horrible will
hit me down the line?

Test cluster configuration
--
Cassandra version: 1.2.19
Number of nodes: 3
Keyspace: NetworkTopologyStrategy:  {DC1: 1, DC2:1, DC3: 1}

/ Jonas



signature.asc
Description: OpenPGP digital signature

Understanding what is key and partition key

2014-12-16 Thread Chamila Wijayarathna

Hello all,

I have read a lot about Cassandra and I read about key-value pairs,
partition keys, clustering keys, etc..
Is key mentioned in key-value pair and partition key refers to same or are
they different?

CREATE TABLE corpus.bigram_time_category_ordered_frequency (
id bigint,
word1 varchar,
word2 varchar,
year int,
category varchar,
frequency int,
PRIMARY KEY((year, category),frequency,word1,word2));


In this schema, I know (year, category) is the compound partition key and
frequency is the clustering key. What is the key here?


Thank You!

-- 
*Chamila Dilshan Wijayarathna,*
SMIEEE, SMIESL,
Undergraduate,
Department of Computer Science and Engineering,
University of Moratuwa.

Re: Understanding what is key and partition key

2014-12-16 Thread Jack Krupansky

Correction: year and category form a “composite partition key”.

frequency, word1, and word2 are “clustering columns”.

The combination of a partition key with clustering columns is a “compound 
primary key”.

Every CQL row will have a partition key by definition, and may optionally have 
clustering columns.

“The key” should just be a synonym for “primary key”, although sometimes people 
are loosely speaking about “the partition” (which should be “the partition 
key”) rather than the CQL “row”.

-- Jack Krupansky

From: Chamila Wijayarathna 
Sent: Tuesday, December 16, 2014 8:03 AM
To: user@cassandra.apache.org 
Subject: Understanding what is key and partition key

Hello all,  

I have read a lot about Cassandra and I read about key-value pairs, partition 
keys, clustering keys, etc.. 
Is key mentioned in key-value pair and partition key refers to same or are they 
different?

CREATE TABLE corpus.bigram_time_category_ordered_frequency (
id bigint,
word1 varchar,
word2 varchar,
year int,
category varchar,
frequency int,
PRIMARY KEY((year, category),frequency,word1,word2)
);
In this schema, I know (year, category) is the compound partition key and 
frequency is the clustering key. What is the key here?


Thank You! 


-- 

Chamila Dilshan Wijayarathna,
SMIEEE, SMIESL,
Undergraduate,
Department of Computer Science and Engineering,
University of Moratuwa.

Re: Understanding what is key and partition key

2014-12-16 Thread Chamila Wijayarathna

Hi Jack,

So what will be the keys and values of the following CF instance?

year | category | frequency | word1| word2   | id
--+--+---+--+-+---
 2014 |N | 1 |සියළුම | යුද්ධ |   664
 2014 |N | 1 |එච් |   කාණ්ඩය | 12526
 2014 |N | 1 |ගජබා | සුපර්ක්‍රොස් | 25779
 2014 |N | 1 |  බී|   කාණ්ඩය | 12505

Thank You!

On Tue, Dec 16, 2014 at 6:45 PM, Jack Krupansky 
wrote:
>
>   Correction: year and category form a “composite partition key”.
>
> frequency, word1, and word2 are “clustering columns”.
>
> The combination of a partition key with clustering columns is a “compound
> primary key”.
>
> Every CQL row will have a partition key by definition, and may optionally
> have clustering columns.
>
> “The key” should just be a synonym for “primary key”, although sometimes
> people are loosely speaking about “the partition” (which should be “the
> partition key”) rather than the CQL “row”.
>
> -- Jack Krupansky
>
>  *From:* Chamila Wijayarathna 
> *Sent:* Tuesday, December 16, 2014 8:03 AM
> *To:* user@cassandra.apache.org
> *Subject:* Understanding what is key and partition key
>
>  Hello all,
>
> I have read a lot about Cassandra and I read about key-value pairs,
> partition keys, clustering keys, etc..
> Is key mentioned in key-value pair and partition key refers to same or are
> they different?
>
>
> CREATE TABLE corpus.bigram_time_category_ordered_frequency (
> id bigint,
> word1 varchar,
> word2 varchar,
> year int,
> category varchar,
> frequency int,
> PRIMARY KEY((year, category),frequency,word1,word2));
>
>
> In this schema, I know (year, category) is the compound partition key and
> frequency is the clustering key. What is the key here?
>
>
> Thank You!
>
> --
> *Chamila Dilshan Wijayarathna,*
> SMIEEE, SMIESL,
> Undergraduate,
> Department of Computer Science and Engineering,
> University of Moratuwa.
>


-- 
*Chamila Dilshan Wijayarathna,*
SMIEEE, SMIESL,
Undergraduate,
Department of Computer Science and Engineering,
University of Moratuwa.

Re: Understanding what is key and partition key

2014-12-16 Thread Jens Rantil

For the first row, the key is: (2014, N, 1, සියළුම, යුද්ධ) and the value-part 
is (664).




Cheers,

Jens


———
Jens Rantil
Backend engineer
Tink AB

Email: jens.ran...@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

On Tue, Dec 16, 2014 at 2:25 PM, Chamila Wijayarathna
 wrote:

> Hi Jack,
> So what will be the keys and values of the following CF instance?
> year | category | frequency | word1| word2   | id
> --+--+---+--+-+---
>  2014 |N | 1 |සියළුම | යුද්ධ |   664
>  2014 |N | 1 |එච් |   කාණ්ඩය | 12526
>  2014 |N | 1 |ගජබා | සුපර්ක්‍රොස් | 25779
>  2014 |N | 1 |  බී|   කාණ්ඩය | 12505
> Thank You!
> On Tue, Dec 16, 2014 at 6:45 PM, Jack Krupansky 
> wrote:
>>
>>   Correction: year and category form a “composite partition key”.
>>
>> frequency, word1, and word2 are “clustering columns”.
>>
>> The combination of a partition key with clustering columns is a “compound
>> primary key”.
>>
>> Every CQL row will have a partition key by definition, and may optionally
>> have clustering columns.
>>
>> “The key” should just be a synonym for “primary key”, although sometimes
>> people are loosely speaking about “the partition” (which should be “the
>> partition key”) rather than the CQL “row”.
>>
>> -- Jack Krupansky
>>
>>  *From:* Chamila Wijayarathna 
>> *Sent:* Tuesday, December 16, 2014 8:03 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Understanding what is key and partition key
>>
>>  Hello all,
>>
>> I have read a lot about Cassandra and I read about key-value pairs,
>> partition keys, clustering keys, etc..
>> Is key mentioned in key-value pair and partition key refers to same or are
>> they different?
>>
>>
>> CREATE TABLE corpus.bigram_time_category_ordered_frequency (
>> id bigint,
>> word1 varchar,
>> word2 varchar,
>> year int,
>> category varchar,
>> frequency int,
>> PRIMARY KEY((year, category),frequency,word1,word2));
>>
>>
>> In this schema, I know (year, category) is the compound partition key and
>> frequency is the clustering key. What is the key here?
>>
>>
>> Thank You!
>>
>> --
>> *Chamila Dilshan Wijayarathna,*
>> SMIEEE, SMIESL,
>> Undergraduate,
>> Department of Computer Science and Engineering,
>> University of Moratuwa.
>>
> -- 
> *Chamila Dilshan Wijayarathna,*
> SMIEEE, SMIESL,
> Undergraduate,
> Department of Computer Science and Engineering,
> University of Moratuwa.

Re: Understanding what is key and partition key

2014-12-16 Thread Chamila Wijayarathna

Hi Jens,

Thank You!

On Tue, Dec 16, 2014 at 7:03 PM, Jens Rantil  wrote:
>
> For the first row, the key is: (2014, N, 1, සියළුම, යුද්ධ) and the
> value-part is (664).
>
> Cheers,
> Jens
>
> ——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
>
>
> On Tue, Dec 16, 2014 at 2:25 PM, Chamila Wijayarathna <
> cdwijayarat...@gmail.com> wrote:
>
>> Hi Jack,
>>
>> So what will be the keys and values of the following CF instance?
>>
>>  year | category | frequency | word1| word2   | id
>> --+--+---+--+-+---
>>  2014 |N | 1 |සියළුම | යුද්ධ |   664
>>  2014 |N | 1 |එච් |   කාණ්ඩය | 12526
>>  2014 |N | 1 |ගජබා | සුපර්ක්‍රොස් | 25779
>>  2014 |N | 1 |  බී|   කාණ්ඩය | 12505
>>
>> Thank You!
>>
>> On Tue, Dec 16, 2014 at 6:45 PM, Jack Krupansky 
>> wrote:
>>>
>>>   Correction: year and category form a “composite partition key”.
>>>
>>> frequency, word1, and word2 are “clustering columns”.
>>>
>>> The combination of a partition key with clustering columns is a
>>> “compound primary key”.
>>>
>>> Every CQL row will have a partition key by definition, and may
>>> optionally have clustering columns.
>>>
>>> “The key” should just be a synonym for “primary key”, although sometimes
>>> people are loosely speaking about “the partition” (which should be “the
>>> partition key”) rather than the CQL “row”.
>>>
>>> -- Jack Krupansky
>>>
>>>  *From:* Chamila Wijayarathna 
>>>  *Sent:* Tuesday, December 16, 2014 8:03 AM
>>>  *To:* user@cassandra.apache.org
>>>  *Subject:* Understanding what is key and partition key
>>>
>>>   Hello all,
>>>
>>> I have read a lot about Cassandra and I read about key-value pairs,
>>> partition keys, clustering keys, etc..
>>> Is key mentioned in key-value pair and partition key refers to same or
>>> are they different?
>>>
>>>
>>> CREATE TABLE corpus.bigram_time_category_ordered_frequency (
>>> id bigint,
>>> word1 varchar,
>>> word2 varchar,
>>> year int,
>>> category varchar,
>>> frequency int,
>>> PRIMARY KEY((year, category),frequency,word1,word2));
>>>
>>>
>>> In this schema, I know (year, category) is the compound partition key
>>> and frequency is the clustering key. What is the key here?
>>>
>>>
>>> Thank You!
>>>
>>> --
>>> *Chamila Dilshan Wijayarathna,*
>>> SMIEEE, SMIESL,
>>> Undergraduate,
>>> Department of Computer Science and Engineering,
>>> University of Moratuwa.
>>>
>>
>>
>> --
>> *Chamila Dilshan Wijayarathna,*
>> SMIEEE, SMIESL,
>> Undergraduate,
>> Department of Computer Science and Engineering,
>> University of Moratuwa.
>>
>
>

-- 
*Chamila Dilshan Wijayarathna,*
SMIEEE, SMIESL,
Undergraduate,
Department of Computer Science and Engineering,
University of Moratuwa.

Defining DataSet.json for cassandra-unit testing

2014-12-16 Thread Chamila Wijayarathna

Hello all,

I am trying to test my application using cassandra-unit with following
schema and data given below.

CREATE TABLE corpus.bigram_time_category_ordered_frequency (
id bigint,
word1 varchar,
word2 varchar,
year int,
category varchar,
frequency int,
PRIMARY KEY((year, category),frequency,word1,word2));

year | category | frequency | word1| word2   | id
--+--+---+--+-+---
 2014 |N | 1 |සියළුම | යුද්ධ |   664
 2014 |N | 1 |එච් |   කාණ්ඩය | 12526
 2014 |N | 1 |ගජබා | සුපර්ක්‍රොස් | 25779
 2014 |N | 1 |  බී|   කාණ්ඩය | 12505

Since this has a compound primary key, I am not clear with how to define
dataset.json [1] for this CF. Can somebody help me on how to do that?

Thank You!

1.
https://github.com/jsevellec/cassandra-unit/wiki/What-can-you-set-into-a-dataSet

-- 
*Chamila Dilshan Wijayarathna,*
SMIEEE, SMIESL,
Undergraduate,
Department of Computer Science and Engineering,
University of Moratuwa.

Re: batch_size_warn_threshold_in_kb

2014-12-16 Thread Eric Stevens

> You are, of course, free to use batches in your application

I'm not looking to justify the use of batches, I'm looking for the path
forward that will give us the Best Results™ both near and long term, for
some definition of Best (which would be a balance of client throughput and
cluster pressure). If individual writes are best for us, that's what I
want to do. If batches are best for us, that's what I want to do.

I'm just struggling that I'm not able to reproduce your advice
experimentally, and it's not just a few percent difference, it's 5x to 8x
difference. It's really difficult for me to adopt advice blindly when it
differs from my own observations by such a substantial amount. That means
something is wrong either with my observations or with the advice, and I
would really like to know which. I'm not trying to be argumentative or
push for a particular approach, I'm trying to resolve an inconsistency.

RE your questions: I'm sorry this turns into a wall of text, simple
questions about parallelism and distributed systems rarely can be
adequately answered in just a few words. I'm trying to be open and
transparent about my testing approach because I want to find out where the
disconnect is here. At the same time I'm trying to bridge the knowledge
gap since I'm working with parallelism toolset with which you're not
familiar, and that could obviously have a substantial impact on the
results. Hopefully someone else in the community familiar with Scala will
notice this and provide feedback that I'm not making a fundamental mistake.

1) My original runs were in EC2 being driven by a different server than the
Cassandra cluster, but in the same AZ as one of the Cassandra
servers (typical 3-AZ setup for Cassandra). All four instances (3x C*, 1x
test driver) were i2.2xl, so have gigabit network between them.

2) The system was under some moderate other load, this is our test cluster
that takes a steady stream of simulated data to provide other developers
with something to work against. That load is quite constant and doesn't
work these servers particularly hard - only a few thousand records per
second typically. Load averages between 1 and 3 most of the time.

Unfortunately I'm not successful getting cassandra-stress talking to this
cluster because of ssl configuration (it doesn't seem to actually pay
attention to -ts and -tspw command line flags). I can find out if our ops
guys would be ok with turning off ssl for a while, but as that would break
our other applications using the same cluster, and may block our other
engineers as a result. So it has farther reaching implications than just
being something I can happily turn on or off at whim.

I'm curious how you would expect the performance of my stress tool to
differ when the cluster was being overworked - could you explain what you
anticipate the change in results to look like? I.e. would single-writes
remain about constant for performance while batches would degrade in
performance?

3) Well I specifically attempt to control for this by testing three
different concurrency models, these were named by me "parallel," "scatter,"
and "traverse" (just aliases to make it easier to control the driver). You
can see the code between the different approaches here - they are pretty
similar to each other, but probably involve some knowledge of how
concurrency works in Scala to really appreciate the differences:
https://gist.github.com/MightyE/1c98912fca104f6138fc/a7db68e72f99ac1215fcfb096d69391ee285c080#file-testsuite-L181-L203

I know you're not a Scala guy, so I'll explain roughly what they do, but
the point is that I'm trying hard to control for just having chosen a bad
concurrency model:

scatter -> Take all of the Statements and call executeAsync() on them as
*fast* the Session will let me. This is the Unintelligent Brute Force
approach, and it's definitely not how I would model a typical production
application as it doesn't attempt to respond to system pressure at all, and
it's trying to gobble up as many resources as it can. Use the Scala
Futures system to combine the the set of async calls into a single Future
that completes when all the futures returned from executeAsync() have
completed.

traverse -> Give all of the Statements to the Scala Futures system and tell
it to call executeAsync() on them all at the rate that it thinks is
appropriate. This would be much closer to my recommendation on how to
model a production application, because in a real application, there's more
than a single class of work to be done, and the Futures system schedules
both this work and other work intelligently and configurably. It gives us
a single awaitable Future that completes when it has finished all of its
work and all of the async calls have been completed. You guys are using
Netty for your native protocol, and Netty offers true event driven
concurrency which gets along famously well with Scala's Futures system.

parallel -> Use a Scala Parallel collection

does consistency=ALL for deletes obviate the need for tombstones?

2014-12-16 Thread Ian Rose

Howdy all,

Our use of cassandra unfortunately makes use of lots of deletes.  Yes, I
know that C* is not well suited to this kind of workload, but that's where
we are, and before I go looking for an entirely new data layer I would
rather explore whether C* could be tuned to work well for us.

However, deletions are never driven by users in our app - deletions always
occur by backend processes to "clean up" data after it has been processed,
and thus they do not need to be 100% available.  So this made me think,
what if I did the following?

   - gc_grace_seconds = 0, which ensures that tombstones are never created
   - replication factor = 3
   - for writes that are inserts, consistency = QUORUM, which ensures that
   writes can proceed even if 1 replica is slow/down
   - for deletes, consistency = ALL, which ensures that when we delete a
   record it disappears entirely (no need for tombstones)
   - for reads, consistency = QUORUM

Also, I should clarify that our data essentially append only, so I don't
need to worry about inconsistencies created by partial updates (e.g. value
gets changed on one machine but not another).  Sometimes there will be
duplicate writes, but I think that should be fine since the value is always
identical.

Any red flags with this approach?  Has anyone tried it and have experiences
to share?  Also, I *think* that this means that I don't need to run
repairs, which from an ops perspective is great.

Thanks, as always,
- Ian

Re: does consistency=ALL for deletes obviate the need for tombstones?

2014-12-16 Thread Eric Stevens

No, deletes are always written as a tombstone no matter the consistency.
This is because data at rest is written to sstables which are immutable
once written. The tombstone marks that a record in another sstable is now
deleted, and so a read of that value should be treated as if it doesn't
exist.

When sstables are later compacted, several sstables are merged into one and
any overlapping values between the tables are condensed into one. Values
which have a tombstone can be excluded from the new sstable. GC grace
period indicates how long a tombstone should be kept after all underlying
values have been compacted away so that the deleted value can't be
resurrected if a node rejoins the cluster which knew that value.
On Dec 16, 2014 8:23 AM, "Ian Rose"  wrote:

> Howdy all,
>
> Our use of cassandra unfortunately makes use of lots of deletes.  Yes, I
> know that C* is not well suited to this kind of workload, but that's where
> we are, and before I go looking for an entirely new data layer I would
> rather explore whether C* could be tuned to work well for us.
>
> However, deletions are never driven by users in our app - deletions always
> occur by backend processes to "clean up" data after it has been processed,
> and thus they do not need to be 100% available.  So this made me think,
> what if I did the following?
>
>- gc_grace_seconds = 0, which ensures that tombstones are never created
>- replication factor = 3
>- for writes that are inserts, consistency = QUORUM, which ensures
>that writes can proceed even if 1 replica is slow/down
>- for deletes, consistency = ALL, which ensures that when we delete a
>record it disappears entirely (no need for tombstones)
>- for reads, consistency = QUORUM
>
> Also, I should clarify that our data essentially append only, so I don't
> need to worry about inconsistencies created by partial updates (e.g. value
> gets changed on one machine but not another).  Sometimes there will be
> duplicate writes, but I think that should be fine since the value is always
> identical.
>
> Any red flags with this approach?  Has anyone tried it and have
> experiences to share?  Also, I *think* that this means that I don't need to
> run repairs, which from an ops perspective is great.
>
> Thanks, as always,
> - Ian
>
>

Re: does consistency=ALL for deletes obviate the need for tombstones?

2014-12-16 Thread Robert Wille

Tombstones have to be created. The SSTables are immutable, so the data cannot 
be deleted. Therefore, a tombstone is required. The value you deleted will be 
physically removed during compaction.

My workload sounds similar to yours in some respects, and I was able to get C* 
working for me. I have large chunks of data which I periodically replace. I 
write the new data, update a reference, and then delete the old data. I 
designed my schema to be tombstone-friendly, and C* works great. For some of my 
tables I am able to delete entire partitions. Because of the reference that I 
updated, I never try to access the old data, and therefore the tombstones for 
these partitions are never read. The old data simply has to wait for 
compaction. Other tables require deleting records within partitions. These 
tombstones do get read, so there are performance implications. I was able to 
design my schema so that no partition ever has more than a few tombstones (one 
for each generation of deleted data, which is usually no more than one).

Hope this helps.

Robert

On Dec 16, 2014, at 8:22 AM, Ian Rose 
mailto:ianr...@fullstory.com>> wrote:

Howdy all,

Our use of cassandra unfortunately makes use of lots of deletes.  Yes, I know 
that C* is not well suited to this kind of workload, but that's where we are, 
and before I go looking for an entirely new data layer I would rather explore 
whether C* could be tuned to work well for us.

However, deletions are never driven by users in our app - deletions always 
occur by backend processes to "clean up" data after it has been processed, and 
thus they do not need to be 100% available.  So this made me think, what if I 
did the following?

  *   gc_grace_seconds = 0, which ensures that tombstones are never created
  *   replication factor = 3
  *   for writes that are inserts, consistency = QUORUM, which ensures that 
writes can proceed even if 1 replica is slow/down
  *   for deletes, consistency = ALL, which ensures that when we delete a 
record it disappears entirely (no need for tombstones)
  *   for reads, consistency = QUORUM

Also, I should clarify that our data essentially append only, so I don't need 
to worry about inconsistencies created by partial updates (e.g. value gets 
changed on one machine but not another).  Sometimes there will be duplicate 
writes, but I think that should be fine since the value is always identical.

Any red flags with this approach?  Has anyone tried it and have experiences to 
share?  Also, I *think* that this means that I don't need to run repairs, which 
from an ops perspective is great.

Thanks, as always,
- Ian

Re: does consistency=ALL for deletes obviate the need for tombstones?

2014-12-16 Thread Ian Rose

Ah, makes sense.  Thanks for the explanations!

- Ian


On Tue, Dec 16, 2014 at 10:53 AM, Robert Wille  wrote:
>
>  Tombstones have to be created. The SSTables are immutable, so the data
> cannot be deleted. Therefore, a tombstone is required. The value you
> deleted will be physically removed during compaction.
>
>  My workload sounds similar to yours in some respects, and I was able to
> get C* working for me. I have large chunks of data which I periodically
> replace. I write the new data, update a reference, and then delete the old
> data. I designed my schema to be tombstone-friendly, and C* works great.
> For some of my tables I am able to delete entire partitions. Because of the
> reference that I updated, I never try to access the old data, and therefore
> the tombstones for these partitions are never read. The old data simply has
> to wait for compaction. Other tables require deleting records within
> partitions. These tombstones do get read, so there are performance
> implications. I was able to design my schema so that no partition ever has
> more than a few tombstones (one for each generation of deleted data, which
> is usually no more than one).
>
>  Hope this helps.
>
>  Robert
>
>  On Dec 16, 2014, at 8:22 AM, Ian Rose  wrote:
>
>  Howdy all,
>
>  Our use of cassandra unfortunately makes use of lots of deletes.  Yes, I
> know that C* is not well suited to this kind of workload, but that's where
> we are, and before I go looking for an entirely new data layer I would
> rather explore whether C* could be tuned to work well for us.
>
>  However, deletions are never driven by users in our app - deletions
> always occur by backend processes to "clean up" data after it has been
> processed, and thus they do not need to be 100% available.  So this made me
> think, what if I did the following?
>
>- gc_grace_seconds = 0, which ensures that tombstones are never created
>- replication factor = 3
>- for writes that are inserts, consistency = QUORUM, which ensures
>that writes can proceed even if 1 replica is slow/down
>- for deletes, consistency = ALL, which ensures that when we delete a
>record it disappears entirely (no need for tombstones)
>- for reads, consistency = QUORUM
>
> Also, I should clarify that our data essentially append only, so I don't
> need to worry about inconsistencies created by partial updates (e.g. value
> gets changed on one machine but not another).  Sometimes there will be
> duplicate writes, but I think that should be fine since the value is always
> identical.
>
>  Any red flags with this approach?  Has anyone tried it and have
> experiences to share?  Also, I *think* that this means that I don't need to
> run repairs, which from an ops perspective is great.
>
>  Thanks, as always,
> - Ian
>
>
>

Re: does consistency=ALL for deletes obviate the need for tombstones?

2014-12-16 Thread Jack Krupansky

When you say “no need for tombstones”, did you actually read that somewhere or 
were you just speculating? If the former, where exactly?

-- Jack Krupansky

From: Ian Rose 
Sent: Tuesday, December 16, 2014 10:22 AM
To: user 
Subject: does consistency=ALL for deletes obviate the need for tombstones?

Howdy all, 

Our use of cassandra unfortunately makes use of lots of deletes.  Yes, I know 
that C* is not well suited to this kind of workload, but that's where we are, 
and before I go looking for an entirely new data layer I would rather explore 
whether C* could be tuned to work well for us.

However, deletions are never driven by users in our app - deletions always 
occur by backend processes to "clean up" data after it has been processed, and 
thus they do not need to be 100% available.  So this made me think, what if I 
did the following?
  a.. gc_grace_seconds = 0, which ensures that tombstones are never created 
  b.. replication factor = 3 
  c.. for writes that are inserts, consistency = QUORUM, which ensures that 
writes can proceed even if 1 replica is slow/down 
  d.. for deletes, consistency = ALL, which ensures that when we delete a 
record it disappears entirely (no need for tombstones) 
  e.. for reads, consistency = QUORUM
Also, I should clarify that our data essentially append only, so I don't need 
to worry about inconsistencies created by partial updates (e.g. value gets 
changed on one machine but not another).  Sometimes there will be duplicate 
writes, but I think that should be fine since the value is always identical.

Any red flags with this approach?  Has anyone tried it and have experiences to 
share?  Also, I *think* that this means that I don't need to run repairs, which 
from an ops perspective is great.

Thanks, as always,
- Ian

Re: Hinted handoff not working

2014-12-16 Thread Robert Wille

Nope. I added millions of records and several GB to the cluster while one node 
was down, and then ran "nodetool flush system hints" on a couple of nodes that 
were up, and system/hints has less than 200K in it.

Here’s the relevant part of "nodetool cfstats system.hints":

Keyspace: system
Read Count: 28572
Read Latency: 0.01806502869942601 ms.
Write Count: 351
Write Latency: 0.04547008547008547 ms.
Pending Tasks: 0
Table: hints
SSTable count: 1
Space used (live), bytes: 7446
Space used (total), bytes: 80062
SSTable Compression Ratio: 0.2651441528992549
Number of keys (estimate): 128
Memtable cell count: 1
Memtable data size, bytes: 1740

The hints are definitely not being stored.

Robert

On Dec 14, 2014, at 11:44 PM, Jens Rantil 
mailto:jens.ran...@tink.se>> wrote:

Hi Robert ,

Maybe you need to flush your memtables to actually see the disk usage increase? 
This applies to both hosts.

Cheers,
Jens




On Sun, Dec 14, 2014 at 3:52 PM, Robert Wille 
mailto:rwi...@fold3.com>> wrote:

I have a cluster with RF=3. If I shut down one node, add a bunch of data to the 
cluster, I don’t see a bunch of records added to system.hints. Also, du of 
/var/lib/cassandra/data/system/hints of the nodes that are up shows that hints 
aren’t being stored. When I start the down node, its data doesn’t grow until I 
run repair, which then takes a really long time because it is significantly out 
of date. Is there some magic setting I cannot find in the documentation to 
enable hinted handoff? I’m running 2.0.11. Any insights would be greatly 
appreciated.

Thanks

Robert

Re: Cassandra Maintenance Best practices

2014-12-16 Thread Neha Trivedi

Hi Jonathan,QUORUM = (sum_of_replication_factors / 2) + 1, For us Quorum =
(2/2) +1 = 2.

Default CL is ONE and RF=2 with Two Nodes in the cluster.(I am little
confused, what is my read CL and what is my WRITE CL?)

So, does it mean that for every WRITE it will write in both the nodes?

and For every READ, it will read from both nodes and give back to client?

DOWNGRADERETRYPOLICY will downgrade the CL if a node is down?

Regards

Neha

On Wed, Dec 10, 2014 at 1:00 PM, Jonathan Haddad  wrote:
>
> I did a presentation on diagnosing performance problems in production at
> the US & Euro summits, in which I covered quite a few tools & preventative
> measures you should know when running a production cluster.  You may find
> it useful:
> http://rustyrazorblade.com/2014/09/cassandra-summit-recap-diagnosing-problems-in-production/
>
> On ops center - I recommend it.  It gives you a nice dashboard.  I don't
> think it's completely comprehensive (but no tool really is) but it gets you
> 90% of the way there.
>
> It's a good idea to run repairs, especially if you're doing deletes or
> querying at CL=ONE.  I assume you're not using quorum, because on RF=2
> that's the same as CL=ALL.
>
> I recommend at least RF=3 because if you lose 1 server, you're on the edge
> of data loss.
>
>
> On Tue Dec 09 2014 at 7:19:32 PM Neha Trivedi 
> wrote:
>
>> Hi,
>> We have Two Node Cluster Configuration in production with RF=2.
>>
>> Which means that the data is written in both the clusters and it's
>> running for about a month now and has good amount of data.
>>
>> Questions?
>> 1. What are the best practices for maintenance?
>> 2. Is OPScenter required to be installed or I can manage with nodetool
>> utility?
>> 3. Is is necessary to run repair weekly?
>>
>> thanks
>> regards
>> Neha
>>
>

Comprehensive documentation on Cassandra Data modelling

2014-12-16 Thread Jason Kania

Hi,
I have been having a few exchanges with contributors to the project around what 
is possible with Cassandra and a common response that comes up when I describe 
functionality as broken or missing is that I am not modelling my data 
correctly. Unfortunately, I cannot seem to find comprehensive documentation on 
modelling with Cassandra. In particular, I am finding myself modelling by 
restriction rather than what I would like to do.

Does such documentations exist? If not, is there any effort to create such 
documentation?The DataStax documentation on data modelling is far too weak to 
be meaningful.

In particular, I am caught because:
1) I want to search on a specific column to make updates to it after further 
processing; ie I don't know its value on first insert
2) If I want to search on a column, it has to be part of the primary key3) If a 
column is part of the primary key, it cannot be edited so I have a circular 
dependency
Thanks,
Jason

55 matches

Mail list logo