Composite Column Question

[Martin Arrowsmith]

Dear Cassandra Experts,

Are the number of composite attributes fixed for each column family ?

I have been doing : "create column family MyCF with comparator =
'CompositeType(IntegerType, UTF8Type)'

And this creates a composite { integer:string }

Hector complains when I give a 3rd attribute.

If "unstatic" composite columns are possible, what would be the CLI command
to create such a column family, and how can it be implemented ?

Best wishes,

Martin

Can I slice on composite indexes?

[Maxim Potekhin]

Let's say I have rows with composite columns Like

("key1", {('xyz', 'abc'): 'colval1'},  {('xyz', 'def'): 'colval2'})
("key2", {('ble', 'meh'): 'otherval'})

Is it possible to create a composite type index such that I can query on 
'xyz'

and get the first two columns?

Thanks

Maxim

Routine nodetool repair

[Blake Starkenburg]

I have been playing around with Cassandra for a few months now. Starting to
explore more of the routine maintenance and backup strategies and I have a
general question about nodetool repair. After reading the following page:
http://www.datastax.com/docs/0.8/operations/cluster_management it has
occurred to me that for these past few months I have NOT DONE any cleanup
or repair commands on a test 2-node cluster (and their has been quite a few
deletes, writes, etc.).

For some reason I was under the assumption that Cassandra handled the
tombstone records from deletes automatically? Should I still run nodetool
repair and if so, what about old deletes which occurred months ago?

Thank You!

Re: about decommissioned node that returns

[aaron morton]

A node will only request data from other nodes on start up if it is 
bootstrapping. Bootstrapping happens when there is no system data and the 
auto_bootstrap: yaml config is set true, normally nodes only bootstrap once in 
their life. 

If you bring a node back for whatever reason you will want too run repair, see 
also http://wiki.apache.org/cassandra/Operations#Replacing_a_Dead_Node

Cheers

  
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 10:13 AM, Carlos Pérez Miguel wrote:

> What happens with the data stored for that token range already in the
> cluster? will the restarted node actualize it from any of the
> replicas?
> 
> I readed in the maillist archives that in case of token collision, the
> cluster may not work correctly. But if the restarted node takes always
> ownership of its token it means that no collision is possible, isn't
> it?
> 
> Thank you
> 
> Carlos Pérez Miguel
> 
> 
> 
> 2011/12/20 Brandon Williams :
>> On Tue, Dec 20, 2011 at 2:24 PM, aaron morton  
>> wrote:
>>> Sorry, got that a little wrong.
>>> 
>>> At startup the node will use the higher of the current seconds since epoch
>>> or the stored generation number.
>> 
>> Technically stored generation + 1 so it's always increased on a restart.
>> 
>> -Brandon

Re: Doubts related to composite type column names/values

[aaron morton]

+1 use them if you can. 

Also you can reverse the sort order on components in the type, that can make 
some common queries faster. 

Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 9:49 AM, Guy Incognito wrote:

> afaik composite lets you do sorting in a way that would be 
> difficult/impossible with string concatenation.
> 
> eg  with the string ascending, and the integer descending.
> 
> if i had composites available (which i don't b/c we are on 0.7), i would use 
> them over string concatenation.  string concatenation is a pain.
> 
> On 20/12/2011 20:33, Maxim Potekhin wrote:
>> 
>> Thank you Aaron! As long as I have plain strings, would you say that I would 
>> do almost as well with catenation?
>> 
>> Of course I realize that mixed types are a very different case where the 
>> composite is very useful.
>> 
>> Thanks
>> 
>> Maxim
>> 
>> 
>> On 12/20/2011 2:44 PM, aaron morton wrote:
>>> 
>>> Component values are compared in a type aware fashion, an Integer is an 
>>> Integer. Not a 10 character zero padded string. 
>>> 
>>> You can also slice on the components. Just like with string concat, but 
>>> nicer.  . e.g. If you app is storing comments for a thing, and the column 
>>> names have the form  or   you can slice 
>>> for all properties of a comment or all properties for comments between two 
>>> comment_id's
>>> 
>>> Finally, the client library knows what's going on. 
>>> 
>>> Hope that helps.
>>> 
>>> -
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>> 
>>> On 21/12/2011, at 7:43 AM, Maxim Potekhin wrote:
>>> 
 With regards to static, what are major benefits as it compares with
 string catenation (with some convenient separator inserted)?
 
 Thanks
 
 Maxim
 
 
 On 12/20/2011 1:39 PM, Richard Low wrote:
> On Tue, Dec 20, 2011 at 5:28 PM, Ertio Lew  wrote:
>> With regard to the composite columns stuff in Cassandra, I have the
>> following doubts :
>> 
>> 1. What is the storage overhead of the composite type column 
>> names/values,
> The values are the same.  For each dimension, there is 3 bytes overhead.
> 
>> 2. what exactly is the difference between the DynamicComposite and Static
>> Composite ?
> Static composite type has the types of each dimension specified in the
> column family definition, so all names within that column family have
> the same type.  Dynamic composite type lets you specify the type for
> each column, so they can be different.  There is extra storage
> overhead for this and care must be taken to ensure all column names
> remain comparable.
> 
 
>>> 
>> 
> 

Counter read requests spread across replicas ?

[Philippe]

Hello,
5 nodes running 0.8.7/0.8.9, RF=3, BOP, counter columns inside super
columns. Read queries are multigetslices of super columns inside of which I
read every column for processing (20-30 at most), using Hector with default
settings.
Watching tpstat on the 3 nodes holding the data being most often queries, I
see the pending count increase only on the "main replica" and I see heavy
CPU load and network load only on that node. The other nodes seem to be
doing very little.

Aren't counter read requests supposed to be round-robin across replicas ?
I'm confused as to why the nodes don't exhibit the same load.

Thanks

Re: setStrategy_options syntax in thrift

[A J]

Thanks, that worked.

On Tue, Dec 20, 2011 at 4:08 PM, Dave Brosius wrote:

>
> KsDef ksDef = new KsDef();
> Map options = new HashMap();
> options.put("replication_factor", "2");
> ksDef.setStrategy_options(options);
>
>
>
> *- Original Message -*
> *From:* "A J" 
> *Sent:* Tue, December 20, 2011 16:03
> *Subject:* Re: setStrategy_options syntax in thrift
>
> I am new to java. Can you specify the exact syntax for replication_factor=2 ?
>
> Thanks.
>
> On Tue, Dec 20, 2011 at 1:50 PM, aaron morton  
> wrote:> It looks like you tried to pass the string "{replication_factor:2}">> 
> You need to pas a Map type , where the the key is the option> 
> and the value is the option value.>> Cheers>> -> Aaron 
> Morton> Freelance Developer> @aaronmorton> http://www.thelastpickle.com>> On 
> 20/12/2011, at 12:02 PM, A J wrote:>> What is the syntax of 
> setStrategy_options in thrift.>> The following fails:>> Util.java:22:> 
> setStrategy_options(java.util.Map)> in 
> org.apache.cassandra.thrift.KsDef cannot be applied to> (java.lang.String)>   
>  newKs.setStrategy_options("{replication_factor:2}");>>
>
>

Re: java thrift error

[A J]

The following worked:
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharacterCodingException;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;
import java.nio.charset.CodingErrorAction;
.
public static Charset charset = Charset.forName("UTF-8");
public static CharsetEncoder encoder = charset.newEncoder();
public static CharsetDecoder decoder = charset.newDecoder();

public static ByteBuffer str_to_bb(String msg){
  try{
return encoder.encode(CharBuffer.wrap(msg));
  }catch(Exception e){e.printStackTrace();}
  return null;
}

and then instead of "count_key".getBytes("UTF-8")
do
str_to_bb("count_key")

On Tue, Dec 20, 2011 at 4:03 PM, Dave Brosius wrote:

> A ByteBuffer is not a byte[] to convert a String to a ByteBuffer do
> something like
>
> public static ByteBuffer toByteBuffer(String value) throws 
> UnsupportedEncodingException{return 
> ByteBuffer.wrap(value.getBytes("UTF-8"));}
>
>
>
> see http://wiki.apache.org/cassandra/ThriftExamples
>
>
> *- Original Message -*
> *From:* "A J" 
> *Sent:* Tue, December 20, 2011 15:52
> *Subject:* java thrift error
>
> The following syntax :
> import org.apache.cassandra.thrift.*;
> .
> .
> ColumnOrSuperColumn col = client.get("count_key".getBytes("UTF-8"),
> cp, ConsistencyLevel.QUORUM);
>
>
> is giving the error:
> get(java.nio.ByteBuffer,org.apache.cassandra.thrift.ColumnPath,org.apache.cassandra.thrift.ConsistencyLevel)
> in org.apache.cassandra.thrift.Cassandra.Client cannot be applied to
> (byte[],org.apache.cassandra.thrift.ColumnPath,org.apache.cassandra.thrift.ConsistencyLevel)
>
>
> Any idea on how to cast?
>
> Thanks.
>
>

Re: about decommissioned node that returns

[Carlos Pérez Miguel]

What happens with the data stored for that token range already in the
cluster? will the restarted node actualize it from any of the
replicas?

I readed in the maillist archives that in case of token collision, the
cluster may not work correctly. But if the restarted node takes always
ownership of its token it means that no collision is possible, isn't
it?

Thank you

Carlos Pérez Miguel



2011/12/20 Brandon Williams :
> On Tue, Dec 20, 2011 at 2:24 PM, aaron morton  wrote:
>> Sorry, got that a little wrong.
>>
>> At startup the node will use the higher of the current seconds since epoch
>> or the stored generation number.
>
> Technically stored generation + 1 so it's always increased on a restart.
>
> -Brandon

Re: setStrategy_options syntax in thrift

[Dave Brosius]

 KsDef ksDef = new KsDef();Map>;String, String;String, String

Re: java thrift error

[Dave Brosius]

A ByteBuffer is not a byte[] to convert a String to a ByteBuffer do something 
likepublic static ByteBuffer toByteBuffer(String value) 
throws UnsupportedEncodingException
{
return ByteBuffer.wrap(value.getBytes("UTF-8"));
} see http://wiki.apache.org/cassandra/ThriftExamples - Original 
Message -From: "A J" >;s5a...@gmail.com

Re: setStrategy_options syntax in thrift

[A J]

I am new to java. Can you specify the exact syntax for replication_factor=2 ?

Thanks.

On Tue, Dec 20, 2011 at 1:50 PM, aaron morton  wrote:
> It looks like you tried to pass the string "{replication_factor:2}"
>
> You need to pas a Map type , where the the key is the option
> and the value is the option value.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/12/2011, at 12:02 PM, A J wrote:
>
> What is the syntax of setStrategy_options in thrift.
>
> The following fails:
>
> Util.java:22:
> setStrategy_options(java.util.Map)
> in org.apache.cassandra.thrift.KsDef cannot be applied to
> (java.lang.String)
>    newKs.setStrategy_options("{replication_factor:2}");
>
>

Re: questions on datastax opscenter

[Tyler Hobbs]

On Tue, Dec 20, 2011 at 2:19 PM, Tyler Hobbs  wrote:

> On Tue, Dec 20, 2011 at 1:49 PM, Feng Qu  wrote:
>
>> I have two questions for community version of opscenter
>>
>> 1) does it work with multiple cassandra cluster?
>> 2) are agent ports 61620/61621 configurable? these ports are blocked in
>> our env.
>>
>> Thanks in advance
>>
>> Feng Qu
>>
>
> They are configurable. If you add the following lines to opscenterd.conf
> (replacing the default port numbers with your own) and restart opscenterd,
> the change should take effect:
> [agents]
> https_port = 61621
> incoming_port = 61620
>
>
Actually, I left out an important step. You will need to update the config
for each agent and restart them as well.

The config file you will need to edit is address.yaml. If the agents were
installed automatically by opscenter, or if you installed the agent
packages (either the rpm or deb files) manually, it will be located at
/var/lib/opscenter-agent/conf/address.yaml.

Add the following line to address.yaml, replacing 61620 with your custom
port:

stomp_port: 61620

This port number should match the "incoming_port" number in
opscenterd.conf.

-- 
Tyler Hobbs
DataStax 

java thrift error

[A J]

The following syntax :
import org.apache.cassandra.thrift.*;
.
.
ColumnOrSuperColumn col = client.get("count_key".getBytes("UTF-8"),
cp, ConsistencyLevel.QUORUM);


is giving the error:
get(java.nio.ByteBuffer,org.apache.cassandra.thrift.ColumnPath,org.apache.cassandra.thrift.ConsistencyLevel)
in org.apache.cassandra.thrift.Cassandra.Client cannot be applied to
(byte[],org.apache.cassandra.thrift.ColumnPath,org.apache.cassandra.thrift.ConsistencyLevel)


Any idea on how to cast?

Thanks.

Re: Doubts related to composite type column names/values

[Guy Incognito]

afaik composite lets you do sorting in a way that would be 
difficult/impossible with string concatenation.


eg  with the string ascending, and the integer descending.

if i had composites available (which i don't b/c we are on 0.7), i would 
use them over string concatenation.  string concatenation is a pain.


On 20/12/2011 20:33, Maxim Potekhin wrote:
Thank you Aaron! As long as I have plain strings, would you say that I 
would do almost as well with catenation?


Of course I realize that mixed types are a very different case where 
the composite is very useful.


Thanks

Maxim


On 12/20/2011 2:44 PM, aaron morton wrote:
Component values are compared in a type aware fashion, an Integer is 
an Integer. Not a 10 character zero padded string.


You can also slice on the components. Just like with string concat, 
but nicer.  . e.g. If you app is storing comments for a thing, and 
the column names have the form  or String> you can slice for all properties of a comment or all 
properties for comments between two comment_id's


Finally, the client library knows what's going on.

Hope that helps.

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 7:43 AM, Maxim Potekhin wrote:


With regards to static, what are major benefits as it compares with
string catenation (with some convenient separator inserted)?

Thanks

Maxim


On 12/20/2011 1:39 PM, Richard Low wrote:
On Tue, Dec 20, 2011 at 5:28 PM, Ertio Lew>  wrote:

With regard to the composite columns stuff in Cassandra, I have the
following doubts :

1. What is the storage overhead of the composite type column 
names/values,
The values are the same.  For each dimension, there is 3 bytes 
overhead.


2. what exactly is the difference between the DynamicComposite and 
Static

Composite ?

Static composite type has the types of each dimension specified in the
column family definition, so all names within that column family have
the same type.  Dynamic composite type lets you specify the type for
each column, so they can be different.  There is extra storage
overhead for this and care must be taken to ensure all column names
remain comparable.

Re: Doubts related to composite type column names/values

[Maxim Potekhin]

Thank you Aaron! As long as I have plain strings, would you say that I 
would do almost as well with catenation?


Of course I realize that mixed types are a very different case where the 
composite is very useful.


Thanks

Maxim


On 12/20/2011 2:44 PM, aaron morton wrote:
Component values are compared in a type aware fashion, an Integer is 
an Integer. Not a 10 character zero padded string.


You can also slice on the components. Just like with string concat, 
but nicer.  . e.g. If you app is storing comments for a thing, and the 
column names have the form  or  
you can slice for all properties of a comment or all properties for 
comments between two comment_id's


Finally, the client library knows what's going on.

Hope that helps.

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 7:43 AM, Maxim Potekhin wrote:


With regards to static, what are major benefits as it compares with
string catenation (with some convenient separator inserted)?

Thanks

Maxim


On 12/20/2011 1:39 PM, Richard Low wrote:
On Tue, Dec 20, 2011 at 5:28 PM, Ertio Lew>  wrote:

With regard to the composite columns stuff in Cassandra, I have the
following doubts :

1. What is the storage overhead of the composite type column 
names/values,

The values are the same.  For each dimension, there is 3 bytes overhead.

2. what exactly is the difference between the DynamicComposite and 
Static

Composite ?

Static composite type has the types of each dimension specified in the
column family definition, so all names within that column family have
the same type.  Dynamic composite type lets you specify the type for
each column, so they can be different.  There is extra storage
overhead for this and care must be taken to ensure all column names
remain comparable.

Re: about decommissioned node that returns

[Brandon Williams]

On Tue, Dec 20, 2011 at 2:24 PM, aaron morton  wrote:
> Sorry, got that a little wrong.
>
> At startup the node will use the higher of the current seconds since epoch
> or the stored generation number.

Technically stored generation + 1 so it's always increased on a restart.

-Brandon

Re: about decommissioned node that returns

[aaron morton]

Sorry, got that a little wrong. 

At startup the node will use the higher of the current seconds since epoch or 
the stored generation number. 

So if you restart the old node it should always take ownership of it's token. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 8:31 AM, aaron morton wrote:

>> - What happens in Cassandra with a failed node that has been
>> decommissioned if it returns to the cluster with old data and its old
>> token?
> If it starts normally it *may* take ownership of the token away from the 
> previous node. The Gossip Generation is compared and the server with the 
> higher value is the one that keeps the token. Normally the generation is 
> seeded with the current time when the server first starts and is then 
> increased everytime something significant happens, like a server restart. 
> 
> So if you restarted the old node with the same system CFs, maybe the old node 
> would get ownership of the token. If you restart it after removing the system 
> CFs it will most likely take ownership.
> 
>> - Will it actualize the data using ReadRepair or the bootstrapping process?
> 
> Bootstrap only runs the first time the server starts, technically when data 
> is not present in the system CF's. So if you restart without changing the 
> system CF's use a repair.
> 
>> - And what if it has changed its IP?
> Does not matter. Tokens are the thing. 
> 
> Best thing would be to bring the node back into the the ring as a new clean 
> node. 
> 
> Hope that helps. 
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 21/12/2011, at 5:24 AM, Carlos Pérez Miguel wrote:
> 
>> Hello,
>> 
>> I have several questions about decommissioned nodes that I would like
>> to clarify:
>> 
>> - What happens in Cassandra with a failed node that has been
>> decommissioned if it returns to the cluster with old data and its old
>> token?
>> - Will it actualize the data using ReadRepair or the bootstrapping process?
>> - Or it will be ignored or reconnected with another token?
>> - And what if it has changed its IP?
>> 
>> Thank you
>> 
>> Carlos Pérez Miguel
> 

Re: questions on datastax opscenter

[Tyler Hobbs]

On Tue, Dec 20, 2011 at 1:49 PM, Feng Qu  wrote:

> I have two questions for community version of opscenter
>
> 1) does it work with multiple cassandra cluster?
> 2) are agent ports 61620/61621 configurable? these ports are blocked in
> our env.
>
> Thanks in advance
>
> Feng Qu
>

They are configurable. If you add the following lines to opscenterd.conf
(replacing the default port numbers with your own) and restart opscenterd,
the change should take effect:
[agents]
https_port = 61621
incoming_port = 61620

-- 
Tyler Hobbs
DataStax 

Re: Choosing a Partitioner Type for Random java.util.UUID Row Keys

[Bryce Allen]

I wasn't aware of CompositeColumns, thanks for the tip. However I think
it still doesn't allow me to do the query I need - basically I need to
do a timestamp range query, limiting only to certain file names at
each timestamp. With BOP and a separate row for each timestamp,
prefixed by a random UUID, and file names as column names, I can do this
query. With CompositeColumns, I can only query one contiguous range, so
I'd have to know the timestamps before hand to limit the file names. I
can resolve this using indexes, but on paper it looks like this would be
significantly slower (it would take me 5 round trips instead of 3 to
complete each query, and the query is made multiple times on every
single client request).

The two down sides I've seen listed for BOP are balancing issues and
hotspots. I can understand why RP is recommended, from the balancing
issues alone. However these aren't problems for my application. Is
there anything else I am missing? Does the Cassandra team plan on
continuing to support BOP? I haven't completely ruled out RP, but I
like having BOP as an option, it opens up interesting modeling
alternatives that I think have real advantages for some
(if uncommon) applications.

Thanks,
Bryce

On Wed, 21 Dec 2011 08:08:16 +1300
aaron morton  wrote:
> Bryce, 
>   Have you considered using CompositeColumns and a standard CF?
> Row key is the UUID column name is (timestamp : dir_entry) you can
> then slice all columns with a particular time stamp. 
> 
>   Even if you have a random key, I would use the RP unless you
> have an extreme use case. 
> 
>  Cheers
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 21/12/2011, at 3:06 AM, Bryce Allen wrote:
> 
> > I think it comes down to how much you benefit from row range scans,
> > and how confident you are that going forward all data will continue
> > to use random row keys.
> > 
> > I'm considering using BOP as a way of working around the non indexes
> > super column limitation. In my current schema, row keys are random
> > UUIDs, super column names are timestamps, and columns contain a
> > snapshot in time of directory contents, and could be quite large. If
> > instead I use row keys that are (uuid)-(timestamp), and use a
> > standard column family, I can do a row range query and select only
> > specific columns. I'm still evaluating if I can do this with BOP -
> > ideally the token would just use the first 128 bits of the key, and
> > I haven't found any documentation on how it compares keys of
> > different length.
> > 
> > Another trick with BOP is to use MD5(rowkey)-rowkey for data that
> > has non uniform row keys. I think it's reasonable to use if most
> > data is uniform and benefits from range scans, but a few things are
> > added that aren't/don't. This trick does make the keys larger,
> > which increases storage cost and IO load, so it's probably a bad
> > idea if a significant subset of the data requires it.
> > 
> > Disclaimer - I wrote that wiki article to fill in a documentation
> > gap, since there were no examples of BOP and I wasted a lot of time
> > before I noticed the hex byte array vs decimal distinction for
> > specifying the initial tokens (which to be fair is documented, just
> > easy to miss on a skim). I'm also new to cassandra, I'm just
> > describing what makes sense to me "on paper". FWIW I confirmed that
> > random UUIDs (type 4) row keys really do evenly distribute when
> > using BOP.
> > 
> > -Bryce
> > 
> > On Mon, 19 Dec 2011 19:01:00 -0800
> > Drew Kutcharian  wrote:
> >> Hey Guys,
> >> 
> >> I just came across
> >> http://wiki.apache.org/cassandra/ByteOrderedPartitioner and it got
> >> me thinking. If the row keys are java.util.UUID which are generated
> >> randomly (and securely), then what type of partitioner would be the
> >> best? Since the key values are already random, would it make a
> >> difference to use RandomPartitioner or one can use
> >> ByteOrderedPartitioner or OrderPreservingPartitioning as well and
> >> get the same result?
> >> 
> >> -- Drew
> >> 
> 


signature.asc
Description: PGP signature

questions on datastax opscenter

[Feng Qu]

I have two questions for community version of opscenter

1) does it work with multiple cassandra cluster?
2) are agent ports 61620/61621 configurable? these ports are blocked in our 
env. 

Thanks in advance
 
Feng Qu

Re: Doubts related to composite type column names/values

[aaron morton]

Component values are compared in a type aware fashion, an Integer is an 
Integer. Not a 10 character zero padded string. 

You can also slice on the components. Just like with string concat, but nicer.  
. e.g. If you app is storing comments for a thing, and the column names have 
the form  or   you can slice for all 
properties of a comment or all properties for comments between two comment_id's

Finally, the client library knows what's going on. 

Hope that helps.

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 7:43 AM, Maxim Potekhin wrote:

> With regards to static, what are major benefits as it compares with
> string catenation (with some convenient separator inserted)?
> 
> Thanks
> 
> Maxim
> 
> 
> On 12/20/2011 1:39 PM, Richard Low wrote:
>> On Tue, Dec 20, 2011 at 5:28 PM, Ertio Lew  wrote:
>>> With regard to the composite columns stuff in Cassandra, I have the
>>> following doubts :
>>> 
>>> 1. What is the storage overhead of the composite type column names/values,
>> The values are the same.  For each dimension, there is 3 bytes overhead.
>> 
>>> 2. what exactly is the difference between the DynamicComposite and Static
>>> Composite ?
>> Static composite type has the types of each dimension specified in the
>> column family definition, so all names within that column family have
>> the same type.  Dynamic composite type lets you specify the type for
>> each column, so they can be different.  There is extra storage
>> overhead for this and care must be taken to ensure all column names
>> remain comparable.
>> 
> 

Re: about decommissioned node that returns

[aaron morton]

> - What happens in Cassandra with a failed node that has been
> decommissioned if it returns to the cluster with old data and its old
> token?
If it starts normally it *may* take ownership of the token away from the 
previous node. The Gossip Generation is compared and the server with the higher 
value is the one that keeps the token. Normally the generation is seeded with 
the current time when the server first starts and is then increased everytime 
something significant happens, like a server restart. 

So if you restarted the old node with the same system CFs, maybe the old node 
would get ownership of the token. If you restart it after removing the system 
CFs it will most likely take ownership.

> - Will it actualize the data using ReadRepair or the bootstrapping process?

Bootstrap only runs the first time the server starts, technically when data is 
not present in the system CF's. So if you restart without changing the system 
CF's use a repair.

> - And what if it has changed its IP?
Does not matter. Tokens are the thing. 

Best thing would be to bring the node back into the the ring as a new clean 
node. 

Hope that helps. 

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 5:24 AM, Carlos Pérez Miguel wrote:

> Hello,
> 
> I have several questions about decommissioned nodes that I would like
> to clarify:
> 
> - What happens in Cassandra with a failed node that has been
> decommissioned if it returns to the cluster with old data and its old
> token?
> - Will it actualize the data using ReadRepair or the bootstrapping process?
> - Or it will be ignored or reconnected with another token?
> - And what if it has changed its IP?
> 
> Thank you
> 
> Carlos Pérez Miguel

Re: memory estimate for each key in the key cache

[aaron morton]

If you get the sizing wrong there area couple of emergency pressure valves in 
the config…
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L113

Watch for log messages with "Cassandra is now reducing cache sizes to free up 
memory." in them. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 5:21 AM, Kent Tong wrote:

>>  It is not telling you to multiply your key size by 10-12, it is telling you 
>> to
>> multiply the output of the nodetool cfstats reported "key cache size" by 
>> 10-12.
> 
> 
> The  "key cache size" reported is actually the number of keys in the key 
> cache. So,
> it is the same thing as suggesting each key takes 10-12 bytes.

Re: Choosing a Partitioner Type for Random java.util.UUID Row Keys

[aaron morton]

Bryce, 
Have you considered using CompositeColumns and a standard CF? Row key 
is the UUID column name is (timestamp : dir_entry) you can then slice all 
columns with a particular time stamp. 

Even if you have a random key, I would use the RP unless you have an 
extreme use case. 

 Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/12/2011, at 3:06 AM, Bryce Allen wrote:

> I think it comes down to how much you benefit from row range scans, and
> how confident you are that going forward all data will continue to use
> random row keys.
> 
> I'm considering using BOP as a way of working around the non indexes
> super column limitation. In my current schema, row keys are random
> UUIDs, super column names are timestamps, and columns contain a
> snapshot in time of directory contents, and could be quite large. If
> instead I use row keys that are (uuid)-(timestamp), and use a standard
> column family, I can do a row range query and select only specific
> columns. I'm still evaluating if I can do this with BOP - ideally the
> token would just use the first 128 bits of the key, and I haven't found
> any documentation on how it compares keys of different length.
> 
> Another trick with BOP is to use MD5(rowkey)-rowkey for data that has
> non uniform row keys. I think it's reasonable to use if most data is
> uniform and benefits from range scans, but a few things are added that
> aren't/don't. This trick does make the keys larger, which increases
> storage cost and IO load, so it's probably a bad idea if a significant
> subset of the data requires it.
> 
> Disclaimer - I wrote that wiki article to fill in a documentation gap,
> since there were no examples of BOP and I wasted a lot of time before I
> noticed the hex byte array vs decimal distinction for specifying the
> initial tokens (which to be fair is documented, just easy to miss on a
> skim). I'm also new to cassandra, I'm just describing what makes sense
> to me "on paper". FWIW I confirmed that random UUIDs (type 4) row keys
> really do evenly distribute when using BOP.
> 
> -Bryce
> 
> On Mon, 19 Dec 2011 19:01:00 -0800
> Drew Kutcharian  wrote:
>> Hey Guys,
>> 
>> I just came across
>> http://wiki.apache.org/cassandra/ByteOrderedPartitioner and it got me
>> thinking. If the row keys are java.util.UUID which are generated
>> randomly (and securely), then what type of partitioner would be the
>> best? Since the key values are already random, would it make a
>> difference to use RandomPartitioner or one can use
>> ByteOrderedPartitioner or OrderPreservingPartitioning as well and get
>> the same result?
>> 
>> -- Drew
>> 

Re: Counters != Counts

[aaron morton]

Yes you can do a rolling restart. For more info on the counters architecture 
see the presentation here 
http://www.datastax.com/events/cassandrasf2011/presentations

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 20/12/2011, at 10:10 PM, Alain RODRIGUEZ wrote:

> By the way, I wonder if a rolling restart is still possible when I am using 
> counters ? Don't we lose the SPOF too, considering that counters are wrong 
> every time we retry to write them with the client ?
> 
> 
> 2011/12/16 Alain RODRIGUEZ 
> Can we have a hope that counters will be replayed as safely as a classical 
> data someday ? Do someone still work on jiras like 
> issues.apache.org/jira/browse/CASSANDRA-2495 ? I thought that replaying a 
> write from the client didn't lead to over-counts contrary to the internal 
> cassandra replay from commitlog.
> 
> I just made a new connection pool with retries / 2 and timeouts * 4. I hope 
> it will improve the accuracy of my counters.
> 
> Anyways, thank you for answering that fast.
> 
> Alain 
> 
> 2011/12/16 Tyler Hobbs 
> Probably quite a few of them are coming from automatic retries by phpcassa.  
> When working with counters, I recommend minimizing retries and/or increasing 
> timeouts.  Usually this means you want to use a separate connection pool with 
> different settings just for counters.
> 
> By the way, this advice applies to other clients as well.
> 
> 
> On Wed, Dec 14, 2011 at 10:29 AM, Alain RODRIGUEZ  wrote:
> Hi everybody.
> 
> I'm using a lot of counters to make statistics on a 4 nodes cluster (ec2 
> m1.small) with phpcassa (cassandra v1.0.2).
> 
> I store some events and increment counters at the same time.
> 
> Counters give me over-counts compared with the count of every corresponding 
> events.
> 
> I sure that my non-counters counts are good.
> 
> I'm not sure why these over-counts happen, but I heard that recovering from 
> commitlogs can produce this.
> I have some timeouts on phpcassa which are written in my apache logs while a 
> compaction is running. However I am always able to write at Quorum, so I 
> guess I shouldn't have to recover from cassandra commitlogs. 
> 
> Where can these over-counts come from ?
> 
> Alain
>  
> 
> 
> 
> -- 
> Tyler Hobbs
> DataStax
> 
> 
> 

Re: setStrategy_options syntax in thrift

[aaron morton]

It looks like you tried to pass the string "{replication_factor:2}" 

You need to pas a Map type , where the the key is the option 
and the value is the option value.

Cheers
 
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 20/12/2011, at 12:02 PM, A J wrote:

> What is the syntax of setStrategy_options in thrift.
> 
> The following fails:
> 
> Util.java:22: 
> setStrategy_options(java.util.Map)
> in org.apache.cassandra.thrift.KsDef cannot be applied to
> (java.lang.String)
>newKs.setStrategy_options("{replication_factor:2}");

Re: Doubts related to composite type column names/values

[Maxim Potekhin]

With regards to static, what are major benefits as it compares with
string catenation (with some convenient separator inserted)?

Thanks

Maxim


On 12/20/2011 1:39 PM, Richard Low wrote:

On Tue, Dec 20, 2011 at 5:28 PM, Ertio Lew  wrote:

With regard to the composite columns stuff in Cassandra, I have the
following doubts :

1. What is the storage overhead of the composite type column names/values,

The values are the same.  For each dimension, there is 3 bytes overhead.


2. what exactly is the difference between the DynamicComposite and Static
Composite ?

Static composite type has the types of each dimension specified in the
column family definition, so all names within that column family have
the same type.  Dynamic composite type lets you specify the type for
each column, so they can be different.  There is extra storage
overhead for this and care must be taken to ensure all column names
remain comparable.

Re: Doubts related to composite type column names/values

[Richard Low]

On Tue, Dec 20, 2011 at 5:28 PM, Ertio Lew  wrote:
> With regard to the composite columns stuff in Cassandra, I have the
> following doubts :
>
> 1. What is the storage overhead of the composite type column names/values,

The values are the same.  For each dimension, there is 3 bytes overhead.

> 2. what exactly is the difference between the DynamicComposite and Static
> Composite ?

Static composite type has the types of each dimension specified in the
column family definition, so all names within that column family have
the same type.  Dynamic composite type lets you specify the type for
each column, so they can be different.  There is extra storage
overhead for this and care must be taken to ensure all column names
remain comparable.

-- 
Richard Low
Acunu | http://www.acunu.com | @acunu

Doubts related to composite type column names/values

[Ertio Lew]

With regard to the composite columns stuff in Cassandra, I have the
following doubts :

1. What is the storage overhead of the composite type column names/values,
and

2. what exactly is the difference between the DynamicComposite and Static
Composite ?

about decommissioned node that returns

[Carlos Pérez Miguel]

Hello,

I have several questions about decommissioned nodes that I would like
to clarify:

- What happens in Cassandra with a failed node that has been
decommissioned if it returns to the cluster with old data and its old
token?
- Will it actualize the data using ReadRepair or the bootstrapping process?
- Or it will be ignored or reconnected with another token?
- And what if it has changed its IP?

Thank you

Carlos Pérez Miguel

Re: memory estimate for each key in the key cache

[Kent Tong]

> It is not telling you to multiply your key size by 10-12, it is telling you to
> multiply the output of the nodetool cfstats reported "key cache size" by 
> 10-12.


The  "key cache size" reported is actually the number of keys in the key cache. 
So,
it is the same thing as suggesting each key takes 10-12 bytes.

Re: Choosing a Partitioner Type for Random java.util.UUID Row Keys

[Bryce Allen]

I think it comes down to how much you benefit from row range scans, and
how confident you are that going forward all data will continue to use
random row keys.

I'm considering using BOP as a way of working around the non indexes
super column limitation. In my current schema, row keys are random
UUIDs, super column names are timestamps, and columns contain a
snapshot in time of directory contents, and could be quite large. If
instead I use row keys that are (uuid)-(timestamp), and use a standard
column family, I can do a row range query and select only specific
columns. I'm still evaluating if I can do this with BOP - ideally the
token would just use the first 128 bits of the key, and I haven't found
any documentation on how it compares keys of different length.

Another trick with BOP is to use MD5(rowkey)-rowkey for data that has
non uniform row keys. I think it's reasonable to use if most data is
uniform and benefits from range scans, but a few things are added that
aren't/don't. This trick does make the keys larger, which increases
storage cost and IO load, so it's probably a bad idea if a significant
subset of the data requires it.

Disclaimer - I wrote that wiki article to fill in a documentation gap,
since there were no examples of BOP and I wasted a lot of time before I
noticed the hex byte array vs decimal distinction for specifying the
initial tokens (which to be fair is documented, just easy to miss on a
skim). I'm also new to cassandra, I'm just describing what makes sense
to me "on paper". FWIW I confirmed that random UUIDs (type 4) row keys
really do evenly distribute when using BOP.

-Bryce

On Mon, 19 Dec 2011 19:01:00 -0800
Drew Kutcharian  wrote:
> Hey Guys,
> 
> I just came across
> http://wiki.apache.org/cassandra/ByteOrderedPartitioner and it got me
> thinking. If the row keys are java.util.UUID which are generated
> randomly (and securely), then what type of partitioner would be the
> best? Since the key values are already random, would it make a
> difference to use RandomPartitioner or one can use
> ByteOrderedPartitioner or OrderPreservingPartitioning as well and get
> the same result?
> 
> -- Drew
> 


signature.asc
Description: PGP signature

Re: Choosing a Partitioner Type for Random java.util.UUID Row Keys

[Filipe Gonçalves]

Generally, RandomPartitioner is the recommended one.
If you already provide randomized keys it doesn't make much of a
difference, the nodes should be balanced with any partitioner.
However, unless you have UUID in all keys of all column families
(highly unlikely) ByteOrderedPartitioner and
OrderPreservingPartitioning will lead to hotspots and unbalanced
rings.

2011/12/20 Drew Kutcharian :
> Hey Guys,
>
> I just came
> across http://wiki.apache.org/cassandra/ByteOrderedPartitioner and it got me
> thinking. If the row keys are java.util.UUID which are generated randomly
> (and securely), then what type of partitioner would be the best? Since the
> key values are already random, would it make a difference to use
> RandomPartitioner or one can use ByteOrderedPartitioner or
> OrderPreservingPartitioning as well and get the same result?
>
> -- Drew
>



-- 
Filipe Gonçalves

RE: Drop schema Called "Index"

[Michael Vaknine]

Any way I was able to delete the keyspace using python pycassa 

 

Thanks

Michael

 

From: aaron morton [mailto:aa...@thelastpickle.com] 
Sent: Monday, December 19, 2011 9:39 PM
To: user@cassandra.apache.org
Subject: Re: Drop schema Called "Index"

 

Which keyspace are you trying to delete ? 

 

What happens when you use drop ; in cassandra-cli ? 

 

If nothing happens is anything logged in the server side logs  ? 

 

Cheers

 

-

Aaron Morton

Freelance Developer

@aaronmorton

http://www.thelastpickle.com

 

On 19/12/2011, at 11:48 PM, Michael Vaknine wrote:





Hi,

 

I had Cassandra version 0.7.6 and a schema called Index.

When I upgraded to 1.0.3 I realized that I cant create this keyspace maybe
because Index has become a reserved word?

I tried 'Index' and "Index" and still get an error so I created a new
keyspace instead.

Now I am not able to delete this.

 

Can someone tell me how can I delete this keyspace?

 

Thanks

Michael

 

Re: Counters != Counts

[Alain RODRIGUEZ]

By the way, I wonder if a rolling restart is still possible when I am using
counters ? Don't we lose the SPOF too, considering that counters are wrong
every time we retry to write them with the client ?


2011/12/16 Alain RODRIGUEZ 

> Can we have a hope that counters will be replayed as safely as a classical
> data someday ? Do someone still work on jiras like
> issues.apache.org/jira/browse/CASSANDRA-2495 ? I thought that replaying a
> write from the client didn't lead to over-counts contrary to the internal
> cassandra replay from commitlog.
>
> I just made a new connection pool with retries / 2 and timeouts * 4. I
> hope it will improve the accuracy of my counters.
>
> Anyways, thank you for answering that fast.
>
> Alain
>
> 2011/12/16 Tyler Hobbs 
>
>> Probably quite a few of them are coming from automatic retries by
>> phpcassa.  When working with counters, I recommend minimizing retries
>> and/or increasing timeouts.  Usually this means you want to use a separate
>> connection pool with different settings just for counters.
>>
>> By the way, this advice applies to other clients as well.
>>
>>
>> On Wed, Dec 14, 2011 at 10:29 AM, Alain RODRIGUEZ wrote:
>>
>>> Hi everybody.
>>>
>>> I'm using a lot of counters to make statistics on a 4 nodes cluster (ec2
>>> m1.small) with phpcassa (cassandra v1.0.2).
>>>
>>> I store some events and increment counters at the same time.
>>>
>>> Counters give me over-counts compared with the count of every
>>> corresponding events.
>>>
>>> I sure that my non-counters counts are good.
>>>
>>> I'm not sure why these over-counts happen, but I heard that recovering
>>> from commitlogs can produce this.
>>> I have some timeouts on phpcassa which are written in my apache logs
>>> while a compaction is running. However I am always able to write at Quorum,
>>> so I guess I shouldn't have to recover from cassandra commitlogs.
>>>
>>> Where can these over-counts come from ?
>>>
>>> Alain
>>>
>>>
>>
>>
>>
>> --
>> Tyler Hobbs
>> DataStax 
>>
>>
>