Re: Cassandra 1.1.6 - New node bootstrap not completing

2013-10-29 Thread Narendra Sharma
Thanks Robert. I didn't realize that some of the keyspaces (not all and esp. the biggest one I was focusing on) had RF > 2. I wasted 3 days on it. Thanks again for the pointers. I will try again and share the results. On Wed, Oct 30, 2013 at 12:28 AM, Robert Coli wrote: > On Tue, Oct 29, 2013

heap issues - looking for advices on gc tuning

2013-10-29 Thread Piavlo
Hi, Below I try to give a full picture to the problem I'm facing. This is a 12 node cluster, running on ec2 with m2.xlarge instances (17G ram , 2 cpus). Cassandra version is 1.0.8 Cluster normally having between 3000 - 1500 reads per second (depends on time of the day) and 1700 - 800 writes p

ReadCount change rate is different across nodes

2013-10-29 Thread Daning Wang
We are running 1.2.5 on 8 nodes(256 tokens). all the nodes are running on same type of machine. and db size is about same. but recently we checked ReadCount stats through jmx, and found that some nodes got 3 times change rate(we have calculated the changes per minute) than others. We are using h

Re: Recovering from mystical cassandra errors.

2013-10-29 Thread Robert Coli
On Tue, Oct 29, 2013 at 4:40 PM, Jacob Rhoden wrote: > Im testing out Cassandra with a 2.0.1 install [and having this issue where > nodes won't start after being stopped] > Someone else having what looks like this same issue (in C* 2.0.2 iirc) came into #cassandra yesterday. I encourage you to

Re: not even number of keys per CFs in fully balanced cluster with random partitioner

2013-10-29 Thread Robert Coli
On Tue, Oct 29, 2013 at 4:39 PM, Robert Coli wrote: > On Tue, Oct 29, 2013 at 4:09 PM, Piavlo wrote: > >> I can't understand how statistically it's possible that with random >> partitioner some nodes have x2 more keys than others with 50-100 millions >> of keys per node. >> > > 1) you have very

Recovering from mystical cassandra errors.

2013-10-29 Thread Jacob Rhoden
Hi Guys, Im testing out Cassandra with a 2.0.1 install, just like with 1.2.8 I last tested, from time to time I am coming across errors that prevent Cassandra starting up. For example: Yesterday I installed this instance, created and removed some “tables” using cql3 and inserted/updated/trunca

Re: not even number of keys per CFs in fully balanced cluster with random partitioner

2013-10-29 Thread Robert Coli
On Tue, Oct 29, 2013 at 4:09 PM, Piavlo wrote: > I can't understand how statistically it's possible that with random > partitioner some nodes have x2 more keys than others with 50-100 millions > of keys per node. > 1) you have very high row variance and really really bad luck with the distributi

not even number of keys per CFs in fully balanced cluster with random partitioner

2013-10-29 Thread Piavlo
Hi, There is a 12 node cluster , still stuck on 1.0.8. All nodes in the cluster ring are balanced. Using random partitioner. All CFs use compression. Data size on nodes varies from 40G to 75G. This variance is not due to the bigger nodes having more uncompacted sstables than others. Most bigges

Re: Cassandra 1.1.6 - New node bootstrap not completing

2013-10-29 Thread Robert Coli
On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma wrote: > We had a cluster of 4 nodes in AWS. The average load on each node was > approx 750GB. We added 4 new nodes. It is now more than 30 hours and the > node is still in JOINING mode. > Specifically I am analyzing the one with IP 10.3.1.29. The

Cassandra 1.1.6 - New node bootstrap not completing

2013-10-29 Thread Narendra Sharma
We had a cluster of 4 nodes in AWS. The average load on each node was approx 750GB. We added 4 new nodes. It is now more than 30 hours and the node is still in JOINING mode. Specifically I am analyzing the one with IP 10.3.1.29. There is no compaction or streaming or index building happening. $ ./

RE: OpsCenter not connecting to Cluster

2013-10-29 Thread Pieter Callewaert
Hi Nigel, I've currently have a simular problem. However, it has only been reproduced on Ubuntu... Are you using hsha as rpc_server_type? http://stackoverflow.com/questions/19633980/adding-cluster-error-creating-cluster-call-to-cluster-configs-timed-out is a guy with the same problem, showing h

Re: Example Cassandra 2.0 trigger classes?

2013-10-29 Thread Robert Coli
On Mon, Oct 28, 2013 at 7:24 PM, Jacob Rhoden wrote: > 1. Before I slog my way through the documentation on using triggers? Is > anyone aware of any real world usage sample code or blogs with examples on > this? > I feel compelled to mention that currently, triggers are explicitly "experimental"

Re: IllegalStateException when bootstrapping new nodes

2013-10-29 Thread Robert Coli
On Tue, Oct 29, 2013 at 7:38 AM, Cyril Scetbon wrote: > We didn't find the reason why it didn't work but we are wondering if it's > a BUG. We are using Cassandra 1.2.2 and we resolved the issue with a > rolling restart of other nodes in the same DC and maybe with some luck... > We don't know if it

OpsCenter not connecting to Cluster

2013-10-29 Thread Nigel LEACH
Cassandra 2.0.1 OpsCenter 3.2.2 Java 1.7.0_25 RHEL 6.4 This is a new test cluster with just three nodes, two seed nodes, SSL turned off, and GossipingPropertyFileSnitch. Pretty much out of the box environment, with both Cassandra and OpsCenter via DataStax yum repository. Cassandra seems fine,

Re: Too many open files (Cassandra 2.0.1)

2013-10-29 Thread Jon Haddad
In general, my understanding is that memory mapped files use a lot of open file handlers. We raise all our DBs to unlimited open files. On Oct 29, 2013, at 8:30 AM, Pieter Callewaert wrote: > Investigated a bit more: > > -I can reproduce it, happened already on several nodes when I

Re: IllegalStateException when bootstrapping new nodes

2013-10-29 Thread Cyril Scetbon
Sorry but as the link is bad here is the good one : http://www.sendspace.com/file/7p81lz -- Cyril SCETBON On 29 Oct 2013, at 15:38, Cyril Scetbon wrote: > Hi, > > We added yesterday 4 nodes in one of our data-centers but met an exception on > 2 of them : > > java.lang.IllegalStateException:

RE: Too many open files (Cassandra 2.0.1)

2013-10-29 Thread Pieter Callewaert
Investigated a bit more: -I can reproduce it, happened already on several nodes when I do some stress testing (5 select's spread over multiple threads) -Unexpected exception in the selector loop. Seems not related with the Too many open files, it just happens. -It'

Re: [RELEASE] Apache Cassandra 2.0.2 released

2013-10-29 Thread Chris Burroughs
On 10/28/2013 06:20 AM, Sylvain Lebresne wrote: [2]:http://goo.gl/uEtkmb (NEWS.txt) https://wiki.apache.org/cassandra/Metrics has been updated with a reference to the new Configurable metrics reporting.

IllegalStateException when bootstrapping new nodes

2013-10-29 Thread Cyril Scetbon
Hi, We added yesterday 4 nodes in one of our data-centers but met an exception on 2 of them : java.lang.IllegalStateException: unable to find sufficient sources for streaming range (6358673204990085385,6363273598121398976] at org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(Rang

Re: CQL & selecting individual items from a map

2013-10-29 Thread Keith Freeman
There's some rationale here: http://mail-archives.apache.org/mod_mbox/cassandra-user/201305.mbox/%3CCAENxBwx6pcSA=cWn=dkw_52k5odw5f3xigj-zn_4bwfth+4...@mail.gmail.com%3E And I'm sure part of the reason is the 64k size limit: maps (and sets and lists) are limited to 64k total size (http://wiki.

Re: Query a datacenter

2013-10-29 Thread srmore
Thanks Rob that helps ! On Fri, Oct 25, 2013 at 7:34 PM, Robert Coli wrote: > On Fri, Oct 25, 2013 at 2:47 PM, srmore wrote: > >> I don't know whether this is possible but was just curious, can you query >> for the data in the remote datacenter with a CL.ONE ? >> > > A coordinator at CL.ONE pi

Too many open files (Cassandra 2.0.1)

2013-10-29 Thread Pieter Callewaert
Hi, I've noticed some nodes in our cluster are dying after some period of time. WARN [New I/O server boss #17] 2013-10-29 12:22:20,725 Slf4JLogger.java (line 76) Failed to accept a connection. java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native M

RE: Read repair

2013-10-29 Thread Baskar Duraikannu
Aaron Rack1 goes down and some writes happen in quorum against rack 2 and 3. Hinted handoff is set to 30 mins. After couple of hours rack1 comes back and rack2 goes down. Hinted handoff will play but will not cover all of the writes because of 30 min setting. Now for rows inserted for about 1 h

Too many open files with Cassandra 1.2.11

2013-10-29 Thread Oleg Dulin
Got this error: WARN [Thread-8] 2013-10-29 02:58:24,565 CustomTThreadPoolServer.java (line 122) Transport error occurred during acceptance of message. 2 org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files 3 at org.apache.cassandra.thri