Re: Uneven distribution using RP

2010-06-23 Thread Eric Evans
On Tue, 2010-06-22 at 17:47 -0400, James Golick wrote: > It's also flushing memtables really quickly for a particular CF. Like, > really quickly. Like, one every minute. I increased the thresholds by > 10x and it's still going fast. What is MemtableFlushAfterMinutes set to? -- Eric Evans eev...@

Re: Uneven distribution using RP

2010-06-23 Thread aaron morton
That sounds odd: - do you have an idea of how many inserts per second / minute you think you are sending? - Is is just one node thats flushing mem tables fast? - Can you send the column family for the super busy CF (nodetool cfstats) - Can you send the storage-conf.xml - Can you send the nodetool

Re: Uneven distribution using RP

2010-06-22 Thread James Golick
It's compacting at a ridiculously fast rate. The pending compactions have been growing for a while. It's also flushing memtables really quickly for a particular CF. Like, really quickly. Like, one every minute. I increased the thresholds by 10x and it's still going fast. On Tue, Jun 22, 2010 at 5

Re: Uneven distribution using RP

2010-06-22 Thread Jeremy Dunck
On Tue, Jun 22, 2010 at 4:08 PM, James Golick wrote: > Turns out that this is due to a larger proportion of the wide rows in the > system being located on that node. I moved its token over a little to > compensate for it, but it doesn't seem to have helped at this point. > What's confusing about t

Re: Uneven distribution using RP

2010-06-22 Thread James Golick
Turns out that this is due to a larger proportion of the wide rows in the system being located on that node. I moved its token over a little to compensate for it, but it doesn't seem to have helped at this point. What's confusing about this is that RF=3 and no other node's load is growing as quick

Re: Uneven distribution using RP

2010-06-22 Thread James Golick
RackUnaware, currently On Tue, Jun 22, 2010 at 1:26 PM, Robert Coli wrote: > On 6/22/10 10:07 AM, James Golick wrote: > >> This node's load is now growing at a ridiculous rate. It is at 105GB, with >> the next most loaded node at 70.63GB. >> >> Given that RF=3, I would assume that the replicas'

Re: Uneven distribution using RP

2010-06-22 Thread Robert Coli
On 6/22/10 10:07 AM, James Golick wrote: This node's load is now growing at a ridiculous rate. It is at 105GB, with the next most loaded node at 70.63GB. Given that RF=3, I would assume that the replicas' nodes would grow relatively quickly too? What Replica Placement Strategy are you using (R

Re: Uneven distribution using RP

2010-06-22 Thread James Golick
This node's load is now growing at a ridiculous rate. It is at 105GB, with the next most loaded node at 70.63GB. Given that RF=3, I would assume that the replicas' nodes would grow relatively quickly too? On Mon, Jun 21, 2010 at 6:44 AM, aaron morton wrote: > According to http://wiki.apache.org/

Re: Uneven distribution using RP

2010-06-21 Thread aaron morton
According to http://wiki.apache.org/cassandra/Operations nodetool repair is used to perform a major compaction and compare data between the nodes, repairing any conflicts. Not sure that would improve the load balance, though it may reduce some wasted space on the nodes. nodetool loadbalance wil

Re: Uneven distribution using RP

2010-06-20 Thread James Golick
I know, but that's not a big enough difference to warrant the huge amount of difference in load. On Sun, Jun 20, 2010 at 10:53 AM, Jordan Pittier - Rezel wrote: > Node 1 should have token 42535295865117307932921825928971026432 and node > 3 127605887595351923798765477786913079296 according to the

Re: Uneven distribution using RP

2010-06-20 Thread Jordan Pittier - Rezel
Node 1 should have token 42535295865117307932921825928971026432 and node 3 127605887595351923798765477786913079296 according to the formula i * (2**127 / 4) for i=1..4 On Sun, Jun 20, 2010 at 4:31 PM, James Golick wrote: > I ran cleanup on all of them and the distribution looked roughly even aft

Re: Uneven distribution using RP

2010-06-20 Thread James Golick
I ran cleanup on all of them and the distribution looked roughly even after that, but a couple of days later, it's looking pretty uneven. On Sun, Jun 20, 2010 at 10:21 AM, Jordan Pittier - Rezel wrote: > Hi, > Have you tried nodetool repair (or cleanup) on your nodes ? > > > On Sun, Jun 20, 2010

Re: Uneven distribution using RP

2010-06-20 Thread Jordan Pittier - Rezel
Hi, Have you tried nodetool repair (or cleanup) on your nodes ? On Sun, Jun 20, 2010 at 4:16 PM, James Golick wrote: > I just increased my cluster from 2 to 4 nodes, and RF=2 to RF=3, using RP. > > The tokens seem pretty even on the ring, but two of the nodes are far more > heavily loaded than t

Uneven distribution using RP

2010-06-20 Thread James Golick
I just increased my cluster from 2 to 4 nodes, and RF=2 to RF=3, using RP. The tokens seem pretty even on the ring, but two of the nodes are far more heavily loaded than the others. I understand that there are a variety of possible reasons for this, but I'm wondering whether anybody has suggestion