Re: frequent node up/downs

2012-07-06 Thread feedly team
responses below. thanks! On Fri, Jul 6, 2012 at 3:09 PM, aaron morton wrote: > It looks like this happens when there is a promotion failure. > > > Java Heap is full. > Memory is fragmented. > Use C for web scale. > unfortunately i became too dumb to use C around 2004. camping accident. > > Also

Re: frequent node up/downs

2012-07-06 Thread aaron morton
> It looks like this happens when there is a promotion failure. Java Heap is full. Memory is fragmented. Use C for web scale. > Also is it normal to see the "Heap is xx full. You may need to reduce > memtable and/or cache sizes" message quite often? I haven't turned on row > caches or chan

Re: frequent node up/downs

2012-07-06 Thread feedly team
I reduced the load and the problem hasn't been happening as much. After enabling gc logging, I see messages mentioning promotion failed when the pauses happen. It looks like this happens when there is a promotion failure. From reading on the web it looks like I could try reducing the CMSInitiatingO

Re: frequent node up/downs

2012-07-04 Thread aaron morton
> What accounts for the much larger virtual number? some kind of off-heap > memory? http://wiki.apache.org/cassandra/FAQ#mmap > I'm a little puzzled as to why I would get such long pauses without swapping. The two are not related. On startup the JVM memory is locked so it will not swap, from t

Re: frequent node up/downs

2012-07-02 Thread feedly team
Couple more details. I confirmed that swap space is not being used (free -m shows 0 swap) and cassandra.log has a message like "JNA mlockall successful". top shows the process having 9g in resident memory but 21.6g in virtual...What accounts for the much larger virtual number? some kind of off-heap

Re: frequent node up/downs

2012-07-02 Thread feedly team
Yeah I noticed the leap second problem and ran the suggested fix, but I have been facing these problems before Saturday and still see the occasional failures after running the fix. Thanks. On Mon, Jul 2, 2012 at 11:17 AM, Marcus Both wrote: > Yeah! Look that. > > http://arstechnica.com/business

Re: frequent node up/downs

2012-07-02 Thread Marcus Both
Yeah! Look that. http://arstechnica.com/business/2012/07/one-day-later-the-leap-second-v-the-internet-scorecard/ I had the same problem. The solution was rebooting. On Mon, 2 Jul 2012 11:08:57 -0400 feedly team wrote: > Hello, >I recently set up a 2 node cassandra cluster on dedicated hardwa

RE: frequent node up/downs

2012-07-02 Thread Pieter Callewaert
Hi, Had the same problem this morning, seems related to the leap second bug. Rebooting the nodes fixed it for me, but there seems to be a fix also without rebooting the server. Kind regards, Pieter From: feedly team [mailto:feedly...@gmail.com] Sent: maandag 2 juli 2012 17:09 To: user@cassandra