Re: Nodes stopping

I'm switching the instances to machines with 61G of RAM, in this case would you still recommend using 8G of heap space?Here is a gist of my heap settings from jvm.optionshttps://gist.github.com/dlsteuer/40e80280029897e6bb5fd12f2a86cbbe On May 11 2017, at 3:08

Re: Nodes stopping

Hi Daniel, Could you paste the exact GC options in use? Also 30 GB is not much. I would not use more than 8 GB for the JVM and probably CMS in those conditions for what it is worth. The thing is if memtables, bloom filter, caches, indexes, etc are off heap, then you probably ran out of Native

Re: Nodes stopping

Thank you, it's an Out of memory crash according to dmesg. I have the heap size set to 15G in the jvm.options for cassandra, and there is 30G on the machine. On May 11 2017, at 2:22 pm, Cogumelos Maravilha wrote:

Re: Nodes stopping

I had 2 nodes go down today, here is the ERRORs from the system log on both nodeshttps://gist.github.com/dlsteuer/28c610bc733a2bff22c8d3953ef8c218For some context, I'm trying to get regular repairs going but am having issues with it. On May 11 2017, at 2:10 pm,

Re: Nodes stopping

Can you grep ERROR system.log On 11-05-2017 21:52, Daniel Steuernol wrote: > There is nothing in the system log about it being drained or shutdown, > I'm not sure how else it would be pre-empted. No one else on the team > is on the servers and I haven't been shutting them down. There also is >

Re: Nodes stopping

Maybe this article helps you. http://stackoverflow.com/questions/26285133/who-sends-a-sigkill-to-my-process-mysteriously-on-ubuntu-server On Thu, May 11, 2017 at 1:52 PM, Daniel Steuernol wrote: > There is nothing in the system log about it being drained or shutdown, I'm

Re: Nodes stopping

There is nothing in the system log about it being drained or shutdown, I'm not sure how else it would be pre-empted. No one else on the team is on the servers and I haven't been shutting them down. There also is no java memory dump on the server either. It appears that the process just died.

Re: Nodes stopping

What do you mean by "no obvious error in the logs", do you see node was drained or shutdown. Are you sure, no other process is calling nodetool drain or shutdown, OR pre-empting cassandra process? On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol wrote: > > I have a 6

Nodes stopping

I have a 6 node cassandra cluster running, and frequently a node will go down with no obvious error in the logs. This is starting to happen quite often, almost daily now. Any suggestions on how to track down what is causing the node to stop?