Re: rebuild constantly fails, 3.11
the nodes have 32G ram, there are no other processes running. Thanks for the info about the g1gc. I used bootstrap resume to finish the bootstrap, then added another two nodes. This worked, but I saw in munin a constantly rising memory consumption during streaming, while on the other nodes there was a constantly amount of memory used. Nothing was killed this time, since the streaming took much less time to complete. I think I'll switch to cms on the problem node and compare the memory usage and compare the nodetool info output. Michael Am 11.08.2017 um 17:55 schrieb kurt greaves: > How much memory do these machines have? Typically we've found that G1 > isn't worth it until you get to around 24G heaps, and even at that it's > not really better than CMS. You could try CMS with an 8G heap and 2G new > size. > > However as the oom is only happening on one node have you ensured there > are no extra processes running on that node that could be consuming > extra memory? Note that the oom killer will kill the process with the > highest oom score, which generally corresponds to the process using the > most memory, but not necessarily the problem. > > Also could you run nodetool info on the problem node and 1 other and > dump the output in a gist? It would be interesting to see if there is a > significant difference in off-heap. > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: rebuild constantly fails, 3.11
cc'ing user back in... On 12 Aug. 2017 01:55, "kurt greaves" wrote: > How much memory do these machines have? Typically we've found that G1 > isn't worth it until you get to around 24G heaps, and even at that it's not > really better than CMS. You could try CMS with an 8G heap and 2G new size. > > However as the oom is only happening on one node have you ensured there > are no extra processes running on that node that could be consuming extra > memory? Note that the oom killer will kill the process with the highest oom > score, which generally corresponds to the process using the most memory, > but not necessarily the problem. > > Also could you run nodetool info on the problem node and 1 other and dump > the output in a gist? It would be interesting to see if there is a > significant difference in off-heap. > > On 11 Aug. 2017 17:30, "Micha" wrote: > >> It's an oom issue, the kernel kills the cassandra job. >> The config was to use offheap buffers and 20G java heap, I changed this >> to use heap buffers and 16G java heap. I added a new node yesterday >> which got streams from 4 other nodes. They all succeeded except on the >> one node which failed before. This time again the db was killed by the >> kernel. At the moment I don't know what is the reason here, since the >> nodes are equal. >> >> For me it seems the g1gc is not able to free the memory fast enough. >> The settings were for MaxGCPauseMillis=600 and ParallelGCThreads=10 >> ConcGCThreads=10 which maybe are too high since the node has only 8 >> cores.. >> I changed this ParallelGCThreads=8 and ConcGCThreads=2 as is mentioned >> in the comments of jvm.options >> >> Since the bootstrap of the fifth node did not complete I will start it >> again and check if the memory is still decreasing over time. >> >> >> >> Michael >> >> >> >> On 11.08.2017 01:25, Jeff Jirsa wrote: >> > >> > >> > On 2017-08-08 01:00 (-0700), Micha wrote: >> >> Hi, >> >> >> >> it seems I'm not able to add add 3 node dc to a 3 node dc. After >> >> starting the rebuild on a new node, nodetool netstats show it will >> >> receive 1200 files from node-1 and 5000 from node-2. The stream from >> >> node-1 completes but the stream from node-2 allways fails, after >> sending >> >> ca 4000 files. >> >> >> >> After restarting the rebuild it again starts to send the 5000 files. >> >> The whole cluster is connected via one switch only , no firewall >> >> between, the networks shows no errors. >> >> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. >> >> the logs show no errors. The size of the data is ca 1TB. >> > >> > Is there anything in `dmesg` ? System logs? Nothing? Is node2 running? >> Is node3 running? >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> > For additional commands, e-mail: dev-h...@cassandra.apache.org >> > >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >>
Re: rebuild constantly fails, 3.11
How much memory do these machines have? Typically we've found that G1 isn't worth it until you get to around 24G heaps, and even at that it's not really better than CMS. You could try CMS with an 8G heap and 2G new size. However as the oom is only happening on one node have you ensured there are no extra processes running on that node that could be consuming extra memory? Note that the oom killer will kill the process with the highest oom score, which generally corresponds to the process using the most memory, but not necessarily the problem. Also could you run nodetool info on the problem node and 1 other and dump the output in a gist? It would be interesting to see if there is a significant difference in off-heap. On 11 Aug. 2017 17:30, "Micha" wrote: > It's an oom issue, the kernel kills the cassandra job. > The config was to use offheap buffers and 20G java heap, I changed this > to use heap buffers and 16G java heap. I added a new node yesterday > which got streams from 4 other nodes. They all succeeded except on the > one node which failed before. This time again the db was killed by the > kernel. At the moment I don't know what is the reason here, since the > nodes are equal. > > For me it seems the g1gc is not able to free the memory fast enough. > The settings were for MaxGCPauseMillis=600 and ParallelGCThreads=10 > ConcGCThreads=10 which maybe are too high since the node has only 8 cores.. > I changed this ParallelGCThreads=8 and ConcGCThreads=2 as is mentioned > in the comments of jvm.options > > Since the bootstrap of the fifth node did not complete I will start it > again and check if the memory is still decreasing over time. > > > > Michael > > > > On 11.08.2017 01:25, Jeff Jirsa wrote: > > > > > > On 2017-08-08 01:00 (-0700), Micha wrote: > >> Hi, > >> > >> it seems I'm not able to add add 3 node dc to a 3 node dc. After > >> starting the rebuild on a new node, nodetool netstats show it will > >> receive 1200 files from node-1 and 5000 from node-2. The stream from > >> node-1 completes but the stream from node-2 allways fails, after sending > >> ca 4000 files. > >> > >> After restarting the rebuild it again starts to send the 5000 files. > >> The whole cluster is connected via one switch only , no firewall > >> between, the networks shows no errors. > >> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. > >> the logs show no errors. The size of the data is ca 1TB. > > > > Is there anything in `dmesg` ? System logs? Nothing? Is node2 running? > Is node3 running? > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: rebuild constantly fails, 3.11
It's an oom issue, the kernel kills the cassandra job. The config was to use offheap buffers and 20G java heap, I changed this to use heap buffers and 16G java heap. I added a new node yesterday which got streams from 4 other nodes. They all succeeded except on the one node which failed before. This time again the db was killed by the kernel. At the moment I don't know what is the reason here, since the nodes are equal. For me it seems the g1gc is not able to free the memory fast enough. The settings were for MaxGCPauseMillis=600 and ParallelGCThreads=10 ConcGCThreads=10 which maybe are too high since the node has only 8 cores.. I changed this ParallelGCThreads=8 and ConcGCThreads=2 as is mentioned in the comments of jvm.options Since the bootstrap of the fifth node did not complete I will start it again and check if the memory is still decreasing over time. Michael On 11.08.2017 01:25, Jeff Jirsa wrote: > > > On 2017-08-08 01:00 (-0700), Micha wrote: >> Hi, >> >> it seems I'm not able to add add 3 node dc to a 3 node dc. After >> starting the rebuild on a new node, nodetool netstats show it will >> receive 1200 files from node-1 and 5000 from node-2. The stream from >> node-1 completes but the stream from node-2 allways fails, after sending >> ca 4000 files. >> >> After restarting the rebuild it again starts to send the 5000 files. >> The whole cluster is connected via one switch only , no firewall >> between, the networks shows no errors. >> The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. >> the logs show no errors. The size of the data is ca 1TB. > > Is there anything in `dmesg` ? System logs? Nothing? Is node2 running? Is > node3 running? > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
RE: rebuild constantly fails, 3.11
I don't know if this is going to help or not but we had a problem disk (was going bad) and would develop bad blocks, requiring raid repair. As nearly as we could guess, that would cause "some sort of timing issue" for nodetool - we were never able to duplicate the exact occurrence on demand. The clue came when the same exact process worked on a different setup that did not use the same disk subsystem. Once we isolated and replaced the degrading disk, the problem went away. Beyond what Jeff suggested, you may want to look at any logs your raid produces for potential clues. Also, if it is always failing when sending the same file, you might want to review what is going on there. I know this is fairly "general" response but I hope it gives you some ideas. Good luck! Bob -Original Message- From: Jeff Jirsa [mailto:jji...@apache.org] Sent: Thursday, August 10, 2017 7:26 PM To: dev@cassandra.apache.org Subject: Re: rebuild constantly fails, 3.11 On 2017-08-08 01:00 (-0700), Micha wrote: > Hi, > > it seems I'm not able to add add 3 node dc to a 3 node dc. After > starting the rebuild on a new node, nodetool netstats show it will > receive 1200 files from node-1 and 5000 from node-2. The stream from > node-1 completes but the stream from node-2 allways fails, after > sending ca 4000 files. > > After restarting the rebuild it again starts to send the 5000 files. > The whole cluster is connected via one switch only , no firewall > between, the networks shows no errors. > The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. > the logs show no errors. The size of the data is ca 1TB. Is there anything in `dmesg` ? System logs? Nothing? Is node2 running? Is node3 running? - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: rebuild constantly fails, 3.11
On 2017-08-08 01:00 (-0700), Micha wrote: > Hi, > > it seems I'm not able to add add 3 node dc to a 3 node dc. After > starting the rebuild on a new node, nodetool netstats show it will > receive 1200 files from node-1 and 5000 from node-2. The stream from > node-1 completes but the stream from node-2 allways fails, after sending > ca 4000 files. > > After restarting the rebuild it again starts to send the 5000 files. > The whole cluster is connected via one switch only , no firewall > between, the networks shows no errors. > The machines have 8 cores, 32GB RAM and two 1TB discs as raid0. > the logs show no errors. The size of the data is ca 1TB. Is there anything in `dmesg` ? System logs? Nothing? Is node2 running? Is node3 running? - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org