Denis does have a point. When we were trying to run using GP2 storage, the cluster would simply lock up for an hour. Once we moved to local SSDs on i3 instances those issues went away (but we needed 2.5 to have the streaming rate hold for up as we had a lot of data loaded). The i3 instances are rated at about 700,000 write IOPS, and we were only getting about 20-30,000 out of GP2. You could separate or combine the WAL and storage, and hardly move the needle. Will describe cluster snapshots on AWS in more detail when we have completed that work.
On Mon, Jul 2, 2018 at 11:20 AM, Denis Mekhanikov <dmekhani...@gmail.com> wrote: > Why did you decide, that cluster is deadlocked in the first place? > > > We've had several deployments in a row fail, apparently due to > deadlocking in the loading process. > What did you see in logs of the failing nodes? > > Denis > > пн, 2 июл. 2018 г. в 17:08, breischl <breis...@gmail.com>: > >> Ah, I had not thought of that, thanks. >> >> Interestingly, going to a smaller cluster seems to have worked around the >> problem. We were running a 44-node cluster using 3 backups of the data. >> Switching to two separate 22-node clusters, each with 1 backup, seems to >> work just fine. Is there some limit to how large a cluster should be? >> >> @smovva - We were using c4.4xl instances, but switched to r4.2xl because >> we >> had spare CPU but kept having memory problems. I suspect that there isn't >> a >> "right" size to use, it just depends on the use case you have. >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> > Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
