I'm testing my 10 nodes cluster under production load and with production data. I was using automated tool which created the nodes (VMs) configured everything and restarted all of them. Everything worked, I mean, I was getting the data I expected, but when I've checked the stats I noticed that I'm running 10 one node clusters. My nodes didn't see each other, they had a separate duplicated set of data on each node. I've stopped all the nodes, cleaned all logs/storage files, and restarted the nodes again. Now I had one cluster with 7 nodes and 3 nodes separate. I've stopped the 3 nodes, cleaned them up, and started them up one by one, they successfully joined the cluster. At the end I've got all 10 nodes working as a single cluster. But I'm afraid that if nodes would get restarted or network would have some problems, I could end up with split cluster again. I use API to start Cache with locators, and all locator's IPs are provided in the config. From the documentation I had impression that Geode would wait till N/2+1 nodes would start before forming the cluster, since the number of locators is preset. But looks like it is not the case. Or should I set some setting to force such behavior?
Thank you, Eugene
