Hi Abhishek, We're running Solr Cloud 8.6 on GKE. 3 node cluster, running 4 cpus (configured) and 8gb of min & max JVM configured, all with anti-affinity so they never exist on the same node. It's got 2 collections of ~13documents each, 6 shards, 3 replicas each, disk usage on each node is ~54gb (we've got all the shards replicated to all nodes)
We're also using a 200gb zonal SSD, which *has* been necessary just so that we've got the right IOPS & bandwidth. (That's approximately 6000 IOPS for read & write each, and 96MB/s for read & write each) Various lessons learnt... You definitely don't want them ever on the same kubernetes node. From a resilience perspective, yes, but also when one SOLR node gets busy, they tend to all get busy, so now you'll have resource contention. Recovery can also get very busy and resource intensive, and again, sitting on the same node is problematic. We also saw the need to move to SSDs because of how IOPS bound we were. Did I mention use SSDs? ;) Good luck! On Mon, Dec 14, 2020 at 5:34 PM Abhishek Mishra <solrmis...@gmail.com> wrote: > Hi Houston, > Sorry for the late reply. Each shard has a 9GB size around. > Yeah, we are providing enough resources to pods. We are currently > using c5.4xlarge. > XMS and XMX is 16GB. The machine is having 32 GB and 16 core. > No, I haven't run it outside Kubernetes. But I do have colleagues who did > the same on 7.2 and didn't face any issue regarding it. > Storage volume is gp2 50GB. > It's not the search query where we are facing inconsistencies or timeouts. > Seems some internal admin APIs sometimes have issues. So while adding new > replica in clusters sometimes result in inconsistencies. Like recovery > takes some time more than one hour. > > Regards, > Abhishek > > On Thu, Dec 10, 2020 at 10:23 AM Houston Putman <houstonput...@gmail.com> > wrote: > > > Hello Abhishek, > > > > It's really hard to provide any advice without knowing any information > > about your setup/usage. > > > > Are you giving your Solr pods enough resources on EKS? > > Have you run Solr in the same configuration outside of kubernetes in the > > past without timeouts? > > What type of storage volumes are you using to store your data? > > Are you using headless services to connect your Solr Nodes, or ingresses? > > > > If this is the first time that you are using this data + Solr > > configuration, maybe it's just that your data within Solr isn't optimized > > for the type of queries that you are doing. > > If you have run it successfully in the past outside of Kubernetes, then I > > would look at the resources that you are giving your pods and the storage > > volumes that you are using. > > If you are using Ingresses, that might be causing slow connections > between > > nodes, or between your client and Solr. > > > > - Houston > > > > On Wed, Dec 9, 2020 at 3:24 PM Abhishek Mishra <solrmis...@gmail.com> > > wrote: > > > > > Hello guys, > > > We are kind of facing some of the issues(Like timeout etc.) which are > > very > > > inconsistent. By any chance can it be related to EKS? We are using solr > > 7.7 > > > and zookeeper 3.4.13. Should we move to ECS? > > > > > > Regards, > > > Abhishek > > > > > >
