FWIW, I have seen Solr exhaust the IOPS burst quota on AWS causing slow replication and high latency for search and indexing operations. You may want to dig into cloud watch metrics and see if you are running into a similar issue. The default IOPS quota on gp2 is very low (100?).
Another thing to check is whether you have DNS TTLs for both positive and negative lookups configured. When nodes go down and come back up in Kubernetes the address of the pod remains the same but the IP can change and the JVM caches DNS lookups. This can cause timeouts. On 12/14/20, Abhishek Mishra <solrmis...@gmail.com> wrote: > Hi Houston, > Sorry for the late reply. Each shard has a 9GB size around. > Yeah, we are providing enough resources to pods. We are currently > using c5.4xlarge. > XMS and XMX is 16GB. The machine is having 32 GB and 16 core. > No, I haven't run it outside Kubernetes. But I do have colleagues who did > the same on 7.2 and didn't face any issue regarding it. > Storage volume is gp2 50GB. > It's not the search query where we are facing inconsistencies or timeouts. > Seems some internal admin APIs sometimes have issues. So while adding new > replica in clusters sometimes result in inconsistencies. Like recovery > takes some time more than one hour. > > Regards, > Abhishek > > On Thu, Dec 10, 2020 at 10:23 AM Houston Putman <houstonput...@gmail.com> > wrote: > >> Hello Abhishek, >> >> It's really hard to provide any advice without knowing any information >> about your setup/usage. >> >> Are you giving your Solr pods enough resources on EKS? >> Have you run Solr in the same configuration outside of kubernetes in the >> past without timeouts? >> What type of storage volumes are you using to store your data? >> Are you using headless services to connect your Solr Nodes, or ingresses? >> >> If this is the first time that you are using this data + Solr >> configuration, maybe it's just that your data within Solr isn't optimized >> for the type of queries that you are doing. >> If you have run it successfully in the past outside of Kubernetes, then I >> would look at the resources that you are giving your pods and the storage >> volumes that you are using. >> If you are using Ingresses, that might be causing slow connections >> between >> nodes, or between your client and Solr. >> >> - Houston >> >> On Wed, Dec 9, 2020 at 3:24 PM Abhishek Mishra <solrmis...@gmail.com> >> wrote: >> >> > Hello guys, >> > We are kind of facing some of the issues(Like timeout etc.) which are >> very >> > inconsistent. By any chance can it be related to EKS? We are using solr >> 7.7 >> > and zookeeper 3.4.13. Should we move to ECS? >> > >> > Regards, >> > Abhishek >> > >> > -- Regards, Shalin Shekhar Mangar.
