Hi Abhishek, Merry Christmas to you too! I think it's really a question regarding your indexing speed NFRs.
Have you had a chance to take a look at your IOPS & write bytes/second graphs for that host & PVC? I'd suggest that's the first thing to go look at, so that you can find out whether you're actually IOPS bound or not. If you are, then it becomes a question of *how* you're indexing, and whether that can be "slowed down" or not. On Thu, Dec 24, 2020 at 5:55 PM Abhishek Mishra <solrmis...@gmail.com> wrote: > Hi Jonathan, > Merry Christmas. > Thanks for the suggestion. To manage IOPS can we do something on > rate-limiting behalf? > > Regards, > Abhishek > > > On Thu, Dec 17, 2020 at 5:07 AM Jonathan Tan <jty....@gmail.com> wrote: > > > Hi Abhishek, > > > > We're running Solr Cloud 8.6 on GKE. > > 3 node cluster, running 4 cpus (configured) and 8gb of min & max JVM > > configured, all with anti-affinity so they never exist on the same node. > > It's got 2 collections of ~13documents each, 6 shards, 3 replicas each, > > disk usage on each node is ~54gb (we've got all the shards replicated to > > all nodes) > > > > We're also using a 200gb zonal SSD, which *has* been necessary just so > that > > we've got the right IOPS & bandwidth. (That's approximately 6000 IOPS for > > read & write each, and 96MB/s for read & write each) > > > > Various lessons learnt... > > You definitely don't want them ever on the same kubernetes node. From a > > resilience perspective, yes, but also when one SOLR node gets busy, they > > tend to all get busy, so now you'll have resource contention. Recovery > can > > also get very busy and resource intensive, and again, sitting on the same > > node is problematic. We also saw the need to move to SSDs because of how > > IOPS bound we were. > > > > Did I mention use SSDs? ;) > > > > Good luck! > > > > On Mon, Dec 14, 2020 at 5:34 PM Abhishek Mishra <solrmis...@gmail.com> > > wrote: > > > > > Hi Houston, > > > Sorry for the late reply. Each shard has a 9GB size around. > > > Yeah, we are providing enough resources to pods. We are currently > > > using c5.4xlarge. > > > XMS and XMX is 16GB. The machine is having 32 GB and 16 core. > > > No, I haven't run it outside Kubernetes. But I do have colleagues who > did > > > the same on 7.2 and didn't face any issue regarding it. > > > Storage volume is gp2 50GB. > > > It's not the search query where we are facing inconsistencies or > > timeouts. > > > Seems some internal admin APIs sometimes have issues. So while adding > new > > > replica in clusters sometimes result in inconsistencies. Like recovery > > > takes some time more than one hour. > > > > > > Regards, > > > Abhishek > > > > > > On Thu, Dec 10, 2020 at 10:23 AM Houston Putman < > houstonput...@gmail.com > > > > > > wrote: > > > > > > > Hello Abhishek, > > > > > > > > It's really hard to provide any advice without knowing any > information > > > > about your setup/usage. > > > > > > > > Are you giving your Solr pods enough resources on EKS? > > > > Have you run Solr in the same configuration outside of kubernetes in > > the > > > > past without timeouts? > > > > What type of storage volumes are you using to store your data? > > > > Are you using headless services to connect your Solr Nodes, or > > ingresses? > > > > > > > > If this is the first time that you are using this data + Solr > > > > configuration, maybe it's just that your data within Solr isn't > > optimized > > > > for the type of queries that you are doing. > > > > If you have run it successfully in the past outside of Kubernetes, > > then I > > > > would look at the resources that you are giving your pods and the > > storage > > > > volumes that you are using. > > > > If you are using Ingresses, that might be causing slow connections > > > between > > > > nodes, or between your client and Solr. > > > > > > > > - Houston > > > > > > > > On Wed, Dec 9, 2020 at 3:24 PM Abhishek Mishra <solrmis...@gmail.com > > > > > > wrote: > > > > > > > > > Hello guys, > > > > > We are kind of facing some of the issues(Like timeout etc.) which > are > > > > very > > > > > inconsistent. By any chance can it be related to EKS? We are using > > solr > > > > 7.7 > > > > > and zookeeper 3.4.13. Should we move to ECS? > > > > > > > > > > Regards, > > > > > Abhishek > > > > > > > > > > > > > > >
