Suggestions required for Reducing update request acknowledgement latency in SolrCloud 9.6.1

[Harshit Sharma]

Hii all

We're looking for advice on reducing the time our indexing client waits for
Solr to acknowledge update requests.

Our indexing throughput is currently limited primarily by the time taken
for each batched update request to complete. Before changing client-side
concurrency or scaling the application, we'd like to understand whether any
Solr-side optimisations could reduce this acknowledgement latency.
Production Environment

   -

   SolrCloud 9.6.1
   -

   10-node cluster on GCP Compute Engine
   -

   Two collections:
   -

      ~180M documents (250+ GB, 65 shards)
      -

      ~70M documents (180+ GB, 65 shards)
      -

   Mostly RF=1
   -

   Implicit routing
   -

   SolrJ client sending batched update requests (3k-4k documents/request)
   -

   Hard commit every *30 minutes* (openSearcher=false)
   -

   Soft commit every *15 minutes*
   -

   ramBufferSizeMB=*200*
   -

   TieredMergePolicy
   -

   Default ConcurrentMergeScheduler
   -

   StandardDirectoryFactory (MMapDirectory)

What we're observing

>From the client side, a blocking update request typically takes:

   -

   ~700 ms for batches of 3,000 documents
   -

   ~1.5 s for batches of 4,000 documents

Occasionally we see several-second outliers, but the median latency remains
consistent even during peak query traffic.

We aim to reduce this latency to increase indexing throughput.
Questions

   1.

   For batched update requests in SolrCloud, what typically contributes
   most to the acknowledgement latency (request distribution, update
   processing, tlog fsync, merges, etc.)?

   2.

   Which Solr metrics would you recommend checking first to identify where
   this time is being spent?

   3.

   Are there any Solr configuration settings that commonly have a
   significant impact on update acknowledgement latency (merge policy, merge
   scheduler, indexing buffer, commit settings, update log configuration,
   etc.)?

   4.

   Has anyone run into similar indexing latency at this scale (60+ shards,
   ~200M documents), and if so, what changes provided the biggest improvement?

We'd appreciate any suggestions or best practices to reduce update request
latency and improve indexing throughput. Please let us know if any
additional configuration details or metrics would be helpful.


Thanks & Regards

Harshit Sharma