Hi Doss,
See a lot of TIMED_WATING connection occurs with high tcp traffic
infrastructure as in a LAMP solution when the Apache server can't
anymore connect to the MySQL/MariaDB database.
In this case, tweak net.ipv4.tcp_tw_reuse is a possible solution (but
never net.ipv4.tcp_tw_recycle as you suggested in your previous post). This
is well explained in this great article
https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux
However, in general and more specifically in your case, I would investigate
the root cause of your issue and do not try to find a workaround.
Can you provide more information about your use case (we know : 3 node SOLR
(8.3.1 NRT) + 3 Node Zookeeper Ensemble) ?
- hardware architecture and sizing
- JVM version / settings
- Solr settings
- collections and queries information
- gc logs or gceasy results
Regards
Dominique
Le lun. 10 août 2020 à 15:43, Doss <itsmed...@gmail.com> a écrit :
> Hi,
>
> In solr 8.3.1 source, I see the following , which I assume could be the
> reason for the issue "Max requests queued per destination 3000 exceeded for
> HttpDestination",
>
> solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
> private static final int MAX_OUTSTANDING_REQUESTS = 1000;
> solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
> available = new Semaphore(MAX_OUTSTANDING_REQUESTS, false);
> solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java:
> return MAX_OUTSTANDING_REQUESTS * 3;
>
> how can I increase this?
>
> On Mon, Aug 10, 2020 at 12:01 AM Doss <itsmed...@gmail.com> wrote:
>
> > Hi,
> >
> > We are having 3 node SOLR (8.3.1 NRT) + 3 Node Zookeeper Ensemble now and
> > then we are facing "Max requests queued per destination 3000 exceeded for
> > HttpDestination"
> >
> > After restart evering thing starts working fine until another problem.
> > Once a problem occurred we are seeing soo many TIMED_WAITING threads
> >
> > Server 1:
> > *7722* Threads are in TIMED_WATING
> >
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@151d5f2f
> > ")
> > Server 2:
> > *4046* Threads are in TIMED_WATING
> >
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1e0205c3
> > ")
> > Server 3:
> > *4210* Threads are in TIMED_WATING
> >
> ("lock":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ee792c0
> > ")
> >
> > Please suggest whether net.ipv4.tcp_tw_reuse=1 will help ? or how can we
> > increase the 3000 limit?
> >
> > Sorry, since I haven't got any response to my previous query, I am
> > creating this as new,
> >
> > Thanks,
> > Mohandoss.
> >
>
