Re: Too many connections in CLOSE_WAIT state on master solr server
Hi Ranveer, You can try this '-Dhttp.maxConnections' out, may resolve the issue. But the root cause I figured may lie with some queries made to solr that are too heavy to have decent turnaround times. As a result the client may close the connection abruptly, resulting in half closed connections. You can also try adding search time out to solr queries: https://issues.apache.org/jira/browse/SOLR-502 On Tue, Jan 10, 2012 at 8:06 AM, Ranveer ranveer.s...@gmail.com wrote: Hi, I am facing same problem. Did -Dhttp.maxConnections resolve the problem ? Please let us know! regards Ranveer On Thursday 15 December 2011 11:30 AM, samarth s wrote: Thanks Erick and Mikhail. I'll try this out. On Wed, Dec 14, 2011 at 7:11 PM, Erick Ericksonerickerick...@gmail.com wrote: I'm guessing (and it's just a guess) that what's happening is that the container is queueing up your requests while waiting for the other connections to close, so Mikhail's suggestion seems like a good idea. Best Erick On Wed, Dec 14, 2011 at 12:28 AM, samarth s samarth.s.seksa...@gmail.com wrote: The updates to the master are user driven, and are needed to be visible quickly. Hence, the high frequency of replication. It may be that too many replication requests are being handled at a time, but why should that result in half closed connections? On Wed, Dec 14, 2011 at 2:47 AM, Erick Ericksonerickerick...@gmail.com wrote: Replicating 40 cores every 20 seconds is just *asking* for trouble. How often do your cores change on the master? How big are they? Is there any chance you just have too many cores replicating at once? Best Erick On Tue, Dec 13, 2011 at 3:52 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: You can try to reuse your connections (prevent them from closing) by specifying -Dhttp.maxConnections=http://download.oracle.com/javase/1.4.2/docs/guide/net/properties.htmlN in jvm startup params. At client JVM!. Number should be chosen considering the number of connection you'd like to keep alive. Let me know if it works for you. On Tue, Dec 13, 2011 at 2:57 PM, samarth ssamarth.s.seksa...@gmail.comwrote: Hi, I am using solr replication and am experiencing a lot of connections in the state CLOSE_WAIT at the master solr server. These disappear after a while, but till then the master solr stops responding. There are about 130 open connections on the master server with the client as the slave m/c and all are in the state CLOSE_WAIT. Also, the client port specified on the master solr server netstat results is not visible in the netstat results on the client (slave solr) m/c. Following is my environment: - 40 cores in the master solr on m/c 1 - 40 cores in the slave solr on m/c 2 - The replication poll interval is 20 seconds. - Replication part in solrconfig.xml in the slave solr: requestHandler name=/replication class=solr.ReplicationHandler lst name=slave !--fully qualified url for the replication handler of master-- str name=masterUrl$mastercorename/replication/str !--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically. But a fetchindex can be triggered from the admin or the http API-- str name=pollInterval00:00:20/str !-- The following values are used when the slave connects to the master to download the index files. Default values implicitly set as 5000ms and 1ms respectively. The user DOES NOT need to specify these unless the bandwidth is extremely low or if there is an extremely high latency-- str name=httpConnTimeout5000/str str name=httpReadTimeout1/str /lst /requestHandler Thanks for any pointers. -- Regards, Samarth -- Sincerely yours Mikhail Khludnev Developer Grid Dynamics tel. 1-415-738-8644 Skype: mkhludnev http://www.griddynamics.com mkhlud...@griddynamics.com -- Regards, Samarth -- Regards, Samarth
Re: Too many connections in CLOSE_WAIT state on master solr server
Hi, I am facing same problem. Did -Dhttp.maxConnections resolve the problem ? Please let us know! regards Ranveer On Thursday 15 December 2011 11:30 AM, samarth s wrote: Thanks Erick and Mikhail. I'll try this out. On Wed, Dec 14, 2011 at 7:11 PM, Erick Ericksonerickerick...@gmail.com wrote: I'm guessing (and it's just a guess) that what's happening is that the container is queueing up your requests while waiting for the other connections to close, so Mikhail's suggestion seems like a good idea. Best Erick On Wed, Dec 14, 2011 at 12:28 AM, samarth s samarth.s.seksa...@gmail.com wrote: The updates to the master are user driven, and are needed to be visible quickly. Hence, the high frequency of replication. It may be that too many replication requests are being handled at a time, but why should that result in half closed connections? On Wed, Dec 14, 2011 at 2:47 AM, Erick Ericksonerickerick...@gmail.com wrote: Replicating 40 cores every 20 seconds is just *asking* for trouble. How often do your cores change on the master? How big are they? Is there any chance you just have too many cores replicating at once? Best Erick On Tue, Dec 13, 2011 at 3:52 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: You can try to reuse your connections (prevent them from closing) by specifying -Dhttp.maxConnections=http://download.oracle.com/javase/1.4.2/docs/guide/net/properties.htmlN in jvm startup params. At client JVM!. Number should be chosen considering the number of connection you'd like to keep alive. Let me know if it works for you. On Tue, Dec 13, 2011 at 2:57 PM, samarth ssamarth.s.seksa...@gmail.comwrote: Hi, I am using solr replication and am experiencing a lot of connections in the state CLOSE_WAIT at the master solr server. These disappear after a while, but till then the master solr stops responding. There are about 130 open connections on the master server with the client as the slave m/c and all are in the state CLOSE_WAIT. Also, the client port specified on the master solr server netstat results is not visible in the netstat results on the client (slave solr) m/c. Following is my environment: - 40 cores in the master solr on m/c 1 - 40 cores in the slave solr on m/c 2 - The replication poll interval is 20 seconds. - Replication part in solrconfig.xml in the slave solr: requestHandler name=/replication class=solr.ReplicationHandler lst name=slave !--fully qualified url for the replication handler of master-- str name=masterUrl$mastercorename/replication/str !--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically. But a fetchindex can be triggered from the admin or the http API-- str name=pollInterval00:00:20/str !-- The following values are used when the slave connects to the master to download the index files. Default values implicitly set as 5000ms and 1ms respectively. The user DOES NOT need to specify these unless the bandwidth is extremely low or if there is an extremely high latency-- str name=httpConnTimeout5000/str str name=httpReadTimeout1/str /lst /requestHandler Thanks for any pointers. -- Regards, Samarth -- Sincerely yours Mikhail Khludnev Developer Grid Dynamics tel. 1-415-738-8644 Skype: mkhludnev http://www.griddynamics.com mkhlud...@griddynamics.com -- Regards, Samarth
Re: Too many connections in CLOSE_WAIT state on master solr server
I'm guessing (and it's just a guess) that what's happening is that the container is queueing up your requests while waiting for the other connections to close, so Mikhail's suggestion seems like a good idea. Best Erick On Wed, Dec 14, 2011 at 12:28 AM, samarth s samarth.s.seksa...@gmail.com wrote: The updates to the master are user driven, and are needed to be visible quickly. Hence, the high frequency of replication. It may be that too many replication requests are being handled at a time, but why should that result in half closed connections? On Wed, Dec 14, 2011 at 2:47 AM, Erick Erickson erickerick...@gmail.com wrote: Replicating 40 cores every 20 seconds is just *asking* for trouble. How often do your cores change on the master? How big are they? Is there any chance you just have too many cores replicating at once? Best Erick On Tue, Dec 13, 2011 at 3:52 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: You can try to reuse your connections (prevent them from closing) by specifying -Dhttp.maxConnections=http://download.oracle.com/javase/1.4.2/docs/guide/net/properties.htmlN in jvm startup params. At client JVM!. Number should be chosen considering the number of connection you'd like to keep alive. Let me know if it works for you. On Tue, Dec 13, 2011 at 2:57 PM, samarth s samarth.s.seksa...@gmail.comwrote: Hi, I am using solr replication and am experiencing a lot of connections in the state CLOSE_WAIT at the master solr server. These disappear after a while, but till then the master solr stops responding. There are about 130 open connections on the master server with the client as the slave m/c and all are in the state CLOSE_WAIT. Also, the client port specified on the master solr server netstat results is not visible in the netstat results on the client (slave solr) m/c. Following is my environment: - 40 cores in the master solr on m/c 1 - 40 cores in the slave solr on m/c 2 - The replication poll interval is 20 seconds. - Replication part in solrconfig.xml in the slave solr: requestHandler name=/replication class=solr.ReplicationHandler lst name=slave !--fully qualified url for the replication handler of master-- str name=masterUrl$mastercorename/replication/str !--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically. But a fetchindex can be triggered from the admin or the http API-- str name=pollInterval00:00:20/str !-- The following values are used when the slave connects to the master to download the index files. Default values implicitly set as 5000ms and 1ms respectively. The user DOES NOT need to specify these unless the bandwidth is extremely low or if there is an extremely high latency-- str name=httpConnTimeout5000/str str name=httpReadTimeout1/str /lst /requestHandler Thanks for any pointers. -- Regards, Samarth -- Sincerely yours Mikhail Khludnev Developer Grid Dynamics tel. 1-415-738-8644 Skype: mkhludnev http://www.griddynamics.com mkhlud...@griddynamics.com -- Regards, Samarth
Re: Too many connections in CLOSE_WAIT state on master solr server
Thanks Erick and Mikhail. I'll try this out. On Wed, Dec 14, 2011 at 7:11 PM, Erick Erickson erickerick...@gmail.com wrote: I'm guessing (and it's just a guess) that what's happening is that the container is queueing up your requests while waiting for the other connections to close, so Mikhail's suggestion seems like a good idea. Best Erick On Wed, Dec 14, 2011 at 12:28 AM, samarth s samarth.s.seksa...@gmail.com wrote: The updates to the master are user driven, and are needed to be visible quickly. Hence, the high frequency of replication. It may be that too many replication requests are being handled at a time, but why should that result in half closed connections? On Wed, Dec 14, 2011 at 2:47 AM, Erick Erickson erickerick...@gmail.com wrote: Replicating 40 cores every 20 seconds is just *asking* for trouble. How often do your cores change on the master? How big are they? Is there any chance you just have too many cores replicating at once? Best Erick On Tue, Dec 13, 2011 at 3:52 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: You can try to reuse your connections (prevent them from closing) by specifying -Dhttp.maxConnections=http://download.oracle.com/javase/1.4.2/docs/guide/net/properties.htmlN in jvm startup params. At client JVM!. Number should be chosen considering the number of connection you'd like to keep alive. Let me know if it works for you. On Tue, Dec 13, 2011 at 2:57 PM, samarth s samarth.s.seksa...@gmail.comwrote: Hi, I am using solr replication and am experiencing a lot of connections in the state CLOSE_WAIT at the master solr server. These disappear after a while, but till then the master solr stops responding. There are about 130 open connections on the master server with the client as the slave m/c and all are in the state CLOSE_WAIT. Also, the client port specified on the master solr server netstat results is not visible in the netstat results on the client (slave solr) m/c. Following is my environment: - 40 cores in the master solr on m/c 1 - 40 cores in the slave solr on m/c 2 - The replication poll interval is 20 seconds. - Replication part in solrconfig.xml in the slave solr: requestHandler name=/replication class=solr.ReplicationHandler lst name=slave !--fully qualified url for the replication handler of master-- str name=masterUrl$mastercorename/replication/str !--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically. But a fetchindex can be triggered from the admin or the http API-- str name=pollInterval00:00:20/str !-- The following values are used when the slave connects to the master to download the index files. Default values implicitly set as 5000ms and 1ms respectively. The user DOES NOT need to specify these unless the bandwidth is extremely low or if there is an extremely high latency-- str name=httpConnTimeout5000/str str name=httpReadTimeout1/str /lst /requestHandler Thanks for any pointers. -- Regards, Samarth -- Sincerely yours Mikhail Khludnev Developer Grid Dynamics tel. 1-415-738-8644 Skype: mkhludnev http://www.griddynamics.com mkhlud...@griddynamics.com -- Regards, Samarth -- Regards, Samarth
Too many connections in CLOSE_WAIT state on master solr server
Hi, I am using solr replication and am experiencing a lot of connections in the state CLOSE_WAIT at the master solr server. These disappear after a while, but till then the master solr stops responding. There are about 130 open connections on the master server with the client as the slave m/c and all are in the state CLOSE_WAIT. Also, the client port specified on the master solr server netstat results is not visible in the netstat results on the client (slave solr) m/c. Following is my environment: - 40 cores in the master solr on m/c 1 - 40 cores in the slave solr on m/c 2 - The replication poll interval is 20 seconds. - Replication part in solrconfig.xml in the slave solr: requestHandler name=/replication class=solr.ReplicationHandler lst name=slave !--fully qualified url for the replication handler of master-- str name=masterUrl$mastercorename/replication/str !--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically. But a fetchindex can be triggered from the admin or the http API-- str name=pollInterval00:00:20/str !-- The following values are used when the slave connects to the master to download the index files. Default values implicitly set as 5000ms and 1ms respectively. The user DOES NOT need to specify these unless the bandwidth is extremely low or if there is an extremely high latency-- str name=httpConnTimeout5000/str str name=httpReadTimeout1/str /lst /requestHandler Thanks for any pointers. -- Regards, Samarth
Re: Too many connections in CLOSE_WAIT state on master solr server
You can try to reuse your connections (prevent them from closing) by specifying -Dhttp.maxConnections=http://download.oracle.com/javase/1.4.2/docs/guide/net/properties.htmlN in jvm startup params. At client JVM!. Number should be chosen considering the number of connection you'd like to keep alive. Let me know if it works for you. On Tue, Dec 13, 2011 at 2:57 PM, samarth s samarth.s.seksa...@gmail.comwrote: Hi, I am using solr replication and am experiencing a lot of connections in the state CLOSE_WAIT at the master solr server. These disappear after a while, but till then the master solr stops responding. There are about 130 open connections on the master server with the client as the slave m/c and all are in the state CLOSE_WAIT. Also, the client port specified on the master solr server netstat results is not visible in the netstat results on the client (slave solr) m/c. Following is my environment: - 40 cores in the master solr on m/c 1 - 40 cores in the slave solr on m/c 2 - The replication poll interval is 20 seconds. - Replication part in solrconfig.xml in the slave solr: requestHandler name=/replication class=solr.ReplicationHandler lst name=slave !--fully qualified url for the replication handler of master-- str name=masterUrl$mastercorename/replication/str !--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically. But a fetchindex can be triggered from the admin or the http API-- str name=pollInterval00:00:20/str !-- The following values are used when the slave connects to the master to download the index files. Default values implicitly set as 5000ms and 1ms respectively. The user DOES NOT need to specify these unless the bandwidth is extremely low or if there is an extremely high latency-- str name=httpConnTimeout5000/str str name=httpReadTimeout1/str /lst /requestHandler Thanks for any pointers. -- Regards, Samarth -- Sincerely yours Mikhail Khludnev Developer Grid Dynamics tel. 1-415-738-8644 Skype: mkhludnev http://www.griddynamics.com mkhlud...@griddynamics.com
Re: Too many connections in CLOSE_WAIT state on master solr server
Replicating 40 cores every 20 seconds is just *asking* for trouble. How often do your cores change on the master? How big are they? Is there any chance you just have too many cores replicating at once? Best Erick On Tue, Dec 13, 2011 at 3:52 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: You can try to reuse your connections (prevent them from closing) by specifying -Dhttp.maxConnections=http://download.oracle.com/javase/1.4.2/docs/guide/net/properties.htmlN in jvm startup params. At client JVM!. Number should be chosen considering the number of connection you'd like to keep alive. Let me know if it works for you. On Tue, Dec 13, 2011 at 2:57 PM, samarth s samarth.s.seksa...@gmail.comwrote: Hi, I am using solr replication and am experiencing a lot of connections in the state CLOSE_WAIT at the master solr server. These disappear after a while, but till then the master solr stops responding. There are about 130 open connections on the master server with the client as the slave m/c and all are in the state CLOSE_WAIT. Also, the client port specified on the master solr server netstat results is not visible in the netstat results on the client (slave solr) m/c. Following is my environment: - 40 cores in the master solr on m/c 1 - 40 cores in the slave solr on m/c 2 - The replication poll interval is 20 seconds. - Replication part in solrconfig.xml in the slave solr: requestHandler name=/replication class=solr.ReplicationHandler lst name=slave !--fully qualified url for the replication handler of master-- str name=masterUrl$mastercorename/replication/str !--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically. But a fetchindex can be triggered from the admin or the http API-- str name=pollInterval00:00:20/str !-- The following values are used when the slave connects to the master to download the index files. Default values implicitly set as 5000ms and 1ms respectively. The user DOES NOT need to specify these unless the bandwidth is extremely low or if there is an extremely high latency-- str name=httpConnTimeout5000/str str name=httpReadTimeout1/str /lst /requestHandler Thanks for any pointers. -- Regards, Samarth -- Sincerely yours Mikhail Khludnev Developer Grid Dynamics tel. 1-415-738-8644 Skype: mkhludnev http://www.griddynamics.com mkhlud...@griddynamics.com
Re: Too many connections in CLOSE_WAIT state on master solr server
The updates to the master are user driven, and are needed to be visible quickly. Hence, the high frequency of replication. It may be that too many replication requests are being handled at a time, but why should that result in half closed connections? On Wed, Dec 14, 2011 at 2:47 AM, Erick Erickson erickerick...@gmail.com wrote: Replicating 40 cores every 20 seconds is just *asking* for trouble. How often do your cores change on the master? How big are they? Is there any chance you just have too many cores replicating at once? Best Erick On Tue, Dec 13, 2011 at 3:52 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: You can try to reuse your connections (prevent them from closing) by specifying -Dhttp.maxConnections=http://download.oracle.com/javase/1.4.2/docs/guide/net/properties.htmlN in jvm startup params. At client JVM!. Number should be chosen considering the number of connection you'd like to keep alive. Let me know if it works for you. On Tue, Dec 13, 2011 at 2:57 PM, samarth s samarth.s.seksa...@gmail.comwrote: Hi, I am using solr replication and am experiencing a lot of connections in the state CLOSE_WAIT at the master solr server. These disappear after a while, but till then the master solr stops responding. There are about 130 open connections on the master server with the client as the slave m/c and all are in the state CLOSE_WAIT. Also, the client port specified on the master solr server netstat results is not visible in the netstat results on the client (slave solr) m/c. Following is my environment: - 40 cores in the master solr on m/c 1 - 40 cores in the slave solr on m/c 2 - The replication poll interval is 20 seconds. - Replication part in solrconfig.xml in the slave solr: requestHandler name=/replication class=solr.ReplicationHandler lst name=slave !--fully qualified url for the replication handler of master-- str name=masterUrl$mastercorename/replication/str !--Interval in which the slave should poll master .Format is HH:mm:ss . If this is absent slave does not poll automatically. But a fetchindex can be triggered from the admin or the http API-- str name=pollInterval00:00:20/str !-- The following values are used when the slave connects to the master to download the index files. Default values implicitly set as 5000ms and 1ms respectively. The user DOES NOT need to specify these unless the bandwidth is extremely low or if there is an extremely high latency-- str name=httpConnTimeout5000/str str name=httpReadTimeout1/str /lst /requestHandler Thanks for any pointers. -- Regards, Samarth -- Sincerely yours Mikhail Khludnev Developer Grid Dynamics tel. 1-415-738-8644 Skype: mkhludnev http://www.griddynamics.com mkhlud...@griddynamics.com -- Regards, Samarth
