Re: 4820 searchers opened?
Le 6 août 2011 à 02:09, Yonik Seeley a écrit : On Fri, Aug 5, 2011 at 7:30 PM, Paul Libbrecht p...@hoplahup.net wrote: my solr is coming to slowly reach its memory limits (8Gb) and the stats displays me a reasonable fieldCache (1800) but 4820 searchers. That sounds a bit much to me, each has been opened in its own time since the last restart about two weeks ago. Definitely sounds like a reference leak. What version are you using? 1.4.1. Is this stock Solr, or do you have any custom request handlers or anything else that could be forgetting to decrement the reference count of the searchers it uses? I have a custom query-handler and a custom response writer. I also use the velocity response-writer (for debug purposes). None store a searcher or params, I believe. I have a query in the query handler that is a thread-local (it's a large preferring query that I add to every query). Could this be the reason? I also have a thread-local that stores a date-formatter. Should I post my config? paul
Re: 4820 searchers opened?
On Sat, Aug 6, 2011 at 11:31 AM, Paul Libbrecht p...@hoplahup.net wrote: Le 6 août 2011 à 02:09, Yonik Seeley a écrit : On Fri, Aug 5, 2011 at 7:30 PM, Paul Libbrecht p...@hoplahup.net wrote: my solr is coming to slowly reach its memory limits (8Gb) and the stats displays me a reasonable fieldCache (1800) but 4820 searchers. That sounds a bit much to me, each has been opened in its own time since the last restart about two weeks ago. Definitely sounds like a reference leak. What version are you using? 1.4.1. Is this stock Solr, or do you have any custom request handlers or anything else that could be forgetting to decrement the reference count of the searchers it uses? I have a custom query-handler and a custom response writer. Do you always retrieve the searcher via SolrQueryRequest.getSearcher()? If so, there should be no problem... but if you call SolrCore.getSearcher(), that is where leaks can happen if you don't decref the reference returned. I also use the velocity response-writer (for debug purposes). None store a searcher or params, I believe. I have a query in the query handler that is a thread-local (it's a large preferring query that I add to every query). Could this be the reason? As long as it's a normal query that has not been rewritten or weighted, it should have no state tied to any particular reader/searcher and you should be fine. -Yonik http://www.lucidimagination.com I also have a thread-local that stores a date-formatter. Should I post my config? paul
Re: 4820 searchers opened?
Le 6 août 2011 à 17:37, Yonik Seeley a écrit : I have a custom query-handler and a custom response writer. Do you always retrieve the searcher via SolrQueryRequest.getSearcher()? If so, there should be no problem... but if you call SolrCore.getSearcher(), that is where leaks can happen if you don't decref the reference returned. I've been using the following: rb.req.getCore().getSearcher().get().getReader() inside query transforming methods. That seems to be exactly what req.getSearcher() does which I had not noticed. Is there a difference. I also use the velocity response-writer (for debug purposes). None store a searcher or params, I believe. I have a query in the query handler that is a thread-local (it's a large preferring query that I add to every query). Could this be the reason? As long as it's a normal query that has not been rewritten or weighted, it should have no state tied to any particular reader/searcher and you should be fine. How would I know if it gets rewritten or weighted? Does something write to these queries somehow so that the reference to the searcher would be held? paul
Re: 4820 searchers opened?
On Sat, Aug 6, 2011 at 1:35 PM, Paul Libbrecht p...@hoplahup.net wrote: Le 6 août 2011 à 17:37, Yonik Seeley a écrit : I have a custom query-handler and a custom response writer. Do you always retrieve the searcher via SolrQueryRequest.getSearcher()? If so, there should be no problem... but if you call SolrCore.getSearcher(), that is where leaks can happen if you don't decref the reference returned. I've been using the following: rb.req.getCore().getSearcher().get().getReader() Bingo! Code should never do core.getSearcher().get() since core.getSearcher returns a reference that must be decremented when you are done. Using req.getSearcher() is much easier since it ensures that the searcher never changes during the scope of a single request and it handles decrementing the reference when the request is closed. -Yonik http://www.lucidimagination.com
Re: 4820 searchers opened?
Le 6 août 2011 à 19:52, Yonik Seeley a écrit : I've been using the following: rb.req.getCore().getSearcher().get().getReader() Bingo! Code should never do core.getSearcher().get() since core.getSearcher returns a reference that must be decremented when you are done. Using req.getSearcher() is much easier since it ensures that the searcher never changes during the scope of a single request and it handles decrementing the reference when the request is closed. When you say decrementing you do also mean dereference? (seems to be from the RefCounted class which would dereference when ref-count is zero) I also see the javadoc of SolrCore.getSearcher() is quite explicit there. I would suggest to add a display of the closed-ness-status of the reader in the stats.jsp. Most likely all these searchers have a closed reader and one would see the bug. paul
Re: 4820 searchers opened?
This is convincing me... I'd like to experiment and close. So, how can I be sure this is the right thing? I would have thought adding a document and committing would have created a Searcher in my current usage but I do not see the reference list actually being enlarged on my development machine. What would create a new one? paul Le 6 août 2011 à 20:04, Paul Libbrecht a écrit : Le 6 août 2011 à 19:52, Yonik Seeley a écrit : I've been using the following: rb.req.getCore().getSearcher().get().getReader() Bingo! Code should never do core.getSearcher().get() since core.getSearcher returns a reference that must be decremented when you are done. Using req.getSearcher() is much easier since it ensures that the searcher never changes during the scope of a single request and it handles decrementing the reference when the request is closed. When you say decrementing you do also mean dereference? (seems to be from the RefCounted class which would dereference when ref-count is zero) I also see the javadoc of SolrCore.getSearcher() is quite explicit there. I would suggest to add a display of the closed-ness-status of the reader in the stats.jsp. Most likely all these searchers have a closed reader and one would see the bug. paul
Re: 4820 searchers opened?
On Sat, Aug 6, 2011 at 2:17 PM, Paul Libbrecht p...@hoplahup.net wrote: This is convincing me... I'd like to experiment and close. So, how can I be sure this is the right thing? I would have thought adding a document and committing would have created a Searcher in my current usage but I do not see the reference list actually being enlarged on my development machine. It is creating a new searcher, but then closing the old searcher after all currently running requests are done using it (that's what the reference counting is for). After the searcher is closed, it's removed from the list. Pay attention to the address of the searcher on the stats page: searcherName : Searcher@7d0ade7e main You should see the address change after a commit. -Yonik http://www.lucidimagination.com
Re: 4820 searchers opened?
Le 6 août 2011 à 20:21, Yonik Seeley a écrit : It is creating a new searcher, but then closing the old searcher after all currently running requests are done using it (that's what the reference counting is for). After the searcher is closed, it's removed from the list. Not if using: rb.req.getCore().getSearcher().get().getReader() right? Pay attention to the address of the searcher on the stats page: searcherName : Searcher@7d0ade7e main You should see the address change after a commit. I saw that one. But how can I see retention? I'm trying to isolate the probable cause of retention and actually see an impact of my (tiniest!) code change. Since each release does take an amount of time of multiple persons and the actual bug (memory hogging) is only reached after a long time, I would like to be sure of things. paul
Re: 4820 searchers opened?
PS: why is RefCounted not using SoftReference?? I think I would not see my bug then.
Re: 4820 searchers opened?
On Sat, Aug 6, 2011 at 2:30 PM, Paul Libbrecht p...@hoplahup.net wrote: Le 6 août 2011 à 20:21, Yonik Seeley a écrit : It is creating a new searcher, but then closing the old searcher after all currently running requests are done using it (that's what the reference counting is for). After the searcher is closed, it's removed from the list. Not if using: rb.req.getCore().getSearcher().get().getReader() right? Pay attention to the address of the searcher on the stats page: searcherName : Searcher@7d0ade7e main You should see the address change after a commit. I saw that one. But how can I see retention? Oh, I see... you want to re-create the bug so you can see when it is fixed? To trigger the bug, you need to hit a code path that uses the getCore().getSearcher().get() code. So first send a request that hits that buggy code, then add a doc and do a commit, and then you should see more than one searcher on the stats page. -Yonik http://www.lucidimagination.com
Re: 4820 searchers opened?
Le 7 août 2011 à 01:10, Yonik Seeley a écrit : Oh, I see... you want to re-create the bug so you can see when it is fixed? To trigger the bug, you need to hit a code path that uses the getCore().getSearcher().get() code. So first send a request that hits that buggy code, then add a doc and do a commit, and then you should see more than one searcher on the stats page. Of course, I am sorry not to have guessed it. And it works: - count searcherName on the stats page - make a query - make a change - count searcherName on the stats page, it is one bigger Thank you much for your patience, I have a feeling we're going to reach months of stability with our next curriki-solr release. paul
Re: 4820 searchers opened?
On Fri, Aug 5, 2011 at 7:30 PM, Paul Libbrecht p...@hoplahup.net wrote: my solr is coming to slowly reach its memory limits (8Gb) and the stats displays me a reasonable fieldCache (1800) but 4820 searchers. That sounds a bit much to me, each has been opened in its own time since the last restart about two weeks ago. Definitely sounds like a reference leak. Is this stock Solr, or do you have any custom request handlers or anything else that could be forgetting to decrement the reference count of the searchers it uses? What version are you using? -Yonik http://www.lucidimagination.com
