Thank you all for your feedback.
This is code that I inherited and the example i gave is intended to
demonstrate the memory leak which based on YourKit is
on java/util/LinkedHashMap$Entry. In short, I'm getting core dumps with
"Detail "java/lang/OutOfMemoryError" "Java heap space" received "
Here is a more detailed layout of the code. This is a crawler that runs
24x7 without any recycle logic in place:
init_data()
while (true)
{
HttpSolrClient client = new HttpSolrClient("
http://localhost:8983/solr/core1 <http://192.168.202.129:8983/solr/core1>/");
<<<< this is real code
see_if_we_have_new_data();
send_new_data_to_solr();
client.close(); <<<< this is real code
sleep_for_a_bit(N); <<<< 'N' can be any positive int
}
By default, our Java program is given 4gb of ram "-Xmx4g" and N is set for
5 min. We had a customer set N to 10 second and we started seeing core
dumps with OOM. As I started to debug, I narrowed the OOM to
HttpSolrClient per my original email.
The follow up answers I got suggest that I move the construction of
HttpSolrClient object outside the while loop which I did (but I also had to
move "client.close()" outside the loop) and the leak is gone.
Give this, is this how HttpSolrClient is suppose to be used? If so, what's
the point of HttpSolrClient.close()?
Another side question. I noticed HttpSolrClient has a setBaseUrl(). Now,
if I call it and give it "http://localhost:8983/solr/core1
<http://192.168.202.129:8983/solr/core1>/" (ntoice the "/" at the end) next
time I use HttpSolrClient to send Solr data, I get back 404. The fix is to
remove the ending "/". This is not how the constructor of HttpSolrClient
behaves; HttpSolrClient will take the URL with or without "/".
In summary, it would be good if someone can confirm f we have a memory leak
in HttpSolrClient if used per my example; if so this is a defect. Also,
can someone confirm the fix I used for this issue: move the constructor of
HttpSolrClient outside the loop and reuse the existing object "client".
Again, thank you all for the quick response it is much appreciated.
Steve
On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson <erickerick...@gmail.com>
wrote:
> Assuming you're not really using code like above and it's a test case....
>
> What's your evidence that memory consumption goes up? Are you sure
> you're not just seeing uncollected garbage?
>
> When I attached Java Mission Control to this program it looked pretty
> scary at first, but the heap allocated after old generation garbage
> collections leveled out to a steady state.
>
>
> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood <wun...@wunderwood.org>
> wrote:
> > Create one HttpSolrClient object for each Solr server you are talking
> to. Reuse it for all requests to that Solr server.
> >
> > It will manage a pool of connections and keep them alive for faster
> communication.
> >
> > I took a look at the JavaDoc and the wiki doc, neither one explains this
> well. I don’t think they even point out what is thread safe.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/ (my blog)
> >
> >
> >> On Jan 30, 2016, at 7:42 AM, Susheel Kumar <susheel2...@gmail.com>
> wrote:
> >>
> >> Hi Steve,
> >>
> >> Can you please elaborate what error you are getting and i didn't
> understand
> >> your code above, that why initiating Solr client object is in loop. In
> >> general creating client instance should be outside the loop and a one
> time
> >> activity during the complete execution of program.
> >>
> >> Thanks,
> >> Susheel
> >>
> >> On Sat, Jan 30, 2016 at 8:15 AM, Steven White <swhite4...@gmail.com>
> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> I'm getting memory leak in my code. I narrowed the code to the
> following
> >>> minimal to cause the leak.
> >>>
> >>> while (true) {
> >>> HttpSolrClient client = new HttpSolrClient("
> >>> http://192.168.202.129:8983/solr/core1";);
> >>> client.close();
> >>> }
> >>>
> >>> Is this a defect or an issue in the way I'm using HttpSolrClient?
> >>>
> >>> I'm on Solr 5.2.1
> >>>
> >>> Thanks.
> >>>
> >>> Steve
> >>>
> >
>
