Re: SolrJ - IOException

[Rozdev29]

Was it stalled due to gc pause?

Sent from my iPhone

On Oct 2, 2012, at 10:02 AM, "balaji.gandhi"  
wrote:

> Hi Toke,
> 
> We encountered this issue again. This time the SOLR servers were stalled. We 
> are at 30 TPS.
> 
> Please let us know any updates in the HTTP issue.
> 
> Thanks,
> Balaji
> 
> Balaji Gandhi, Senior Software Developer, Horizontal Platform Services
> Product Engineering  │  Apollo Group, Inc.
> 1225 W. Washington St.  |  AZ23  |  Tempe, AZ  85281
> Phone: 602.713.2417  |  Email: 
> balaji.gan...@apollogrp.edu
> 
> P Go Green. Don't Print. Moreover soft copies can be indexed by algorithms.
> 
> From: Balaji Gandhi
> Sent: Thursday, September 27, 2012 10:52 AM
> To: 'Toke Eskildsen [via Lucene]'
> Subject: RE: SolrJ - IOException
> 
> Here is the stack trace:-
> 
> org.apache.solr.client.solrj.SolrServerException: IOException occured when 
> talking to server:
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:414)
>  at 
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:182)
>  at 
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
>  at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:122) at 
> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:107) at 
> org.apache.solr.handler.dataimport.thread.task.SolrUploadTask.upload(SolrUploadTask.java:31)
>  at 
> org.apache.solr.handler.dataimport.thread.SolrUploader.run(SolrUploader.java:31)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at 
> java.lang.Thread.run(Unknown Source) Caused by: 
> org.apache.http.NoHttpResponseException: The target server failed to respond 
> at 
> org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:101)
>  at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
>  at 
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:282)
>  at 
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
>  at 
> org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:216)
>  at 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
>  at 
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
>  at 
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:647)
>  at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464)
>  at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
>  at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
>  at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
>  at 
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:353)
>  ... 9 more
> 
> Balaji Gandhi, Senior Software Developer, Horizontal Platform Services
> Product Engineering  │  Apollo Group, Inc.
> 1225 W. Washington St.  |  AZ23  |  Tempe, AZ  85281
> Phone: 602.713.2417  |  Email: 
> balaji.gan...@apollogrp.edu
> 
> P Go Green. Don't Print. Moreover soft copies can be indexed by algorithms.
> 
> From: Toke Eskildsen [via Lucene] 
> [mailto:ml-node+s472066n4010082...@n3.nabble.com]
> Sent: Tuesday, September 25, 2012 12:19 AM
> To: Balaji Gandhi
> Subject: Re: SolrJ - IOException
> 
> On Tue, 2012-09-25 at 01:50 +0200, balaji.gandhi wrote:
>> I am encountering this error randomly (under load) when posting to Solr
>> using SolrJ.
>> 
>> Has anyone encountered a similar error?
>> 
>> org.apache.solr.client.solrj.SolrServerException: IOException occured when
>> talking to server at: http://localhost:8080/solr/profile at
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:414)
> [...]
> 
> This looks suspiciously like a potential bug in the HTTP keep-alive flow
> that we encountered some weeks ago. I am guessing that you are issuing
> more than 100 separate updates/second. Could you please provide the full
> stack trace?
> 
> 
> If you reply to this email, your message will be added to the discussion 
> below:
> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4010082.html
> To unsubscribe from SolrJ - IOException, click 
> here.
> NAML

Re: SOLR memory usage jump in JVM

[Rozdev29]

I have used this setting to reduce gc pauses with CMS - java 6 u23

XX:+ParallelRefProcEnabled

With this setting, jvm does gc of weakrefs with multiple threads and pauses are 
low.

Please use this option only when you have multiple cores.

For me, CMS gives better results

Sent from my iPhone

On Sep 19, 2012, at 8:50 AM, Walter Underwood  wrote:

> Ooh, that is a nasty one. Is this JDK 7 only or also in 6?
> 
> It looks like the "-XX:ConcGCThreads=1" option is a workaround, is that right?
> 
> We've had some 1.6 JVMs behave in the same way that bug describes, but I 
> haven't verified it is because of finalizer problems.
> 
> wunder
> 
> On Sep 19, 2012, at 5:43 AM, Erick Erickson wrote:
> 
>> Two in one morning
>> 
>> The JVM bug I'm familiar with is here:
>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7112034
>> 
>> FWIW,
>> Erick
>> 
>> On Wed, Sep 19, 2012 at 8:20 AM, Shawn Heisey  wrote:
>>> On 9/18/2012 9:29 PM, Lance Norskog wrote:
 
 There is a known JVM garbage collection bug that causes this. It has to do
 with reclaiming Weak references, I think in WeakHashMap. Concurrent garbage
 collection collides with this bug and the result is that old field cache
 data is retained after closing the index. The bug is more common with more
 processors doing GC simultaneously.
 
 The symptom is that when you run a monitor, the memory usage rises to a
 peak, drops to a floor, rises again in the classic sawtooth pattern. When
 the GC bug happens, the ceiling becomes the floor, and the sawtooth goes
 from the new floor to a new ceiling. The two sizes are the same. So, 2G to
 5G, over and over, suddenly it is 5G to 8G, over and over.
 
 The bug is fixed in recent Java 7 releases. I'm sorry, but I cannot find
 the bug number.
>>> 
>>> 
>>> I think I ran into this when I was looking at memory usage on my SolrJ
>>> indexing program.  Under Java6, memory usage in jconsole (remotely via JMX)
>>> was fairly constant long-term (aside from the unavoidable sawtooth).  When I
>>> ran it under Java 7u3, it would continually grow, slowly ... but if I
>>> measured it with jstat on the Linux commandline rather than remotely via
>>> jconsole under windows, memory usage was consistent over time, just like
>>> under java6 with the remote jconsole.  After looking at heap dumps and
>>> scratching my head a lot, I finally concluded that I did not have a memory
>>> leak, there was a problem with remote JMX monitoring in java7.  Glad to hear
>>> I was not imagining it, and that it's fixed now.
>>> 
>>> Thanks,
>>> Shawn
>>> 
> 
> --
> Walter Underwood
> wun...@wunderwood.org
> 
> 
>