timeouts when update sent to non-Leader

[Oakley, Craig (NIH/NLM/NCBI) [C]]

We have a problem on a 3.5gig collection running Solr7.4 (we will soon upgrade 
to Solr8.5.2)

Users were often encountering timeout errors of the type shown below

My colleague found a blog post at 
https://stackoverflow.com/questions/62075353/solr-8-4-getting-async-exception-during-distributed-update-java-io-ioexception
 which prompted him to ask the users to direct their updates to the host which 
has the Leader replica rather than the host that has the non-Leader replica.

This seems to have provided a temporary solution; but should not SolrCloud be 
able to handle updates sent to any node of the SolrCloud? Generally our 
experience has been that SolrCloud is able to handle this as advertised; but 
has anyone else encountered (and perhaps corrected) this phenomenon?


In case it is helpful, the users' error message is below:

Error: (CSolrClientException::Http error) 



  -1
  656637


  
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException
  
  Async exception during distributed update: Read timed 
out
  org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
 Async exception during distributed update: Read timed out
at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:944)
at 
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1946)
at 
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2539)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at 
org.eclipse

Debugging indexing timeouts

[fredsearch157]

Hi all,

A couple of months ago, I migrated my solr deployment off of some legacy 
hardware (old spinning disks), and onto much newer hardware (SSD's, newer 
processors). While I am seeing much improved search performance since this 
move, I am also seeing intermittent indexing timeouts for 10-15 min periods 
about once a day or so (both from my indexing code and between replicas), which 
were not happening before. I have been scratching my head trying to figure out 
why, but have thus far been unsuccessful. I was hoping someone on here could 
maybe offer some thoughts as to how to further debug.

Some information about my setup:
-Solr Cloud 8.3, running on linux
-2 nodes, 1 shard (2 replicas) per collection
-Handful of collections, maxing out in the 10s of millions of docs per 
collection. Less than 100 million docs total
-Nodes are 8 CPU cores with SSD storage. 64 GB of RAM on server, heap size of 
26 GB.
-Relatively aggressive NRT tuning (hard commit 60 sec, soft commit 15 sec).
-Multi-threaded indexing process using SolrJ CloudSolrClient, sending updates 
in batches of ~1000 docs
-Indexing and querying is done constantly throughout the day

The indexing process, heap sizes, and soft/hard commit intervals were carefully 
tuned for my original setup, and were working flawlessly until the hardware 
change. It's only since the move to faster hardware/SSDs that I am now seeing 
timeouts during indexing (maybe counter-intuitively).

My first thought was that I was having stop the world GC pauses which were 
causing the timeouts, but when I captured GC logs during one of the timeout 
windows and ran it through a log analyzer, there were no issues detected. 
Largest GC pause was under 1 second. I monitor the heap continuously, and I 
always sit between 15-20 GB of 26 GB used...so I don't think that my heap is 
too small necessarily.

My next thought was that maybe it had to do with segment merges happening in 
the background, causing indexing to block. I am using the dynamic defaults for 
the merge scheduler, which almost certainly changed when I moved hardware 
(since now it is detecting a non-spinning disk, and my understanding is that 
the max concurrent merges is set based on this). I have been unable to confirm 
this though. I do not see any merge warnings or errors in the logs, and I have 
thus far been able to catch it in action to try and confirm via a thread dump.

Interestingly, when I did take a thread dump during normal execution, I noticed 
that one of my nodes has a huge number of running threads (~1700) compared to 
the other node (~150). Most of the threads are updateExecutor threads that 
appear to be permanently in a waiting state. I'm not sure what causes the node 
to get into this state, or if it is related to the timeouts at all.

I have thus far been unable to replicate the issue in a test environment, so 
it's hard to trial and error possible solutions. Does anyone have any 
suggestions on what could be causing these timeouts all of a sudden, or tips on 
how to debug further?

Thanks!

Re: Solr standalone timeouts after upgrading to SOLR 7

[Walter Underwood]

Just set Xms and Xmx the same. The server will be running for weeks,
so allocate the memory and get on with it.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 3, 2019, at 11:38 AM, ndra  wrote:
> 
>> I don’t think having the initial heap larger than the max heap is a legal
> configuration.
>> I have no idea what that would do.
> 
> Sorry my wording was poor. I meant if, instead of my initial HEAP of 512MB,
> it was closer to say 6 or 8GB or equal to my Max allowed of 10GB.
> 
> Appreciate the info though.
> 
> Thanks Watler
> 
> Nick
> 
> On Thu, Oct 3, 2019 at 10:57 AM Walter Underwood 
> wrote:
> 
>> I don’t think having the initial heap larger than the max heap is a legal
>> configuration.
>> I have no idea what that would do.
>> 
>> Modern GCs have a separate area for short-lived allocations. When that
>> fills up,
>> a minor GC happens. As allocations survive several minor GCs, they are
>> moved
>> to the long-lived space, which is managed by major GCs. This is called a
>> generational
>> collector.
>> 
>> In Solr, nearly all allocations are for a single request, so those are
>> garbage after the
>> response is sent. Most of the action will be in the new allocation
>> (short-lived) space.
>> Cache allocations get moved to the long-lived space.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>>> On Oct 3, 2019, at 10:11 AM, ndra  wrote:
>>> 
 When the heap is out of free space that
 can be recovered with minor GC, the JVM will increase the size if
>> possible.
 Once it is at max, it will do a major GC.
>>> 
>>> Thanks Walter,
>>> 
>>> One more quick question about the above, so if the initial HEAP was
>> larger
>>> (or equal to the max as you suggested earlier) will the GCs minor GCs be
>>> less frequent because the HEAP still has available space?
>>> 
>>> Unfortunately the system was rebooted by a infrastructure person prior to
>>> looking at `/admin/system/info` endpoint to look at the stats. Is there
>>> another way get those statistics (there is no JVM monitoring in place
>> from
>>> our INF team so I am starting this all from scratch right now.
>>> 
>>> Nick
>>> 
>>> 
>>> On Thu, Oct 3, 2019 at 9:44 AM Walter Underwood 
>>> wrote:
>>> 
> On Oct 3, 2019, at 9:31 AM, ndra  wrote:
> 
> I was under the impression that by allocating a smaller initial HEAP I
> could avoid having a larger GCs but if I am understanding what you all
 are
> suggesting, the smaller initial HEAP is requiring more full GCs to
 maintian
> a HEAP closer to  512MB. Is that correct?
 
 I don’t think either of those are true. When the heap is out of free
>> space
 that
 can be recovered with minor GC, the JVM will increase the size if
>> possible.
 Once it is at max, it will do a major GC.
 
 We use these settings in solr.in.sh:
 
 SOLR_HEAP=8g
 # Use G1 GC  -- wunder 2017-01-23
 # Settings from https://wiki.apache.org/solr/ShawnHeisey
 GC_TUNE=" \
 -XX:+UseG1GC \
 -XX:+ParallelRefProcEnabled \
 -XX:G1HeapRegionSize=8m \
 -XX:MaxGCPauseMillis=200 \
 -XX:+UseLargePages \
 -XX:+AggressiveOpts \
 "
 wunder
 Walter Underwood
 wun...@wunderwood.org
 http://observer.wunderwood.org/  (my blog)
 
 
>> 
>> 

Re: Solr standalone timeouts after upgrading to SOLR 7

[ndra]

> I don’t think having the initial heap larger than the max heap is a legal
configuration.
> I have no idea what that would do.

Sorry my wording was poor. I meant if, instead of my initial HEAP of 512MB,
it was closer to say 6 or 8GB or equal to my Max allowed of 10GB.

Appreciate the info though.

Thanks Watler

Nick

On Thu, Oct 3, 2019 at 10:57 AM Walter Underwood 
wrote:

> I don’t think having the initial heap larger than the max heap is a legal
> configuration.
> I have no idea what that would do.
>
> Modern GCs have a separate area for short-lived allocations. When that
> fills up,
> a minor GC happens. As allocations survive several minor GCs, they are
> moved
> to the long-lived space, which is managed by major GCs. This is called a
> generational
> collector.
>
> In Solr, nearly all allocations are for a single request, so those are
> garbage after the
> response is sent. Most of the action will be in the new allocation
> (short-lived) space.
> Cache allocations get moved to the long-lived space.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Oct 3, 2019, at 10:11 AM, ndra  wrote:
> >
> >> When the heap is out of free space that
> >> can be recovered with minor GC, the JVM will increase the size if
> possible.
> >> Once it is at max, it will do a major GC.
> >
> > Thanks Walter,
> >
> > One more quick question about the above, so if the initial HEAP was
> larger
> > (or equal to the max as you suggested earlier) will the GCs minor GCs be
> > less frequent because the HEAP still has available space?
> >
> > Unfortunately the system was rebooted by a infrastructure person prior to
> > looking at `/admin/system/info` endpoint to look at the stats. Is there
> > another way get those statistics (there is no JVM monitoring in place
> from
> > our INF team so I am starting this all from scratch right now.
> >
> > Nick
> >
> >
> > On Thu, Oct 3, 2019 at 9:44 AM Walter Underwood 
> > wrote:
> >
> >>> On Oct 3, 2019, at 9:31 AM, ndra  wrote:
> >>>
> >>> I was under the impression that by allocating a smaller initial HEAP I
> >>> could avoid having a larger GCs but if I am understanding what you all
> >> are
> >>> suggesting, the smaller initial HEAP is requiring more full GCs to
> >> maintian
> >>> a HEAP closer to  512MB. Is that correct?
> >>
> >> I don’t think either of those are true. When the heap is out of free
> space
> >> that
> >> can be recovered with minor GC, the JVM will increase the size if
> possible.
> >> Once it is at max, it will do a major GC.
> >>
> >> We use these settings in solr.in.sh:
> >>
> >> SOLR_HEAP=8g
> >> # Use G1 GC  -- wunder 2017-01-23
> >> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> >> GC_TUNE=" \
> >> -XX:+UseG1GC \
> >> -XX:+ParallelRefProcEnabled \
> >> -XX:G1HeapRegionSize=8m \
> >> -XX:MaxGCPauseMillis=200 \
> >> -XX:+UseLargePages \
> >> -XX:+AggressiveOpts \
> >> "
> >> wunder
> >> Walter Underwood
> >> wun...@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>
>
>

Re: Solr standalone timeouts after upgrading to SOLR 7

[Walter Underwood]

I don’t think having the initial heap larger than the max heap is a legal 
configuration.
I have no idea what that would do.

Modern GCs have a separate area for short-lived allocations. When that fills up,
a minor GC happens. As allocations survive several minor GCs, they are moved 
to the long-lived space, which is managed by major GCs. This is called a 
generational
collector.

In Solr, nearly all allocations are for a single request, so those are garbage 
after the
response is sent. Most of the action will be in the new allocation 
(short-lived) space.
Cache allocations get moved to the long-lived space.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 3, 2019, at 10:11 AM, ndra  wrote:
> 
>> When the heap is out of free space that
>> can be recovered with minor GC, the JVM will increase the size if possible.
>> Once it is at max, it will do a major GC.
> 
> Thanks Walter,
> 
> One more quick question about the above, so if the initial HEAP was larger
> (or equal to the max as you suggested earlier) will the GCs minor GCs be
> less frequent because the HEAP still has available space?
> 
> Unfortunately the system was rebooted by a infrastructure person prior to
> looking at `/admin/system/info` endpoint to look at the stats. Is there
> another way get those statistics (there is no JVM monitoring in place from
> our INF team so I am starting this all from scratch right now.
> 
> Nick
> 
> 
> On Thu, Oct 3, 2019 at 9:44 AM Walter Underwood 
> wrote:
> 
>>> On Oct 3, 2019, at 9:31 AM, ndra  wrote:
>>> 
>>> I was under the impression that by allocating a smaller initial HEAP I
>>> could avoid having a larger GCs but if I am understanding what you all
>> are
>>> suggesting, the smaller initial HEAP is requiring more full GCs to
>> maintian
>>> a HEAP closer to  512MB. Is that correct?
>> 
>> I don’t think either of those are true. When the heap is out of free space
>> that
>> can be recovered with minor GC, the JVM will increase the size if possible.
>> Once it is at max, it will do a major GC.
>> 
>> We use these settings in solr.in.sh:
>> 
>> SOLR_HEAP=8g
>> # Use G1 GC  -- wunder 2017-01-23
>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> "
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 

Re: Solr standalone timeouts after upgrading to SOLR 7

[ndra]

> When the heap is out of free space that
>can be recovered with minor GC, the JVM will increase the size if possible.
>Once it is at max, it will do a major GC.

Thanks Walter,

One more quick question about the above, so if the initial HEAP was larger
(or equal to the max as you suggested earlier) will the GCs minor GCs be
less frequent because the HEAP still has available space?

Unfortunately the system was rebooted by a infrastructure person prior to
looking at `/admin/system/info` endpoint to look at the stats. Is there
another way get those statistics (there is no JVM monitoring in place from
our INF team so I am starting this all from scratch right now.

Nick


On Thu, Oct 3, 2019 at 9:44 AM Walter Underwood 
wrote:

> > On Oct 3, 2019, at 9:31 AM, ndra  wrote:
> >
> > I was under the impression that by allocating a smaller initial HEAP I
> > could avoid having a larger GCs but if I am understanding what you all
> are
> > suggesting, the smaller initial HEAP is requiring more full GCs to
> maintian
> > a HEAP closer to  512MB. Is that correct?
>
> I don’t think either of those are true. When the heap is out of free space
> that
> can be recovered with minor GC, the JVM will increase the size if possible.
> Once it is at max, it will do a major GC.
>
> We use these settings in solr.in.sh:
>
> SOLR_HEAP=8g
> # Use G1 GC  -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
>

Re: Solr standalone timeouts after upgrading to SOLR 7

[Walter Underwood]

> On Oct 3, 2019, at 9:31 AM, ndra  wrote:
> 
> I was under the impression that by allocating a smaller initial HEAP I
> could avoid having a larger GCs but if I am understanding what you all are
> suggesting, the smaller initial HEAP is requiring more full GCs to maintian
> a HEAP closer to  512MB. Is that correct?

I don’t think either of those are true. When the heap is out of free space that 
can be recovered with minor GC, the JVM will increase the size if possible.
Once it is at max, it will do a major GC.

We use these settings in solr.in.sh:

SOLR_HEAP=8g
# Use G1 GC  -- wunder 2017-01-23
# Settings from https://wiki.apache.org/solr/ShawnHeisey
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

Re: Solr standalone timeouts after upgrading to SOLR 7

[ndra]

Thank you all for the responses. I do have a question about moving to G1.
Is there any plans for SOLR to make this the default JAVA_OPT for SOLR? We
have been running CMS without any issue prior to 7.


The primary change we made between 6->7 was to increase the max HEAP size
and increase the Xms as well (went from 1 and 8 to 2 and 10). We saw
similar issues with these new values so I reduced the Xms based on my
understanding of this (from
https://lucene.apache.org/solr/guide/7_7/jvm-settings.html)

> The larger the heap the longer it takes to do garbage collection. This
can mean minor, random pauses or, in extreme cases, "freeze the world"
pauses of a minute or more. As a practical matter, this >can become a
serious problem for heap sizes that exceed about two gigabytes, even if far
more physical memory is available. On robust hardware, you may get better
results running multiple JVMs, >rather than just one with a large memory
heap. Some specialized JVM implementations may have customized garbage
collection algorithms that do better with large heaps. Consult your JVM
vendor’s >documentation.


I was under the impression that by allocating a smaller initial HEAP I
could avoid having a larger GCs but if I am understanding what you all are
suggesting, the smaller initial HEAP is requiring more full GCs to maintian
a HEAP closer to  512MB. Is that correct?

Nick

On Thu, Oct 3, 2019 at 8:24 AM Jörn Franke  wrote:

> As the others said: move to the newer GC. I would also use this
> opportunity to work with the default Java options of Solr 7 and then tune
> again. If you change major versions you should always review the GC
> settings if they still make sense.
>
> > Am 02.10.2019 um 23:14 schrieb Solr User :
> >
> > Hello all,
> >
> > We recently moved to SOLR 7 from SOLR 6 about 2 weeks ago. Once each week
> > (including today) we experienced query timeout issues with corresponding
> GC
> > events. There was a spike in CPU up to 66% which is not something we
> > previously saw w/ Solr 6. From the SOLR logs it looks like something
> inside
> > the JVM has happend, SOLR is reporting closed connections from Jetty. Our
> > data size is relatively small but we do run 5 cores within the one Jetty
> > instance. There index sizes are anywhere between 200Mb to 2GB
> >
> > Our memory consumption is relatively low:
> >
> > "free":"296.1 MB",
> >
> >  "total":"569.6 MB",
> >
> >  "max":"9.6 GB",
> >
> >  "used":"273.5 MB (%2.8)",
> >
> >
> >
> > We had a spike in traffic about 5 minutes prior to some longer GC events
> > (similar situation last week).
> >
> > Any help would be appreciated. Below is my current system info along
> with a
> > GC log snippet and the corresponding SOLR log error.
> >
> > *System info:*
> > AMZ2 linux
> > 8 core 32 GB Mem
> > *Java:* 1.8.0_222-ea 25.222-b03
> > *Solr: *solr-spec-version":"7.7.2"
> > *Start options: *
> > "-Xms512m",
> >"-Xmx10g",
> >"-XX:NewRatio=3",
> >"-XX:SurvivorRatio=4",
> >"-XX:TargetSurvivorRatio=90",
> >"-XX:MaxTenuringThreshold=8",
> >"-XX:+UseConcMarkSweepGC",
> >"-XX:ConcGCThreads=4",
> >"-XX:ParallelGCThreads=4",
> >"-XX:+CMSScavengeBeforeRemark",
> >"-XX:PretenureSizeThreshold=64m",
> >"-XX:+UseCMSInitiatingOccupancyOnly",
> >"-XX:CMSInitiatingOccupancyFraction=50",
> >"-XX:CMSMaxAbortablePrecleanTime=6000",
> >"-XX:+CMSParallelRemarkEnabled",
> >"-XX:+ParallelRefProcEnabled",
> >"-XX:-OmitStackTraceInFastThrow",
> >"-verbose:gc",
> >"-XX:+PrintHeapAtGC",
> >"-XX:+PrintGCDetails",
> >"-XX:+PrintGCDateStamps",
> >"-XX:+PrintGCTimeStamps",
> >"-XX:+PrintTenuringDistribution",
> >"-XX:+PrintGCApplicationStoppedTime",
> >"-XX:+UseGCLogFileRotation",
> >"-XX:NumberOfGCLogFiles=9",
> >"-XX:GCLogFileSize=20M",
> >"-Xss256k",
> >"-Dsolr.log.muteconsole"
> >
> > Here is an example of from the GC log:
> >
> > 2019-10-02T16:03:15.888+: 265318.624: [Full GC (Allocation
> > Failure) 2019-10-02T16:03:15.888+: 265318.624:
> > [CMS2019-10-02T16:03:16.134+: 26
> > 5318.870: [CMS-concurrent-mark: 1.773/1.783 secs] [Times: user=13.14
> > sys=0.00, real=1.78 secs]
> > (concurrent mode failure): 7864319K->7864319K(7864320K), 9.5890129
> > secs] 10048895K->8863021K(10048896K), [Metaspace:
> > 53159K->53159K(1097728K)], 9.5892061 secs] [Times: user=10.31
> > sys=0.00, real=9.59 secs]
> > Heap after GC invocations=296656 (full 546):
> > par new generation   total 2184576K, used 998701K
> > [0x00054000, 0x0005e000, 0x0005e000)
> >  eden space 1747712K,  57% used [0x00054000,
> > 0x00057cf4b4f0, 0x0005aaac)
> >  from space 436864K,   0% used [0x0005aaac,
> > 0x0005aaac, 0x0005c556)
> >  to   space 436864K,   0% used [0x0005c556,
> > 0x0005c556, 0x0005e000)

Re: Solr standalone timeouts after upgrading to SOLR 7

[Jörn Franke]

As the others said: move to the newer GC. I would also use this opportunity to 
work with the default Java options of Solr 7 and then tune again. If you change 
major versions you should always review the GC settings if they still make 
sense.

> Am 02.10.2019 um 23:14 schrieb Solr User :
> 
> Hello all,
> 
> We recently moved to SOLR 7 from SOLR 6 about 2 weeks ago. Once each week
> (including today) we experienced query timeout issues with corresponding GC
> events. There was a spike in CPU up to 66% which is not something we
> previously saw w/ Solr 6. From the SOLR logs it looks like something inside
> the JVM has happend, SOLR is reporting closed connections from Jetty. Our
> data size is relatively small but we do run 5 cores within the one Jetty
> instance. There index sizes are anywhere between 200Mb to 2GB
> 
> Our memory consumption is relatively low:
> 
> "free":"296.1 MB",
> 
>  "total":"569.6 MB",
> 
>  "max":"9.6 GB",
> 
>  "used":"273.5 MB (%2.8)",
> 
> 
> 
> We had a spike in traffic about 5 minutes prior to some longer GC events
> (similar situation last week).
> 
> Any help would be appreciated. Below is my current system info along with a
> GC log snippet and the corresponding SOLR log error.
> 
> *System info:*
> AMZ2 linux
> 8 core 32 GB Mem
> *Java:* 1.8.0_222-ea 25.222-b03
> *Solr: *solr-spec-version":"7.7.2"
> *Start options: *
> "-Xms512m",
>"-Xmx10g",
>"-XX:NewRatio=3",
>"-XX:SurvivorRatio=4",
>"-XX:TargetSurvivorRatio=90",
>"-XX:MaxTenuringThreshold=8",
>"-XX:+UseConcMarkSweepGC",
>"-XX:ConcGCThreads=4",
>"-XX:ParallelGCThreads=4",
>"-XX:+CMSScavengeBeforeRemark",
>"-XX:PretenureSizeThreshold=64m",
>"-XX:+UseCMSInitiatingOccupancyOnly",
>"-XX:CMSInitiatingOccupancyFraction=50",
>"-XX:CMSMaxAbortablePrecleanTime=6000",
>"-XX:+CMSParallelRemarkEnabled",
>"-XX:+ParallelRefProcEnabled",
>"-XX:-OmitStackTraceInFastThrow",
>"-verbose:gc",
>"-XX:+PrintHeapAtGC",
>"-XX:+PrintGCDetails",
>"-XX:+PrintGCDateStamps",
>"-XX:+PrintGCTimeStamps",
>"-XX:+PrintTenuringDistribution",
>"-XX:+PrintGCApplicationStoppedTime",
>"-XX:+UseGCLogFileRotation",
>"-XX:NumberOfGCLogFiles=9",
>"-XX:GCLogFileSize=20M",
>"-Xss256k",
>"-Dsolr.log.muteconsole"
> 
> Here is an example of from the GC log:
> 
> 2019-10-02T16:03:15.888+: 265318.624: [Full GC (Allocation
> Failure) 2019-10-02T16:03:15.888+: 265318.624:
> [CMS2019-10-02T16:03:16.134+: 26
> 5318.870: [CMS-concurrent-mark: 1.773/1.783 secs] [Times: user=13.14
> sys=0.00, real=1.78 secs]
> (concurrent mode failure): 7864319K->7864319K(7864320K), 9.5890129
> secs] 10048895K->8863021K(10048896K), [Metaspace:
> 53159K->53159K(1097728K)], 9.5892061 secs] [Times: user=10.31
> sys=0.00, real=9.59 secs]
> Heap after GC invocations=296656 (full 546):
> par new generation   total 2184576K, used 998701K
> [0x00054000, 0x0005e000, 0x0005e000)
>  eden space 1747712K,  57% used [0x00054000,
> 0x00057cf4b4f0, 0x0005aaac)
>  from space 436864K,   0% used [0x0005aaac,
> 0x0005aaac, 0x0005c556)
>  to   space 436864K,   0% used [0x0005c556,
> 0x0005c556, 0x0005e000)
> concurrent mark-sweep generation total 7864320K, used 7864319K
> [0x0005e000, 0x0007c000, 0x0007c000)
> Metaspace   used 53159K, capacity 54766K, committed 55148K,
> reserved 1097728K
>  class spaceused 5589K, capacity 5950K, committed 6000K, reserved 1048576K
> }
> 2019-10-02T16:03:25.477+: 265328.214: Total time for which
> application threads were stopped: 9.5906157 seconds, Stopping threads
> took: 0.0001274 seconds
> *With the following from the SOLR log: *
> 
> [   x:core] o.a.s.s.HttpSolrCall Unable to write response, client
> closed connection or we are s
> 
> hutting down
> 
> org.eclipse.jetty.io.EofException: Closed
> 
>at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:665)
> ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
> 
>at 
> org.apache.solr.servlet.ServletOutputStreamWrapper.write(ServletOutputStreamWrapper.java:126)
> ~[solr-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c
> 
> 1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48]
> 
>at 
> org.apache.solr.response.QueryResponseWriterUtil$1.write(QueryResponseWriterUtil.java:54)
> ~[solr-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fef
> 
> c589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48]
> 
>at java.io.OutputStream.write(OutputStream.java:116) ~[?:1.8.0_222-ea]
> 
>at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> ~[?:1.8.0_222-ea]
> 
>at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> ~[?:1.8.0_222-ea]
> 
>at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> ~[?:1.8.0_222-e

Re: Solr standalone timeouts after upgrading to SOLR 7

[Walter Underwood]

Always make Xmx and Xms the same. The heap will increase to the max before a 
major GC, so avoid the pauses to grow it.

Use the G1 collector. CMS is really obsolete. We’ve had G1 in prod for at least 
three years.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 2, 2019, at 11:07 PM, Paras Lehana  wrote:
> 
> Hi concerned (please use signature),
> 
> The timeouts could be because of the "stop the world" kinda behaviour of
> longer GC pause. When we shifted our Auto-Suggest from Redis to Solr, I
> faced the similar GC pause issue while stress testing. This article
> <https://cwiki.apache.org/confluence/display/solr/ShawnHeisey#ShawnHeisey-GCTuningforSolr>
> by Shawn Heisey helped me a lot to experiment with different setups - I
> tried combinations of different collectors, heap and other configs.
> 
> I think your -Xms and -Xmx values have too much of the relative difference.
> Your heap could be getting filled up to the max 10G during peak times and
> then the longer GC pauses would have made pending queries go timeout. By
> seeing your current memory usage, I recommend you to decrease the Xmx. Try
> with same values first or maybe 4,6G. If you think 10G is okay, then
> consider increasing your Xms to maybe 6-8G? Since this is a sort of
> experiment you have to perform and I can only recommend things as per my
> experience, I strongly recommend you to read Shawn's article.
> 
> On Thu, 3 Oct 2019 at 02:44, Solr User  wrote:
> 
>> Hello all,
>> 
>> We recently moved to SOLR 7 from SOLR 6 about 2 weeks ago. Once each week
>> (including today) we experienced query timeout issues with corresponding GC
>> events. There was a spike in CPU up to 66% which is not something we
>> previously saw w/ Solr 6. From the SOLR logs it looks like something inside
>> the JVM has happend, SOLR is reporting closed connections from Jetty. Our
>> data size is relatively small but we do run 5 cores within the one Jetty
>> instance. There index sizes are anywhere between 200Mb to 2GB
>> 
>> Our memory consumption is relatively low:
>> 
>> "free":"296.1 MB",
>> 
>>  "total":"569.6 MB",
>> 
>>  "max":"9.6 GB",
>> 
>>  "used":"273.5 MB (%2.8)",
>> 
>> 
>> 
>> We had a spike in traffic about 5 minutes prior to some longer GC events
>> (similar situation last week).
>> 
>> Any help would be appreciated. Below is my current system info along with a
>> GC log snippet and the corresponding SOLR log error.
>> 
>> *System info:*
>> AMZ2 linux
>> 8 core 32 GB Mem
>> *Java:* 1.8.0_222-ea 25.222-b03
>> *Solr: *solr-spec-version":"7.7.2"
>> *Start options: *
>> "-Xms512m",
>>"-Xmx10g",
>>"-XX:NewRatio=3",
>>"-XX:SurvivorRatio=4",
>>"-XX:TargetSurvivorRatio=90",
>>"-XX:MaxTenuringThreshold=8",
>>"-XX:+UseConcMarkSweepGC",
>>"-XX:ConcGCThreads=4",
>>"-XX:ParallelGCThreads=4",
>>"-XX:+CMSScavengeBeforeRemark",
>>"-XX:PretenureSizeThreshold=64m",
>>"-XX:+UseCMSInitiatingOccupancyOnly",
>>"-XX:CMSInitiatingOccupancyFraction=50",
>>"-XX:CMSMaxAbortablePrecleanTime=6000",
>>"-XX:+CMSParallelRemarkEnabled",
>>"-XX:+ParallelRefProcEnabled",
>>"-XX:-OmitStackTraceInFastThrow",
>>"-verbose:gc",
>>"-XX:+PrintHeapAtGC",
>>"-XX:+PrintGCDetails",
>>"-XX:+PrintGCDateStamps",
>>"-XX:+PrintGCTimeStamps",
>>"-XX:+PrintTenuringDistribution",
>>"-XX:+PrintGCApplicationStoppedTime",
>>"-XX:+UseGCLogFileRotation",
>>"-XX:NumberOfGCLogFiles=9",
>>"-XX:GCLogFileSize=20M",
>>"-Xss256k",
>>"-Dsolr.log.muteconsole"
>> 
>> Here is an example of from the GC log:
>> 
>> 2019-10-02T16:03:15.888+: 265318.624: [Full GC (Allocation
>> Failure) 2019-10-02T16:03:15.888+: 265318.624:
>> [CMS2019-10-02T16:03:16.134+: 26
>> 5318.870: [CMS-concurrent-mark: 1.773/1.783 secs] [Times: user=13.14
>> sys=0.00, real=1.78 secs]
>> (concurrent mode failure): 7864319K->7864319K(7864320K), 9.5890129
&g

Re: Solr standalone timeouts after upgrading to SOLR 7

[Paras Lehana]

Hi concerned (please use signature),

The timeouts could be because of the "stop the world" kinda behaviour of
longer GC pause. When we shifted our Auto-Suggest from Redis to Solr, I
faced the similar GC pause issue while stress testing. This article
<https://cwiki.apache.org/confluence/display/solr/ShawnHeisey#ShawnHeisey-GCTuningforSolr>
by Shawn Heisey helped me a lot to experiment with different setups - I
tried combinations of different collectors, heap and other configs.

I think your -Xms and -Xmx values have too much of the relative difference.
Your heap could be getting filled up to the max 10G during peak times and
then the longer GC pauses would have made pending queries go timeout. By
seeing your current memory usage, I recommend you to decrease the Xmx. Try
with same values first or maybe 4,6G. If you think 10G is okay, then
consider increasing your Xms to maybe 6-8G? Since this is a sort of
experiment you have to perform and I can only recommend things as per my
experience, I strongly recommend you to read Shawn's article.

On Thu, 3 Oct 2019 at 02:44, Solr User  wrote:

> Hello all,
>
> We recently moved to SOLR 7 from SOLR 6 about 2 weeks ago. Once each week
> (including today) we experienced query timeout issues with corresponding GC
> events. There was a spike in CPU up to 66% which is not something we
> previously saw w/ Solr 6. From the SOLR logs it looks like something inside
> the JVM has happend, SOLR is reporting closed connections from Jetty. Our
> data size is relatively small but we do run 5 cores within the one Jetty
> instance. There index sizes are anywhere between 200Mb to 2GB
>
> Our memory consumption is relatively low:
>
> "free":"296.1 MB",
>
>   "total":"569.6 MB",
>
>   "max":"9.6 GB",
>
>   "used":"273.5 MB (%2.8)",
>
>
>
> We had a spike in traffic about 5 minutes prior to some longer GC events
> (similar situation last week).
>
> Any help would be appreciated. Below is my current system info along with a
> GC log snippet and the corresponding SOLR log error.
>
> *System info:*
> AMZ2 linux
> 8 core 32 GB Mem
> *Java:* 1.8.0_222-ea 25.222-b03
> *Solr: *solr-spec-version":"7.7.2"
> *Start options: *
> "-Xms512m",
> "-Xmx10g",
> "-XX:NewRatio=3",
> "-XX:SurvivorRatio=4",
> "-XX:TargetSurvivorRatio=90",
> "-XX:MaxTenuringThreshold=8",
> "-XX:+UseConcMarkSweepGC",
> "-XX:ConcGCThreads=4",
> "-XX:ParallelGCThreads=4",
> "-XX:+CMSScavengeBeforeRemark",
> "-XX:PretenureSizeThreshold=64m",
> "-XX:+UseCMSInitiatingOccupancyOnly",
> "-XX:CMSInitiatingOccupancyFraction=50",
> "-XX:CMSMaxAbortablePrecleanTime=6000",
> "-XX:+CMSParallelRemarkEnabled",
> "-XX:+ParallelRefProcEnabled",
> "-XX:-OmitStackTraceInFastThrow",
> "-verbose:gc",
> "-XX:+PrintHeapAtGC",
> "-XX:+PrintGCDetails",
> "-XX:+PrintGCDateStamps",
> "-XX:+PrintGCTimeStamps",
> "-XX:+PrintTenuringDistribution",
> "-XX:+PrintGCApplicationStoppedTime",
> "-XX:+UseGCLogFileRotation",
> "-XX:NumberOfGCLogFiles=9",
> "-XX:GCLogFileSize=20M",
> "-Xss256k",
> "-Dsolr.log.muteconsole"
>
> Here is an example of from the GC log:
>
> 2019-10-02T16:03:15.888+: 265318.624: [Full GC (Allocation
> Failure) 2019-10-02T16:03:15.888+: 265318.624:
> [CMS2019-10-02T16:03:16.134+: 26
> 5318.870: [CMS-concurrent-mark: 1.773/1.783 secs] [Times: user=13.14
> sys=0.00, real=1.78 secs]
>  (concurrent mode failure): 7864319K->7864319K(7864320K), 9.5890129
> secs] 10048895K->8863021K(10048896K), [Metaspace:
> 53159K->53159K(1097728K)], 9.5892061 secs] [Times: user=10.31
> sys=0.00, real=9.59 secs]
> Heap after GC invocations=296656 (full 546):
>  par new generation   total 2184576K, used 998701K
> [0x00054000, 0x0005e000, 0x0005e000)
>   eden space 1747712K,  57% used [0x00054000,
> 0x00057cf4b4f0, 0x0005aaac)
>   from space 436864K,   0% used [0x0005aaac,
> 0x0005aaac, 0x0005c556)
>   to   space 436864K,   0% used [0x0005c556,
> 0x0005c556, 0x0005e000)
>  concurrent mark-sweep generation total 7864320K, used 7864319K
> [0x0005e000, 0x0

Solr standalone timeouts after upgrading to SOLR 7

[Solr User]

Hello all,

We recently moved to SOLR 7 from SOLR 6 about 2 weeks ago. Once each week
(including today) we experienced query timeout issues with corresponding GC
events. There was a spike in CPU up to 66% which is not something we
previously saw w/ Solr 6. From the SOLR logs it looks like something inside
the JVM has happend, SOLR is reporting closed connections from Jetty. Our
data size is relatively small but we do run 5 cores within the one Jetty
instance. There index sizes are anywhere between 200Mb to 2GB

Our memory consumption is relatively low:

"free":"296.1 MB",

  "total":"569.6 MB",

  "max":"9.6 GB",

  "used":"273.5 MB (%2.8)",



We had a spike in traffic about 5 minutes prior to some longer GC events
(similar situation last week).

Any help would be appreciated. Below is my current system info along with a
GC log snippet and the corresponding SOLR log error.

*System info:*
AMZ2 linux
8 core 32 GB Mem
*Java:* 1.8.0_222-ea 25.222-b03
*Solr: *solr-spec-version":"7.7.2"
*Start options: *
"-Xms512m",
"-Xmx10g",
"-XX:NewRatio=3",
"-XX:SurvivorRatio=4",
"-XX:TargetSurvivorRatio=90",
"-XX:MaxTenuringThreshold=8",
"-XX:+UseConcMarkSweepGC",
"-XX:ConcGCThreads=4",
"-XX:ParallelGCThreads=4",
"-XX:+CMSScavengeBeforeRemark",
"-XX:PretenureSizeThreshold=64m",
"-XX:+UseCMSInitiatingOccupancyOnly",
"-XX:CMSInitiatingOccupancyFraction=50",
"-XX:CMSMaxAbortablePrecleanTime=6000",
"-XX:+CMSParallelRemarkEnabled",
"-XX:+ParallelRefProcEnabled",
"-XX:-OmitStackTraceInFastThrow",
"-verbose:gc",
"-XX:+PrintHeapAtGC",
"-XX:+PrintGCDetails",
"-XX:+PrintGCDateStamps",
"-XX:+PrintGCTimeStamps",
"-XX:+PrintTenuringDistribution",
"-XX:+PrintGCApplicationStoppedTime",
"-XX:+UseGCLogFileRotation",
"-XX:NumberOfGCLogFiles=9",
"-XX:GCLogFileSize=20M",
"-Xss256k",
"-Dsolr.log.muteconsole"

Here is an example of from the GC log:

2019-10-02T16:03:15.888+: 265318.624: [Full GC (Allocation
Failure) 2019-10-02T16:03:15.888+: 265318.624:
[CMS2019-10-02T16:03:16.134+: 26
5318.870: [CMS-concurrent-mark: 1.773/1.783 secs] [Times: user=13.14
sys=0.00, real=1.78 secs]
 (concurrent mode failure): 7864319K->7864319K(7864320K), 9.5890129
secs] 10048895K->8863021K(10048896K), [Metaspace:
53159K->53159K(1097728K)], 9.5892061 secs] [Times: user=10.31
sys=0.00, real=9.59 secs]
Heap after GC invocations=296656 (full 546):
 par new generation   total 2184576K, used 998701K
[0x00054000, 0x0005e000, 0x0005e000)
  eden space 1747712K,  57% used [0x00054000,
0x00057cf4b4f0, 0x0005aaac)
  from space 436864K,   0% used [0x0005aaac,
0x0005aaac, 0x0005c556)
  to   space 436864K,   0% used [0x0005c556,
0x0005c556, 0x0005e000)
 concurrent mark-sweep generation total 7864320K, used 7864319K
[0x0005e000, 0x0007c000, 0x0007c000)
 Metaspace   used 53159K, capacity 54766K, committed 55148K,
reserved 1097728K
  class spaceused 5589K, capacity 5950K, committed 6000K, reserved 1048576K
}
2019-10-02T16:03:25.477+: 265328.214: Total time for which
application threads were stopped: 9.5906157 seconds, Stopping threads
took: 0.0001274 seconds
*With the following from the SOLR log: *

[   x:core] o.a.s.s.HttpSolrCall Unable to write response, client
closed connection or we are s

hutting down

org.eclipse.jetty.io.EofException: Closed

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:665)
~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]

at 
org.apache.solr.servlet.ServletOutputStreamWrapper.write(ServletOutputStreamWrapper.java:126)
~[solr-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c

1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48]

at 
org.apache.solr.response.QueryResponseWriterUtil$1.write(QueryResponseWriterUtil.java:54)
~[solr-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fef

c589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48]

at java.io.OutputStream.write(OutputStream.java:116) ~[?:1.8.0_222-ea]

at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
~[?:1.8.0_222-ea]

at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
~[?:1.8.0_222-ea]

at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
~[?:1.8.0_222-ea]

at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
~[?:1.8.0_222-ea]

at org.apache.solr.common.util.FastWriter.flush(FastWriter.java:140)
~[solr-solrj-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94
- j

anhoy - 2019-05-28 23:37:52]

at 
org.apache.solr.common.util.FastWriter.flushBuffer(FastWriter.java:154)
~[solr-solrj-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b

94 - janhoy - 2019-05-28 23:37:52]

at 
org.apache.solr.respons

Fwd: Solr standalone timeouts after upgrading to SOLR 7

[Solr User]

Hello all,

We recently moved to SOLR 7 from SOLR 6 about 2 weeks ago. Once each week
(including today) we experienced query timeout issues with corresponding GC
events. There was a spike in CPU up to 66% which is not something we
previously saw w/ Solr 6. From the SOLR logs it looks like something inside
the JVM has happend, SOLR is reporting closed connections from Jetty. Our
data size is relatively small but we do run 5 cores within the one Jetty
instance. There index sizes are anywhere between 200Mb to 2GB

Our memory consumption is relatively low:

"free":"296.1 MB",

  "total":"569.6 MB",

  "max":"9.6 GB",

  "used":"273.5 MB (%2.8)",



We had a spike in traffic about 5 minutes prior to some longer GC events
(similar situation last week).

Any help would be appreciated. Below is my current system info along with a
GC log snippet and the corresponding SOLR log error.

*System info:*
AMZ2 linux
8 core 32 GB Mem
*Java:* 1.8.0_222-ea 25.222-b03
*Solr: *solr-spec-version":"7.7.2"
*Start options: *
"-Xms512m",
"-Xmx10g",
"-XX:NewRatio=3",
"-XX:SurvivorRatio=4",
"-XX:TargetSurvivorRatio=90",
"-XX:MaxTenuringThreshold=8",
"-XX:+UseConcMarkSweepGC",
"-XX:ConcGCThreads=4",
"-XX:ParallelGCThreads=4",
"-XX:+CMSScavengeBeforeRemark",
"-XX:PretenureSizeThreshold=64m",
"-XX:+UseCMSInitiatingOccupancyOnly",
"-XX:CMSInitiatingOccupancyFraction=50",
"-XX:CMSMaxAbortablePrecleanTime=6000",
"-XX:+CMSParallelRemarkEnabled",
"-XX:+ParallelRefProcEnabled",
"-XX:-OmitStackTraceInFastThrow",
"-verbose:gc",
"-XX:+PrintHeapAtGC",
"-XX:+PrintGCDetails",
"-XX:+PrintGCDateStamps",
"-XX:+PrintGCTimeStamps",
"-XX:+PrintTenuringDistribution",
"-XX:+PrintGCApplicationStoppedTime",
"-XX:+UseGCLogFileRotation",
"-XX:NumberOfGCLogFiles=9",
"-XX:GCLogFileSize=20M",
"-Xss256k",
"-Dsolr.log.muteconsole"

Here is an example of from the GC log:

2019-10-02T16:03:15.888+: 265318.624: [Full GC (Allocation
Failure) 2019-10-02T16:03:15.888+: 265318.624:
[CMS2019-10-02T16:03:16.134+: 26
5318.870: [CMS-concurrent-mark: 1.773/1.783 secs] [Times: user=13.14
sys=0.00, real=1.78 secs]
 (concurrent mode failure): 7864319K->7864319K(7864320K), 9.5890129
secs] 10048895K->8863021K(10048896K), [Metaspace:
53159K->53159K(1097728K)], 9.5892061 secs] [Times: user=10.31
sys=0.00, real=9.59 secs]
Heap after GC invocations=296656 (full 546):
 par new generation   total 2184576K, used 998701K
[0x00054000, 0x0005e000, 0x0005e000)
  eden space 1747712K,  57% used [0x00054000,
0x00057cf4b4f0, 0x0005aaac)
  from space 436864K,   0% used [0x0005aaac,
0x0005aaac, 0x0005c556)
  to   space 436864K,   0% used [0x0005c556,
0x0005c556, 0x0005e000)
 concurrent mark-sweep generation total 7864320K, used 7864319K
[0x0005e000, 0x0007c000, 0x0007c000)
 Metaspace   used 53159K, capacity 54766K, committed 55148K,
reserved 1097728K
  class spaceused 5589K, capacity 5950K, committed 6000K, reserved 1048576K
}
2019-10-02T16:03:25.477+: 265328.214: Total time for which
application threads were stopped: 9.5906157 seconds, Stopping threads
took: 0.0001274 seconds
*With the following from the SOLR log: *

[   x:core] o.a.s.s.HttpSolrCall Unable to write response, client
closed connection or we are s

hutting down

org.eclipse.jetty.io.EofException: Closed

at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:665)
~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]

at 
org.apache.solr.servlet.ServletOutputStreamWrapper.write(ServletOutputStreamWrapper.java:126)
~[solr-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c

1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48]

at 
org.apache.solr.response.QueryResponseWriterUtil$1.write(QueryResponseWriterUtil.java:54)
~[solr-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fef

c589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48]

at java.io.OutputStream.write(OutputStream.java:116) ~[?:1.8.0_222-ea]

at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
~[?:1.8.0_222-ea]

at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
~[?:1.8.0_222-ea]

at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
~[?:1.8.0_222-ea]

at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
~[?:1.8.0_222-ea]

at org.apache.solr.common.util.FastWriter.flush(FastWriter.java:140)
~[solr-solrj-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94
- j

anhoy - 2019-05-28 23:37:52]

at 
org.apache.solr.common.util.FastWriter.flushBuffer(FastWriter.java:154)
~[solr-solrj-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b

94 - janhoy - 2019-05-28 23:37:52]

at 
org.apache.solr.respons

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Upon further investigation on this issue, I see the below log lines during
the indexing process:

2019-06-06 22:24:56.203 INFO  (qtp1169794610-5652)
[c:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623
s:shard22 r:core_node87
x:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623_shard22_replica_n84]
org.apache.solr.update.LoggingInfoStream [FP][qtp1169794610-5652]: trigger
flush: activeBytes=352402600 deleteBytes=279 vs limit=104857600
2019-06-06 22:24:56.203 INFO  (qtp1169794610-5652)
[c:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623
s:shard22 r:core_node87
x:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623_shard22_replica_n84]
org.apache.solr.update.LoggingInfoStream [FP][qtp1169794610-5652]: thread
state has 352402600 bytes; docInRAM=1
2019-06-06 22:24:56.204 INFO  (qtp1169794610-5652)
[c:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623
s:shard22 r:core_node87
x:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623_shard22_replica_n84]
org.apache.solr.update.LoggingInfoStream [FP][qtp1169794610-5652]: 1 in-use
non-flushing threads states
2019-06-06 22:24:56.204 INFO  (qtp1169794610-5652)
[c:UM_IndexServer_MailArchiv_Spelle_66AC8340-4734-438A-9D1D-A84B659B1623
s:shard22 r:core_node87

I have the below questions:
1) The log line which says "thread state has 352402600 bytes; docInRAM=1 ",
does it mean that the buffer was flushed to disk with only one huge
document ?
2) If yes, does this flush create a segment with just one document ?
3) Heap dump analysis shows large (>350 MB) instances of
DocumentWritersPerThread. Does one instance of this class correspond to one
document?


Help is much appreciated.

Thanks,
Rahul


On Fri, Jul 5, 2019 at 2:11 AM Rahul Goswami  wrote:

> Shawn,Erick,
> Thank you for the explanation. The merge scheduler params make sense now.
>
> Thanks,
> Rahul
>
> On Wed, Jul 3, 2019 at 11:30 AM Erick Erickson 
> wrote:
>
>> Two more tidbits to add to Shawn’s explanation:
>>
>> There are heuristics built in to ConcurrentMergeScheduler.
>> From the Javadocs:
>> * If it's an SSD,
>> *  {@code maxThreadCount} is set to {@code max(1, min(4,
>> cpuCoreCount/2))},
>> *  otherwise 1.  Note that detection only currently works on
>> *  Linux; other platforms will assume the index is not on an SSD.
>>
>> Second, TieredMergePolicy (the default) merges in “tiers” that
>> are of similar size. So you can have multiple merges going on
>> at the same time on disjoint sets of segments.
>>
>> Best,
>> Erick
>>
>> > On Jul 3, 2019, at 7:54 AM, Shawn Heisey  wrote:
>> >
>> > On 7/2/2019 10:53 PM, Rahul Goswami wrote:
>> >> Hi Shawn,
>> >> Thank you for the detailed suggestions. Although, I would like to
>> >> understand the maxMergeCount and maxThreadCount params better. The
>> >> documentation
>> >> <
>> https://lucene.apache.org/solr/guide/7_3/indexconfig-in-solrconfig.html#mergescheduler
>> >
>> >> mentions
>> >> that
>> >> maxMergeCount : The maximum number of simultaneous merges that are
>> allowed.
>> >> maxThreadCount : The maximum number of simultaneous merge threads that
>> >> should be running at once
>> >> Since one thread can only do 1 merge at any given point of time, how
>> does
>> >> maxMergeCount being greater than maxThreadCount help anyway? I am
>> having
>> >> difficulty wrapping my head around this, and would appreciate if you
>> could
>> >> help clear it for me.
>> >
>> > The maxMergeCount setting controls the number of merges that can be
>> *scheduled* at the same time.  As soon as that number of merges is reached,
>> the indexing thread(s) will be paused until the number of merges in the
>> schedule drops below this number.  This ensures that no more merges will be
>> scheduled.
>> >
>> > By setting maxMergeCount higher than the number of merges that are
>> expected in the schedule, you can ensure that indexing will never be
>> paused.  It would require very atypical merge policy settings for the
>> number of scheduled merges to ever reach six.  On my own indexing, I
>> reached three scheduled merges quite frequently.  The default setting for
>> maxMergeCount is three.
>> >
>> > The maxThreadCount setting controls how many of the scheduled merges
>> will be simultaneously executed. With index data on standard spinning
>> disks, you do not want to increase this number beyond 1, or you will have a
>> performance problem due to thrashing disk heads.  If your data is on SSD,
>> you can make it larger than 1.
>> >
>> > Thanks,
>> > Shawn
>>
>>

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Shawn,Erick,
Thank you for the explanation. The merge scheduler params make sense now.

Thanks,
Rahul

On Wed, Jul 3, 2019 at 11:30 AM Erick Erickson 
wrote:

> Two more tidbits to add to Shawn’s explanation:
>
> There are heuristics built in to ConcurrentMergeScheduler.
> From the Javadocs:
> * If it's an SSD,
> *  {@code maxThreadCount} is set to {@code max(1, min(4, cpuCoreCount/2))},
> *  otherwise 1.  Note that detection only currently works on
> *  Linux; other platforms will assume the index is not on an SSD.
>
> Second, TieredMergePolicy (the default) merges in “tiers” that
> are of similar size. So you can have multiple merges going on
> at the same time on disjoint sets of segments.
>
> Best,
> Erick
>
> > On Jul 3, 2019, at 7:54 AM, Shawn Heisey  wrote:
> >
> > On 7/2/2019 10:53 PM, Rahul Goswami wrote:
> >> Hi Shawn,
> >> Thank you for the detailed suggestions. Although, I would like to
> >> understand the maxMergeCount and maxThreadCount params better. The
> >> documentation
> >> <
> https://lucene.apache.org/solr/guide/7_3/indexconfig-in-solrconfig.html#mergescheduler
> >
> >> mentions
> >> that
> >> maxMergeCount : The maximum number of simultaneous merges that are
> allowed.
> >> maxThreadCount : The maximum number of simultaneous merge threads that
> >> should be running at once
> >> Since one thread can only do 1 merge at any given point of time, how
> does
> >> maxMergeCount being greater than maxThreadCount help anyway? I am having
> >> difficulty wrapping my head around this, and would appreciate if you
> could
> >> help clear it for me.
> >
> > The maxMergeCount setting controls the number of merges that can be
> *scheduled* at the same time.  As soon as that number of merges is reached,
> the indexing thread(s) will be paused until the number of merges in the
> schedule drops below this number.  This ensures that no more merges will be
> scheduled.
> >
> > By setting maxMergeCount higher than the number of merges that are
> expected in the schedule, you can ensure that indexing will never be
> paused.  It would require very atypical merge policy settings for the
> number of scheduled merges to ever reach six.  On my own indexing, I
> reached three scheduled merges quite frequently.  The default setting for
> maxMergeCount is three.
> >
> > The maxThreadCount setting controls how many of the scheduled merges
> will be simultaneously executed. With index data on standard spinning
> disks, you do not want to increase this number beyond 1, or you will have a
> performance problem due to thrashing disk heads.  If your data is on SSD,
> you can make it larger than 1.
> >
> > Thanks,
> > Shawn
>
>

Re: SolrCloud indexing triggers merges and timeouts

[Erick Erickson]

Two more tidbits to add to Shawn’s explanation:

There are heuristics built in to ConcurrentMergeScheduler.
From the Javadocs:
* If it's an SSD,
*  {@code maxThreadCount} is set to {@code max(1, min(4, cpuCoreCount/2))},
*  otherwise 1.  Note that detection only currently works on
*  Linux; other platforms will assume the index is not on an SSD.

Second, TieredMergePolicy (the default) merges in “tiers” that
are of similar size. So you can have multiple merges going on
at the same time on disjoint sets of segments.

Best,
Erick

> On Jul 3, 2019, at 7:54 AM, Shawn Heisey  wrote:
> 
> On 7/2/2019 10:53 PM, Rahul Goswami wrote:
>> Hi Shawn,
>> Thank you for the detailed suggestions. Although, I would like to
>> understand the maxMergeCount and maxThreadCount params better. The
>> documentation
>> 
>> mentions
>> that
>> maxMergeCount : The maximum number of simultaneous merges that are allowed.
>> maxThreadCount : The maximum number of simultaneous merge threads that
>> should be running at once
>> Since one thread can only do 1 merge at any given point of time, how does
>> maxMergeCount being greater than maxThreadCount help anyway? I am having
>> difficulty wrapping my head around this, and would appreciate if you could
>> help clear it for me.
> 
> The maxMergeCount setting controls the number of merges that can be 
> *scheduled* at the same time.  As soon as that number of merges is reached, 
> the indexing thread(s) will be paused until the number of merges in the 
> schedule drops below this number.  This ensures that no more merges will be 
> scheduled.
> 
> By setting maxMergeCount higher than the number of merges that are expected 
> in the schedule, you can ensure that indexing will never be paused.  It would 
> require very atypical merge policy settings for the number of scheduled 
> merges to ever reach six.  On my own indexing, I reached three scheduled 
> merges quite frequently.  The default setting for maxMergeCount is three.
> 
> The maxThreadCount setting controls how many of the scheduled merges will be 
> simultaneously executed. With index data on standard spinning disks, you do 
> not want to increase this number beyond 1, or you will have a performance 
> problem due to thrashing disk heads.  If your data is on SSD, you can make it 
> larger than 1.
> 
> Thanks,
> Shawn

Re: SolrCloud indexing triggers merges and timeouts

[Shawn Heisey]

On 7/2/2019 10:53 PM, Rahul Goswami wrote:

Hi Shawn,

Thank you for the detailed suggestions. Although, I would like to
understand the maxMergeCount and maxThreadCount params better. The
documentation

mentions
that

maxMergeCount : The maximum number of simultaneous merges that are allowed.
maxThreadCount : The maximum number of simultaneous merge threads that
should be running at once

Since one thread can only do 1 merge at any given point of time, how does
maxMergeCount being greater than maxThreadCount help anyway? I am having
difficulty wrapping my head around this, and would appreciate if you could
help clear it for me.


The maxMergeCount setting controls the number of merges that can be 
*scheduled* at the same time.  As soon as that number of merges is 
reached, the indexing thread(s) will be paused until the number of 
merges in the schedule drops below this number.  This ensures that no 
more merges will be scheduled.


By setting maxMergeCount higher than the number of merges that are 
expected in the schedule, you can ensure that indexing will never be 
paused.  It would require very atypical merge policy settings for the 
number of scheduled merges to ever reach six.  On my own indexing, I 
reached three scheduled merges quite frequently.  The default setting 
for maxMergeCount is three.


The maxThreadCount setting controls how many of the scheduled merges 
will be simultaneously executed.  With index data on standard spinning 
disks, you do not want to increase this number beyond 1, or you will 
have a performance problem due to thrashing disk heads.  If your data is 
on SSD, you can make it larger than 1.


Thanks,
Shawn

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Hi Shawn,

Thank you for the detailed suggestions. Although, I would like to
understand the maxMergeCount and maxThreadCount params better. The
documentation

mentions
that

maxMergeCount : The maximum number of simultaneous merges that are allowed.
maxThreadCount : The maximum number of simultaneous merge threads that
should be running at once

Since one thread can only do 1 merge at any given point of time, how does
maxMergeCount being greater than maxThreadCount help anyway? I am having
difficulty wrapping my head around this, and would appreciate if you could
help clear it for me.

Thanks,
Rahul

On Thu, Jun 13, 2019 at 7:33 AM Shawn Heisey  wrote:

> On 6/6/2019 9:00 AM, Rahul Goswami wrote:
> > *OP Reply* : Total 48 GB per node... I couldn't see another software
> using
> > a lot of memory.
> > I am honestly not sure about the reason for change of directory factory
> to
> > SimpleFSDirectoryFactory. But I was told that with mmap at one point we
> > started to see the shared memory usage on Windows go up significantly,
> > intermittently freezing the system.
> > Could the choice of DirectoryFactory here be a factor for the long
> > updates/frequent merges?
>
> With about 24GB of RAM to cache 1.4TB of index data, you're never going
> to have good performance.  Any query you do is probably going to read
> more than 24GB of data from the index, which means that it cannot come
> from memory, some of it must come from disk, which is incredibly slow
> compared to memory.
>
> MMap is more efficient than "simple" filesystem access.  I do not know
> if you would see markedly better performance, but getting rid of the
> DirectoryFactory config and letting Solr choose its default might help.
>
> > How many total documents (maxDoc, not numDoc) are in that 1.4 TB of
> > space?
> > *OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT
> > numDoc) in that 1.4 TB space
>
> Unless you're doing faceting or grouping on fields with extremely high
> cardinality, which I find to be rarely useful except for data mining,
> 24GB of heap for 12.8 million docs seems very excessive.  I was
> expecting this number to be something like 500 million or more ... that
> small document count must mean each document is HUGE.  Can you take
> steps to reduce the index size, perhaps by setting stored, indexed,
> and/or docValues to "false" on some of your fields, and having your
> application go to the system of record for full details on each
> document?  You will have to reindex after making changes like that.
>
> >> Can you share the GC log that Solr writes?
> > *OP Reply:*  Please find the GC logs and thread dumps at this location
> > https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw
>
> The larger GC log was unrecognized by both gcviwer and gceasy.io ... the
> smaller log shows heap usage about 10GB, but it only covers 10 minutes,
> so it's not really conclusive for diagnosis.  The first thing I can
> suggest to try is to reduce the heap size to 12GB ... but I do not know
> if that's actually going to work.  Indexing might require more memory.
> The idea here is to make more memory available to the OS disk cache ...
> with your index size, you're probably going to need to add memory to the
> system (not the heap).
>
> > Another observation is that the CPU usage reaches around 70% (through
> > manual monitoring) when the indexing starts and the merges are observed.
> It
> > is well below 50% otherwise.
>
> Indexing will increase load, and that increase is often very
> significant.  Adding memory to the system is your best bet for better
> performance.  I'd want 1TB of memory for a 1.4TB index ... but I know
> that memory sizes that high are extremely expensive, and for most
> servers, not even possible.  512GB or 256GB is more attainable, and
> would have better performance than 48GB.
>
> > Also, should something be altered with the mergeScheduler setting ?
> > "mergeScheduler":{
> >  "class":"org.apache.lucene.index.ConcurrentMergeScheduler",
> >  "maxMergeCount":2,
> >  "maxThreadCount":2},
>
> Do not configure maxThreadCount beyond 1 unless your data is on SSD.  It
> will slow things down a lot due to the fact that standard disks must
> move the disk head to read/write from different locations, and head
> moves take time.  SSD can do I/O from any location without pauses, so
> more threads would probably help performance rather than hurt it.
>
> Increase maxMergeCount to 6 -- at 2, large merges will probably stop
> indexing entirely.  With a larger number, Solr can keep indexing even
> when there's a huge segment merge happening.
>
> Thanks,
> Shawn
>

Re: SolrCloud: Configured socket timeouts not reflecting

[Rahul Goswami]

Hi Gus,

Have created a pull request for JIRA 12550
<https://issues.apache.org/jira/browse/SOLR-12550> and updated the affected
Solr version (7.2.1) in the comments. The provided fix is on branch_7_2. I
haven't tried reproducing the issue on the latest version, but see that the
code for this part is different on the master.

Regards,
Rahul

On Thu, Jun 20, 2019 at 8:22 PM Rahul Goswami  wrote:

> Hi Gus,
> Thanks for the response and referencing the umbrella JIRA for these kind
> of issues. I see that it won't solve the problem since the builder object
> which is used to instantiate a ConcurrentUpdateSolrClient itself doesn't
> contain the timeout values. I did create a local solr-core binary to try
> the patch nevertheless, but it didn't help as I anticipated. I'll update
> the JIRA and submit a patch.
>
> Thank you,
> Rahul
>
> On Thu, Jun 20, 2019 at 11:35 AM Gus Heck  wrote:
>
>> Hi Rahul,
>>
>> Did you try the patch int that issue? Also food for thought:
>> https://issues.apache.org/jira/browse/SOLR-13457
>>
>> -Gus
>>
>> On Tue, Jun 18, 2019 at 5:52 PM Rahul Goswami 
>> wrote:
>>
>> > Hello,
>> >
>> > I was looking into the code to try to get to the root of this issue.
>> Looks
>> > like this is an issue after all (as of 7.2.1 which is the version we are
>> > using), but wanted to confirm on the user list before creating a JIRA. I
>> > found that the soTimeout property of ConcurrentUpdateSolrClient class
>> (in
>> > the code referenced below) remains null and hence the default of 60
>> ms
>> > is set as the timeout in HttpPost class instance variable "method".
>> >
>> >
>> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L334
>> >
>> >
>> > When the call is finally made in the below line, the Httpclient does
>> > contain the configured timeout (as in solr.xml or
>> -DdistribUpdateSoTimeout)
>> > but gets overriden by the hard default of 60 in the "method"
>> parameter
>> > of the execute call.
>> >
>> >
>> >
>> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L348
>> >
>> >
>> > The hard default of 60 is set here:
>> >
>> >
>> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L333
>> >
>> >
>> > I tried to create a local patch with the below fix which works fine:
>> >
>> >
>> https://github.com/apache/lucene-solr/blob/86fe24cbef238d2042d68494bd94e2362a2d996e/solr/core/src/java/org/apache/solr/update/StreamingSolrClients.java#L69
>> >
>> >
>> >
>> > client = new ErrorReportingConcurrentUpdateSolrClient.Builder(url, req,
>> > errors)
>> >   .withHttpClient(httpClient)
>> >   .withQueueSize(100)
>> >   .withSocketTimeout(getSocketTimeout(req))
>> >   .withThreadCount(runnerCount)
>> >   .withExecutorService(updateExecutor)
>> >   .alwaysStreamDeletes()
>> >   .build();
>> >
>> > private int getSocketTimeout(SolrCmdDistributor.Req req) {
>> > if(req==null) {
>> >   return UpdateShardHandlerConfig.DEFAULT_DISTRIBUPDATESOTIMEOUT;
>> > }
>> >
>> > return
>> >
>> >
>> req.cmd.req.getCore().getCoreContainer().getConfig().getUpdateShardHandlerConfig().getDistributedSocketTimeout();
>> >   }
>> >
>> > I found this open JIRA on this issue:
>> >
>> >
>> >
>> https://issues.apache.org/jira/browse/SOLR-12550?jql=text%20~%20%22distribUpdateSoTimeout%22
>> >
>> >
>> > Should I update the JIRA with this ?
>> >
>> > Thanks,
>> > Rahul
>> >
>> >
>> >
>> >
>> > On Thu, Jun 13, 2019 at 12:00 AM Rahul Goswami 
>> > wrote:
>> >
>> > > Hello,
>> > >
>> > > I am running Solr 7.2.1 in cloud mode. To overcome a setup hardware
>> > > bottleneck, I tried to configure distribUpdateSoTimeout and
>> socketTimeout
>> > > to a value greater than the default 10 mins. I did this by passing
>> these
>> > as
>> > > system properties at Solr start up time (-DdistribUpdateSoTimeout and
>> > > -DsocketTimeout  ). The Solr admin UI shows these values in the
>> Dashboard
>> > > args section. As a test, I tried setting each of them to one hour
>> > > (360). However I start seeing socket read timeouts within a few
>> mins.
>> > > Looks like the values are not taking effect. What am I missing? If
>> this
>> > is
>> > > a known issue, is there a JIRA for it ?
>> > >
>> > > Thanks,
>> > > Rahul
>> > >
>> >
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>

Re: SolrCloud: Configured socket timeouts not reflecting

[Rahul Goswami]

Hi Gus,
Thanks for the response and referencing the umbrella JIRA for these kind of
issues. I see that it won't solve the problem since the builder object
which is used to instantiate a ConcurrentUpdateSolrClient itself doesn't
contain the timeout values. I did create a local solr-core binary to try
the patch nevertheless, but it didn't help as I anticipated. I'll update
the JIRA and submit a patch.

Thank you,
Rahul

On Thu, Jun 20, 2019 at 11:35 AM Gus Heck  wrote:

> Hi Rahul,
>
> Did you try the patch int that issue? Also food for thought:
> https://issues.apache.org/jira/browse/SOLR-13457
>
> -Gus
>
> On Tue, Jun 18, 2019 at 5:52 PM Rahul Goswami 
> wrote:
>
> > Hello,
> >
> > I was looking into the code to try to get to the root of this issue.
> Looks
> > like this is an issue after all (as of 7.2.1 which is the version we are
> > using), but wanted to confirm on the user list before creating a JIRA. I
> > found that the soTimeout property of ConcurrentUpdateSolrClient class (in
> > the code referenced below) remains null and hence the default of 60
> ms
> > is set as the timeout in HttpPost class instance variable "method".
> >
> >
> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L334
> >
> >
> > When the call is finally made in the below line, the Httpclient does
> > contain the configured timeout (as in solr.xml or
> -DdistribUpdateSoTimeout)
> > but gets overriden by the hard default of 60 in the "method"
> parameter
> > of the execute call.
> >
> >
> >
> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L348
> >
> >
> > The hard default of 60 is set here:
> >
> >
> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L333
> >
> >
> > I tried to create a local patch with the below fix which works fine:
> >
> >
> https://github.com/apache/lucene-solr/blob/86fe24cbef238d2042d68494bd94e2362a2d996e/solr/core/src/java/org/apache/solr/update/StreamingSolrClients.java#L69
> >
> >
> >
> > client = new ErrorReportingConcurrentUpdateSolrClient.Builder(url, req,
> > errors)
> >   .withHttpClient(httpClient)
> >   .withQueueSize(100)
> >   .withSocketTimeout(getSocketTimeout(req))
> >   .withThreadCount(runnerCount)
> >   .withExecutorService(updateExecutor)
> >   .alwaysStreamDeletes()
> >   .build();
> >
> > private int getSocketTimeout(SolrCmdDistributor.Req req) {
> > if(req==null) {
> >   return UpdateShardHandlerConfig.DEFAULT_DISTRIBUPDATESOTIMEOUT;
> > }
> >
> > return
> >
> >
> req.cmd.req.getCore().getCoreContainer().getConfig().getUpdateShardHandlerConfig().getDistributedSocketTimeout();
> >   }
> >
> > I found this open JIRA on this issue:
> >
> >
> >
> https://issues.apache.org/jira/browse/SOLR-12550?jql=text%20~%20%22distribUpdateSoTimeout%22
> >
> >
> > Should I update the JIRA with this ?
> >
> > Thanks,
> > Rahul
> >
> >
> >
> >
> > On Thu, Jun 13, 2019 at 12:00 AM Rahul Goswami 
> > wrote:
> >
> > > Hello,
> > >
> > > I am running Solr 7.2.1 in cloud mode. To overcome a setup hardware
> > > bottleneck, I tried to configure distribUpdateSoTimeout and
> socketTimeout
> > > to a value greater than the default 10 mins. I did this by passing
> these
> > as
> > > system properties at Solr start up time (-DdistribUpdateSoTimeout and
> > > -DsocketTimeout  ). The Solr admin UI shows these values in the
> Dashboard
> > > args section. As a test, I tried setting each of them to one hour
> > > (360). However I start seeing socket read timeouts within a few
> mins.
> > > Looks like the values are not taking effect. What am I missing? If this
> > is
> > > a known issue, is there a JIRA for it ?
> > >
> > > Thanks,
> > > Rahul
> > >
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>

Re: SolrCloud: Configured socket timeouts not reflecting

[Gus Heck]

Hi Rahul,

Did you try the patch int that issue? Also food for thought:
https://issues.apache.org/jira/browse/SOLR-13457

-Gus

On Tue, Jun 18, 2019 at 5:52 PM Rahul Goswami  wrote:

> Hello,
>
> I was looking into the code to try to get to the root of this issue. Looks
> like this is an issue after all (as of 7.2.1 which is the version we are
> using), but wanted to confirm on the user list before creating a JIRA. I
> found that the soTimeout property of ConcurrentUpdateSolrClient class (in
> the code referenced below) remains null and hence the default of 60 ms
> is set as the timeout in HttpPost class instance variable "method".
>
> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L334
>
>
> When the call is finally made in the below line, the Httpclient does
> contain the configured timeout (as in solr.xml or -DdistribUpdateSoTimeout)
> but gets overriden by the hard default of 60 in the "method" parameter
> of the execute call.
>
>
> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L348
>
>
> The hard default of 60 is set here:
>
> https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L333
>
>
> I tried to create a local patch with the below fix which works fine:
>
> https://github.com/apache/lucene-solr/blob/86fe24cbef238d2042d68494bd94e2362a2d996e/solr/core/src/java/org/apache/solr/update/StreamingSolrClients.java#L69
>
>
>
> client = new ErrorReportingConcurrentUpdateSolrClient.Builder(url, req,
> errors)
>   .withHttpClient(httpClient)
>   .withQueueSize(100)
>   .withSocketTimeout(getSocketTimeout(req))
>   .withThreadCount(runnerCount)
>   .withExecutorService(updateExecutor)
>   .alwaysStreamDeletes()
>   .build();
>
> private int getSocketTimeout(SolrCmdDistributor.Req req) {
> if(req==null) {
>   return UpdateShardHandlerConfig.DEFAULT_DISTRIBUPDATESOTIMEOUT;
> }
>
> return
>
> req.cmd.req.getCore().getCoreContainer().getConfig().getUpdateShardHandlerConfig().getDistributedSocketTimeout();
>   }
>
> I found this open JIRA on this issue:
>
>
> https://issues.apache.org/jira/browse/SOLR-12550?jql=text%20~%20%22distribUpdateSoTimeout%22
>
>
> Should I update the JIRA with this ?
>
> Thanks,
> Rahul
>
>
>
>
> On Thu, Jun 13, 2019 at 12:00 AM Rahul Goswami 
> wrote:
>
> > Hello,
> >
> > I am running Solr 7.2.1 in cloud mode. To overcome a setup hardware
> > bottleneck, I tried to configure distribUpdateSoTimeout and socketTimeout
> > to a value greater than the default 10 mins. I did this by passing these
> as
> > system properties at Solr start up time (-DdistribUpdateSoTimeout and
> > -DsocketTimeout  ). The Solr admin UI shows these values in the Dashboard
> > args section. As a test, I tried setting each of them to one hour
> > (360). However I start seeing socket read timeouts within a few mins.
> > Looks like the values are not taking effect. What am I missing? If this
> is
> > a known issue, is there a JIRA for it ?
> >
> > Thanks,
> > Rahul
> >
>


-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: SolrCloud: Configured socket timeouts not reflecting

[Rahul Goswami]

Hello,

I was looking into the code to try to get to the root of this issue. Looks
like this is an issue after all (as of 7.2.1 which is the version we are
using), but wanted to confirm on the user list before creating a JIRA. I
found that the soTimeout property of ConcurrentUpdateSolrClient class (in
the code referenced below) remains null and hence the default of 60 ms
is set as the timeout in HttpPost class instance variable "method".
https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L334


When the call is finally made in the below line, the Httpclient does
contain the configured timeout (as in solr.xml or -DdistribUpdateSoTimeout)
but gets overriden by the hard default of 60 in the "method" parameter
of the execute call.

https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L348


The hard default of 60 is set here:
https://github.com/apache/lucene-solr/blob/e6f6f352cfc30517235822b3deed83df1ee144c6/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L333


I tried to create a local patch with the below fix which works fine:
https://github.com/apache/lucene-solr/blob/86fe24cbef238d2042d68494bd94e2362a2d996e/solr/core/src/java/org/apache/solr/update/StreamingSolrClients.java#L69



client = new ErrorReportingConcurrentUpdateSolrClient.Builder(url, req,
errors)
  .withHttpClient(httpClient)
  .withQueueSize(100)
  .withSocketTimeout(getSocketTimeout(req))
  .withThreadCount(runnerCount)
  .withExecutorService(updateExecutor)
  .alwaysStreamDeletes()
  .build();

private int getSocketTimeout(SolrCmdDistributor.Req req) {
if(req==null) {
  return UpdateShardHandlerConfig.DEFAULT_DISTRIBUPDATESOTIMEOUT;
}

return
req.cmd.req.getCore().getCoreContainer().getConfig().getUpdateShardHandlerConfig().getDistributedSocketTimeout();
  }

I found this open JIRA on this issue:

https://issues.apache.org/jira/browse/SOLR-12550?jql=text%20~%20%22distribUpdateSoTimeout%22


Should I update the JIRA with this ?

Thanks,
Rahul




On Thu, Jun 13, 2019 at 12:00 AM Rahul Goswami 
wrote:

> Hello,
>
> I am running Solr 7.2.1 in cloud mode. To overcome a setup hardware
> bottleneck, I tried to configure distribUpdateSoTimeout and socketTimeout
> to a value greater than the default 10 mins. I did this by passing these as
> system properties at Solr start up time (-DdistribUpdateSoTimeout and
> -DsocketTimeout  ). The Solr admin UI shows these values in the Dashboard
> args section. As a test, I tried setting each of them to one hour
> (360). However I start seeing socket read timeouts within a few mins.
> Looks like the values are not taking effect. What am I missing? If this is
> a known issue, is there a JIRA for it ?
>
> Thanks,
> Rahul
>

Re: SolrCloud indexing triggers merges and timeouts

[Shawn Heisey]

On 6/6/2019 9:00 AM, Rahul Goswami wrote:

*OP Reply* : Total 48 GB per node... I couldn't see another software using
a lot of memory.
I am honestly not sure about the reason for change of directory factory to
SimpleFSDirectoryFactory. But I was told that with mmap at one point we
started to see the shared memory usage on Windows go up significantly,
intermittently freezing the system.
Could the choice of DirectoryFactory here be a factor for the long
updates/frequent merges?


With about 24GB of RAM to cache 1.4TB of index data, you're never going 
to have good performance.  Any query you do is probably going to read 
more than 24GB of data from the index, which means that it cannot come 
from memory, some of it must come from disk, which is incredibly slow 
compared to memory.


MMap is more efficient than "simple" filesystem access.  I do not know 
if you would see markedly better performance, but getting rid of the 
DirectoryFactory config and letting Solr choose its default might help.



How many total documents (maxDoc, not numDoc) are in that 1.4 TB of
space?
*OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT
numDoc) in that 1.4 TB space


Unless you're doing faceting or grouping on fields with extremely high 
cardinality, which I find to be rarely useful except for data mining, 
24GB of heap for 12.8 million docs seems very excessive.  I was 
expecting this number to be something like 500 million or more ... that 
small document count must mean each document is HUGE.  Can you take 
steps to reduce the index size, perhaps by setting stored, indexed, 
and/or docValues to "false" on some of your fields, and having your 
application go to the system of record for full details on each 
document?  You will have to reindex after making changes like that.



Can you share the GC log that Solr writes?

*OP Reply:*  Please find the GC logs and thread dumps at this location
https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw


The larger GC log was unrecognized by both gcviwer and gceasy.io ... the 
smaller log shows heap usage about 10GB, but it only covers 10 minutes, 
so it's not really conclusive for diagnosis.  The first thing I can 
suggest to try is to reduce the heap size to 12GB ... but I do not know 
if that's actually going to work.  Indexing might require more memory. 
The idea here is to make more memory available to the OS disk cache ... 
with your index size, you're probably going to need to add memory to the 
system (not the heap).



Another observation is that the CPU usage reaches around 70% (through
manual monitoring) when the indexing starts and the merges are observed. It
is well below 50% otherwise.


Indexing will increase load, and that increase is often very 
significant.  Adding memory to the system is your best bet for better 
performance.  I'd want 1TB of memory for a 1.4TB index ... but I know 
that memory sizes that high are extremely expensive, and for most 
servers, not even possible.  512GB or 256GB is more attainable, and 
would have better performance than 48GB.



Also, should something be altered with the mergeScheduler setting ?
"mergeScheduler":{
 "class":"org.apache.lucene.index.ConcurrentMergeScheduler",
 "maxMergeCount":2,
 "maxThreadCount":2},


Do not configure maxThreadCount beyond 1 unless your data is on SSD.  It 
will slow things down a lot due to the fact that standard disks must 
move the disk head to read/write from different locations, and head 
moves take time.  SSD can do I/O from any location without pauses, so 
more threads would probably help performance rather than hurt it.


Increase maxMergeCount to 6 -- at 2, large merges will probably stop 
indexing entirely.  With a larger number, Solr can keep indexing even 
when there's a huge segment merge happening.


Thanks,
Shawn

SolrCloud: Configured socket timeouts not reflecting

[Rahul Goswami]

Hello,

I am running Solr 7.2.1 in cloud mode. To overcome a setup hardware
bottleneck, I tried to configure distribUpdateSoTimeout and socketTimeout
to a value greater than the default 10 mins. I did this by passing these as
system properties at Solr start up time (-DdistribUpdateSoTimeout and
-DsocketTimeout  ). The Solr admin UI shows these values in the Dashboard
args section. As a test, I tried setting each of them to one hour
(360). However I start seeing socket read timeouts within a few mins.
Looks like the values are not taking effect. What am I missing? If this is
a known issue, is there a JIRA for it ?

Thanks,
Rahul

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Updating the thread with further findings:

So turns out that the nodes hosting Solr are VMs with Virtual disks.
Additionally, a Windows system process (the infamous PID 4) is hogging a
lot of disk. This is indicated by disk reponse times in excess of 100 ms
and a disk drive queue length of 5 which would be considered very high. The
indexing is running in two parallel threads each sending a batch of 50 docs
per request. I would like to believe this is not too high (?). The docs are
not too heavy either, only containing metadata fields.So the disk IO seems
to be the bottleneck at this point, causing commits and merges to take more
time than they should. This is causing update routing to the leader replica
to take more than 10 mins, resulting into read time outs, and eventually
failed updates.
I could not find anything alarming in the GC logs I shared earlier.

Will update the thread with more findings as I have them and the attempted
solutions. At this point I am considering increasing the Jetty timeout and
increasing the distribUpdateConnTimeout to a higher value to let the
indexing proceed slowly but successfully. In the meantime, would greatly
appreciate any other ideas/measures.

Thanks,
Rahul


On Thu, Jun 6, 2019 at 11:00 AM Rahul Goswami  wrote:

> Thank you for your responses. Please find additional details about the
> setup below:
>
> We are using Solr 7.2.1
>
> > I have a solrcloud setup on Windows server with below config:
> > 3 nodes,
> > 24 shards with replication factor 2
> > Each node hosts 16 cores.
>
> 16 CPU cores, or 16 Solr cores?  The info may not be all that useful
> either way, but just in case, it should be clarified.
>
> *OP Reply:* 16 Solr cores (i.e. replicas)
>
> > Index size is 1.4 TB per node
> > Xms 8 GB , Xmx 24 GB
> > Directory factory used is SimpleFSDirectoryFactory
>
> How much total memory in the server?  Is there other software using
> significant levels of memory?
>
> *OP Reply* : Total 48 GB per node... I couldn't see another software
> using a lot of memory.
> I am honestly not sure about the reason for change of directory factory to
> SimpleFSDirectoryFactory. But I was told that with mmap at one point we
> started to see the shared memory usage on Windows go up significantly,
> intermittently freezing the system.
> Could the choice of DirectoryFactory here be a factor for the long
> updates/frequent merges?
>
> > How many total documents (maxDoc, not numDoc) are in that 1.4 TB of
> space?
> *OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT
> numDoc) in that 1.4 TB space
>
> > Can you share the GC log that Solr writes?
> *OP Reply:*  Please find the GC logs and thread dumps at this location
> https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw
>
> Another observation is that the CPU usage reaches around 70% (through
> manual monitoring) when the indexing starts and the merges are observed. It
> is well below 50% otherwise.
>
> Also, should something be altered with the mergeScheduler setting ?
> "mergeScheduler":{
> "class":"org.apache.lucene.index.ConcurrentMergeScheduler",
> "maxMergeCount":2,
> "maxThreadCount":2},
>
> Thanks,
> Rahul
>
>
> On Wed, Jun 5, 2019 at 4:24 PM Shawn Heisey  wrote:
>
>> On 6/5/2019 9:39 AM, Rahul Goswami wrote:
>> > I have a solrcloud setup on Windows server with below config:
>> > 3 nodes,
>> > 24 shards with replication factor 2
>> > Each node hosts 16 cores.
>>
>> 16 CPU cores, or 16 Solr cores?  The info may not be all that useful
>> either way, but just in case, it should be clarified.
>>
>> > Index size is 1.4 TB per node
>> > Xms 8 GB , Xmx 24 GB
>> > Directory factory used is SimpleFSDirectoryFactory
>>
>> How much total memory in the server?  Is there other software using
>> significant levels of memory?
>>
>> Why did you opt to change the DirectoryFactory away from Solr's default?
>>   The default is chosen with care ... any other choice will probably
>> result in lower performance.  The default in recent versions of Solr is
>> NRTCachingDirectoryFactory, which uses MMap for file access.
>>
>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>
>> The screenshot described here might become useful for more in-depth
>> troubleshooting:
>>
>>
>> https://wiki.apache.org/solr/SolrPerformanceProblems#Process_listing_on_Windows
>>
>> How many total documents (maxDoc, not numDoc) are in that 1.4 TB of space?
>>
>> > The cloud is all nice and green for the most part. Only when we start

Re: SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Thank you for your responses. Please find additional details about the
setup below:

We are using Solr 7.2.1

> I have a solrcloud setup on Windows server with below config:
> 3 nodes,
> 24 shards with replication factor 2
> Each node hosts 16 cores.

16 CPU cores, or 16 Solr cores?  The info may not be all that useful
either way, but just in case, it should be clarified.

*OP Reply:* 16 Solr cores (i.e. replicas)

> Index size is 1.4 TB per node
> Xms 8 GB , Xmx 24 GB
> Directory factory used is SimpleFSDirectoryFactory

How much total memory in the server?  Is there other software using
significant levels of memory?

*OP Reply* : Total 48 GB per node... I couldn't see another software using
a lot of memory.
I am honestly not sure about the reason for change of directory factory to
SimpleFSDirectoryFactory. But I was told that with mmap at one point we
started to see the shared memory usage on Windows go up significantly,
intermittently freezing the system.
Could the choice of DirectoryFactory here be a factor for the long
updates/frequent merges?

> How many total documents (maxDoc, not numDoc) are in that 1.4 TB of
space?
*OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT
numDoc) in that 1.4 TB space

> Can you share the GC log that Solr writes?
*OP Reply:*  Please find the GC logs and thread dumps at this location
https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw

Another observation is that the CPU usage reaches around 70% (through
manual monitoring) when the indexing starts and the merges are observed. It
is well below 50% otherwise.

Also, should something be altered with the mergeScheduler setting ?
"mergeScheduler":{
"class":"org.apache.lucene.index.ConcurrentMergeScheduler",
"maxMergeCount":2,
"maxThreadCount":2},

Thanks,
Rahul


On Wed, Jun 5, 2019 at 4:24 PM Shawn Heisey  wrote:

> On 6/5/2019 9:39 AM, Rahul Goswami wrote:
> > I have a solrcloud setup on Windows server with below config:
> > 3 nodes,
> > 24 shards with replication factor 2
> > Each node hosts 16 cores.
>
> 16 CPU cores, or 16 Solr cores?  The info may not be all that useful
> either way, but just in case, it should be clarified.
>
> > Index size is 1.4 TB per node
> > Xms 8 GB , Xmx 24 GB
> > Directory factory used is SimpleFSDirectoryFactory
>
> How much total memory in the server?  Is there other software using
> significant levels of memory?
>
> Why did you opt to change the DirectoryFactory away from Solr's default?
>   The default is chosen with care ... any other choice will probably
> result in lower performance.  The default in recent versions of Solr is
> NRTCachingDirectoryFactory, which uses MMap for file access.
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> The screenshot described here might become useful for more in-depth
> troubleshooting:
>
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#Process_listing_on_Windows
>
> How many total documents (maxDoc, not numDoc) are in that 1.4 TB of space?
>
> > The cloud is all nice and green for the most part. Only when we start
> > indexing, within a few seconds, I start seeing Read timeouts and socket
> > write errors and replica recoveries thereafter. We are indexing in 2
> > parallel threads in batches of 50 docs per update request. After
> examining
> > the thread dump, I see segment merges happening. My understanding is that
> > this is the cause, and the timeouts and recoveries are the symptoms. Is
> my
> > understanding correct? If yes, what steps could I take to help the
> > situation. I do see that the difference between "Num Docs" and "Max Docs"
> > is about 20%.
>
> Segment merges are a completely normal part of Lucene's internal
> operation.  They should never cause problems like you have described.
>
> My best guess is that a 24GB heap is too small.  Or possibly WAY too
> large, although with the index size you have mentioned, that seems
> unlikely.
>
> Can you share the GC log that Solr writes?  The problem should occur
> during the timeframe covered by the log, and the log should be as large
> as possible.  You'll need to use a file sharing site -- attaching it to
> an email is not going to work.
>
> What version of Solr?
>
> Thanks,
> Shawn
>

Re: SolrCloud indexing triggers merges and timeouts

[Shawn Heisey]

On 6/5/2019 9:39 AM, Rahul Goswami wrote:

I have a solrcloud setup on Windows server with below config:
3 nodes,
24 shards with replication factor 2
Each node hosts 16 cores.


16 CPU cores, or 16 Solr cores?  The info may not be all that useful 
either way, but just in case, it should be clarified.



Index size is 1.4 TB per node
Xms 8 GB , Xmx 24 GB
Directory factory used is SimpleFSDirectoryFactory


How much total memory in the server?  Is there other software using 
significant levels of memory?


Why did you opt to change the DirectoryFactory away from Solr's default? 
 The default is chosen with care ... any other choice will probably 
result in lower performance.  The default in recent versions of Solr is 
NRTCachingDirectoryFactory, which uses MMap for file access.


http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

The screenshot described here might become useful for more in-depth 
troubleshooting:


https://wiki.apache.org/solr/SolrPerformanceProblems#Process_listing_on_Windows

How many total documents (maxDoc, not numDoc) are in that 1.4 TB of space?


The cloud is all nice and green for the most part. Only when we start
indexing, within a few seconds, I start seeing Read timeouts and socket
write errors and replica recoveries thereafter. We are indexing in 2
parallel threads in batches of 50 docs per update request. After examining
the thread dump, I see segment merges happening. My understanding is that
this is the cause, and the timeouts and recoveries are the symptoms. Is my
understanding correct? If yes, what steps could I take to help the
situation. I do see that the difference between "Num Docs" and "Max Docs"
is about 20%.


Segment merges are a completely normal part of Lucene's internal 
operation.  They should never cause problems like you have described.


My best guess is that a 24GB heap is too small.  Or possibly WAY too 
large, although with the index size you have mentioned, that seems unlikely.


Can you share the GC log that Solr writes?  The problem should occur 
during the timeframe covered by the log, and the log should be as large 
as possible.  You'll need to use a file sharing site -- attaching it to 
an email is not going to work.


What version of Solr?

Thanks,
Shawn

Re: SolrCloud indexing triggers merges and timeouts

[Walter Underwood]

Yes, set Xmx and Xms the same.

We run an 8 GB heap for all our clusters. Unless you are doing some really
memory-intensive stuff like faceting, 8 GB should be fine.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Jun 5, 2019, at 1:05 PM, Gus Heck  wrote:
> 
> Probably not a solution, but so.ething I notice off the bat... generally
> you want Xmx and Xms set to the same value so the jvm doesn't have to spend
> time asking for more and more memory, and also reduce the chance that the
> memory is not available by the time solr needs it.
> 
> On Wed, Jun 5, 2019, 11:39 AM Rahul Goswami  wrote:
> 
>> Hello,
>> I have a solrcloud setup on Windows server with below config:
>> 3 nodes,
>> 24 shards with replication factor 2
>> Each node hosts 16 cores.
>> 
>> Index size is 1.4 TB per node
>> Xms 8 GB , Xmx 24 GB
>> Directory factory used is SimpleFSDirectoryFactory
>> 
>> The cloud is all nice and green for the most part. Only when we start
>> indexing, within a few seconds, I start seeing Read timeouts and socket
>> write errors and replica recoveries thereafter. We are indexing in 2
>> parallel threads in batches of 50 docs per update request. After examining
>> the thread dump, I see segment merges happening. My understanding is that
>> this is the cause, and the timeouts and recoveries are the symptoms. Is my
>> understanding correct? If yes, what steps could I take to help the
>> situation. I do see that the difference between "Num Docs" and "Max Docs"
>> is about 20%.
>> 
>> Would appreciate your help.
>> 
>> Thanks,
>> Rahul
>> 

Re: SolrCloud indexing triggers merges and timeouts

[Gus Heck]

Probably not a solution, but so.ething I notice off the bat... generally
you want Xmx and Xms set to the same value so the jvm doesn't have to spend
time asking for more and more memory, and also reduce the chance that the
memory is not available by the time solr needs it.

On Wed, Jun 5, 2019, 11:39 AM Rahul Goswami  wrote:

> Hello,
> I have a solrcloud setup on Windows server with below config:
> 3 nodes,
> 24 shards with replication factor 2
> Each node hosts 16 cores.
>
> Index size is 1.4 TB per node
> Xms 8 GB , Xmx 24 GB
> Directory factory used is SimpleFSDirectoryFactory
>
> The cloud is all nice and green for the most part. Only when we start
> indexing, within a few seconds, I start seeing Read timeouts and socket
> write errors and replica recoveries thereafter. We are indexing in 2
> parallel threads in batches of 50 docs per update request. After examining
> the thread dump, I see segment merges happening. My understanding is that
> this is the cause, and the timeouts and recoveries are the symptoms. Is my
> understanding correct? If yes, what steps could I take to help the
> situation. I do see that the difference between "Num Docs" and "Max Docs"
> is about 20%.
>
> Would appreciate your help.
>
> Thanks,
> Rahul
>

SolrCloud indexing triggers merges and timeouts

[Rahul Goswami]

Hello,
I have a solrcloud setup on Windows server with below config:
3 nodes,
24 shards with replication factor 2
Each node hosts 16 cores.

Index size is 1.4 TB per node
Xms 8 GB , Xmx 24 GB
Directory factory used is SimpleFSDirectoryFactory

The cloud is all nice and green for the most part. Only when we start
indexing, within a few seconds, I start seeing Read timeouts and socket
write errors and replica recoveries thereafter. We are indexing in 2
parallel threads in batches of 50 docs per update request. After examining
the thread dump, I see segment merges happening. My understanding is that
this is the cause, and the timeouts and recoveries are the symptoms. Is my
understanding correct? If yes, what steps could I take to help the
situation. I do see that the difference between "Num Docs" and "Max Docs"
is about 20%.

Would appreciate your help.

Thanks,
Rahul

Re: Configuring ZK Timeouts

[Shawn Heisey]

On 7/26/2018 8:58 AM, solrnoobie wrote:
> We are having problems with zk / solr node recovery and we are encountering
> this issue:
>
>  [   ] o.a.z.ClientCnxn Client session timed out, have not heard from server
> in 5003ms
>
> We have set the solr.xml zkClientTimeout to 30 secs.

The internal default for zkClientTimeout is 15 seconds, and recent
example configurations use a value of 30 seconds.  This value is used by
Solr to configure the zookeeper client session timeout.

Assuming that this log entry is found in the Solr server log ... it
appears that zkClientTimeout is set to 5 seconds.  If Solr were ignoring
its configuration entirely, the session timeout would be 15 seconds, not
5 seconds ... so I believe that Solr *is* paying attention to some kind
of config.

Can you share your full solr.xml file as well as everything in "JVM
Args" on the admin UI dashboard?

What version of Solr is it?

The default value of 15 seconds is a relative eternity to a program like
Solr.  Unless system resources are inadequate for handling the load, I
would even expect 5 seconds to be plenty of time.

Thanks,
Shawn

Configuring ZK Timeouts

[solrnoobie]

We are having problems with zk / solr node recovery and we are encountering
this issue:

 [   ] o.a.z.ClientCnxn Client session timed out, have not heard from server
in 5003ms

We have set the solr.xml zkClientTimeout to 30 secs.


What are we missing here?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

solr LTR edismax features very slow and cause timeouts

[lekha_dipika]

I am trying to use the SOLR LTR plugin to train a machine learning model for
my application. One of the features I require for my model is edismax score
of my query over a set of index fields. 

The way I have created the feature is as follows:

{  "store":"sampleFeatureStore",
   "name" : "edismax_feature",
   "class" : "org.apache.solr.ltr.feature.SolrFeature",
   "params" : { "q" : "{!edismax qf=\"field1 field2 field3 field4\"
}${user_query}" }
 }


field1 ... field4 are text fields with some analyzers performing
tokenization, lowercasing etc. 

While this feature gives me the edismax scores I desire, however, on load
testing my entire setup, presence of these features increased latency
conderably and I could see timeouts in the logs for many queries.  (The
request took too long to iterate over terms) I also saw my CPU usage and JVM
memory use increase.

Without this feature things were stable and load testing was fine. 

I have 2 collections in my solr setup with 25M and 2M docs respectively.
Load testing involved sending 200 queries per minute.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-LTR-edismax-features-very-slow-and-cause-timeouts-tp4335059.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr Timeouts during query aggregation

[Peter Lee]

Greetings,

I'm having a hard time locating information about how solr handles timeouts (if 
it DOES handle them) during a search across multiple shards in a solr cloud 
configuration.

I have found information and have confirmed through empirical testing as to how 
solr handles timeouts on each shard when a query is made to a collection. What 
I have NOT been able to find is information or settings related to the time it 
takes Solr to aggregate the results returned from multiple shards before 
returning the response to the user. Does Solr not have any sort of timeout on 
this operation?

For clarity, I'll repeat my question and try to explain it in more detail.

If I send a query to a solr cloud setup that has 6 shards, the query will be 
sent to each of the 6 shards that will each return some number of hits. The 
answers from each of the shards is sent back to the server that originally 
caught the query, and that original server must then aggregate the data from 
all of the different shards to produce a single set of hits to return to the 
user. I see how to use "timeAllowed" to limit the time of the search on each 
shard...but I was wondering if there was a separate timeout for the 
"aggregation" step just before the response is returned.

I am asking this question because our existing search technology has this 
behavior and setting, and I and I am trying to determine if there is a related 
feature within the solr technology. At this point, since I have not seen any 
documentation nor configuration settings for this feature, I am ready to take 
it as truth that Solr does NOT include this functionality. However, I thought I 
should ask the mailing list to see if I've missed something.

Thank you!

Peter S. Lee, Software Engineer Lead
ProQuest | 789 E. Eisenhower Parkway | Ann Arbor, MI, 48106-1346 USA
www.proquest.com

Re: Timeouts for create_collection

[Andrej van der Zee]

Hi,

I completely started over again. Now I get the following error upon
create_collection:

solr@ip-172-31-11-63:/opt/solr$ ./bin/solr create_collection -c connects
-replicationFactor 2

Connecting to ZooKeeper at 172.31.11.65:2181 ...
Re-using existing configuration directory connects

Creating new collection 'connects' using command:
http://localhost:8984/solr/admin/collections?action=CREATE&name=connects&numShards=1&replicationFactor=2&maxShardsPerNode=1&collection.configName=connects

ERROR: Failed to create collection 'connects' due to:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://172.31.11.63:8984/solr: Error CREATEing SolrCore
'connects_shard1_replica1': Unable to create core
[connects_shard1_replica1] Caused by: Can't find resource 'solrconfig.xml'
in classpath or '/configs/connects', cwd=/opt/solr/server


When I look in Zookeeker, I can see that solrconfig.xml is in
/configs/connects/conf/. Why can't the Solr instance find it? Am I missing
some crucial configuration setting?

Thanks,
Andrej

Timeouts for create_collection

[Andrej van der Zee]

Hi,

I am newby to Solr and I am having difficulties setting up a cluster with a
single Zookeeper instance and two Solr instances. The Solr intances both
successfully establish sessions with the Zookeeper and I am able to upload
collection configs to Zookeeper, but somehow creating a collection from one
of the Solr instances timeouts:

core@ip-172-31-11-63:/opt/solr# ./bin/solr create_collection -c connects
-replicationFactor 2

Connecting to ZooKeeper at 172.31.11.65:2181 ...
Re-using existing configuration directory connects

Creating new collection 'connects' using command:
http://localhost:8984/solr/admin/collections?action=CREATE&name=connects&numShards=1&replicationFactor=2&maxShardsPerNode=1&collection.configName=connects

ERROR: Failed to create collection 'connects' due to: create the collection
time out:180s


Another thing that got my attention is that the /clusterstate.json is
empty, even after establishing sessions from the Solr instances. I am not
sure if it is related or that the clusterstate is supposed to be empty
until I successfully create a collection?

Thanks,
Andrej

Re: More Robust Search Timeouts (to Kill Zombie Queries)?

[Steve Davids]

I have also experienced a similar problem on our cluster, I went ahead and 
opened SOLR-5986 to track the issue. I know Apache Blur has implemented a 
mechanism to kill these long running term enumerations, would be fantastic if 
Solr can get a similar mechanism.

-Steve

On Apr 15, 2014, at 5:23 AM, Salman Akram  
wrote:

> Looking at this, sharding seems to be best and simple option to handle such
> queries.
> 
> 
> On Wed, Apr 2, 2014 at 1:26 AM, Mikhail Khludnev > wrote:
> 
>> Hello Salman,
>> Let's me drop few thoughts on
>> 
>> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E
>> 
>> There two aspects of this question:
>> 1. dealing with long running processing (thread divergence actions
>> http://docs.oracle.com/javase/specs/jls/se5.0/html/memory.html#65310) and
>> 2. an actual time checking.
>> "terminating" or "aborting" thread (2.) are just a way to tracking time
>> externally, and send interrupt() which the thread should react on, which
>> they don't do now, and we returning to the core issue (1.)
>> 
>> Solr's time allowed is to the proper way to handle this things, the only
>> problem is that expect that the only core search is long running, but in
>> your case rewriting MultiTermQuery-s takes a huge time.
>> Let's consider this problem. First of all MultiTermQuery.rewrite() is the
>> nearly design issue, after heavy rewrite occurs, it's thrown away, after
>> search is done. I think the most straightforward way is to address this
>> issue by caching these expensive queries. Solr does it well
>> http://wiki.apache.org/solr/CommonQueryParameters#fq However, only for
>> http://en.wikipedia.org/wiki/Conjunctive_normal_form like queries, there
>> is
>> a workaround allows to cache disjunction legs see
>> http://blog.griddynamics.com/2014/01/segmented-filter-cache-in-solr.html
>> If you still want to run expensively rewritten queries you need to
>> implement timeout check (similar to TimeLimitingCollector) for TermsEnum
>> returned from MultiTermQuery.getTermsEnum(), wrapping an actual TermsEnums
>> is the good way, to apply queries injecting time limiting wrapper
>> TermsEnum, you might consider override methods like
>> SolrQueryParserBase.newWildcardQuery(Term) or post process the query three
>> after parsing.
>> 
>> 
>> 
>> On Mon, Mar 31, 2014 at 2:24 PM, Salman Akram <
>> salman.ak...@northbaysolutions.net> wrote:
>> 
>>> Anyone?
>>> 
>>> 
>>> On Wed, Mar 26, 2014 at 7:55 PM, Salman Akram <
>>> salman.ak...@northbaysolutions.net> wrote:
>>> 
 With reference to this thread<
>>> 
>> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E
>>> I
>>> wanted to know if there was any response to that or if Chris Harris
 himself can comment on what he ended up doing, that would be great!
 
 
 --
 Regards,
 
 Salman Akram
 
 
>>> 
>>> 
>>> --
>>> Regards,
>>> 
>>> Salman Akram
>>> 
>> 
>> 
>> 
>> --
>> Sincerely yours
>> Mikhail Khludnev
>> Principal Engineer,
>> Grid Dynamics
>> 
>> 
>> 
>> 
> 
> 
> 
> -- 
> Regards,
> 
> Salman Akram

Re: More Robust Search Timeouts (to Kill Zombie Queries)?

[Salman Akram]

Looking at this, sharding seems to be best and simple option to handle such
queries.


On Wed, Apr 2, 2014 at 1:26 AM, Mikhail Khludnev  wrote:

> Hello Salman,
> Let's me drop few thoughts on
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E
>
> There two aspects of this question:
> 1. dealing with long running processing (thread divergence actions
> http://docs.oracle.com/javase/specs/jls/se5.0/html/memory.html#65310) and
> 2. an actual time checking.
> "terminating" or "aborting" thread (2.) are just a way to tracking time
> externally, and send interrupt() which the thread should react on, which
> they don't do now, and we returning to the core issue (1.)
>
> Solr's time allowed is to the proper way to handle this things, the only
> problem is that expect that the only core search is long running, but in
> your case rewriting MultiTermQuery-s takes a huge time.
> Let's consider this problem. First of all MultiTermQuery.rewrite() is the
> nearly design issue, after heavy rewrite occurs, it's thrown away, after
> search is done. I think the most straightforward way is to address this
> issue by caching these expensive queries. Solr does it well
> http://wiki.apache.org/solr/CommonQueryParameters#fq However, only for
> http://en.wikipedia.org/wiki/Conjunctive_normal_form like queries, there
> is
> a workaround allows to cache disjunction legs see
> http://blog.griddynamics.com/2014/01/segmented-filter-cache-in-solr.html
> If you still want to run expensively rewritten queries you need to
> implement timeout check (similar to TimeLimitingCollector) for TermsEnum
> returned from MultiTermQuery.getTermsEnum(), wrapping an actual TermsEnums
> is the good way, to apply queries injecting time limiting wrapper
> TermsEnum, you might consider override methods like
> SolrQueryParserBase.newWildcardQuery(Term) or post process the query three
> after parsing.
>
>
>
> On Mon, Mar 31, 2014 at 2:24 PM, Salman Akram <
> salman.ak...@northbaysolutions.net> wrote:
>
> > Anyone?
> >
> >
> > On Wed, Mar 26, 2014 at 7:55 PM, Salman Akram <
> > salman.ak...@northbaysolutions.net> wrote:
> >
> > > With reference to this thread<
> >
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E
> >I
> > wanted to know if there was any response to that or if Chris Harris
> > > himself can comment on what he ended up doing, that would be great!
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Salman Akram
> > >
> > >
> >
> >
> > --
> > Regards,
> >
> > Salman Akram
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> 
>  
>



-- 
Regards,

Salman Akram

Re: More Robust Search Timeouts (to Kill Zombie Queries)?

[Mikhail Khludnev]

Hello Salman,
Let's me drop few thoughts on
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E

There two aspects of this question:
1. dealing with long running processing (thread divergence actions
http://docs.oracle.com/javase/specs/jls/se5.0/html/memory.html#65310) and
2. an actual time checking.
"terminating" or "aborting" thread (2.) are just a way to tracking time
externally, and send interrupt() which the thread should react on, which
they don't do now, and we returning to the core issue (1.)

Solr's time allowed is to the proper way to handle this things, the only
problem is that expect that the only core search is long running, but in
your case rewriting MultiTermQuery-s takes a huge time.
Let's consider this problem. First of all MultiTermQuery.rewrite() is the
nearly design issue, after heavy rewrite occurs, it's thrown away, after
search is done. I think the most straightforward way is to address this
issue by caching these expensive queries. Solr does it well
http://wiki.apache.org/solr/CommonQueryParameters#fq However, only for
http://en.wikipedia.org/wiki/Conjunctive_normal_form like queries, there is
a workaround allows to cache disjunction legs see
http://blog.griddynamics.com/2014/01/segmented-filter-cache-in-solr.html
If you still want to run expensively rewritten queries you need to
implement timeout check (similar to TimeLimitingCollector) for TermsEnum
returned from MultiTermQuery.getTermsEnum(), wrapping an actual TermsEnums
is the good way, to apply queries injecting time limiting wrapper
TermsEnum, you might consider override methods like
SolrQueryParserBase.newWildcardQuery(Term) or post process the query three
after parsing.



On Mon, Mar 31, 2014 at 2:24 PM, Salman Akram <
salman.ak...@northbaysolutions.net> wrote:

> Anyone?
>
>
> On Wed, Mar 26, 2014 at 7:55 PM, Salman Akram <
> salman.ak...@northbaysolutions.net> wrote:
>
> > With reference to this thread<
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E>I
> wanted to know if there was any response to that or if Chris Harris
> > himself can comment on what he ended up doing, that would be great!
> >
> >
> > --
> > Regards,
> >
> > Salman Akram
> >
> >
>
>
> --
> Regards,
>
> Salman Akram
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics


 

Re: More Robust Search Timeouts (to Kill Zombie Queries)?

[Luis Lebolo]

I got responses, but no easy solution to allow me to directly cancel a
request. The responses did point to:

   - timeAllowed query parameter that returns partial results -
   
https://cwiki.apache.org/confluence/display/solr/Common+Query+Parameters#CommonQueryParameters-ThetimeAllowedParameter
   - A possible hack that I never followed through -
   
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201401.mbox/%3CCANGii8eaSouePGxa7JfvOBhrnJUL++Ct4rQha2pxMefvaWhH=g...@mail.gmail.com%3E

Maybe one of those will help you? If they do, make sure to report back!

-Luis


On Tue, Apr 1, 2014 at 3:13 AM, Salman Akram <
salman.ak...@northbaysolutions.net> wrote:

> So you too never got any response...
>
>
> On Mon, Mar 31, 2014 at 6:57 PM, Luis Lebolo 
> wrote:
>
> > Hi Salman,
> >
> > I was interested in something similar, take a look at the following
> thread:
> >
> >
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201401.mbox/%3CCADSoL-i04aYrsOo2%3DGcaFqsQ3mViF%2Bhn24ArDtT%3D7kpALtVHzA%40mail.gmail.com%3E#archives
> >
> > I never followed through, however.
> >
> > -Luis
> >
> >
> > On Mon, Mar 31, 2014 at 6:24 AM, Salman Akram <
> > salman.ak...@northbaysolutions.net> wrote:
> >
> > > Anyone?
> > >
> > >
> > > On Wed, Mar 26, 2014 at 7:55 PM, Salman Akram <
> > > salman.ak...@northbaysolutions.net> wrote:
> > >
> > > > With reference to this thread<
> > >
> >
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E
> > >I
> > > wanted to know if there was any response to that or if Chris Harris
> > > > himself can comment on what he ended up doing, that would be great!
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Salman Akram
> > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Salman Akram
> > >
> >
>
>
>
> --
> Regards,
>
> Salman Akram
>

Re: More Robust Search Timeouts (to Kill Zombie Queries)?

[Salman Akram]

So you too never got any response...


On Mon, Mar 31, 2014 at 6:57 PM, Luis Lebolo  wrote:

> Hi Salman,
>
> I was interested in something similar, take a look at the following thread:
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201401.mbox/%3CCADSoL-i04aYrsOo2%3DGcaFqsQ3mViF%2Bhn24ArDtT%3D7kpALtVHzA%40mail.gmail.com%3E#archives
>
> I never followed through, however.
>
> -Luis
>
>
> On Mon, Mar 31, 2014 at 6:24 AM, Salman Akram <
> salman.ak...@northbaysolutions.net> wrote:
>
> > Anyone?
> >
> >
> > On Wed, Mar 26, 2014 at 7:55 PM, Salman Akram <
> > salman.ak...@northbaysolutions.net> wrote:
> >
> > > With reference to this thread<
> >
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E
> >I
> > wanted to know if there was any response to that or if Chris Harris
> > > himself can comment on what he ended up doing, that would be great!
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Salman Akram
> > >
> > >
> >
> >
> > --
> > Regards,
> >
> > Salman Akram
> >
>



-- 
Regards,

Salman Akram

Re: More Robust Search Timeouts (to Kill Zombie Queries)?

[Luis Lebolo]

Hi Salman,

I was interested in something similar, take a look at the following thread:
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201401.mbox/%3CCADSoL-i04aYrsOo2%3DGcaFqsQ3mViF%2Bhn24ArDtT%3D7kpALtVHzA%40mail.gmail.com%3E#archives

I never followed through, however.

-Luis


On Mon, Mar 31, 2014 at 6:24 AM, Salman Akram <
salman.ak...@northbaysolutions.net> wrote:

> Anyone?
>
>
> On Wed, Mar 26, 2014 at 7:55 PM, Salman Akram <
> salman.ak...@northbaysolutions.net> wrote:
>
> > With reference to this thread<
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200903.mbox/%3c856ac15f0903272054q2dbdbd19kea3c5ba9e105b...@mail.gmail.com%3E>I
> wanted to know if there was any response to that or if Chris Harris
> > himself can comment on what he ended up doing, that would be great!
> >
> >
> > --
> > Regards,
> >
> > Salman Akram
> >
> >
>
>
> --
> Regards,
>
> Salman Akram
>

Re: More Robust Search Timeouts (to Kill Zombie Queries)?

[Salman Akram]

Anyone?


On Wed, Mar 26, 2014 at 7:55 PM, Salman Akram <
salman.ak...@northbaysolutions.net> wrote:

> With reference to this 
> threadI
>  wanted to know if there was any response to that or if Chris Harris
> himself can comment on what he ended up doing, that would be great!
>
>
> --
> Regards,
>
> Salman Akram
>
>


-- 
Regards,

Salman Akram

More Robust Search Timeouts (to Kill Zombie Queries)?

[Salman Akram]

With reference to this
threadI
wanted to know if there was any response to that or if Chris Harris
himself can comment on what he ended up doing, that would be great!


-- 
Regards,

Salman Akram

Re: Two problems (missing updates and timeouts)

[Erick Erickson]

For your first problem I'd be looking at the solr logs and verifying that
1> the update was sent
2> no stack traces are thrown
3> You probably already know all about commits, but just in case the commit
interval is passed.

For your second problem, I'm not quite sure where you're setting these
timeouts. SolrJ?

Best
Erick


On Sat, Mar 23, 2013 at 4:23 PM, Aaron Jensen  wrote:

> Hi all,
>
> I'm having two problem with our solr implementation. I don't have a lot of
> detail about them because we're just starting to get into diagnosing them.
> I'm hoping for some help with that diagnosis, ideas, tips, whatever.
>
> Our stack:
>
> Rails
> Sunspot Solr
> sunspot_index_queue
> two solr servers, master and slave, all traffic currently going to master,
> slave is just a replication slave/backup.
>
>
> The first and biggest problem is that we occasionally "lose" updates.
> Something will get added to the database, it will trigger a solr update,
> but then we can't search for that thing. It's just gone. indexing that
> thing again will have it show up. There are a number of moving parts in our
> stack and this is a relatively new problem. It was working fine for 1.5
> years without a problem. We're considering adding a delayed job that will
> index anything that is newly created a second after it is created just to
> "be sure" but this is a giant hack. Any ideas around this would be helpful.
>
>
>
> The second problem is that we get occasional timeouts. These don't happen
> very often, maybe 5-7/day. Solr is serving at most like 350 requests per
> minute. Our timeouts are set to 2 seconds on read and 1 second on open.
> Average response time is around 20ms. It doesn't seem like any requests
> should be timing out but they are. I have no idea how to debug it either.
> Any ideas?
>
> Thanks,
>
> Aaron
>
>

Two problems (missing updates and timeouts)

[Aaron Jensen]

Hi all, 

I'm having two problem with our solr implementation. I don't have a lot of 
detail about them because we're just starting to get into diagnosing them. I'm 
hoping for some help with that diagnosis, ideas, tips, whatever.

Our stack:

Rails
Sunspot Solr
sunspot_index_queue
two solr servers, master and slave, all traffic currently going to master, 
slave is just a replication slave/backup.


The first and biggest problem is that we occasionally "lose" updates. Something 
will get added to the database, it will trigger a solr update, but then we 
can't search for that thing. It's just gone. indexing that thing again will 
have it show up. There are a number of moving parts in our stack and this is a 
relatively new problem. It was working fine for 1.5 years without a problem. 
We're considering adding a delayed job that will index anything that is newly 
created a second after it is created just to "be sure" but this is a giant 
hack. Any ideas around this would be helpful.



The second problem is that we get occasional timeouts. These don't happen very 
often, maybe 5-7/day. Solr is serving at most like 350 requests per minute. Our 
timeouts are set to 2 seconds on read and 1 second on open. Average response 
time is around 20ms. It doesn't seem like any requests should be timing out but 
they are. I have no idea how to debug it either. Any ideas?

Thanks,

Aaron 

Re: Shard timeouts on large (1B docs) Solr cluster

[Marc Sturlese]

timeAllowed can be used outside distributed search. It is used by the
TimeL¡mitingCollector. When the search time is equal to timeAllowed it will
stop searching and will return the results that could find till then.
This can be a problem when using incremental indexing. Lucene starts
searching from "the bottom" and new docs are inserted on the top, so,
timeAllowed could cause that new docs never appear on the search results.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Shard-timeouts-on-large-1B-docs-Solr-cluster-tp3691229p3713263.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shard timeouts on large (1B docs) Solr cluster

[Jay Hill]

i'm changing the params to socketTimeout and connTimeout and will test this
afternoon. client timeout was actually removed today, which helped a bit.

what about the other params, "timeAllowed" and "partialResults". my
expectation was that these were specifically for distributed search,
meaning if a response wasn't received w/in the timeAllowed, and if
partialResults is true, then that shard would not be waited on for results.
is that correct?

thanks,
-jay


On Thu, Jan 26, 2012 at 2:23 PM, Jay Hill  wrote:

> We're on the trunk:
> 4.0-2011-10-26_08-46-59 1189079 - hudson - 2011-10-26 08:51:47
>
> Client timeouts are set to 4 seconds.
>
> Thanks,
> -Jay
>
>
> On Thu, Jan 26, 2012 at 1:40 PM, Mark Miller wrote:
>
>>
>> On Jan 26, 2012, at 1:28 PM, Jay Hill wrote:
>>
>> >
>> > I've tried setting the following config in our default req. handler:
>> > 2000
>> > 2000
>> >
>>
>>
>> What version are you using Jay? At least on trunk, I took a look and it
>> appears at some point these where renamed to socketTimeout and connTimeout.
>>
>> What about a timeout on your clients?
>>
>> - Mark Miller
>> lucidimagination.com
>>
>>
>

Re: Shard timeouts on large (1B docs) Solr cluster

[Jay Hill]

We're on the trunk:
4.0-2011-10-26_08-46-59 1189079 - hudson - 2011-10-26 08:51:47

Client timeouts are set to 4 seconds.

Thanks,
-Jay

On Thu, Jan 26, 2012 at 1:40 PM, Mark Miller  wrote:

>
> On Jan 26, 2012, at 1:28 PM, Jay Hill wrote:
>
> >
> > I've tried setting the following config in our default req. handler:
> > 2000
> > 2000
> >
>
>
> What version are you using Jay? At least on trunk, I took a look and it
> appears at some point these where renamed to socketTimeout and connTimeout.
>
> What about a timeout on your clients?
>
> - Mark Miller
> lucidimagination.com
>
>

Re: Shard timeouts on large (1B docs) Solr cluster

[Mark Miller]

On Jan 26, 2012, at 1:28 PM, Jay Hill wrote:

> 
> I've tried setting the following config in our default req. handler:
> 2000
> 2000
> 


What version are you using Jay? At least on trunk, I took a look and it appears 
at some point these where renamed to socketTimeout and connTimeout.

What about a timeout on your clients?

- Mark Miller
lucidimagination.com

Shard timeouts on large (1B docs) Solr cluster

[Jay Hill]

I'm on a project where we have 1B docs sharded across 20 servers. We're not
in production yet and we're doing load tests now. We're sending load to hit
100qps per server. As the load increases we're seeing query times
sporadically increasing to 10 seconds, 20 seconds, etc. at times. What
we're trying to do is set a shard timeout so that responses longer than 2
seconds are discarded. We can live with less results in these cases. We're
not replicating yet as we want to see how the 20 shards perform first (plus
we're waiting on the massive amount of hardware)

I've tried setting the following config in our default req. handler:
2000
2000

I've just added these, and am testing now, but this doesn't look promising
either:
2000
true

Couldn't find much on the wiki about these params - I'm looking for more
details about how these work. I'll be happy to update the wiki with more
details based on the discussion here.

Any details about exactly how I can achieve my goal of timing out and
disregarding queries longer that 2 seconds would be greatly appreciated.

The index is insanely lean - no stored fields, no norms, no stop words,
etc. RAM buffer is 128, and we're using the standard "search" req. handler.
Essentially we're running Solr as a nosql data store, which suits this
project, but we need responses to be no longer than 2 seconds at the max.

Thanks,
-Jay

Intermittent connection timeouts to Solr server using SolrNet

[Ian Grainger]

Hi - I have also posted this question on SO:
http://stackoverflow.com/questions/8741080/intermittent-connection-timeouts-to-solr-server-using-solrnet


I have a production webserver hosting a search webpage, which uses SolrNet
to connect to another machine which hosts the Solr search server (on a
subnet which is in the same room, so no network problems). All is fine >90%
of the time, but I consistently get a small number of The operation has
timed out errors.

I've increased the timeout in the SolrNet init to *30* seconds (!)

SolrNet.Startup.Init(
new SolrNet.Impl.SolrConnection(
System.Configuration.ConfigurationManager.AppSettings["URL"]
) {Timeout = 3}
);

...but all that happened is I started getting this message instead of Unable
to connect to the remote server which I was seeing before. It seems to have
made no difference to the amount of timeout errors.

I can see *nothing* in *any* log (believe me I've looked!) and clearly my
configuration is correct because it works most of the time. Anyone any
ideas how I can find more information on this problem?

Thanks!


-- 
Ian

i...@isfluent.com 
+44 (0)1223 257903

Timeouts in distributed search using Solr + Zookeeper

[Cinquini, Luca (3880)]

Hi,
we are looking at Solr+Zookeeper as the architecture for enabling 
federated searches among geographically distributed data centers.

I wonder if anybody can comment on what is the status of enabling timeouts with 
respect to distributed searches in a Solr-Zookeeper environment. Specifially, 
following the example C) in the Solr Cloud wiki:

http://wiki.apache.org/solr/SolrCloud

it seems like the system is resilient to any Solr server (out of 4) being 
unavailable, but if both Solr servers serving the same shard go down, then a 
distributed query results in error, instead or returning partial results. Is 
there any special configuration that needs to be set for the Solr and/or 
Zookeepers servers, or any request parameter that needs to be added, to make 
the distributed query just return results from the only available shard ? Or 
maybe this feature is not yet operational ?

thanks a lot,
Luca

Re: Solr Timeouts

[Shalin Shekhar Mangar]

On Wed, Oct 7, 2009 at 2:19 AM, Giovanni Fernandez-Kincade <
gfernandez-kinc...@capitaliq.com> wrote:

>
> What does the "maxCommitsToKeep"(from SolrDeletionPolicy in SolrConfig.xml)
> parameter actually do? Increasing this value seems to have helped a little,
> but I'm wary of cranking it without having a better understanding of what it
> does.
>
>
maxCommitsToKeep is the number of commit points (a point-in-time snapshot of
the index) to keep from getting deleted. But deletion of commit points only
happens on startup or when someone calls commit/optimize.

-- 
Regards,
Shalin Shekhar Mangar.

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

Yeah that's exactly right Mark.

What does the "maxCommitsToKeep"(from SolrDeletionPolicy in SolrConfig.xml) 
parameter actually do? Increasing this value seems to have helped a little, but 
I'm wary of cranking it without having a better understanding of what it does.

-Original Message-
From: Mark Miller [mailto:markrmil...@gmail.com]
Sent: Tuesday, October 06, 2009 4:44 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Timeouts

It sounds like he is indexing on a local disk, but reading the files to
be index from NFS - which would be fine.

You can get Lucene indexes to work on NFS (though still not recommended)
, but you need to use a custom IndexDeletionPolicy to keep older commit
points around longer and be sure not to use NIOFSDirectory.

Feak, Todd wrote:
> I seem to recall hearing something about *not* putting a Solr index directory 
> on an NFS mount. Might want to search on that.
>
> That, of course, doesn't have anything to do with commits showing up 
> unexpectedly in stack traces, per your original email.
>
> -Todd
>
> -Original Message-
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
> Sent: Tuesday, October 06, 2009 12:39 PM
> To: solr-user@lucene.apache.org; yo...@lucidimagination.com
> Subject: RE: Solr Timeouts
>
> That thread was blocking for an hour while all other threads were idle or 
> blocked.
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Tuesday, October 06, 2009 3:07 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
> This specific thread was blocked for an hour?
> If so, I'd echo Lance... this is a local disk right?
>
> -Yonik
> http://www.lucidimagination.com
>
>
> On Mon, Oct 5, 2009 at 2:11 PM, Giovanni Fernandez-Kincade
>  wrote:
>
>> I just grabbed another stack trace for a thread that has been similarly 
>> blocking for over an hour. Notice that there is no Commit in this one:
>>
>> http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
>> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
>> org.apache.lucene.index.SegmentTermEnum.next()
>> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
>> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
>> org.apache.lucene.index.TermInfosReader.get(Term)
>> org.apache.lucene.index.SegmentTermDocs.seek(Term)
>> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
>> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
>> org.apache.lucene.index.IndexWriter.applyDeletes()
>> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
>> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
>> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
>> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
>> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
>>  AddUpdateCommand)
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
>>  SolrQueryResponse, ContentStream)
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>>  SolrQueryResponse)
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
>> SolrQueryResponse)
>> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
>>  SolrQueryResponse)
>> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
>> SolrQueryResponse)
>> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
>> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
>> ServletResponse, FilterChain)
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
>>  ServletResponse)
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
>> ServletResponse)
>> org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
>> org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
>> org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
>> org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
>&

Re: Solr Timeouts

[Mark Miller]

It sounds like he is indexing on a local disk, but reading the files to
be index from NFS - which would be fine.

You can get Lucene indexes to work on NFS (though still not recommended)
, but you need to use a custom IndexDeletionPolicy to keep older commit
points around longer and be sure not to use NIOFSDirectory.

Feak, Todd wrote:
> I seem to recall hearing something about *not* putting a Solr index directory 
> on an NFS mount. Might want to search on that.
>
> That, of course, doesn't have anything to do with commits showing up 
> unexpectedly in stack traces, per your original email.
>
> -Todd
>
> -Original Message-
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
> Sent: Tuesday, October 06, 2009 12:39 PM
> To: solr-user@lucene.apache.org; yo...@lucidimagination.com
> Subject: RE: Solr Timeouts
>
> That thread was blocking for an hour while all other threads were idle or 
> blocked.
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Tuesday, October 06, 2009 3:07 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
> This specific thread was blocked for an hour?
> If so, I'd echo Lance... this is a local disk right?
>
> -Yonik
> http://www.lucidimagination.com
>
>
> On Mon, Oct 5, 2009 at 2:11 PM, Giovanni Fernandez-Kincade
>  wrote:
>   
>> I just grabbed another stack trace for a thread that has been similarly 
>> blocking for over an hour. Notice that there is no Commit in this one:
>>
>> http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
>> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
>> org.apache.lucene.index.SegmentTermEnum.next()
>> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
>> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
>> org.apache.lucene.index.TermInfosReader.get(Term)
>> org.apache.lucene.index.SegmentTermDocs.seek(Term)
>> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
>> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
>> org.apache.lucene.index.IndexWriter.applyDeletes()
>> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
>> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
>> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
>> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
>> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
>>  AddUpdateCommand)
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
>> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
>>  SolrQueryResponse, ContentStream)
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>>  SolrQueryResponse)
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
>> SolrQueryResponse)
>> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
>>  SolrQueryResponse)
>> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
>> SolrQueryResponse)
>> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
>> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
>> ServletResponse, FilterChain)
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
>>  ServletResponse)
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
>> ServletResponse)
>> org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
>> org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
>> org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
>> org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
>> org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
>> org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
>> org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
>>  Object[])
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket, 
>> TcpConnection,

RE: Solr Timeouts

[Feak, Todd]

I seem to recall hearing something about *not* putting a Solr index directory 
on an NFS mount. Might want to search on that.

That, of course, doesn't have anything to do with commits showing up 
unexpectedly in stack traces, per your original email.

-Todd

-Original Message-
From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
Sent: Tuesday, October 06, 2009 12:39 PM
To: solr-user@lucene.apache.org; yo...@lucidimagination.com
Subject: RE: Solr Timeouts

That thread was blocking for an hour while all other threads were idle or 
blocked.

-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: Tuesday, October 06, 2009 3:07 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Timeouts

This specific thread was blocked for an hour?
If so, I'd echo Lance... this is a local disk right?

-Yonik
http://www.lucidimagination.com


On Mon, Oct 5, 2009 at 2:11 PM, Giovanni Fernandez-Kincade
 wrote:
> I just grabbed another stack trace for a thread that has been similarly 
> blocking for over an hour. Notice that there is no Commit in this one:
>
> http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
> org.apache.lucene.index.TermInfosReader.get(Term)
> org.apache.lucene.index.SegmentTermDocs.seek(Term)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
> org.apache.lucene.index.IndexWriter.applyDeletes()
> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
> org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
>  AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
>  SolrQueryResponse, ContentStream)
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>  SolrQueryResponse)
> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
> SolrQueryResponse)
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
>  SolrQueryResponse)
> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
> SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
> ServletResponse, FilterChain)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
>  ServletResponse)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
> ServletResponse)
> org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
> org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
> org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
> org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
> org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
> org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
> org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
>  Object[])
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket, 
> TcpConnection, Object[])
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[])
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
> java.lang.Thread.run()
>
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Monday, October 05, 2009 1:18 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
> OK... next step is to verify that SolrCell doesn't have a bug that
> causes it to commit.
> I'll try and verify today unless someone else beats me to it.
>
> -Yonik
> http://www.lucidimagination.com
>
> On Mon, Oct 5, 2009 at 1:04 PM, Giovann

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

That thread was blocking for an hour while all other threads were idle or 
blocked.

-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: Tuesday, October 06, 2009 3:07 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Timeouts

This specific thread was blocked for an hour?
If so, I'd echo Lance... this is a local disk right?

-Yonik
http://www.lucidimagination.com


On Mon, Oct 5, 2009 at 2:11 PM, Giovanni Fernandez-Kincade
 wrote:
> I just grabbed another stack trace for a thread that has been similarly 
> blocking for over an hour. Notice that there is no Commit in this one:
>
> http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
> org.apache.lucene.index.TermInfosReader.get(Term)
> org.apache.lucene.index.SegmentTermDocs.seek(Term)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
> org.apache.lucene.index.IndexWriter.applyDeletes()
> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
> org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
>  AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
>  SolrQueryResponse, ContentStream)
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>  SolrQueryResponse)
> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
> SolrQueryResponse)
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
>  SolrQueryResponse)
> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
> SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
> ServletResponse, FilterChain)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
>  ServletResponse)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
> ServletResponse)
> org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
> org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
> org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
> org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
> org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
> org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
> org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
>  Object[])
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket, 
> TcpConnection, Object[])
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[])
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
> java.lang.Thread.run()
>
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Monday, October 05, 2009 1:18 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
> OK... next step is to verify that SolrCell doesn't have a bug that
> causes it to commit.
> I'll try and verify today unless someone else beats me to it.
>
> -Yonik
> http://www.lucidimagination.com
>
> On Mon, Oct 5, 2009 at 1:04 PM, Giovanni Fernandez-Kincade
>  wrote:
>> I'm fairly certain that all of the indexing jobs are calling SOLR with 
>> commit=false. They all construct the indexing URLs using a CLR function I 
>> wrote, which takes in a Commit parameter, which is always set to false.
>>
>> Also, I don't see any calls to commit in the Tomcat logs (whereas normally 
>> when I make a commit call I do).
>>
>> This suggests that Solr is doing it automatically, b

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

Yeah this is Java 1.6.

The indexes are being written to a local disk, but they files being indexed 
live on a NFS.

-Original Message-
From: Lance Norskog [mailto:goks...@gmail.com]
Sent: Tuesday, October 06, 2009 2:59 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Timeouts

Is this Java 1.5? There are known threading bugs in 1.5 that were
fixed in Java 1.6. Also, there was one short series of 1.6 releases
that wrote bogus Lucene index files.

So, make sure you use the latest Java 1.6 release.

Also, I hope this is a local disk. Some shops try running over NFS or
Windows file sharing and this often does not work well.

Lance



On 10/6/09, Giovanni Fernandez-Kincade  wrote:
> Is it possible that deletions are triggering these commits? Some of the
> documents that I'm making indexing requests for already exist in the index,
> so they would result in deletions. I tried messing with some of these
> parameters but I'm still running into the same problem:
>
> 
>   
>   false
>   
>   100
>   
> 
>
> This is happening like every 30-40minutes and it's really hampering the
> indexing progress...
>
>
> -Original Message-
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
> Sent: Monday, October 05, 2009 2:11 PM
> To: solr-user@lucene.apache.org; yo...@lucidimagination.com
> Subject: RE: Solr Timeouts
>
> I just grabbed another stack trace for a thread that has been similarly
> blocking for over an hour. Notice that there is no Commit in this one:
>
> http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
> org.apache.lucene.index.TermInfosReader.get(Term)
> org.apache.lucene.index.SegmentTermDocs.seek(Term)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
> org.apache.lucene.index.IndexWriter.applyDeletes()
> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
> org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
> AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
> SolrQueryResponse, ContentStream)
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
> SolrQueryResponse)
> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest,
> SolrQueryResponse)
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
> SolrQueryResponse)
> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest,
> SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest,
> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest,
> ServletResponse, FilterChain)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
> ServletResponse)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
> ServletResponse)
> org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
> org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
> org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
> org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
> org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
> org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
> org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
> Object[])
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket,
> TcpConnection, Object[])
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[])
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
> java.lang.Thread.run()
>
>
> -Original Message-
> From: ysee..

Re: Solr Timeouts

[Yonik Seeley]

This specific thread was blocked for an hour?
If so, I'd echo Lance... this is a local disk right?

-Yonik
http://www.lucidimagination.com


On Mon, Oct 5, 2009 at 2:11 PM, Giovanni Fernandez-Kincade
 wrote:
> I just grabbed another stack trace for a thread that has been similarly 
> blocking for over an hour. Notice that there is no Commit in this one:
>
> http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
> org.apache.lucene.index.TermInfosReader.get(Term)
> org.apache.lucene.index.SegmentTermDocs.seek(Term)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
> org.apache.lucene.index.IndexWriter.applyDeletes()
> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
> org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
>  AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
>  SolrQueryResponse, ContentStream)
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>  SolrQueryResponse)
> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
> SolrQueryResponse)
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
>  SolrQueryResponse)
> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
> SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
> ServletResponse, FilterChain)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
>  ServletResponse)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
> ServletResponse)
> org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
> org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
> org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
> org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
> org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
> org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
> org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
>  Object[])
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket, 
> TcpConnection, Object[])
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[])
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
> java.lang.Thread.run()
>
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Monday, October 05, 2009 1:18 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
> OK... next step is to verify that SolrCell doesn't have a bug that
> causes it to commit.
> I'll try and verify today unless someone else beats me to it.
>
> -Yonik
> http://www.lucidimagination.com
>
> On Mon, Oct 5, 2009 at 1:04 PM, Giovanni Fernandez-Kincade
>  wrote:
>> I'm fairly certain that all of the indexing jobs are calling SOLR with 
>> commit=false. They all construct the indexing URLs using a CLR function I 
>> wrote, which takes in a Commit parameter, which is always set to false.
>>
>> Also, I don't see any calls to commit in the Tomcat logs (whereas normally 
>> when I make a commit call I do).
>>
>> This suggests that Solr is doing it automatically, but the extract handler 
>> doesn't seem to be the problem:
>>  > class="org.apache.solr.handler.extraction.ExtractingRequestHandler" 
>> startup="lazy">
>>    
>>      ignored_
>>      fileData
>>    
>>  
&g

Re: Solr Timeouts

[Lance Norskog]

Is this Java 1.5? There are known threading bugs in 1.5 that were
fixed in Java 1.6. Also, there was one short series of 1.6 releases
that wrote bogus Lucene index files.

So, make sure you use the latest Java 1.6 release.

Also, I hope this is a local disk. Some shops try running over NFS or
Windows file sharing and this often does not work well.

Lance



On 10/6/09, Giovanni Fernandez-Kincade  wrote:
> Is it possible that deletions are triggering these commits? Some of the
> documents that I'm making indexing requests for already exist in the index,
> so they would result in deletions. I tried messing with some of these
> parameters but I'm still running into the same problem:
>
> 
>   
>   false
>   
>   100
>   
> 
>
> This is happening like every 30-40minutes and it's really hampering the
> indexing progress...
>
>
> -Original Message-
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
> Sent: Monday, October 05, 2009 2:11 PM
> To: solr-user@lucene.apache.org; yo...@lucidimagination.com
> Subject: RE: Solr Timeouts
>
> I just grabbed another stack trace for a thread that has been similarly
> blocking for over an hour. Notice that there is no Commit in this one:
>
> http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
> org.apache.lucene.index.TermInfosReader.get(Term)
> org.apache.lucene.index.SegmentTermDocs.seek(Term)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
> org.apache.lucene.index.IndexWriter.applyDeletes()
> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
> org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
> org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
> AddUpdateCommand)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
> SolrQueryResponse, ContentStream)
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
> SolrQueryResponse)
> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest,
> SolrQueryResponse)
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
> SolrQueryResponse)
> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest,
> SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest,
> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest,
> ServletResponse, FilterChain)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
> ServletResponse)
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
> ServletResponse)
> org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
> org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
> org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
> org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
> org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
> org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
> org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
> Object[])
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket,
> TcpConnection, Object[])
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[])
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
> java.lang.Thread.run()
>
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Monday, October 05, 2009 1:18 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
> OK... next step is to verify that SolrCell doesn't have a bug that
> causes it to commit.
> I&

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

Is it possible that deletions are triggering these commits? Some of the 
documents that I'm making indexing requests for already exist in the index, so 
they would result in deletions. I tried messing with some of these parameters 
but I'm still running into the same problem:


  
  false
  
  100
  


This is happening like every 30-40minutes and it's really hampering the 
indexing progress...


-Original Message-
From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
Sent: Monday, October 05, 2009 2:11 PM
To: solr-user@lucene.apache.org; yo...@lucidimagination.com
Subject: RE: Solr Timeouts

I just grabbed another stack trace for a thread that has been similarly 
blocking for over an hour. Notice that there is no Commit in this one:

http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
org.apache.lucene.index.SegmentTermEnum.next()
org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
org.apache.lucene.index.TermInfosReader.get(Term, boolean)
org.apache.lucene.index.TermInfosReader.get(Term)
org.apache.lucene.index.SegmentTermDocs.seek(Term)
org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
org.apache.lucene.index.IndexWriter.applyDeletes()
org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
 AddUpdateCommand)
org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
 SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
 ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
ServletResponse)
org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
 Object[])
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket, TcpConnection, 
Object[])
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[])
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
java.lang.Thread.run()


-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: Monday, October 05, 2009 1:18 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Timeouts

OK... next step is to verify that SolrCell doesn't have a bug that
causes it to commit.
I'll try and verify today unless someone else beats me to it.

-Yonik
http://www.lucidimagination.com

On Mon, Oct 5, 2009 at 1:04 PM, Giovanni Fernandez-Kincade
 wrote:
> I'm fairly certain that all of the indexing jobs are calling SOLR with 
> commit=false. They all construct the indexing URLs using a CLR function I 
> wrote, which takes in a Commit parameter, which is always set to false.
>
> Also, I don't see any calls to commit in the Tomcat logs (whereas normally 
> when I make a commit call I do).
>
> This suggests that Solr is doing it automatically, but the extract handler 
> doesn't seem to be the problem:
>   class="org.apache.solr.handler.extraction.ExtractingRequestHandler" 
> startup="lazy">
>
>  ignored_
>

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

I just grabbed another stack trace for a thread that has been similarly 
blocking for over an hour. Notice that there is no Commit in this one:

http-8080-Processor67 [RUNNABLE] CPU time: 1:02:05
org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
org.apache.lucene.index.SegmentTermEnum.next()
org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
org.apache.lucene.index.TermInfosReader.get(Term, boolean)
org.apache.lucene.index.TermInfosReader.get(Term)
org.apache.lucene.index.SegmentTermDocs.seek(Term)
org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
org.apache.lucene.index.IndexWriter.applyDeletes()
org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
org.apache.lucene.index.IndexWriter.updateDocument(Term, Document, Analyzer)
org.apache.lucene.index.IndexWriter.updateDocument(Term, Document)
org.apache.solr.update.DirectUpdateHandler2.addDoc(AddUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(AddUpdateCommand)
org.apache.solr.handler.extraction.ExtractingDocumentLoader.doAdd(SolrContentHandler,
 AddUpdateCommand)
org.apache.solr.handler.extraction.ExtractingDocumentLoader.addDoc(SolrContentHandler)
org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(SolrQueryRequest,
 SolrQueryResponse, ContentStream)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
 ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
ServletResponse)
org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
 Object[])
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket, TcpConnection, 
Object[])
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[])
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
java.lang.Thread.run()


-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: Monday, October 05, 2009 1:18 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Timeouts

OK... next step is to verify that SolrCell doesn't have a bug that
causes it to commit.
I'll try and verify today unless someone else beats me to it.

-Yonik
http://www.lucidimagination.com

On Mon, Oct 5, 2009 at 1:04 PM, Giovanni Fernandez-Kincade
 wrote:
> I'm fairly certain that all of the indexing jobs are calling SOLR with 
> commit=false. They all construct the indexing URLs using a CLR function I 
> wrote, which takes in a Commit parameter, which is always set to false.
>
> Also, I don't see any calls to commit in the Tomcat logs (whereas normally 
> when I make a commit call I do).
>
> This suggests that Solr is doing it automatically, but the extract handler 
> doesn't seem to be the problem:
>   class="org.apache.solr.handler.extraction.ExtractingRequestHandler" 
> startup="lazy">
>    
>      ignored_
>      fileData
>    
>  
>
>
> There is no external config file specified, and I don't see anything about 
> commits here.
>
> I've tried setting up more detailed indexer logging but haven't been able to 
> get it to work:
> true
>
> I tried relative and absolute paths, but no dice so far.
>
> Any other ideas?
>
> -Gio.
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Monday, October 05, 2009 12:52 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
>> This is what one of my SOL

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

Thanks for your help!

-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: Monday, October 05, 2009 1:18 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Timeouts

OK... next step is to verify that SolrCell doesn't have a bug that
causes it to commit.
I'll try and verify today unless someone else beats me to it.

-Yonik
http://www.lucidimagination.com

On Mon, Oct 5, 2009 at 1:04 PM, Giovanni Fernandez-Kincade
 wrote:
> I'm fairly certain that all of the indexing jobs are calling SOLR with 
> commit=false. They all construct the indexing URLs using a CLR function I 
> wrote, which takes in a Commit parameter, which is always set to false.
>
> Also, I don't see any calls to commit in the Tomcat logs (whereas normally 
> when I make a commit call I do).
>
> This suggests that Solr is doing it automatically, but the extract handler 
> doesn't seem to be the problem:
>   class="org.apache.solr.handler.extraction.ExtractingRequestHandler" 
> startup="lazy">
>    
>      ignored_
>      fileData
>    
>  
>
>
> There is no external config file specified, and I don't see anything about 
> commits here.
>
> I've tried setting up more detailed indexer logging but haven't been able to 
> get it to work:
> true
>
> I tried relative and absolute paths, but no dice so far.
>
> Any other ideas?
>
> -Gio.
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Monday, October 05, 2009 12:52 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
>> This is what one of my SOLR requests look like:
>>
>> http://titans:8080/solr/update/extract/?literal.versionId=684936&literal.filingDate=1997-12-04T00:00:00Z&literal.formTypeId=95&literal.companyId=3567904&literal.sourceId=0&resource.name=684936.txt&commit=false
>
> Have you verified that all of your indexing jobs (you said you had 4
> or 5) have commit=false?
>
> Also make sure that your extract handler doesn't have a default of
> something that could cause a commit - like commitWithin or something.
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
> On Mon, Oct 5, 2009 at 12:44 PM, Giovanni Fernandez-Kincade
>  wrote:
>> Is there somewhere other than solrConfig.xml that the autoCommit feature is 
>> enabled? I've looked through that file and found autocommit to be commented 
>> out:
>>
>>
>>
>> 
>>
>>
>>
>
>>
>>
>>
>> -Original Message-
>> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
>> Sent: Monday, October 05, 2009 12:40 PM
>> To: solr-user@lucene.apache.org
>> Subject: RE: Solr Timeouts
>>
>>
>>
>> Actually, ignore my other response.
>>
>>
>>
>> I believe you are committing, whether you know it or not.
>>
>>
>>
>> This is in your provided stack trace
>>
>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
>>  SolrParams, boolean) 
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>>  SolrQueryResponse)
>>
>>
>>
>> I think Yonik gave you additional information for how to make it faster.
>>
>>
>>
>> -Todd
>>
>>
>>
>> -Original Message-
>>
>> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
>>
>> Sent: Monday, October 05, 2009 9:30 AM
>>
>> To: solr-user@lucene.apache.org
>>
>> Subject: RE: Solr Timeouts
>>
>>
>>
>> I'm not committing at all actually - I'm waiting for all 6 million to be 
>> done.
>>
>>
>>
>> -Original Message-
>>
>> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
>>
>> Sent: Monday, October 05, 2009 12:10 PM
>>
>> To: solr-user@lucene.apache.org
>>
>> Subject: RE: Solr Timeouts
>>
>>
>>
>> How often are you committing?
>>
>>
>>
>> Every time you commit, Solr will close the old index and open the new one. 
>> If you are doing this in parallel from multiple jobs (4-5 you mention) then 
>> eventually the server gets behind and you start to pile up commit requests. 
>> Once this starts to happen, it will cascade out of control if the rate of 
>> commits isn't slowed.
>>
>>
>>
>> -Todd
>>
>>
>>
>> 
>>
>> From: Giovanni Fernandez-Kinc

Re: Solr Timeouts

[Yonik Seeley]

OK... next step is to verify that SolrCell doesn't have a bug that
causes it to commit.
I'll try and verify today unless someone else beats me to it.

-Yonik
http://www.lucidimagination.com

On Mon, Oct 5, 2009 at 1:04 PM, Giovanni Fernandez-Kincade
 wrote:
> I'm fairly certain that all of the indexing jobs are calling SOLR with 
> commit=false. They all construct the indexing URLs using a CLR function I 
> wrote, which takes in a Commit parameter, which is always set to false.
>
> Also, I don't see any calls to commit in the Tomcat logs (whereas normally 
> when I make a commit call I do).
>
> This suggests that Solr is doing it automatically, but the extract handler 
> doesn't seem to be the problem:
>   class="org.apache.solr.handler.extraction.ExtractingRequestHandler" 
> startup="lazy">
>    
>      ignored_
>      fileData
>    
>  
>
>
> There is no external config file specified, and I don't see anything about 
> commits here.
>
> I've tried setting up more detailed indexer logging but haven't been able to 
> get it to work:
> true
>
> I tried relative and absolute paths, but no dice so far.
>
> Any other ideas?
>
> -Gio.
>
> -Original Message-
> From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
> Sent: Monday, October 05, 2009 12:52 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Timeouts
>
>> This is what one of my SOLR requests look like:
>>
>> http://titans:8080/solr/update/extract/?literal.versionId=684936&literal.filingDate=1997-12-04T00:00:00Z&literal.formTypeId=95&literal.companyId=3567904&literal.sourceId=0&resource.name=684936.txt&commit=false
>
> Have you verified that all of your indexing jobs (you said you had 4
> or 5) have commit=false?
>
> Also make sure that your extract handler doesn't have a default of
> something that could cause a commit - like commitWithin or something.
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
> On Mon, Oct 5, 2009 at 12:44 PM, Giovanni Fernandez-Kincade
>  wrote:
>> Is there somewhere other than solrConfig.xml that the autoCommit feature is 
>> enabled? I've looked through that file and found autocommit to be commented 
>> out:
>>
>>
>>
>> 
>>
>>
>>
>
>>
>>
>>
>> -Original Message-
>> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
>> Sent: Monday, October 05, 2009 12:40 PM
>> To: solr-user@lucene.apache.org
>> Subject: RE: Solr Timeouts
>>
>>
>>
>> Actually, ignore my other response.
>>
>>
>>
>> I believe you are committing, whether you know it or not.
>>
>>
>>
>> This is in your provided stack trace
>>
>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
>>  SolrParams, boolean) 
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>>  SolrQueryResponse)
>>
>>
>>
>> I think Yonik gave you additional information for how to make it faster.
>>
>>
>>
>> -Todd
>>
>>
>>
>> -Original Message-
>>
>> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
>>
>> Sent: Monday, October 05, 2009 9:30 AM
>>
>> To: solr-user@lucene.apache.org
>>
>> Subject: RE: Solr Timeouts
>>
>>
>>
>> I'm not committing at all actually - I'm waiting for all 6 million to be 
>> done.
>>
>>
>>
>> -Original Message-
>>
>> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
>>
>> Sent: Monday, October 05, 2009 12:10 PM
>>
>> To: solr-user@lucene.apache.org
>>
>> Subject: RE: Solr Timeouts
>>
>>
>>
>> How often are you committing?
>>
>>
>>
>> Every time you commit, Solr will close the old index and open the new one. 
>> If you are doing this in parallel from multiple jobs (4-5 you mention) then 
>> eventually the server gets behind and you start to pile up commit requests. 
>> Once this starts to happen, it will cascade out of control if the rate of 
>> commits isn't slowed.
>>
>>
>>
>> -Todd
>>
>>
>>
>> 
>>
>> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
>>
>> Sent: Monday, October 05, 2009 9:04 AM
>>
>> To: solr-user@lucene.apache.org
>>
>> Subject: Solr Timeouts
>>
>>
>>
>>

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

I'm fairly certain that all of the indexing jobs are calling SOLR with 
commit=false. They all construct the indexing URLs using a CLR function I 
wrote, which takes in a Commit parameter, which is always set to false.

Also, I don't see any calls to commit in the Tomcat logs (whereas normally when 
I make a commit call I do). 

This suggests that Solr is doing it automatically, but the extract handler 
doesn't seem to be the problem:
  

  ignored_
  fileData

  


There is no external config file specified, and I don't see anything about 
commits here. 

I've tried setting up more detailed indexer logging but haven't been able to 
get it to work:
true

I tried relative and absolute paths, but no dice so far. 

Any other ideas?

-Gio.

-Original Message-
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: Monday, October 05, 2009 12:52 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Timeouts

> This is what one of my SOLR requests look like:
>
> http://titans:8080/solr/update/extract/?literal.versionId=684936&literal.filingDate=1997-12-04T00:00:00Z&literal.formTypeId=95&literal.companyId=3567904&literal.sourceId=0&resource.name=684936.txt&commit=false

Have you verified that all of your indexing jobs (you said you had 4
or 5) have commit=false?

Also make sure that your extract handler doesn't have a default of
something that could cause a commit - like commitWithin or something.

-Yonik
http://www.lucidimagination.com



On Mon, Oct 5, 2009 at 12:44 PM, Giovanni Fernandez-Kincade
 wrote:
> Is there somewhere other than solrConfig.xml that the autoCommit feature is 
> enabled? I've looked through that file and found autocommit to be commented 
> out:
>
>
>
> 
>
>
>

>
>
>
> -Original Message-
> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
> Sent: Monday, October 05, 2009 12:40 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Solr Timeouts
>
>
>
> Actually, ignore my other response.
>
>
>
> I believe you are committing, whether you know it or not.
>
>
>
> This is in your provided stack trace
>
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
>  SolrParams, boolean) 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>  SolrQueryResponse)
>
>
>
> I think Yonik gave you additional information for how to make it faster.
>
>
>
> -Todd
>
>
>
> -Original Message-
>
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
>
> Sent: Monday, October 05, 2009 9:30 AM
>
> To: solr-user@lucene.apache.org
>
> Subject: RE: Solr Timeouts
>
>
>
> I'm not committing at all actually - I'm waiting for all 6 million to be done.
>
>
>
> -Original Message-
>
> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
>
> Sent: Monday, October 05, 2009 12:10 PM
>
> To: solr-user@lucene.apache.org
>
> Subject: RE: Solr Timeouts
>
>
>
> How often are you committing?
>
>
>
> Every time you commit, Solr will close the old index and open the new one. If 
> you are doing this in parallel from multiple jobs (4-5 you mention) then 
> eventually the server gets behind and you start to pile up commit requests. 
> Once this starts to happen, it will cascade out of control if the rate of 
> commits isn't slowed.
>
>
>
> -Todd
>
>
>
> 
>
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
>
> Sent: Monday, October 05, 2009 9:04 AM
>
> To: solr-user@lucene.apache.org
>
> Subject: Solr Timeouts
>
>
>
> Hi,
>
> I'm attempting to index approximately 6 million HTML/Text files using SOLR 
> 1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. 
> I've fired up 4-5 different jobs that are making indexing requests using the 
> ExtractionRequestHandler, and everything works well for about 30-40 minutes, 
> after which all indexing requests start timing out. I profiled the server and 
> found that all of the threads are getting blocked by this call to flush the 
> Lucene index to disk (see below).
>
>
>
> This leads me to a few questions:
>
>
>
> 1.       Is this normal?
>
>
>
> 2.       Can I reduce the frequency with which this happens somehow? I've 
> greatly increased the indexing options in SolrConfig.xml (attached here) to 
> no avail.
>
>
>
> 3.       During these flushes, resource utilization (CPU, I/O, Memory 
> Consumption) is significantly down compared to when requests are being 
> handled

RE: Solr Timeouts

[Walter Underwood]

How long is your timeout? Maybe it should be longer, since this is normal
Solr behavior. --wunder

-Original Message-
From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com] 
Sent: Monday, October 05, 2009 9:45 AM
To: solr-user@lucene.apache.org
Subject: RE: Solr Timeouts

Is there somewhere other than solrConfig.xml that the autoCommit feature is
enabled? I've looked through that file and found autocommit to be commented
out:







This is what one of my SOLR requests look like:



http://titans:8080/solr/update/extract/?literal.versionId=684936&literal.fil
ingDate=1997-12-04T00:00:00Z&literal.formTypeId=95&literal.companyId=3567904
&literal.sourceId=0&resource.name=684936.txt&commit=false



-Original Message-
From: Feak, Todd [mailto:todd.f...@smss.sony.com]
Sent: Monday, October 05, 2009 12:40 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr Timeouts



Actually, ignore my other response.



I believe you are committing, whether you know it or not.



This is in your provided stack trace

org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProces
sor, SolrParams, boolean)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQuery
Request, SolrQueryResponse)



I think Yonik gave you additional information for how to make it faster.



-Todd



-Original Message-

From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]

Sent: Monday, October 05, 2009 9:30 AM

To: solr-user@lucene.apache.org

Subject: RE: Solr Timeouts



I'm not committing at all actually - I'm waiting for all 6 million to be
done.



-Original Message-

From: Feak, Todd [mailto:todd.f...@smss.sony.com]

Sent: Monday, October 05, 2009 12:10 PM

To: solr-user@lucene.apache.org

Subject: RE: Solr Timeouts



How often are you committing?



Every time you commit, Solr will close the old index and open the new one.
If you are doing this in parallel from multiple jobs (4-5 you mention) then
eventually the server gets behind and you start to pile up commit requests.
Once this starts to happen, it will cascade out of control if the rate of
commits isn't slowed.



-Todd





From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]

Sent: Monday, October 05, 2009 9:04 AM

To: solr-user@lucene.apache.org

Subject: Solr Timeouts



Hi,

I'm attempting to index approximately 6 million HTML/Text files using SOLR
1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM.
I've fired up 4-5 different jobs that are making indexing requests using the
ExtractionRequestHandler, and everything works well for about 30-40 minutes,
after which all indexing requests start timing out. I profiled the server
and found that all of the threads are getting blocked by this call to flush
the Lucene index to disk (see below).



This leads me to a few questions:



1.   Is this normal?



2.   Can I reduce the frequency with which this happens somehow? I've
greatly increased the indexing options in SolrConfig.xml (attached here) to
no avail.



3.   During these flushes, resource utilization (CPU, I/O, Memory
Consumption) is significantly down compared to when requests are being
handled. Is there any way to make this index go faster? I have plenty of
bandwidth on the machine.



I appreciate any insight you can provide. We're currently using MS SQL 2005
as our full-text solution and are pretty much miserable. So far SOLR has
been a great experience.



Thanks,

Gio.



http-8080-Processor21 [RUNNABLE] CPU time: 9:51

java.io.RandomAccessFile.seek(long)

org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(by
te[], int, int)

org.apache.lucene.store.BufferedIndexInput.refill()

org.apache.lucene.store.BufferedIndexInput.readByte()

org.apache.lucene.store.IndexInput.readVInt()

org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)

org.apache.lucene.index.SegmentTermEnum.next()

org.apache.lucene.index.SegmentTermEnum.scanTo(Term)

org.apache.lucene.index.TermInfosReader.get(Term, boolean)

org.apache.lucene.index.TermInfosReader.get(Term)

org.apache.lucene.index.SegmentTermDocs.seek(Term)

org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)

org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)

org.apache.lucene.index.IndexWriter.applyDeletes()

org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)

org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)

org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)

org.apache.lucene.index.IndexWriter.closeInternal(boolean)

org.apache.lucene.index.IndexWriter.close(boolean)

org.apache.lucene.index.IndexWriter.close()

org.apache.solr.update.SolrIndexWriter.close()

org.apache.solr.update.DirectUpdateHandler2.closeWriter()

o

Re: Solr Timeouts

[Yonik Seeley]

> This is what one of my SOLR requests look like:
>
> http://titans:8080/solr/update/extract/?literal.versionId=684936&literal.filingDate=1997-12-04T00:00:00Z&literal.formTypeId=95&literal.companyId=3567904&literal.sourceId=0&resource.name=684936.txt&commit=false

Have you verified that all of your indexing jobs (you said you had 4
or 5) have commit=false?

Also make sure that your extract handler doesn't have a default of
something that could cause a commit - like commitWithin or something.

-Yonik
http://www.lucidimagination.com



On Mon, Oct 5, 2009 at 12:44 PM, Giovanni Fernandez-Kincade
 wrote:
> Is there somewhere other than solrConfig.xml that the autoCommit feature is 
> enabled? I've looked through that file and found autocommit to be commented 
> out:
>
>
>
> 
>
>
>

>
>
>
> -Original Message-
> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
> Sent: Monday, October 05, 2009 12:40 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Solr Timeouts
>
>
>
> Actually, ignore my other response.
>
>
>
> I believe you are committing, whether you know it or not.
>
>
>
> This is in your provided stack trace
>
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
>  SolrParams, boolean) 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>  SolrQueryResponse)
>
>
>
> I think Yonik gave you additional information for how to make it faster.
>
>
>
> -Todd
>
>
>
> -Original Message-----
>
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
>
> Sent: Monday, October 05, 2009 9:30 AM
>
> To: solr-user@lucene.apache.org
>
> Subject: RE: Solr Timeouts
>
>
>
> I'm not committing at all actually - I'm waiting for all 6 million to be done.
>
>
>
> -Original Message-
>
> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
>
> Sent: Monday, October 05, 2009 12:10 PM
>
> To: solr-user@lucene.apache.org
>
> Subject: RE: Solr Timeouts
>
>
>
> How often are you committing?
>
>
>
> Every time you commit, Solr will close the old index and open the new one. If 
> you are doing this in parallel from multiple jobs (4-5 you mention) then 
> eventually the server gets behind and you start to pile up commit requests. 
> Once this starts to happen, it will cascade out of control if the rate of 
> commits isn't slowed.
>
>
>
> -Todd
>
>
>
> 
>
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
>
> Sent: Monday, October 05, 2009 9:04 AM
>
> To: solr-user@lucene.apache.org
>
> Subject: Solr Timeouts
>
>
>
> Hi,
>
> I'm attempting to index approximately 6 million HTML/Text files using SOLR 
> 1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. 
> I've fired up 4-5 different jobs that are making indexing requests using the 
> ExtractionRequestHandler, and everything works well for about 30-40 minutes, 
> after which all indexing requests start timing out. I profiled the server and 
> found that all of the threads are getting blocked by this call to flush the 
> Lucene index to disk (see below).
>
>
>
> This leads me to a few questions:
>
>
>
> 1.       Is this normal?
>
>
>
> 2.       Can I reduce the frequency with which this happens somehow? I've 
> greatly increased the indexing options in SolrConfig.xml (attached here) to 
> no avail.
>
>
>
> 3.       During these flushes, resource utilization (CPU, I/O, Memory 
> Consumption) is significantly down compared to when requests are being 
> handled. Is there any way to make this index go faster? I have plenty of 
> bandwidth on the machine.
>
>
>
> I appreciate any insight you can provide. We're currently using MS SQL 2005 
> as our full-text solution and are pretty much miserable. So far SOLR has been 
> a great experience.
>
>
>
> Thanks,
>
> Gio.
>
>
>
> http-8080-Processor21 [RUNNABLE] CPU time: 9:51
>
> java.io.RandomAccessFile.seek(long)
>
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
>  int, int)
>
> org.apache.lucene.store.BufferedIndexInput.refill()
>
> org.apache.lucene.store.BufferedIndexInput.readByte()
>
> org.apache.lucene.store.IndexInput.readVInt()
>
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
>
> org.apache.lucene.index.SegmentTermEnum.next()
>
> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
>
&

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

Is there somewhere other than solrConfig.xml that the autoCommit feature is 
enabled? I've looked through that file and found autocommit to be commented out:







This is what one of my SOLR requests look like:



http://titans:8080/solr/update/extract/?literal.versionId=684936&literal.filingDate=1997-12-04T00:00:00Z&literal.formTypeId=95&literal.companyId=3567904&literal.sourceId=0&resource.name=684936.txt&commit=false



-Original Message-
From: Feak, Todd [mailto:todd.f...@smss.sony.com]
Sent: Monday, October 05, 2009 12:40 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr Timeouts



Actually, ignore my other response.



I believe you are committing, whether you know it or not.



This is in your provided stack trace

org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
 SolrParams, boolean) 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
 SolrQueryResponse)



I think Yonik gave you additional information for how to make it faster.



-Todd



-Original Message-

From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]

Sent: Monday, October 05, 2009 9:30 AM

To: solr-user@lucene.apache.org

Subject: RE: Solr Timeouts



I'm not committing at all actually - I'm waiting for all 6 million to be done.



-Original Message-

From: Feak, Todd [mailto:todd.f...@smss.sony.com]

Sent: Monday, October 05, 2009 12:10 PM

To: solr-user@lucene.apache.org

Subject: RE: Solr Timeouts



How often are you committing?



Every time you commit, Solr will close the old index and open the new one. If 
you are doing this in parallel from multiple jobs (4-5 you mention) then 
eventually the server gets behind and you start to pile up commit requests. 
Once this starts to happen, it will cascade out of control if the rate of 
commits isn't slowed.



-Todd





From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]

Sent: Monday, October 05, 2009 9:04 AM

To: solr-user@lucene.apache.org

Subject: Solr Timeouts



Hi,

I'm attempting to index approximately 6 million HTML/Text files using SOLR 
1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. I've 
fired up 4-5 different jobs that are making indexing requests using the 
ExtractionRequestHandler, and everything works well for about 30-40 minutes, 
after which all indexing requests start timing out. I profiled the server and 
found that all of the threads are getting blocked by this call to flush the 
Lucene index to disk (see below).



This leads me to a few questions:



1.   Is this normal?



2.   Can I reduce the frequency with which this happens somehow? I've 
greatly increased the indexing options in SolrConfig.xml (attached here) to no 
avail.



3.   During these flushes, resource utilization (CPU, I/O, Memory 
Consumption) is significantly down compared to when requests are being handled. 
Is there any way to make this index go faster? I have plenty of bandwidth on 
the machine.



I appreciate any insight you can provide. We're currently using MS SQL 2005 as 
our full-text solution and are pretty much miserable. So far SOLR has been a 
great experience.



Thanks,

Gio.



http-8080-Processor21 [RUNNABLE] CPU time: 9:51

java.io.RandomAccessFile.seek(long)

org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
 int, int)

org.apache.lucene.store.BufferedIndexInput.refill()

org.apache.lucene.store.BufferedIndexInput.readByte()

org.apache.lucene.store.IndexInput.readVInt()

org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)

org.apache.lucene.index.SegmentTermEnum.next()

org.apache.lucene.index.SegmentTermEnum.scanTo(Term)

org.apache.lucene.index.TermInfosReader.get(Term, boolean)

org.apache.lucene.index.TermInfosReader.get(Term)

org.apache.lucene.index.SegmentTermDocs.seek(Term)

org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)

org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)

org.apache.lucene.index.IndexWriter.applyDeletes()

org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)

org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)

org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)

org.apache.lucene.index.IndexWriter.closeInternal(boolean)

org.apache.lucene.index.IndexWriter.close(boolean)

org.apache.lucene.index.IndexWriter.close()

org.apache.solr.update.SolrIndexWriter.close()

org.apache.solr.update.DirectUpdateHandler2.closeWriter()

org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)

org.apache.solr.update.processor.RunUpdateProcessor.processCommit(CommitUpdateCommand)

org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
 SolrParams, boolean)

org.apache.solr.handler.ContentStr

RE: Solr Timeouts

[Feak, Todd]

Actually, ignore my other response. 

I believe you are committing, whether you know it or not. 

This is in your provided stack trace
org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
 SolrParams, boolean) 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
 SolrQueryResponse)

I think Yonik gave you additional information for how to make it faster.

-Todd

-Original Message-
From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com] 
Sent: Monday, October 05, 2009 9:30 AM
To: solr-user@lucene.apache.org
Subject: RE: Solr Timeouts

I'm not committing at all actually - I'm waiting for all 6 million to be done. 

-Original Message-
From: Feak, Todd [mailto:todd.f...@smss.sony.com] 
Sent: Monday, October 05, 2009 12:10 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr Timeouts

How often are you committing?

Every time you commit, Solr will close the old index and open the new one. If 
you are doing this in parallel from multiple jobs (4-5 you mention) then 
eventually the server gets behind and you start to pile up commit requests. 
Once this starts to happen, it will cascade out of control if the rate of 
commits isn't slowed.

-Todd


From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
Sent: Monday, October 05, 2009 9:04 AM
To: solr-user@lucene.apache.org
Subject: Solr Timeouts

Hi,
I'm attempting to index approximately 6 million HTML/Text files using SOLR 
1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. I've 
fired up 4-5 different jobs that are making indexing requests using the 
ExtractionRequestHandler, and everything works well for about 30-40 minutes, 
after which all indexing requests start timing out. I profiled the server and 
found that all of the threads are getting blocked by this call to flush the 
Lucene index to disk (see below).

This leads me to a few questions:

1.   Is this normal?

2.   Can I reduce the frequency with which this happens somehow? I've 
greatly increased the indexing options in SolrConfig.xml (attached here) to no 
avail.

3.   During these flushes, resource utilization (CPU, I/O, Memory 
Consumption) is significantly down compared to when requests are being handled. 
Is there any way to make this index go faster? I have plenty of bandwidth on 
the machine.

I appreciate any insight you can provide. We're currently using MS SQL 2005 as 
our full-text solution and are pretty much miserable. So far SOLR has been a 
great experience.

Thanks,
Gio.

http-8080-Processor21 [RUNNABLE] CPU time: 9:51
java.io.RandomAccessFile.seek(long)
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
 int, int)
org.apache.lucene.store.BufferedIndexInput.refill()
org.apache.lucene.store.BufferedIndexInput.readByte()
org.apache.lucene.store.IndexInput.readVInt()
org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
org.apache.lucene.index.SegmentTermEnum.next()
org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
org.apache.lucene.index.TermInfosReader.get(Term, boolean)
org.apache.lucene.index.TermInfosReader.get(Term)
org.apache.lucene.index.SegmentTermDocs.seek(Term)
org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
org.apache.lucene.index.IndexWriter.applyDeletes()
org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
org.apache.lucene.index.IndexWriter.closeInternal(boolean)
org.apache.lucene.index.IndexWriter.close(boolean)
org.apache.lucene.index.IndexWriter.close()
org.apache.solr.update.SolrIndexWriter.close()
org.apache.solr.update.DirectUpdateHandler2.closeWriter()
org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(CommitUpdateCommand)
org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
 SolrParams, boolean)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
 ServletResponse)
org.apache.catalina.core.Applic

RE: Solr Timeouts

[Feak, Todd]

Ok. Guess that isn't a problem. :)

A second consideration... I could see lock contention being an issue with 
multiple clients indexing at once. Is there any disadvantage to serializing the 
clients to remove lock contention?

-Todd

-Original Message-
From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com] 
Sent: Monday, October 05, 2009 9:30 AM
To: solr-user@lucene.apache.org
Subject: RE: Solr Timeouts

I'm not committing at all actually - I'm waiting for all 6 million to be done. 

-Original Message-
From: Feak, Todd [mailto:todd.f...@smss.sony.com] 
Sent: Monday, October 05, 2009 12:10 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr Timeouts

How often are you committing?

Every time you commit, Solr will close the old index and open the new one. If 
you are doing this in parallel from multiple jobs (4-5 you mention) then 
eventually the server gets behind and you start to pile up commit requests. 
Once this starts to happen, it will cascade out of control if the rate of 
commits isn't slowed.

-Todd


From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
Sent: Monday, October 05, 2009 9:04 AM
To: solr-user@lucene.apache.org
Subject: Solr Timeouts

Hi,
I'm attempting to index approximately 6 million HTML/Text files using SOLR 
1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. I've 
fired up 4-5 different jobs that are making indexing requests using the 
ExtractionRequestHandler, and everything works well for about 30-40 minutes, 
after which all indexing requests start timing out. I profiled the server and 
found that all of the threads are getting blocked by this call to flush the 
Lucene index to disk (see below).

This leads me to a few questions:

1.   Is this normal?

2.   Can I reduce the frequency with which this happens somehow? I've 
greatly increased the indexing options in SolrConfig.xml (attached here) to no 
avail.

3.   During these flushes, resource utilization (CPU, I/O, Memory 
Consumption) is significantly down compared to when requests are being handled. 
Is there any way to make this index go faster? I have plenty of bandwidth on 
the machine.

I appreciate any insight you can provide. We're currently using MS SQL 2005 as 
our full-text solution and are pretty much miserable. So far SOLR has been a 
great experience.

Thanks,
Gio.

http-8080-Processor21 [RUNNABLE] CPU time: 9:51
java.io.RandomAccessFile.seek(long)
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
 int, int)
org.apache.lucene.store.BufferedIndexInput.refill()
org.apache.lucene.store.BufferedIndexInput.readByte()
org.apache.lucene.store.IndexInput.readVInt()
org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
org.apache.lucene.index.SegmentTermEnum.next()
org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
org.apache.lucene.index.TermInfosReader.get(Term, boolean)
org.apache.lucene.index.TermInfosReader.get(Term)
org.apache.lucene.index.SegmentTermDocs.seek(Term)
org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
org.apache.lucene.index.IndexWriter.applyDeletes()
org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
org.apache.lucene.index.IndexWriter.closeInternal(boolean)
org.apache.lucene.index.IndexWriter.close(boolean)
org.apache.lucene.index.IndexWriter.close()
org.apache.solr.update.SolrIndexWriter.close()
org.apache.solr.update.DirectUpdateHandler2.closeWriter()
org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(CommitUpdateCommand)
org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
 SolrParams, boolean)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
 ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
ServletResponse)
org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
org.apache.catalina.core.

Re: Solr Timeouts

[Yonik Seeley]

On Mon, Oct 5, 2009 at 12:30 PM, Giovanni Fernandez-Kincade
 wrote:
> I'm not committing at all actually - I'm waiting for all 6 million to be done.

You either have solr auto commit set up, or a client is issuing a commit.

-Yonik
http://www.lucidimagination.com



> -Original Message-
> From: Feak, Todd [mailto:todd.f...@smss.sony.com]
> Sent: Monday, October 05, 2009 12:10 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Solr Timeouts
>
> How often are you committing?
>
> Every time you commit, Solr will close the old index and open the new one. If 
> you are doing this in parallel from multiple jobs (4-5 you mention) then 
> eventually the server gets behind and you start to pile up commit requests. 
> Once this starts to happen, it will cascade out of control if the rate of 
> commits isn't slowed.
>
> -Todd
>
> 
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
> Sent: Monday, October 05, 2009 9:04 AM
> To: solr-user@lucene.apache.org
> Subject: Solr Timeouts
>
> Hi,
> I'm attempting to index approximately 6 million HTML/Text files using SOLR 
> 1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. 
> I've fired up 4-5 different jobs that are making indexing requests using the 
> ExtractionRequestHandler, and everything works well for about 30-40 minutes, 
> after which all indexing requests start timing out. I profiled the server and 
> found that all of the threads are getting blocked by this call to flush the 
> Lucene index to disk (see below).
>
> This leads me to a few questions:
>
> 1.       Is this normal?
>
> 2.       Can I reduce the frequency with which this happens somehow? I've 
> greatly increased the indexing options in SolrConfig.xml (attached here) to 
> no avail.
>
> 3.       During these flushes, resource utilization (CPU, I/O, Memory 
> Consumption) is significantly down compared to when requests are being 
> handled. Is there any way to make this index go faster? I have plenty of 
> bandwidth on the machine.
>
> I appreciate any insight you can provide. We're currently using MS SQL 2005 
> as our full-text solution and are pretty much miserable. So far SOLR has been 
> a great experience.
>
> Thanks,
> Gio.
>
> http-8080-Processor21 [RUNNABLE] CPU time: 9:51
> java.io.RandomAccessFile.seek(long)
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
>  int, int)
> org.apache.lucene.store.BufferedIndexInput.refill()
> org.apache.lucene.store.BufferedIndexInput.readByte()
> org.apache.lucene.store.IndexInput.readVInt()
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
> org.apache.lucene.index.TermInfosReader.get(Term)
> org.apache.lucene.index.SegmentTermDocs.seek(Term)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
> org.apache.lucene.index.IndexWriter.applyDeletes()
> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
> org.apache.lucene.index.IndexWriter.closeInternal(boolean)
> org.apache.lucene.index.IndexWriter.close(boolean)
> org.apache.lucene.index.IndexWriter.close()
> org.apache.solr.update.SolrIndexWriter.close()
> org.apache.solr.update.DirectUpdateHandler2.closeWriter()
> org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(CommitUpdateCommand)
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
>  SolrParams, boolean)
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
>  SolrQueryResponse)
> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
> SolrQueryResponse)
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
>  SolrQueryResponse)
> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
> SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
> ServletResponse, FilterChain)
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
>  Servlet

RE: Solr Timeouts

[Giovanni Fernandez-Kincade]

I'm not committing at all actually - I'm waiting for all 6 million to be done. 

-Original Message-
From: Feak, Todd [mailto:todd.f...@smss.sony.com] 
Sent: Monday, October 05, 2009 12:10 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr Timeouts

How often are you committing?

Every time you commit, Solr will close the old index and open the new one. If 
you are doing this in parallel from multiple jobs (4-5 you mention) then 
eventually the server gets behind and you start to pile up commit requests. 
Once this starts to happen, it will cascade out of control if the rate of 
commits isn't slowed.

-Todd


From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
Sent: Monday, October 05, 2009 9:04 AM
To: solr-user@lucene.apache.org
Subject: Solr Timeouts

Hi,
I'm attempting to index approximately 6 million HTML/Text files using SOLR 
1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. I've 
fired up 4-5 different jobs that are making indexing requests using the 
ExtractionRequestHandler, and everything works well for about 30-40 minutes, 
after which all indexing requests start timing out. I profiled the server and 
found that all of the threads are getting blocked by this call to flush the 
Lucene index to disk (see below).

This leads me to a few questions:

1.   Is this normal?

2.   Can I reduce the frequency with which this happens somehow? I've 
greatly increased the indexing options in SolrConfig.xml (attached here) to no 
avail.

3.   During these flushes, resource utilization (CPU, I/O, Memory 
Consumption) is significantly down compared to when requests are being handled. 
Is there any way to make this index go faster? I have plenty of bandwidth on 
the machine.

I appreciate any insight you can provide. We're currently using MS SQL 2005 as 
our full-text solution and are pretty much miserable. So far SOLR has been a 
great experience.

Thanks,
Gio.

http-8080-Processor21 [RUNNABLE] CPU time: 9:51
java.io.RandomAccessFile.seek(long)
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
 int, int)
org.apache.lucene.store.BufferedIndexInput.refill()
org.apache.lucene.store.BufferedIndexInput.readByte()
org.apache.lucene.store.IndexInput.readVInt()
org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
org.apache.lucene.index.SegmentTermEnum.next()
org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
org.apache.lucene.index.TermInfosReader.get(Term, boolean)
org.apache.lucene.index.TermInfosReader.get(Term)
org.apache.lucene.index.SegmentTermDocs.seek(Term)
org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
org.apache.lucene.index.IndexWriter.applyDeletes()
org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
org.apache.lucene.index.IndexWriter.closeInternal(boolean)
org.apache.lucene.index.IndexWriter.close(boolean)
org.apache.lucene.index.IndexWriter.close()
org.apache.solr.update.SolrIndexWriter.close()
org.apache.solr.update.DirectUpdateHandler2.closeWriter()
org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(CommitUpdateCommand)
org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
 SolrParams, boolean)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
 ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
ServletResponse)
org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.

Re: Solr Timeouts

[Yonik Seeley]

On Mon, Oct 5, 2009 at 12:03 PM, Giovanni Fernandez-Kincade
 wrote:
> Hi,
>
> I’m attempting to index approximately 6 million HTML/Text files using SOLR
> 1.4/Tomcat6 on Windows Server 2003 x64. I’m running 64 bit Tomcat and JVM.
> I’ve fired up 4-5 different jobs that are making indexing requests using the
> ExtractionRequestHandler, and everything works well for about 30-40 minutes,
> after which all indexing requests start timing out. I profiled the server
> and found that all of the threads are getting blocked by this call to flush
> the Lucene index to disk (see below).
>
>
>
> This leads me to a few questions:
>
> 1. Is this normal?

Yes... one can't currently add documents when the first part of a
commit is going on (closing the IndexWriter).  The threads will
normally block and then resume after the writer has been successfully
closed.  This is normally fine and you can work around it by
increasing the servlet container timeout.

Due to advances in Lucene, this restriction will probably be lifted in
the next version of Solr (1.5)

> 2. Can I reduce the frequency with which this happens somehow? I’ve
> greatly increased the indexing options in SolrConfig.xml (attached here) to
> no avail.

It looks like Solr is committing because you told it to?

> 3. During these flushes, resource utilization (CPU, I/O, Memory
> Consumption) is significantly down compared to when requests are being
> handled. Is there any way to make this index go faster? I have plenty of
> bandwidth on the machine.

Don't commit until you're done a big indexing run?
If you're using SolrJ, use the StreamingUpdateSolrServer it's much faster!

-Yonik
http://www.lucidimagination.com


> I appreciate any insight you can provide. We’re currently using MS SQL 2005
> as our full-text solution and are pretty much miserable. So far SOLR has
> been a great experience.
>
>
>
> Thanks,
>
> Gio.
>
>
>
> http-8080-Processor21 [RUNNABLE] CPU time: 9:51
>
> java.io.RandomAccessFile.seek(long)
>
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
> int, int)
>
> org.apache.lucene.store.BufferedIndexInput.refill()
>
> org.apache.lucene.store.BufferedIndexInput.readByte()
>
> org.apache.lucene.store.IndexInput.readVInt()
>
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
>
> org.apache.lucene.index.SegmentTermEnum.next()
>
> org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
>
> org.apache.lucene.index.TermInfosReader.get(Term, boolean)
>
> org.apache.lucene.index.TermInfosReader.get(Term)
>
> org.apache.lucene.index.SegmentTermDocs.seek(Term)
>
> org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
>
> org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
>
> org.apache.lucene.index.IndexWriter.applyDeletes()
>
> org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
>
> org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
>
> org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
>
> org.apache.lucene.index.IndexWriter.closeInternal(boolean)
>
> org.apache.lucene.index.IndexWriter.close(boolean)
>
> org.apache.lucene.index.IndexWriter.close()
>
> org.apache.solr.update.SolrIndexWriter.close()
>
> org.apache.solr.update.DirectUpdateHandler2.closeWriter()
>
> org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
>
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(CommitUpdateCommand)
>
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
> SolrParams, boolean)
>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
> SolrQueryResponse)
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest,
> SolrQueryResponse)
>
> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
> SolrQueryResponse)
>
> org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest,
> SolrQueryResponse)
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest,
> SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest,
> ServletResponse, FilterChain)
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
> ServletResponse)
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
> ServletResponse)
>
> org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
>
> org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
>
> org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
>
> org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
>
> org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
>
> org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
>
> org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
>
> org.apache.coyote

RE: Solr Timeouts

[Feak, Todd]

How often are you committing?

Every time you commit, Solr will close the old index and open the new one. If 
you are doing this in parallel from multiple jobs (4-5 you mention) then 
eventually the server gets behind and you start to pile up commit requests. 
Once this starts to happen, it will cascade out of control if the rate of 
commits isn't slowed.

-Todd


From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
Sent: Monday, October 05, 2009 9:04 AM
To: solr-user@lucene.apache.org
Subject: Solr Timeouts

Hi,
I'm attempting to index approximately 6 million HTML/Text files using SOLR 
1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. I've 
fired up 4-5 different jobs that are making indexing requests using the 
ExtractionRequestHandler, and everything works well for about 30-40 minutes, 
after which all indexing requests start timing out. I profiled the server and 
found that all of the threads are getting blocked by this call to flush the 
Lucene index to disk (see below).

This leads me to a few questions:

1.   Is this normal?

2.   Can I reduce the frequency with which this happens somehow? I've 
greatly increased the indexing options in SolrConfig.xml (attached here) to no 
avail.

3.   During these flushes, resource utilization (CPU, I/O, Memory 
Consumption) is significantly down compared to when requests are being handled. 
Is there any way to make this index go faster? I have plenty of bandwidth on 
the machine.

I appreciate any insight you can provide. We're currently using MS SQL 2005 as 
our full-text solution and are pretty much miserable. So far SOLR has been a 
great experience.

Thanks,
Gio.

http-8080-Processor21 [RUNNABLE] CPU time: 9:51
java.io.RandomAccessFile.seek(long)
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
 int, int)
org.apache.lucene.store.BufferedIndexInput.refill()
org.apache.lucene.store.BufferedIndexInput.readByte()
org.apache.lucene.store.IndexInput.readVInt()
org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
org.apache.lucene.index.SegmentTermEnum.next()
org.apache.lucene.index.SegmentTermEnum.scanTo(Term)
org.apache.lucene.index.TermInfosReader.get(Term, boolean)
org.apache.lucene.index.TermInfosReader.get(Term)
org.apache.lucene.index.SegmentTermDocs.seek(Term)
org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int)
org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos)
org.apache.lucene.index.IndexWriter.applyDeletes()
org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean)
org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean)
org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean)
org.apache.lucene.index.IndexWriter.closeInternal(boolean)
org.apache.lucene.index.IndexWriter.close(boolean)
org.apache.lucene.index.IndexWriter.close()
org.apache.solr.update.SolrIndexWriter.close()
org.apache.solr.update.DirectUpdateHandler2.closeWriter()
org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand)
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(CommitUpdateCommand)
org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProcessor,
 SolrParams, boolean)
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(SolrQueryRequest,
 SolrQueryResponse)
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, 
SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, 
SolrRequestHandler, SolrQueryRequest, SolrQueryResponse)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, 
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
 ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, 
ServletResponse)
org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream)
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(TcpConnection,
 Object[])
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket, TcpConnection, 
Object[])
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[])
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
java.lang.Thread.run()

Re: timeouts

[James liu]

*Collins:

*i don't know what u wanna say?

-- 
regards
j.L ( I live in Shanghai, China)

More Robust Search Timeouts (to Kill Zombie Queries)?

[Chris Harris]

I've noticed that some of my queries take so long (5 min+) that by the
time they return, there is no longer any plausible use for the search
results. I've started calling these zombie queries because, well, they
should be dead, but they just won't die. Instead, they stick around,
wasting my Solr box's CPU, RAM, and I/O resources, and potentially
causing more legitimate queries to stack up. (Regarding "stacking up",
see SOLR-138.)

I may be able to prevent some of this by optimizing my index settings
and by disallowing certain things, such as prefix wildcard queries
(e.g. *ar). However, right now I'm most interested in figuring out how
to get more robust server-side search timeouts in Solr. This would
seem to provide a good balance between these goals:

1) I would like to allow users to attempt to run potentially expensive
queries, such as queries with lots of wildcards or ranges
2) I would like to make sure that potentially expensive queries don't
turn into zombies -- especially long-lasting zombies

For example, I think some of my users might be willing to wait a
minute or two for certain classes of search to complete. But after
that point, I'd really like to say enough is enough.

[Background]

While my load is pretty low (it's not a public-facing site), some of
my queries are monsters that can take, say, over 5 minutes. (I don't
know how much longer than 5 minutes they might take. Some of them
might take hours, for all I know, if allowed to run to completion!)

The biggest cuprit queries currently seem to be wildcard queries. This
is made worse by how I've allowed prefix wildcard searches on an index
with a large # of terms. (This is made worse yet by doing word bigram
indexing.)

I've implemented the "timeAllowed" search time out support feature
introduced in SOLR-502, and this does catch some searches that would
have become zombies. (Some proximity searches, for example.) But the
timeAllowed mechanism does not catch everything. And, as I understand
it, it's powerless to do anything about, say, wildcard expansions that
are taking forever.

The question is how to proceed.

[Option 1: Wait for someone to bring timeAllowed support to more parts
of Solr search]

This might be nice. I sort of assume it will happen eventually. I kind
of want a more immediate solution, though. Any thoughts on how hard it
would be to add the timeout to, say, wildcard expansion? I haven't
figured out if I know enough about Solr yet to work on this myself.

[Option 2: Add gross timeout support to StandardRequestHandler?]

What if I modified StandardRequestHandler so that, when it was invoked,
the following would happen:

* spawn new thread t to do the stuff that StandardRequestHandlerStuff
would normally do
* start thread t
* sleep, waiting for either the thread t to finish or for a timer go off
* after waking up, look whether the timer went off. if so, then
terminate thread t

This would kill any runaway zombie queries. But maybe it would also
have horrible side-effects. Is it wishful thinking to believe that
this might not screw up referencing counting, or create deadlocks, or
anything else?

[Option 3: Servlet container-level Solutions?]

I thought Jetty and friends would might have an option along the lines of
"if a request is taking longer than x seconds, then abort the thread
handling it". This seems troublesome in practice, though:

1) I can't find a servlet container with documentation clearly stating
that this is possible.
2) I played with Jetty, and maxIdleTime sounded like it *might* cause
this behavior, but experiments suggest otherwise.
3) This behavior sounds dangerous, especially unless you can convince
the servlet container to only abort index-reading threads, while
leaving index-writing threads alone.

Thanks for any advice,
Chris

RE: timeouts

[Chris Hostetter]

: Tomcat is using about 98mb memory, mysql is about 500mb. Tomcat
: completely freezes up - can't do anything other than restart the
: service.

a thread dump from the jvm running tomcat would probably be helpful in 
figuring out what's going on

: timing out well before getting to the commit. As it's a public search
: people will still be hitting the search whilst this is going on, but
: it's not massive search traffic. A few thousand requests spread over a
: full day.

i'm confused ... are these people executing searches still getting 
results?  ... that seems at odds with your "completley freezes up" 
statement.


-Hoss

Re: timeouts

[Noble Paul നോബിള്‍ नोब्ळ्]

I may be a bit off the mark. It seems that DataImportHandler may be
able to do this very easily for you.
http://wiki.apache.org/solr/DataImportHandler#jdbcdatasource


On Fri, Oct 24, 2008 at 6:28 PM, Simon Collins
<[EMAIL PROTECTED]> wrote:
> Hi
>
>
>
> We're running solr on a win 2k3 box under tomcat with about 100,000 records.
> When doing large updates of records via solr sharp, solr completely freezes
> and doesn't come back until we restart tomcat.
>
>
>
> This has only started happening since putting mysql on the same box (as a
> source of the data to update from).
>
>
>
> Are there any known issues with running solr and mysql on the same box? When
> it's frozen, the cpu usage is around 1-2% not exactly out of resources!
>
>
>
> Am i best using something else instead of tomcat? We're still trialling solr
> (presently, used for our main site search www.shoe-shop.com and search and
> navigation for our microsites ). It's an excellent search product, but I
> don't want to fork out on new hardware for it just yet – until i know how
> more about the performance and which environment i'm best to go for
> (win/linux).
>
>
>
> If anyone has any suggestions/needs more info, i'd be extremely grateful.
>
>
>
> Thanks
> Simon
>
> 
> Simon Collins
> Systems Analyst
>
> Telephone: 01904 606 867
> Fax Number: 01904 528 791
> shoe-shop.com ltd
> Catherine House
> Northminster Business Park
> Upper Poppleton, YORK
> YO26 6QU
>
> www.shoe-shop.com
>
> 
>
> This message (and any associated files) is intended only for the use of the
> individual or entity to which it is addressed and may contain information
> that is confidential, subject to copyright or constitutes a trade secret. If
> you are not the intended recipient you are hereby notified that any
> dissemination, copying or distribution of this message, or files associated
> with this message, is strictly prohibited. If you have received this message
> in error, please notify us immediately by replying to the message and
> deleting it from your computer. Messages sent to and from us may be
> monitored.
>
> Internet communications cannot be guaranteed to be secure or error-free as
> information could be intercepted, corrupted, lost, destroyed, arrive late or
> incomplete, or contain viruses. Therefore, we do not accept responsibility
> for any errors or omissions that are present in this message, or any
> attachment, that have arisen as a result of e-mail transmission. If
> verification is required, please request a hard-copy version. Any views or
> opinions presented are solely those of the author and do not necessarily
> represent those of the company. (PAVD001)
>
> Shoe-shop.com Limited is a company registered in England and Wales with
> company number 03817232. Vat Registration GB 734 256 241. Registered Office
> Catherine House, Northminster Business Park, Upper Poppleton, YORK, YO26
> 6QU.
>
> This message has been scanned for malware by SurfControl plc.
> www.surfcontrol.com



-- 
--Noble Paul

RE: timeouts

[Simon Collins]

Tomcat is using about 98mb memory, mysql is about 500mb. Tomcat
completely freezes up - can't do anything other than restart the
service.

It's freezing when it's adding new documents (we add using solrSharp).
Once the documents are added, we do a comit through solrsharp. It's
timing out well before getting to the commit. As it's a public search
people will still be hitting the search whilst this is going on, but
it's not massive search traffic. A few thousand requests spread over a
full day.

I can't find anything logged anywhere as a reason - do I need to turn on
specific logging anywhere? I might just bite the bullet and shift them
onto different boxes





Simon Collins
Systems Analyst

Telephone: 01904 606 867
Fax Number: 01904 528 791

shoe-shop.com ltd
Catherine House
Northminster Business Park
Upper Poppleton, YORK
YO26 6QU
www.shoe-shop.com


This message (and any associated files) is intended only for the use of the 
individual or entity to which it is addressed and may contain information that 
is confidential, subject to copyright or constitutes a trade secret. If you are 
not the intended recipient you are hereby notified that any dissemination, 
copying or distribution of this message, or files associated with this message, 
is strictly prohibited. If you have received this message in error, please 
notify us immediately by replying to the message and deleting it from your 
computer. Messages sent to and from us may be monitored. 

Internet communications cannot be guaranteed to be secure or error-free as 
information could be intercepted, corrupted, lost, destroyed, arrive late or 
incomplete, or contain viruses. Therefore, we do not accept responsibility for 
any errors or omissions that are present in this message, or any attachment, 
that have arisen as a result of e-mail transmission. If verification is 
required, please request a hard-copy version. Any views or opinions presented 
are solely those of the author and do not necessarily represent those of the 
company. (PAVD001) 
Shoe-shop.com Limited is a company registered in England and Wales with company 
number 03817232. Vat Registration GB 734 256 241. Registered Office Catherine 
House, Northminster Business Park, Upper Poppleton, YORK, YO26 6QU.


-Original Message-

From: Grant Ingersoll [mailto:[EMAIL PROTECTED] 
Sent: 24 October 2008 21:05
To: solr-user@lucene.apache.org
Subject: Re: timeouts

What is happening at the time of the freeze?  Are you indexing?   
Searching?  For either of these options, what are the operations that  
are happening?  Is Tomcat and the Solr admin accessible while this  
happens?  Are there any exceptions in the logs?  How much memory are  
you giving it versus MySQL?  I assume you're hitting both, right?  Do  
you have enough memory to do that?  Typically DBs like as much mem as  
they can get, as does Solr (for the most part).

That being said, 100K records is next to nothing, so I doubt Solr is  
taking that much memory.  Please describe a bit more what you are  
doing and seeing.


On Oct 24, 2008, at 8:58 AM, Simon Collins wrote:

> Hi
>
> We're running solr on a win 2k3 box under tomcat with about 100,000  
> records. When doing large updates of records via solr sharp, solr  
> completely freezes and doesn't come back until we restart tomcat.
>
> This has only started happening since putting mysql on the same box  
> (as a source of the data to update from).
>
> Are there any known issues with running solr and mysql on the same  
> box? When it's frozen, the cpu usage is around 1-2% not exactly out  
> of resources!
>
> Am i best using something else instead of tomcat? We're still  
> trialling solr (presently, used for our main site search
www.shoe-shop.com 
>  and search and navigation for our microsites ). It's an excellent  
> search product, but I don't want to fork out on new hardware for it  
> just yet - until i know how more about the performance and which  
> environment i'm best to go for (win/linux).
>
> If anyone has any suggestions/needs more info, i'd be extremely  
> grateful.
>
> Thanks
> Simon
> Simon Collins
> Systems Analyst
>
> Telephone:01904 606 867
> Fax Number:   01904 528 791
>
> shoe-shop.com ltd
> Catherine House
> Northminster Business Park
> Upper Poppleton, YORK
> YO26 6QU
>
> www.shoe-shop.com
>
>
> This message (and any associated files) is intended only for the use  
> of the individual or entity to which it is addressed and may contain  
> information that is confidential, subject to copyright or  
> constitutes a trade secret. If you are not the intended recipient  
> you are hereby notified that any dissemination, copy

Re: timeouts

[Grant Ingersoll]

What is happening at the time of the freeze?  Are you indexing?   
Searching?  For either of these options, what are the operations that  
are happening?  Is Tomcat and the Solr admin accessible while this  
happens?  Are there any exceptions in the logs?  How much memory are  
you giving it versus MySQL?  I assume you're hitting both, right?  Do  
you have enough memory to do that?  Typically DBs like as much mem as  
they can get, as does Solr (for the most part).


That being said, 100K records is next to nothing, so I doubt Solr is  
taking that much memory.  Please describe a bit more what you are  
doing and seeing.



On Oct 24, 2008, at 8:58 AM, Simon Collins wrote:


Hi

We’re running solr on a win 2k3 box under tomcat with about 100,000  
records. When doing large updates of records via solr sharp, solr  
completely freezes and doesn’t come back until we restart tomcat.


This has only started happening since putting mysql on the same box  
(as a source of the data to update from).


Are there any known issues with running solr and mysql on the same  
box? When it’s frozen, the cpu usage is around 1-2% not exactly out  
of resources!


Am i best using something else instead of tomcat? We’re still  
trialling solr (presently, used for our main site search www.shoe-shop.com 
 and search and navigation for our microsites ). It’s an excellent  
search product, but I don’t want to fork out on new hardware for it  
just yet – until i know how more about the performance and which  
environment i’m best to go for (win/linux).


If anyone has any suggestions/needs more info, i’d be extremely  
grateful.


Thanks
Simon
Simon Collins
Systems Analyst

Telephone:  01904 606 867
Fax Number: 01904 528 791

shoe-shop.com ltd
Catherine House
Northminster Business Park
Upper Poppleton, YORK
YO26 6QU

www.shoe-shop.com


This message (and any associated files) is intended only for the use  
of the individual or entity to which it is addressed and may contain  
information that is confidential, subject to copyright or  
constitutes a trade secret. If you are not the intended recipient  
you are hereby notified that any dissemination, copying or  
distribution of this message, or files associated with this message,  
is strictly prohibited. If you have received this message in error,  
please notify us immediately by replying to the message and deleting  
it from your computer. Messages sent to and from us may be monitored.


Internet communications cannot be guaranteed to be secure or error- 
free as information could be intercepted, corrupted, lost,  
destroyed, arrive late or incomplete, or contain viruses. Therefore,  
we do not accept responsibility for any errors or omissions that are  
present in this message, or any attachment, that have arisen as a  
result of e-mail transmission. If verification is required, please  
request a hard-copy version. Any views or opinions presented are  
solely those of the author and do not necessarily represent those of  
the company. (PAVD001)


Shoe-shop.com Limited is a company registered in England and Wales  
with company number 03817232. Vat Registration GB 734 256 241.  
Registered Office Catherine House, Northminster Business Park, Upper  
Poppleton, YORK, YO26 6QU.













This message has been scanned for malware by SurfControl plc. 
www.surfcontrol.com



--
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com


Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

timeouts

[Simon Collins]

Hi

 

We're running solr on a win 2k3 box under tomcat with about 100,000
records. When doing large updates of records via solr sharp, solr
completely freezes and doesn't come back until we restart tomcat.

 

This has only started happening since putting mysql on the same box (as
a source of the data to update from). 

 

Are there any known issues with running solr and mysql on the same box?
When it's frozen, the cpu usage is around 1-2% not exactly out of
resources!

 

Am i best using something else instead of tomcat? We're still trialling
solr (presently, used for our main site search www.shoe-shop.com and
search and navigation for our microsites ). It's an excellent search
product, but I don't want to fork out on new hardware for it just yet -
until i know how more about the performance and which environment i'm
best to go for (win/linux).

 

If anyone has any suggestions/needs more info, i'd be extremely
grateful.

 

Thanks
Simon



Simon Collins
Systems Analyst

Telephone: 01904 606 867
Fax Number: 01904 528 791

shoe-shop.com ltd
Catherine House
Northminster Business Park
Upper Poppleton, YORK
YO26 6QU
www.shoe-shop.com


This message (and any associated files) is intended only for the use of the 
individual or entity to which it is addressed and may contain information that 
is confidential, subject to copyright or constitutes a trade secret. If you are 
not the intended recipient you are hereby notified that any dissemination, 
copying or distribution of this message, or files associated with this message, 
is strictly prohibited. If you have received this message in error, please 
notify us immediately by replying to the message and deleting it from your 
computer. Messages sent to and from us may be monitored. 

Internet communications cannot be guaranteed to be secure or error-free as 
information could be intercepted, corrupted, lost, destroyed, arrive late or 
incomplete, or contain viruses. Therefore, we do not accept responsibility for 
any errors or omissions that are present in this message, or any attachment, 
that have arisen as a result of e-mail transmission. If verification is 
required, please request a hard-copy version. Any views or opinions presented 
are solely those of the author and do not necessarily represent those of the 
company. (PAVD001) 
Shoe-shop.com Limited is a company registered in England and Wales with company 
number 03817232. Vat Registration GB 734 256 241. Registered Office Catherine 
House, Northminster Business Park, Upper Poppleton, YORK, YO26 6QU.


This message has been scanned for malware by SurfControl plc. 
www.surfcontrol.com