Hi Grant,
My stress test is unable to reproduce this exception, either. I'm
adding Wikipedia docs to an index, using a high merge factor, then
opening a new writer with low merge factor (5) and calling optimize.
This forces concurrent merges to run during the optimize.
One more question: are your docs uniform? Or, for example, do some
of them have stored fields while others do not?
Mike
Grant Ingersoll wrote:
On Jun 11, 2008, at 6:00 AM, Michael McCandless wrote:
Grant Ingersoll wrote:
Is more than one thread adding documents to the index?
I don't believe so, but I am trying to reproduce. I've only seen
it once, and don't have a lot of details, other than I noticed it
was on a specific file (.fdt) and was wondering if that was a
factor or not. That is, maybe Paul could reproduce it.
I think your exception differs from Paul's in an important way.
Paul's exception means an entire segment (its CFS file) was
deleted, which is very easily caused by accidentally allowing 2
writers on the index at once. But in your case, SegmentReader
successfully opened the fnm file but then failed on the fdt, so,
your segment wasn't deleted (at least not entirely). So I think
something different caused your exception.
Any changes to the defaults in IndexWriter?
It's the SolrIndexWriter.
OK. But what does your solrconfig.xml look like?
It's pretty much the standard indexing setup that's on trunk.
Merge Factor 10, Max Buffered Docs 1000
After seeing that exception, does IndexReader.open also hit that
exception (ie, is/was the index corrupt)? Or does it only
happen with BG merges?
Not sure, unfortunately, I don't have a lot of info yet. The
background exception happened during an optimize, if that matters
at all
OK. It'd be very useful to know if index was really corrupt
(missing that segment) vs BG merge incorrectly, temporarily,
thought it was supposed to merge that segment.
Is this a largish index?
couple million relatively small records
Like, would there be so many segments that optimize would be
running concurrent merges (> 2*mergeFactor segments)? With
ConcurrentMergeScheduler, optimize is now able to run multiple
merges concurrently, if the index has enough segments.
I don't know. Like I said, the only reason I posted was b/c I
thought it sounded similar to Paul's. I haven't been able to
reproduce it yet.
I'll run some stress tests, focusing on concurrency of merges
during optimize...
Which OS & JRE are you using?
Linux, Java 1.6.0_05
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
