Simon,
I have seen similar errors, and it appears that the commit lock is not
being honored by the IndexWriter. I have two separate processes on my
server, one which indexes and another that searches. I do not see the
error often, but in reviewing 24 hours worth of logs I do see it a
couple of times. I have both processes configured to use the same
directory for the lock files. From the error we have both seen, it
appears that an IndexWriter is merging segments while the IndexSearcher
is reading them during creation of the instance. An IndexSearcher
obtains a commit lock before attempting to read all of the segments of
the index, and an IndexWriter is to also obtain a commit lock before
merging segments and deleting segments. The error of a missing segment
is what leads me to believe that the writer may not be obtaining a
commit lock when merging, but I have not delved into the issue deep
enough to find the cause, which is on my very long to-do list :)
In my application I maintain one searcher for the index, and create a
new one every 5 minutes if the index has changed. If creation of a new
searcher fails, the old instance continues to be used and I try to
create a new one in another 5 minutes. As such it does not affect me
greatly and the only impact I have is that my searcher is a few more
minutes out of date.
Additionally, you can have multiple threads adding documents to a Lucene
index, provided that they are sharing the same instance of IndexWriter,
which is thread-safe internally. Use multiple IndexWriters on the same
index at your own peril. I prefer to control optimizations more tightly
to ensure that only one thread is calling the optimize method at a time
and that no documents are being added to the index at that time. I have
found the performance benefit to using multiple threads on the same
IndexWriter to not be significant, and using a single thread to add
documents and optimize can be easier to maintain and avoid any
concurrency issues with optimization.
Michael
Simon Gartz wrote:
Hi,
I've seen this questions before with slightly different approaches, but
non with sample code. Is this code supposed to work or not, using
Lucene.net v1.9.1.3?
I get follwing error executing the code below. Note that it might take
some time to crash.
System.IO.FileNotFoundException : Could not find file
"C:\DOCUME~1\sgartz\LOCALS~1\Temp\lucene\_1ji.frq".
at System.IO.__Error.WinIOError(Int32 errorCode, String str)
at System.IO.FileStream..ctor(String path, FileMode mode,
FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync,
String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode,
FileAccess access, FileShare share)
at Lucene.Net.Store.Descriptor..ctor(FSIndexInput
enclosingInstance, FileInfo file, FileAccess mode)
at Lucene.Net.Store.FSIndexInput..ctor(FileInfo path)
at Lucene.Net.Store.FSDirectory.OpenInput(String name)
at Lucene.Net.Index.SegmentReader.Initialize(SegmentInfo si)
at Lucene.Net.Index.SegmentReader.Get(Directory dir, SegmentInfo
si, SegmentInfos sis, Boolean closeDir, Boolean ownDir)
at Lucene.Net.Index.SegmentReader.Get(SegmentInfo si)
at Lucene.Net.Index.AnonymousClassWith.DoBody()
at Lucene.Net.Store.With.Run()
at Lucene.Net.Index.IndexReader.Open(Directory directory,
Boolean closeDirectory)
at Lucene.Net.Index.IndexReader.Open(String path)
at Lucene.Net.Search.IndexSearcher..ctor(String path)
c:\projects\x\development\x.test\indexingservice\indexingservicetest.cs(
128,0): at x.Test.IndexingService.IndexingServiceTest.LuceneThreadTest()
private static readonly string directory =
Path.Combine(Path.GetTempPath(), "lucene");
[Test]
public void LuceneThreadTest(){
//Create index
Lucene.Net.Analysis.Analyzer analyzer = new
Lucene.Net.Analysis.SimpleAnalyzer();
Lucene.Net.Index.IndexWriter writer = new
Lucene.Net.Index.IndexWriter(directory, analyzer, true);
writer.Close();
//Start indexing thread
Thread indexThread = new Thread(new ThreadStart(LuceneIndexThread));
indexThread.Start();
while(indexThread.IsAlive){
Console.WriteLine("Searching...");
Lucene.Net.Search.Searcher searcher = new
Lucene.Net.Search.IndexSearcher(directory);
Lucene.Net.Search.Hits hits = null;
Lucene.Net.QueryParsers.QueryParser parser = new
Lucene.Net.QueryParsers.QueryParser("contents", analyzer);
Lucene.Net.Search.Query query = parser.Parse("put");
Console.WriteLine("Query: " + query.ToString("contents"));
hits = searcher.Search(query);
Console.WriteLine(hits.Length() + " total results");
for (int i = 0; i < hits.Length() && i < 10; i++) {
Lucene.Net.Documents.Document d = hits.Doc(i);
Console.WriteLine(i + " " + hits.Score(i) + " " +
d.Get("contents").Length + " " + d.Get("contents").Substring(0, 20));
}
searcher.Close();
Thread.Sleep(1000);
}
}
private void LuceneIndexThread(){
Console.WriteLine("Start indexing files");
for(int j = 0; j < 100; j++){
Lucene.Net.Analysis.Analyzer analyzer = new
Lucene.Net.Analysis.SimpleAnalyzer();
Lucene.Net.Index.IndexWriter writer = new
Lucene.Net.Index.IndexWriter(directory, analyzer, false);
for (int i = 0; i < 100; i++) {
Lucene.Net.Documents.Document d = new
Lucene.Net.Documents.Document();
string content = new string('a', (new Random()).Next(100) * 10);
d.Add(new Lucene.Net.Documents.Field("contents", "Put contents here
" + content, Lucene.Net.Documents.Field.Store.YES,
Lucene.Net.Documents.Field.Index.TOKENIZED));
writer.AddDocument(d);
}
writer.Close();
Console.WriteLine("Indexed 100 documents...");
}
Console.WriteLine("Finished indexing files");
}
Regards
Simon
