Hi Neils, Yes, that's the problem, you hit Windows 2GB application limit. Windows has a 2GB memory limit per application and since your index now over 2GB and you are loading it all into memory, you should expect this kind of a failure.
Why are you loading the entire index into memory? Have you done performance analyses to justify the need to do so? Regards, -- George Aroush -----Original Message----- From: Neils [mailto:[EMAIL PROTECTED] Sent: Sunday, January 21, 2007 3:35 AM To: lucene-net-user@incubator.apache.org Subject: 2GB Filesize, Ramdirectory Hi, I have a 32bit operating system and one indexfiel with a filesize over 2gb (2.3GB). When i try to load index into ramdirectory i get this error: System.IO.IOException: read past EOF bei Lucene.Net.Store.BufferedIndexInput.Refill() bei Lucene.Net.Store.BufferedIndexInput.ReadByte() bei Lucene.Net.Store.IndexInput.ReadVInt() bei Lucene.Net.Index.CompoundFileReader..ctor(Directory dir bei Lucene.Net.Index.SegmentReader.Initialize(SegmentInfo s bei Lucene.Net.Index.SegmentReader.Get(Directory dir, Segme oolean ownDir) bei Lucene.Net.Index.IndexReader.AnonymousClassWith.DoBody( bei Lucene.Net.Store.Lock.With.Run() bei Lucene.Net.Index.IndexReader.Open(Directory directory, bei Lucene.Net.Index.IndexReader.Open(Directory directory) Last week it was working without problems bit indexsize was 1.99 GB. So i think it's related that index is now over 2GB, but i have no clue to solve the problem. Thanks -----Ursprüngliche Nachricht----- Von: Simon Gartz [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] Gesendet: Freitag, 19. Januar 2007 09:50 An: lucene-net-user@incubator.apache.org Betreff: RE: FileNotFoundException, is this code supposed to work? Yes, in the application we are developing yes, but in the unit test in my first post it is only writing, same error occurres. Regards Simon -----Original Message----- From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] Sent: den 18 januari 2007 19:53 To: lucene-net-user@incubator.apache.org Subject: RE: FileNotFoundException, is this code supposed to work? Good to know about the concurrent searching -- just to verify, a search can occur a write/update/delete occurs? I have since created a single thread command queue to solve much like what you mentioned. Thanks, Michael -----Original Message----- From: Torsten Rendelmann [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] Sent: Thursday, January 18, 2007 4:45 AM To: lucene-net-user@incubator.apache.org Subject: RE: FileNotFoundException, is this code supposed to work? AFAIK only index modifications require a lock, while a concurrent search does not. We run into the same problem (concurrent index writers) and "solved" it using a single index mod. thread working with a queue of documents to add/delete/update. > -----Original Message----- > From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] > Sent: Monday, January 15, 2007 7:25 PM > To: lucene-net-user@incubator.apache.org > Subject: RE: FileNotFoundException, is this code supposed to work? > > > I receive these kinds of errors when multiple threads are > searching/writing to the index at the same time. So far, I have had > to resort to .NET locks for key locations in code, such as: > > Dim hits As Hits = Nothing > SyncLock aLock > hits = searcher.Search(query, df1, custSort) > End SyncLock > > SyncLock aLock > Result = reader.DeleteDocuments(New Term("Id", > CStr(Id))) > reader.Close() > End SyncLock > > SyncLock aLock > writer.AddDocument(doc) > writer.Close() > End SyncLock > > > This of course kills our scalability, so I am so interested in knowing > how to address multi-process/multi-threaded access to any particular > index. > > > > - Michael > > > > -----Original Message----- > From: Simon Gartz [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] > Sent: Monday, January 15, 2007 6:54 AM > To: lucene-net-user@incubator.apache.org > Subject: FileNotFoundException, is this code supposed to work? > > Hi, > > I've seen this questions before with slightly different approaches, > but non with sample code. Is this code supposed to work or not, using > Lucene.net v1.9.1.3? > I get follwing error executing the code below. Note that it might take > some time to crash. > > System.IO.FileNotFoundException : Could not find file > "C:\DOCUME~1\sgartz\LOCALS~1\Temp\lucene\_1ji.frq". > at System.IO.__Error.WinIOError(Int32 errorCode, String str) > at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess > access, FileShare share, Int32 bufferSize, Boolean useAsync, String > msgPath, Boolean bFromProxy) > at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess > access, FileShare share) > at Lucene.Net.Store.Descriptor..ctor(FSIndexInput > enclosingInstance, FileInfo file, FileAccess mode) > at Lucene.Net.Store.FSIndexInput..ctor(FileInfo path) > at Lucene.Net.Store.FSDirectory.OpenInput(String name) > at Lucene.Net.Index.SegmentReader.Initialize(SegmentInfo si) > at Lucene.Net.Index.SegmentReader.Get(Directory dir, SegmentInfo si, > SegmentInfos sis, Boolean closeDir, Boolean ownDir) > at Lucene.Net.Index.SegmentReader.Get(SegmentInfo si) > at Lucene.Net.Index.AnonymousClassWith.DoBody() > at Lucene.Net.Store.With.Run() > at Lucene.Net.Index.IndexReader.Open(Directory > directory, Boolean > closeDirectory) > at Lucene.Net.Index.IndexReader.Open(String path) > at Lucene.Net.Search.IndexSearcher..ctor(String path) > > c:\projects\x\development\x.test\indexingservice\indexingservi > cetest.cs( > 128,0): at > x.Test.IndexingService.IndexingServiceTest.LuceneThreadTest() > > > private static readonly string directory = > Path.Combine(Path.GetTempPath(), "lucene"); > > [Test] > public void LuceneThreadTest(){ > > //Create index > Lucene.Net.Analysis.Analyzer analyzer = new > Lucene.Net.Analysis.SimpleAnalyzer(); > Lucene.Net.Index.IndexWriter writer = new > Lucene.Net.Index.IndexWriter(directory, analyzer, true); > writer.Close(); > > //Start indexing thread > Thread indexThread = new Thread(new > ThreadStart(LuceneIndexThread)); > indexThread.Start(); > > while(indexThread.IsAlive){ > Console.WriteLine("Searching..."); > > Lucene.Net.Search.Searcher searcher = new > Lucene.Net.Search.IndexSearcher(directory); > > Lucene.Net.Search.Hits hits = null; > Lucene.Net.QueryParsers.QueryParser parser = new > Lucene.Net.QueryParsers.QueryParser("contents", analyzer); > Lucene.Net.Search.Query query = parser.Parse("put"); > Console.WriteLine("Query: " + query.ToString("contents")); > > hits = searcher.Search(query); > Console.WriteLine(hits.Length() + " total results"); > for (int i = 0; i < hits.Length() && i < 10; i++) { > Lucene.Net.Documents.Document d = hits.Doc(i); > Console.WriteLine(i + " " + hits.Score(i) + " " + > d.Get("contents").Length + " " + d.Get("contents").Substring(0, 20)); > } > searcher.Close(); > Thread.Sleep(1000); > } > } > private void LuceneIndexThread(){ > Console.WriteLine("Start indexing files"); > for(int j = 0; j < 100; j++){ > > Lucene.Net.Analysis.Analyzer analyzer = new > Lucene.Net.Analysis.SimpleAnalyzer(); > Lucene.Net.Index.IndexWriter writer = new > Lucene.Net.Index.IndexWriter(directory, analyzer, false); > > for (int i = 0; i < 100; i++) { > Lucene.Net.Documents.Document d = new > Lucene.Net.Documents.Document(); > string content = new string('a', (new Random()).Next(100) * 10); > d.Add(new Lucene.Net.Documents.Field("contents", "Put contents > here " + content, Lucene.Net.Documents.Field.Store.YES, > Lucene.Net.Documents.Field.Index.TOKENIZED)); > writer.AddDocument(d); > } > writer.Close(); > Console.WriteLine("Indexed 100 documents..."); > } > Console.WriteLine("Finished indexing files"); > } > > > Regards > Simon >
