Lokeya <[EMAIL PROTECTED]> wrote on 21/03/2007 22:09:06: > > Initially I was writing into the Index 7,00,000 times. I chaged the code to > now write only 70 times which means I am putting lot of data in an array > list and add to doc and index at one shot. This is where the improvement > came from. To be precise IndexWriter is now adding document 70 times Vs. > 7,00,000 times.
I was just concluding from the provided code; So now your index should have only 70 documents, right? (Search "granularity" would be 1/70 of collection size, is that really useful?) If now you open the index writer once, add all those 70 documents, call optimize once, close index writer once, that's the right way. > If this is not clear please let me know. I have't pasted the latest code > where I have fixed the lock issue as well. If required I can do that. > > Thanks everyone for quick turnaround and it really helped me a lot. > > > Doron Cohen wrote: > > > > Lokeya <[EMAIL PROTECTED]> wrote on 18/03/2007 13:19:45: > > > >> > >> Yep I did that, and now my code looks as follows. > >> The time taken for indexing one file is now > >> => Elapsed Time in Minutes :: 0.3531 > >> which is really great > > > > I am jumping in late so appologies if I am missing something. > > However I don't understand where this improvement came > > from. (more below) > > > >> > >> //Create an Index under LUCENE > >> IndexWriter writer = new IndexWriter("./LUCENE", new > >> StopStemmingAnalyzer(), > >> false); > >> Document doc = new Document(); > >> > >> //Get Array List Elements and add them as fileds to doc > >> for(int k=0; k < alist_Title.size(); k++) { > >> doc.add(new Field("Title", > >> alist_Title.get(k).toString(), > >> Field.Store.YES, > >> Field.Index.UN_TOKENIZED)); > >> } > >> > >> for(int k=0; k < alist_Descr.size(); k++) { > >> doc.add(new Field("Description", > >> alist_Descr.get(k).toString(), > >> Field.Store.YES, > >> Field.Index.UN_TOKENIZED)); > >> } > >> > >> //Add the document created out of those fields to > >> // the IndexWriter which will create and index > >> writer.addDocument(doc); > >> writer.optimize(); > >> writer.close(); > >> > >> long elapsedTimeMillis = System.currentTimeMillis()-startTime; > >> System.out.println("Elapsed Time for "+ > >> dirName +" :: " + elapsedTimeMillis); > >> float elapsedTimeMin = elapsedTimeMillis/(60*1000F); > >> System.out.println("Elapsed Time in Minutes :: "+ > >> elapsedTimeMin); > > > > the code listed above is, still, for each indexed document, > > doing this: > > > > 1. open index writer > > 2. create a document (and add fields to it) > > 3. add the doc to the index > > 4. optimize the index > > 5. close the index writer > > > > As already mentioned here this is very non optimal. > > So I don't understand where the improvement came from. > > > > It should be done like this: > > > > 1. open/create the index > > 2. for each valid doc input { > > a. create a document (add fields) > > b. add the doc to the index > > } > > 3. optimize the index (optionally) > > 4. close the index > > > > It would probably also help both coding and discussion to capture > > some logic in methods, e.g. > > > > IndexWriter createIndex(); > > IndexWriter openIndex(); > > void closeIndex(); > > Document createDocument(String oai_citeseer_fileName) > > > >> but after processing 4 dumpfiles(which means > >> 40,000 small xml's), I get : > >> > >> caught a class java.io.IOException > >> 40114 with message: Lock obtain timed out: > >> Lock@/tmp/lucene-e36d478e46e88f594d57a03c10ee0b3b-write.lock > >> > >> > >> This is the new issue now.What could be the reason ?. I am surprised > > because > >> I am only writing to Index under ./LUCENE/ evrytime and not doing > > anything > >> with the index(ofcrs to avoid such synchronization issues !) > > > > If the code listed above is getting this exception, and this runs on > > Windows, then perhaps this is related to issue 665 - intensive IO > > activity, write lock acquired and released for each and every added > > document. If so, this too would be avoided by opening the index just once. > > > > Last comment - I did not understand if a single index is used for all > > 70 metadata files, or each one has its own index, totalling 70 indexes. > > I would think it should be only one index for all, and then you better > > avoid calling optimize 70 times - that would waste time too. Better > > organize the code like this: > > > > 1. create/open index > > 2. for each meta data file { > > a. for each "valid element" in meta data file { > > a.1 create document > > a.2 add doc to index > > } > > } > > 3. optimize > > 4. close index > > > > Regards, > > Doron --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]