20,000 queries continuously? Sounds a bit too much. Can you elaborate more what you need to do? Probably you won't need that many queries.
Chew Yee Chuang wrote: > > Hi, > > Thanks for the link provided, actually I've go through those article when > I > developing the index and search function for my application. I haven’t try > profiler yet, but I monitor the CPU usage and notice that whatever index > or > search performing, the CPU usage raise to 100%. Below I will try to > elaborate more on what my application is doing and how I index and search. > > There are many concurrent process running, first, the application will > write > records that received into a text file with tab separated each different > field. Application will point to a new file every 10mins and start writing > to it. So every file will contains only 10mins record, approximate 600,000 > records per file. Then, the indexing process will check whether there is a > text file to be index, if it is, the thread will wake up and start perform > indexing. > > The indexing process will first add documents to RAMDir, Then later, add > RAMDir into FSDir by calling addIndexNoOptimize() when there is 100,000 > documents(32 fields per doc) in RAMDir. There is only 1 IndexWriter(FSDir) > was created but a few IndexWriter(RAMDir) was created during the whole > process. Below are some configuration for IndexWriters that I mentioned:- > > IndexWriter (RAMDir) > - SimpleAnalyzer > - setMaxBufferedDocs(10000) > - Filed.Store.YES > - Field.Index.NO_NORMS > > IndexWriter (FSDir) > - SimpleAnalyzer > - setMergeFactor(20) > - addIndexesNoOptimize() > > For the searching, because there are many queries(20,000) run continuously > to generate the aggregate table for reporting purpose. All this queries is > run in nested loop, and there is only 1 Searcher created, I try searcher > and > filter as well, filter give me better result, but both also utilize lots > of > CPU resources. > > Hope this info will help and sorry for my bad English. > > Thanks > eChuang, Chew > > -----Original Message----- > From: karl wettin [mailto:[EMAIL PROTECTED] > Sent: Tuesday, July 31, 2007 5:54 PM > To: java-user@lucene.apache.org > Subject: Re: High CPU usage duing index and search > > > 31 jul 2007 kl. 05.25 skrev Chew Yee Chuang: >> But just notice that when Lucene performing search or index, >> the CPU usage on my machine raise to 100%, because of this issue, >> some of my >> others backend process will slow down eventually. Just want to know >> does >> anyone face this problem before ? and is it any idea on how to >> overcome this >> problem ? > > Did you run a profiler to see what it is that consume all the resources? > It is very hard to guess based on the information you supplied. Start > here: > > http://wiki.apache.org/lucene-java/BasicsOfPerformance > http://wiki.apache.org/lucene-java/ImproveIndexingSpeed > http://wiki.apache.org/lucene-java/ImproveSearchingSpeed > > > -- > karl > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > No virus found in this incoming message. > Checked by AVG Free Edition. > Version: 7.5.476 / Virus Database: 269.11.0/927 - Release Date: 7/30/2007 > 5:02 PM > > > No virus found in this outgoing message. > Checked by AVG Free Edition. > Version: 7.5.476 / Virus Database: 269.11.0/929 - Release Date: 7/31/2007 > 5:26 PM > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > -- View this message in context: http://www.nabble.com/High-CPU-usage-duing-index-and-search-tf4190756.html#a11962524 Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
