Hi, Thanks for the link provided, actually I've go through those article when I developing the index and search function for my application. I haven’t try profiler yet, but I monitor the CPU usage and notice that whatever index or search performing, the CPU usage raise to 100%. Below I will try to elaborate more on what my application is doing and how I index and search.
There are many concurrent process running, first, the application will write records that received into a text file with tab separated each different field. Application will point to a new file every 10mins and start writing to it. So every file will contains only 10mins record, approximate 600,000 records per file. Then, the indexing process will check whether there is a text file to be index, if it is, the thread will wake up and start perform indexing. The indexing process will first add documents to RAMDir, Then later, add RAMDir into FSDir by calling addIndexNoOptimize() when there is 100,000 documents(32 fields per doc) in RAMDir. There is only 1 IndexWriter(FSDir) was created but a few IndexWriter(RAMDir) was created during the whole process. Below are some configuration for IndexWriters that I mentioned:- IndexWriter (RAMDir) - SimpleAnalyzer - setMaxBufferedDocs(10000) - Filed.Store.YES - Field.Index.NO_NORMS IndexWriter (FSDir) - SimpleAnalyzer - setMergeFactor(20) - addIndexesNoOptimize() For the searching, because there are many queries(20,000) run continuously to generate the aggregate table for reporting purpose. All this queries is run in nested loop, and there is only 1 Searcher created, I try searcher and filter as well, filter give me better result, but both also utilize lots of CPU resources. Hope this info will help and sorry for my bad English. Thanks eChuang, Chew -----Original Message----- From: karl wettin [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 31, 2007 5:54 PM To: java-user@lucene.apache.org Subject: Re: High CPU usage duing index and search 31 jul 2007 kl. 05.25 skrev Chew Yee Chuang: > But just notice that when Lucene performing search or index, > the CPU usage on my machine raise to 100%, because of this issue, > some of my > others backend process will slow down eventually. Just want to know > does > anyone face this problem before ? and is it any idea on how to > overcome this > problem ? Did you run a profiler to see what it is that consume all the resources? It is very hard to guess based on the information you supplied. Start here: http://wiki.apache.org/lucene-java/BasicsOfPerformance http://wiki.apache.org/lucene-java/ImproveIndexingSpeed http://wiki.apache.org/lucene-java/ImproveSearchingSpeed -- karl --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.476 / Virus Database: 269.11.0/927 - Release Date: 7/30/2007 5:02 PM No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.476 / Virus Database: 269.11.0/929 - Release Date: 7/31/2007 5:26 PM --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
