Hi, Make sure you optimise your index, I had queries taking 15sec+ and using a lot of memory. After optimisation I am getting 3 - 6 sec results and better memory usage. The index is 40,000 news articles.
I have also been looking at creating an indexing & search server based on ZSL, which would handle caching, backup, pagination, indexing etc via REST. -----Original Message----- From: Kevin Schroeder [mailto:[EMAIL PROTECTED] Sent: 19 March 2008 21:06 To: 'Eric Marden'; fw-general@lists.zend.com Subject: RE: [fw-general] Speeding up Lucene One thing to note with that is that due to the way the data is handled deep in the code is that if you store a lot of data directly in the index, there is a loop that can get iterated over thousands of times even for small pieces of data. It is generally better to create the index and store a primary key or some other unique identifier of the data being represented and then retrieve the data based off of that unique identifier out of the database. That will minimize the time in that loop, minimize the amount of data being retrieved from the index, minimize the data being serialized and could (potentially) decrease the search response times by a significant amount. Don't know if that's the problem but it's often solved performance problems in the past, in my experience. If you have a profiler (like what's available in Zend Studio) you can take a look at the query and see if there's excessive code iteration in the request. Kevin Schroeder Technical Consultant Zend Technologies, Ltd. www.zend.com > -----Original Message----- > From: Eric Marden [mailto:[EMAIL PROTECTED] > Sent: Wednesday, March 19, 2008 3:53 PM > To: fw-general@lists.zend.com > Subject: RE: [fw-general] Speeding up Lucene > > > Do you have a lot of data in your index, essentially using the index > as a database replacement? > > 20 documents. Index is about 80kb or so. We are using it as DB > replacement, due to the fact that the production set up will have a > mysql cluster behind it (no FULLTEXT index available). However this > will grow when we reimport all of the production data into the new schema. > > > > -- > Eric Marden > > > -----Original Message----- > From: Kevin Schroeder [mailto:[EMAIL PROTECTED] > Sent: Wednesday, March 19, 2008 4:33 PM > To: Eric Marden; fw-general@lists.zend.com > Subject: RE: [fw-general] Speeding up Lucene > > > > > > Kevin Schroeder > Technical Consultant > Zend Technologies, Ltd. > > www.zend.com > > > > > > > > > > -----Original Message----- > > From: Eric Marden [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, March 19, 2008 3:23 PM > > To: fw-general@lists.zend.com > > Subject: RE: [fw-general] Speeding up Lucene > > > > P.S. - This is the only one of my search types that takes a long time. > > All the others can return the page in 3 seconds (timed with Yslow). > > This one is doing it in 23seconds. I've reduced other bottlenecks > > (it was plus 30seconds). > > > > > > > > -- > > Eric Marden -- allpay.net Limited, Fortis et Fides, Whitestone Business Park, Whitestone, Hereford, HR1 3SE. Registered in England No. 02933191. UK VAT Reg. No. 666 9148 88. Telephone: 0870 243 3434, Fax: 0870 243 6041. Website: www.allpay.net Email: [EMAIL PROTECTED] This email, and any files transmitted with it, is confidential and intended solely for the use of the individual or entity to whom it is addressed. If you have received this email in error please notify the allpay.net Information Security Manager at the number above.
