After doing some research I broke down and just updated my Zend Framework. I just installed it not long ago so I didn't think much of it, but then I realized I'm running version 1.6.1 and that Zend is currently on 1.7.8. Upon upgrading the complex fuzzy search that was taking 30 seconds now takes 0.067 seconds. I have no idea what changed in the past few months, and see no mention of performance issues on their issue tracker but all seems well now. Figured I'd post here to give others a heads up if they run into similar issues like I did.
>>> mschrae...@btsb.com 4/2/2009 3:32:01 PM >>> >>> erickerick...@gmail.com 4/2/2009 10:24:42 AM >>> >This seems really odd, especially with an index that size. The >first question is usually "Do you open an IndexReader for >each query?" I'm using the Zend_Search_Lucene implementation so I'm really not sure how it handles the IndexReader. At the top of the script I open the index and do searches on it. Unless Zend is doing something special in the background I'm assuming I'm using the IndexReader on a per-page basis. I haven't been able to find any information on this yet, but from all the examples I've been reading none of them say to keep the index in a session to improve speed. I'll have to get on the zend mailing lists to find out more about best practices. >>> markrmil...@gmail.com 4/2/2009 10:40:59 AM >>> >You might try setting a longer prefix. Fuzzy queries don't scale by the >way. By default they enumerate every unique term. How many unique terms >do you have in the index? I'll look at a longer prefix setting, as the reply above mentioned the improving search speed article. Currently my index has 104076 terms in it.
