And the benefit is faster query times - the socket isn't being setup/cleaned up on every single search request. From the quick testing I did back when I made this change, this made a noticeable difference with Sphinx query times (no numbers at hand, but I think it was at least a 50% improvement, if not closer to 80%).
-- Pat On 11/08/2013, at 3:15 PM, Ngan Pham wrote: > Hm…I was afraid of that… > So, we have 9 application servers with 10-20 processes each. At high traffic > times, we'd be seeing 181 processes for searchd? Won't that blow memory up > like crazy? Is this normally something that everyone deals with? > > Just curious…what are the benefits of persisted connection pools? > > As always, thanks for the quick response Pat! > > - Ngan > On Saturday, August 10, 2013 at 10:11 PM, Pat Allan wrote: > >> Hi Ngan >> >> There's not really any documentation around the changes, I'm afraid… but >> what you're seeing is a process per thread of your app, plus a master daemon >> process, due to the persisted connection pool. >> >> All the logic for this connection pool can be found in >> ThinkingSphinx::Connection: >> https://github.com/pat/thinking-sphinx/blob/v2/lib/thinking_sphinx/connection.rb >> >> The HISTORY file has a list of all relevant changes though - here's the list >> of what's changed between 2.0.14 and 2.1.0 - but this and some delta >> refactoring are the biggest items: >> >> * Removed plugin support - Thinking Sphinx is now gem-only across all >> branches. >> * ThinkingSphinx::Version and the thinking_sphinx:version task have been >> removed - it's a gem, it has a version number. >> * Updating Riddle to 1.5.6 or newer. >> * Requires ActiveRecord ~> 2.1 for TS 1.x releases (earlier versions were >> considered unsupported a few releases ago). >> * Allow custom Riddle controllers - useful for Flying Sphinx to take over >> management of Sphinx daemon/indexing actions. >> * Rejigged delta support to be generic, with local job classes that provide >> a clean, simple interface for third-party libraries. >> * Add hooks for anything that needs to happen before indexing (such as >> clearing out existing delta jobs). >> * Connection pool for all Sphinx client communication, with new connections >> built if there's any connection-related (as opposed to syntax) issues. >> * Multiple-field search conditions can be done with arrays of field names as >> keys in the :conditions hash (Alex Dowad). >> * Removed named capture in regular expressions to maintain MRI 1.8 support >> (Michael Wintrant). >> * Support new JDBC configuration style (Kyle Stevens). >> >> -- >> Pat >> >> On 11/08/2013, at 2:39 PM, Ngan wrote: >> >>> Hi Pat, >>> >>> Thanks for the help. I tried upgrading to 2.1.0...and I'm noticing multiple >>> instance of searched running now. Is that normal? Would you be able to >>> point me to documentation of major changes with 2.1.0? >>> >>> Thanks, >>> Ngan >>> >>> On Friday, August 9, 2013 7:57:51 AM UTC-7, Pat Allan wrote: >>> Hi Ngan >>> >>> In 2.1.0 there's been some patches that deal with these kinds of errors - >>> TS will now retry searches if an error crops up on the client connection >>> (which is also persisted per thread in a connection pool, saving socket >>> setup/teardown time). If an error crops up, a new connection is made, but >>> if the error persists, it'll still get raised… >>> >>> Give 2.1.0 a spin, see if that helps matters. >>> >>> -- >>> Pat >>> >>> On 09/08/2013, at 10:46 PM, Ngan wrote: >>> >>>> Forgot to add the trace: >>>> >>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:438:in `block in >>>> populate' >>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:606:in `call' >>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:606:in >>>> `retry_on_stale_index' >>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:426:in `populate' >>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:104:in `to_a' >>>> >>>> >>>> >>>> On Friday, August 9, 2013 5:45:55 AM UTC-7, Ngan wrote: >>>> Hi, >>>> >>>> We reindex our entire index pretty often (once every 3 minutes) because we >>>> have a pretty small data collection and we don't want to use delayed >>>> delta. I notice however, that every once in a while, when we deploy our >>>> application and it happens to be the same time the reindexing is about to >>>> rotate, we'll get this error "no enabled local indexes to search" every >>>> time we hit sphinx there afterwards. When this happens, we have to restart >>>> our app so that it picks up the new indexes. We are reindexing with rotate >>>> so the existing index should still be there and the rotation should be >>>> seamless. Any ideas on why this happens? And if there's anything to do >>>> about it? >>>> >>>> Thanks, >>>> Ngan >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Thinking Sphinx" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Thinking Sphinx" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/thinking-sphinx. >>> For more options, visit https://groups.google.com/groups/opt_out. >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Thinking Sphinx" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/thinking-sphinx. >> For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/thinking-sphinx. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/groups/opt_out.
