The change was pretty simple:
https://github.com/pat/thinking-sphinx/commit/d5249ea0215128d5d1f916ee42882932ddb86ba7
To use it (for the 2.x releases of Thinking Sphinx):
gem 'thinking-sphinx', '~> 2.1.0',
:git => 'git://github.com/pat/thinking-sphinx.git',
:branch => 'v2',
:ref => '64da4dc7ff'
And then in an initialiser:
ThinkingSphinx.persistence_enabled = false
Will be keen to hear if everything returns to normal for you!
On 12/08/2013, at 12:19 AM, Ngan Pham wrote:
> I see...
>
> So I've tested it out and it's a bit of a disaster for my situation. Some
> search requests are getting dropped and the server that runs sphinx is
> hitting memory limits. I could, in theory, solve the problem by increasing
> ram, how ever we have fallback servers...and it would suck to have to
> increase those as well. I'll do some research on how people handle large
> sphinx requests...like airbnb.
>
> As for my 2 cents: I never felt that there was that much of an issue with the
> cost of opening a connection to sphinx for every request. It is very fast for
> us currently, and shaving 50-80% off something that already only takes
> milliseconds while trading for more memory usage and less scalability isn't
> worth for me specifically. It would awesome if you added a configuration
> option to disable persistent connections! I will buy you a beer! :)
>
>> On Aug 10, 2013, at 10:26 PM, Pat Allan <[email protected]> wrote:
>>
>> And the benefit is faster query times - the socket isn't being setup/cleaned
>> up on every single search request. From the quick testing I did back when I
>> made this change, this made a noticeable difference with Sphinx query times
>> (no numbers at hand, but I think it was at least a 50% improvement, if not
>> closer to 80%).
>>
>> --
>> Pat
>>
>>> On 11/08/2013, at 3:15 PM, Ngan Pham wrote:
>>>
>>> Hm…I was afraid of that…
>>> So, we have 9 application servers with 10-20 processes each. At high
>>> traffic times, we'd be seeing 181 processes for searchd? Won't that blow
>>> memory up like crazy? Is this normally something that everyone deals with?
>>>
>>> Just curious…what are the benefits of persisted connection pools?
>>>
>>> As always, thanks for the quick response Pat!
>>>
>>> - Ngan
>>>> On Saturday, August 10, 2013 at 10:11 PM, Pat Allan wrote:
>>>>
>>>> Hi Ngan
>>>>
>>>> There's not really any documentation around the changes, I'm afraid… but
>>>> what you're seeing is a process per thread of your app, plus a master
>>>> daemon process, due to the persisted connection pool.
>>>>
>>>> All the logic for this connection pool can be found in
>>>> ThinkingSphinx::Connection:
>>>> https://github.com/pat/thinking-sphinx/blob/v2/lib/thinking_sphinx/connection.rb
>>>>
>>>> The HISTORY file has a list of all relevant changes though - here's the
>>>> list of what's changed between 2.0.14 and 2.1.0 - but this and some delta
>>>> refactoring are the biggest items:
>>>>
>>>> * Removed plugin support - Thinking Sphinx is now gem-only across all
>>>> branches.
>>>> * ThinkingSphinx::Version and the thinking_sphinx:version task have been
>>>> removed - it's a gem, it has a version number.
>>>> * Updating Riddle to 1.5.6 or newer.
>>>> * Requires ActiveRecord ~> 2.1 for TS 1.x releases (earlier versions were
>>>> considered unsupported a few releases ago).
>>>> * Allow custom Riddle controllers - useful for Flying Sphinx to take over
>>>> management of Sphinx daemon/indexing actions.
>>>> * Rejigged delta support to be generic, with local job classes that
>>>> provide a clean, simple interface for third-party libraries.
>>>> * Add hooks for anything that needs to happen before indexing (such as
>>>> clearing out existing delta jobs).
>>>> * Connection pool for all Sphinx client communication, with new
>>>> connections built if there's any connection-related (as opposed to syntax)
>>>> issues.
>>>> * Multiple-field search conditions can be done with arrays of field names
>>>> as keys in the :conditions hash (Alex Dowad).
>>>> * Removed named capture in regular expressions to maintain MRI 1.8 support
>>>> (Michael Wintrant).
>>>> * Support new JDBC configuration style (Kyle Stevens).
>>>>
>>>> --
>>>> Pat
>>>>
>>>>> On 11/08/2013, at 2:39 PM, Ngan wrote:
>>>>>
>>>>> Hi Pat,
>>>>>
>>>>> Thanks for the help. I tried upgrading to 2.1.0...and I'm noticing
>>>>> multiple instance of searched running now. Is that normal? Would you be
>>>>> able to point me to documentation of major changes with 2.1.0?
>>>>>
>>>>> Thanks,
>>>>> Ngan
>>>>>
>>>>> On Friday, August 9, 2013 7:57:51 AM UTC-7, Pat Allan wrote:
>>>>> Hi Ngan
>>>>>
>>>>> In 2.1.0 there's been some patches that deal with these kinds of errors -
>>>>> TS will now retry searches if an error crops up on the client connection
>>>>> (which is also persisted per thread in a connection pool, saving socket
>>>>> setup/teardown time). If an error crops up, a new connection is made, but
>>>>> if the error persists, it'll still get raised…
>>>>>
>>>>> Give 2.1.0 a spin, see if that helps matters.
>>>>>
>>>>> --
>>>>> Pat
>>>>>
>>>>>> On 09/08/2013, at 10:46 PM, Ngan wrote:
>>>>>>
>>>>>> Forgot to add the trace:
>>>>>>
>>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:438:in `block
>>>>>> in populate'
>>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:606:in `call'
>>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:606:in
>>>>>> `retry_on_stale_index'
>>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:426:in
>>>>>> `populate'
>>>>>> gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:104:in `to_a'
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Friday, August 9, 2013 5:45:55 AM UTC-7, Ngan wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We reindex our entire index pretty often (once every 3 minutes) because
>>>>>> we have a pretty small data collection and we don't want to use delayed
>>>>>> delta. I notice however, that every once in a while, when we deploy our
>>>>>> application and it happens to be the same time the reindexing is about
>>>>>> to rotate, we'll get this error "no enabled local indexes to search"
>>>>>> every time we hit sphinx there afterwards. When this happens, we have to
>>>>>> restart our app so that it picks up the new indexes. We are reindexing
>>>>>> with rotate so the existing index should still be there and the rotation
>>>>>> should be seamless. Any ideas on why this happens? And if there's
>>>>>> anything to do about it?
>>>>>>
>>>>>> Thanks,
>>>>>> Ngan
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Thinking Sphinx" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to [email protected].
>>>>>> To post to this group, send email to [email protected].
>>>>>> Visit this group at http://groups.google.com/group/thinking-sphinx.
>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Thinking Sphinx" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>>> email to [email protected].
>>>>> To post to this group, send email to [email protected].
>>>>> Visit this group at http://groups.google.com/group/thinking-sphinx.
>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "Thinking Sphinx" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/thinking-sphinx.
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Thinking Sphinx" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/thinking-sphinx.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Thinking Sphinx" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/thinking-sphinx/rPsXVX8wlHU/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/thinking-sphinx.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Thinking Sphinx" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/thinking-sphinx.
> For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"Thinking Sphinx" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/thinking-sphinx.
For more options, visit https://groups.google.com/groups/opt_out.
