Nope, no timing patterns yet.
We've since ripped open Riddle and are outputing raw debug info...
inspecting... and this is what we're getting in this scenario.
SEARCH: RIDDLE RESULT [:words]={"tour"=>{:docs=>1695, :hits=>15802}}
SEARCH: RIDDLE RESULT [:total_found]=208
SEARCH: RIDDLE RESULT [:error]=nil
SEARCH: RIDDLE RESULT={:attribute_names=>["sphinx_internal_id",
"class_crc", "sphinx_deleted", "tour_name_sort", "tour_id_sort",
"selling_company_id", "brochure_id", "brochure_variation_id"],
:matches=>[], :status=>0,
:attributes=>{"brochure_variation_id"=>1073741825, "brochure_id"=>1,
"tour_name_sort"=>3, "selling_company_id"=>1, "sphinx_deleted"=>1,
"class_crc"=>1, "tour_id_sort"=>3, "sphinx_internal_id"=>1},
:total=>208, :time=>0.002, :total_found=>208,
:words=>{"tour"=>{:docs=>1695, :hits=>15802}},
:fields=>["tour_variation_name", "tour_name", "tour_sightseeing",
"tour_intro", "tour_transportation", "tour_travels_to",
"tour_tags_cache", "tour_operating_product_code", "tour_type_name",
"places_cache", "optionals_cache", "highlights_cache", "tour_code",
"tour_id"]}
SEARCH: RIDDLE RESPONSE=#<Riddle::Client::Response:0x7f79a091bf10
@str="\000\000\000\000\000\000\000\016\000\000\000\023tour_variation_name\000\000\000\ttour_name\000\000\000\020tour_sightseeing\000\000\000\ntour_intro\000\000\000\023tour_transportation\000\000\000\017tour_travels_to\000\000\000\017tour_tags_cache\000\000\000\etour_operating_product_code\000\000\000\016tour_type_name\000\000\000\fplaces_cache\000\000\000\017optionals_cache\000\000\000\020highlights_cache\000\000\000\ttour_code\000\000\000\atour_id\000\000\000\b\000\000\000\022sphinx_internal_id\000\000\000\001\000\000\000\tclass_crc\000\000\000\001\000\000\000\016sphinx_deleted\000\000\000\001\000\000\000\016tour_name_sort\000\000\000\003\000\000\000\ftour_id_sort\000\000\000\003\000\000\000\022selling_company_id\000\000\000\001\000\000\000\vbrochure_id\000\000\000\001\000\000\000\025brochure_variation_id@\000\000\001\000\000\000\000\000\000\000\000\000\000\000?\000\000?\000\000\002\000\000\000\001\000\000\000\004tour\000\000\006\237\000\000=\272",
@marker=0>
When it does work... this looks insanely crazy/huge... so I'm going to
concur that sphinx is probably at fault here, which is discouraging.
Guess it's time to take our story to the sphinx mailing list.
Robby
On Tue, Mar 22, 2011 at 6:37 AM, Pat Allan <[email protected]> wrote:
> Any patterns to the timing of the issue? What about load on the server?
> Harder to track that, I guess...
>
> --
> Pat
>
> On 20/03/2011, at 9:41 PM, Robby Russell wrote:
>
>> Thanks for this. Been running in production for a few hours with
>> that... but still producing the same problem. *scratches head*
>>
>> Now to think of some other ways to dissect this problem.
>>
>> On Sun, Mar 20, 2011 at 3:40 PM, Pat Allan <[email protected]> wrote:
>>> Yeah, unless Sphinx is on the app server, it's best to. This should do the
>>> trick:
>>>
>>> production:
>>> version: 0.9.9
>>>
>>> That'll make sure Riddle loads its 0.9.9 additions.
>>>
>>> --
>>> Pat
>>>
>>> On 20/03/2011, at 5:35 PM, Robby Russell wrote:
>>>
>>>> We're not. Should we be?
>>>>
>>>> On Sun, Mar 20, 2011 at 3:25 PM, Pat Allan <[email protected]>
>>>> wrote:
>>>>> In a similar vein, are you setting the Sphinx version number in your
>>>>> config/sphinx.yml file, given that Sphinx isn't on your app servers?
>>>>>
>>>>> --
>>>>> Pat
>>>>>
>>>>> On 20/03/2011, at 5:22 PM, Robby Russell wrote:
>>>>>
>>>>>> I noticed that Riddle had some variant code depending on the version
>>>>>> of the sphinx server. I hadn't restarted the app servers yet, so just
>>>>>> did that and will see if that makes any difference.
>>>>>>
>>>>>> On Sun, Mar 20, 2011 at 3:16 PM, Pat Allan <[email protected]>
>>>>>> wrote:
>>>>>>> Fun indeed.
>>>>>>>
>>>>>>> Have you posted to the Sphinx forum? Given that post you referred to is
>>>>>>> three years old, perhaps there's people with a better idea of what's
>>>>>>> causing it now?
>>>>>>>
>>>>>>> And just to check - the timing of the problem doesn't match the timing
>>>>>>> of your indexing, does it?
>>>>>>>
>>>>>>> --
>>>>>>> Pat
>>>>>>>
>>>>>>> On 20/03/2011, at 4:38 PM, Robby Russell wrote:
>>>>>>>
>>>>>>>> A few hours later, it appears that we're still hitting the same
>>>>>>>> problem with 0.9.9. Fun fun!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 20, 2011 at 11:54 AM, Robby Russell
>>>>>>>> <[email protected]> wrote:
>>>>>>>>> We're adding more memory to the sphinx servers and upgrading to 0.9.9.
>>>>>>>>> Will let you know what we uncover through that process...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Mar 19, 2011 at 3:45 PM, Pat Allan
>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>> Well that's frustrating.
>>>>>>>>>>
>>>>>>>>>> I agree that the fact that it's only cropping up since you've
>>>>>>>>>> switched to TS is a fair reason to suspect TS/Riddle. Can't think
>>>>>>>>>> why it would be so unreliable (and difficult to reproduce).
>>>>>>>>>>
>>>>>>>>>> A shot in the dark, but could you try upgrading to Sphinx 0.9.9 or
>>>>>>>>>> even 1.10-beta?
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Pat
>>>>>>>>>>
>>>>>>>>>> On 19/03/2011, at 5:42 PM, Robby Russell wrote:
>>>>>>>>>>
>>>>>>>>>>> On Sat, Mar 19, 2011 at 12:33 PM, Pat Allan
>>>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>>>> To be honest, this is at the edge of my Sphinx knowledge. I'd
>>>>>>>>>>>> recommend posting a question on the Sphinx forum, talk through how
>>>>>>>>>>>> usually the search works, but sometimes doesn't - and yet results
>>>>>>>>>>>> from Sphinx say there should be results.
>>>>>>>>>>>>
>>>>>>>>>>>> In the meantime, you could add something like the following to
>>>>>>>>>>>> your controller:
>>>>>>>>>>>>
>>>>>>>>>>>> @variations = TourVariation.search 'foo'
>>>>>>>>>>>> if @variations.empty? && @variations.results[:total] > 0
>>>>>>>>>>>> @variations = TourVariation.search 'foo'
>>>>>>>>>>>> end
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Actually, we tried raising an exception, rescuing and forcing it to
>>>>>>>>>>> retry up time five times with a short delay between... and each time
>>>>>>>>>>> it was still returning zero results. When it's not cooperating... it
>>>>>>>>>>> seems like that individual request is just a lost cause.
>>>>>>>>>>>
>>>>>>>>>>> Robby
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>>>> Groups "Thinking Sphinx" group.
>>>>>>>>>>> To post to this group, send email to
>>>>>>>>>>> [email protected].
>>>>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>>>>> [email protected].
>>>>>>>>>>> For more options, visit this group at
>>>>>>>>>>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>>> Groups "Thinking Sphinx" group.
>>>>>>>>>> To post to this group, send email to
>>>>>>>>>> [email protected].
>>>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>>>> [email protected].
>>>>>>>>>> For more options, visit this group at
>>>>>>>>>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Thinking Sphinx" group.
>>>>>>>> To post to this group, send email to [email protected].
>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>> [email protected].
>>>>>>>> For more options, visit this group at
>>>>>>>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Thinking Sphinx" group.
>>>>>>> To post to this group, send email to [email protected].
>>>>>>> To unsubscribe from this group, send email to
>>>>>>> [email protected].
>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Thinking Sphinx" group.
>>>>>> To post to this group, send email to [email protected].
>>>>>> To unsubscribe from this group, send email to
>>>>>> [email protected].
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "Thinking Sphinx" group.
>>>>> To post to this group, send email to [email protected].
>>>>> To unsubscribe from this group, send email to
>>>>> [email protected].
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>>>>
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "Thinking Sphinx" group.
>>>> To post to this group, send email to [email protected].
>>>> To unsubscribe from this group, send email to
>>>> [email protected].
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Thinking Sphinx" group.
>>> To post to this group, send email to [email protected].
>>> To unsubscribe from this group, send email to
>>> [email protected].
>>> For more options, visit this group at
>>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Thinking Sphinx" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/thinking-sphinx?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Thinking Sphinx" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/thinking-sphinx?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Thinking Sphinx" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/thinking-sphinx?hl=en.
