That's the code that manages the attribute updates in searchd's memory - it's all handled by the update API call to Sphinx (lines 26-28 of that file).
Cheers -- Pat On 27/04/2010, at 4:01 AM, agibralter wrote: > Also, I came across update_attribute_values in lib/thinking_sphinx/ > active_record/attribute_updates.rb... how exactly is that used? > > On Apr 26, 12:58 pm, agibralter <[email protected]> wrote: >> Hi Pat, thank you for the response. I'll try to take a stab at a >> patch. Just to be clear, when you say updates just take place in >> searchd's memory, does that mean if searchd restarts, the attributes >> will be stale until the next time the indexer is run? >> >> Also, on a separate note: do you have any thoughts on whether delayed- >> delta or datetime-delta would be more efficient in system that updates >> very often? I.e I'd like to have index updates reflected in a minute >> if possible. Are there any tradeoffs between >> >> "set_property :delta => true" with delayed-delta >> >> and >> >> "set_property :delta => :datetime, :threshold => 2.minutes" with >> datetime-delta? >> >> Thanks again! >> >> -ajg- >> >> On Apr 22, 11:07 pm, Pat Allan <[email protected]> wrote: >> >> >> >>> Hi guys >> >>> Seems there's a few topics being covered here, so it's taking a bit to grok >>> it all, but with regards to MVA attribute updating: if it's something that >>> Sphinx supports, then I'd like TS to support it natively as well. So Nick, >>> if you want to wrap your changes into a patch with specs, that'd be >>> fantastic. >> >>> Obviously, you'll want to check what version of Sphinx is being used, but >>> that's do-able via configuration.version. >> >>> Aaron, what you're wanting to is a little bit more complex, but I like your >>> approaches... maybe the more general hook is better, allowing for complex >>> situations. A patch for that would be fantastic, too. You can invoke >>> updates for Sphinx from any machine - all it actually does is update the >>> attributes within searchd's memory, not the index files - and it's all over >>> the socket anyway, so even if it did change the file, it can be remote. >> >>> Hope this helps clear a couple of things up - would love to see what you >>> guys come up with. And if there's other questions I've not answered, let me >>> know :) >> >>> -- >>> Pat >> >>> On 23/04/2010, at 2:16 AM, agibralter wrote: >> >>>> Hmm I'm not sure I follow... As far as integration with the DSL I was >>>> thinking of something like:http://gist.github.com/375411 >> >>>> But in general, delta indexes aside, if I wanted to do live updates on >>>> attributes I could create some ActiveRecord before_save callbacks that >>>> check for changed attributes and update the Sphinx indexes like so: >>>> http://gist.github.com/375427? >> >>>> Now, my question is, does ThinkingSphinx.update_index need to be >>>> called on a server running searchd? Or can it be called remotely? It >>>> seems like it just uses the Riddle client and can be called from >>>> anywhere... no need to have access to the actual index core files. >> >>>> Thanks again for the help! >> >>>> Best, >>>> Aaron >> >>>> On Apr 22, 6:10 am, Nick Sellen <[email protected]> wrote: >>>>> My extension is only a very small addition to reveal the attribute >>>>> update functionality. It would need quite a bit more work to >>>>> incorporate it into sphinx proper (someone was talking about doing it >>>>> somewhere I read). >> >>>>> It suits my needs well enough for now but it's a fairly manual process >>>>> to run it though. I added it to get around the process of doing >>>>> updates to a lot of records and avoiding this kind of process: >>>>> . add a delayed job for each record to remove the it from the main >>>>> index (well, set the "sphinx_deleted" attribute using client.update) >>>>> . sql to update each single record to delta = 1 >>>>> . and then the job to run the delta index >> >>>>> I was using Model.suspended_delta { # do stuff with Model records } >>>>> but that doesn't set the "sphinx_deleted" attribute so there was >>>>> problems having duplicate entries in the main and delta indexes. >> >>>>> Currently I use Model.update_all calls (which doesn't instantiate the >>>>> model therefore no callbacks) and manually calculated >>>>> ThinkingSphinx.update_index calls. But it's not ideal still because I >>>>> still need to read quite a lot of individual rows from the db to >>>>> update the MVA attributes (as you can't make a call to add or remove a >>>>> value from the array you need to set the whole thing which requires >>>>> fetching it first). >> >>>>> So I think I'm going to go back to running the delta sql for mass >>>>> updates going forwards and using the attribute update methods for >>>>> smaller updates. e.g.: >>>>> Model.update_all("some things I want to update",update conditions) >>>>> Model.update_all('delta = 1',update conditions) >>>>> Model.update_delta_index_and_wait # another addition I've made to >>>>> perform the delta update synchonously >> >>>>> On Apr 21, 10:52 pm, agibralter <[email protected]> wrote: >> >>>>>> Hi Nick, >> >>>>>> This looks awesome! I'm using Sphinx 0.9.9 -- so how exactly would I >>>>>> use it? Set up a callback for my models that call the method in your >>>>>> pastie when their attributes of interest get updated? Or is there a >>>>>> way to hook into ThinkingSphinx's DSL for define_index such that `has >>>>>> attribute` could take an option like :update => true? Does anyone know >>>>>> how Pat feels about incorporating this into TS proper? >> >>>>>> Also, does that update have to happen on the app server with searchd >>>>>> running? >> >>>>>> Thanks a ton! >> >>>>>> -Aaron >> >>>>>> On Apr 20, 4:14 am, Nick Sellen <[email protected]> wrote: >> >>>>>>>> multiple app servers but only one Sphinx/searchd >> >>>>>>> the caveat is "it will only work for a single searchd instance" - so >>>>>>> you should be fine to use the delayed delta with multiple app servers >>>>>>> (but it looks like you're not using delayed job anyway). >> >>>>>>> I use sphinx in a simliar way to you and the most useful information >>>>>>> in my index are the attributes (not fields) and you can actually live >>>>>>> update the index for these. I've stopped doing delta updates where >>>>>>> possible to do the live update instead. I've added a small extension >>>>>>> to thinking sphinx which makes it easier >>>>>>> runhttp://pastebin.com/KgwWREnj. >> >>>>>>> (You need sphinx 0.9.9 if you want to live update MVA attributes >>>>>>> though) >> >>>>>>> Hope this is helpful. >> >>>>>>> Nick >> >>>>>>> On Apr 19, 5:12 pm, agibralter <[email protected]> wrote: >> >>>>>>>> I use Sphinx for displaying filtered, sortable, and searchable lists >>>>>>>> of items in my web app. I use Sphinx even when there is no searching >>>>>>>> involved because ThinkingSphinx's sphinx_scopes make it very easy to >>>>>>>> chain attribute constraints together and because Sphinx is very fast. >>>>>>>> There is an obvious disadvantage though: lists are not displayed in >>>>>>>> realtime. This means I have to leverage delta indexing as much as >>>>>>>> possible... >> >>>>>>>> Because I have multiple app servers but only one Sphinx/searchd >>>>>>>> server, I cannot use the standard delta indexer. I have seen the >>>>>>>> caveat at the bottom >>>>>>>> ofhttp://freelancing-god.github.com/ts/en/deltas.html >>>>>>>> but I can't figure out a good way to send asynchronous delta update >>>>>>>> jobs to my application server with searchd on it. Has anyone figured >>>>>>>> out a good way to do this? Also, I'm not using delayed_job in my >>>>>>>> app... I'm using Workling/Starling, but thinking of moving to >>>>>>>> Resque... has anyone seen a delayed delta that uses Resque? If so, is >>>>>>>> there a way to send the "async_update_deltas" jobs to my app server >>>>>>>> with searchd? >> >>>>>>>> Anyway, as an alternative, I've been using ts-datetime-delta to keep >>>>>>>> the lists as up-to-date as possible, running the rake ts:index:delta >>>>>>>> task every 2 minutes with cron on my server with searchd. I'm thinking >>>>>>>> of moving to 1 minute to be more realtime... This seems much more >>>>>>>> frequent than any examples of ts-datetime-delta I've seen... does this >>>>>>>> sound crazy? Should I try and figure out a way to use the delayed >>>>>>>> delta instead? >> >>>>>>>> Sorry for all the questions! I appreciate any advice! >> >>>>>>>> Best, >>>>>>>> Aaron >> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Thinking Sphinx" group. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit this group >>>>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Thinking Sphinx" group. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected]. >>>>>>> For more options, visit this group >>>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Thinking Sphinx" group. >>>>>> To post to this group, send email to [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group >>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Thinking Sphinx" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group >>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Thinking Sphinx" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group >>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>> -- >>> You received this message because you are... >> >> read more ยป > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/thinking-sphinx?hl=en. > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
