Hi Pat, thank you for the response. I'll try to take a stab at a patch. Just to be clear, when you say updates just take place in searchd's memory, does that mean if searchd restarts, the attributes will be stale until the next time the indexer is run?
Also, on a separate note: do you have any thoughts on whether delayed- delta or datetime-delta would be more efficient in system that updates very often? I.e I'd like to have index updates reflected in a minute if possible. Are there any tradeoffs between "set_property :delta => true" with delayed-delta and "set_property :delta => :datetime, :threshold => 2.minutes" with datetime-delta? Thanks again! -ajg- On Apr 22, 11:07 pm, Pat Allan <[email protected]> wrote: > Hi guys > > Seems there's a few topics being covered here, so it's taking a bit to grok > it all, but with regards to MVA attribute updating: if it's something that > Sphinx supports, then I'd like TS to support it natively as well. So Nick, if > you want to wrap your changes into a patch with specs, that'd be fantastic. > > Obviously, you'll want to check what version of Sphinx is being used, but > that's do-able via configuration.version. > > Aaron, what you're wanting to is a little bit more complex, but I like your > approaches... maybe the more general hook is better, allowing for complex > situations. A patch for that would be fantastic, too. You can invoke updates > for Sphinx from any machine - all it actually does is update the attributes > within searchd's memory, not the index files - and it's all over the socket > anyway, so even if it did change the file, it can be remote. > > Hope this helps clear a couple of things up - would love to see what you guys > come up with. And if there's other questions I've not answered, let me know :) > > -- > Pat > > On 23/04/2010, at 2:16 AM, agibralter wrote: > > > > > > > Hmm I'm not sure I follow... As far as integration with the DSL I was > > thinking of something like:http://gist.github.com/375411 > > > But in general, delta indexes aside, if I wanted to do live updates on > > attributes I could create some ActiveRecord before_save callbacks that > > check for changed attributes and update the Sphinx indexes like so: > >http://gist.github.com/375427? > > > Now, my question is, does ThinkingSphinx.update_index need to be > > called on a server running searchd? Or can it be called remotely? It > > seems like it just uses the Riddle client and can be called from > > anywhere... no need to have access to the actual index core files. > > > Thanks again for the help! > > > Best, > > Aaron > > > On Apr 22, 6:10 am, Nick Sellen <[email protected]> wrote: > >> My extension is only a very small addition to reveal the attribute > >> update functionality. It would need quite a bit more work to > >> incorporate it into sphinx proper (someone was talking about doing it > >> somewhere I read). > > >> It suits my needs well enough for now but it's a fairly manual process > >> to run it though. I added it to get around the process of doing > >> updates to a lot of records and avoiding this kind of process: > >> . add a delayed job for each record to remove the it from the main > >> index (well, set the "sphinx_deleted" attribute using client.update) > >> . sql to update each single record to delta = 1 > >> . and then the job to run the delta index > > >> I was using Model.suspended_delta { # do stuff with Model records } > >> but that doesn't set the "sphinx_deleted" attribute so there was > >> problems having duplicate entries in the main and delta indexes. > > >> Currently I use Model.update_all calls (which doesn't instantiate the > >> model therefore no callbacks) and manually calculated > >> ThinkingSphinx.update_index calls. But it's not ideal still because I > >> still need to read quite a lot of individual rows from the db to > >> update the MVA attributes (as you can't make a call to add or remove a > >> value from the array you need to set the whole thing which requires > >> fetching it first). > > >> So I think I'm going to go back to running the delta sql for mass > >> updates going forwards and using the attribute update methods for > >> smaller updates. e.g.: > >> Model.update_all("some things I want to update",update conditions) > >> Model.update_all('delta = 1',update conditions) > >> Model.update_delta_index_and_wait # another addition I've made to > >> perform the delta update synchonously > > >> On Apr 21, 10:52 pm, agibralter <[email protected]> wrote: > > >>> Hi Nick, > > >>> This looks awesome! I'm using Sphinx 0.9.9 -- so how exactly would I > >>> use it? Set up a callback for my models that call the method in your > >>> pastie when their attributes of interest get updated? Or is there a > >>> way to hook into ThinkingSphinx's DSL for define_index such that `has > >>> attribute` could take an option like :update => true? Does anyone know > >>> how Pat feels about incorporating this into TS proper? > > >>> Also, does that update have to happen on the app server with searchd > >>> running? > > >>> Thanks a ton! > > >>> -Aaron > > >>> On Apr 20, 4:14 am, Nick Sellen <[email protected]> wrote: > > >>>>> multiple app servers but only one Sphinx/searchd > > >>>> the caveat is "it will only work for a single searchd instance" - so > >>>> you should be fine to use the delayed delta with multiple app servers > >>>> (but it looks like you're not using delayed job anyway). > > >>>> I use sphinx in a simliar way to you and the most useful information > >>>> in my index are the attributes (not fields) and you can actually live > >>>> update the index for these. I've stopped doing delta updates where > >>>> possible to do the live update instead. I've added a small extension > >>>> to thinking sphinx which makes it easier runhttp://pastebin.com/KgwWREnj. > > >>>> (You need sphinx 0.9.9 if you want to live update MVA attributes > >>>> though) > > >>>> Hope this is helpful. > > >>>> Nick > > >>>> On Apr 19, 5:12 pm, agibralter <[email protected]> wrote: > > >>>>> I use Sphinx for displaying filtered, sortable, and searchable lists > >>>>> of items in my web app. I use Sphinx even when there is no searching > >>>>> involved because ThinkingSphinx's sphinx_scopes make it very easy to > >>>>> chain attribute constraints together and because Sphinx is very fast. > >>>>> There is an obvious disadvantage though: lists are not displayed in > >>>>> realtime. This means I have to leverage delta indexing as much as > >>>>> possible... > > >>>>> Because I have multiple app servers but only one Sphinx/searchd > >>>>> server, I cannot use the standard delta indexer. I have seen the > >>>>> caveat at the bottom > >>>>> ofhttp://freelancing-god.github.com/ts/en/deltas.html > >>>>> but I can't figure out a good way to send asynchronous delta update > >>>>> jobs to my application server with searchd on it. Has anyone figured > >>>>> out a good way to do this? Also, I'm not using delayed_job in my > >>>>> app... I'm using Workling/Starling, but thinking of moving to > >>>>> Resque... has anyone seen a delayed delta that uses Resque? If so, is > >>>>> there a way to send the "async_update_deltas" jobs to my app server > >>>>> with searchd? > > >>>>> Anyway, as an alternative, I've been using ts-datetime-delta to keep > >>>>> the lists as up-to-date as possible, running the rake ts:index:delta > >>>>> task every 2 minutes with cron on my server with searchd. I'm thinking > >>>>> of moving to 1 minute to be more realtime... This seems much more > >>>>> frequent than any examples of ts-datetime-delta I've seen... does this > >>>>> sound crazy? Should I try and figure out a way to use the delayed > >>>>> delta instead? > > >>>>> Sorry for all the questions! I appreciate any advice! > > >>>>> Best, > >>>>> Aaron > > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups "Thinking Sphinx" group. > >>>>> To post to this group, send email to [email protected]. > >>>>> To unsubscribe from this group, send email to > >>>>> [email protected]. > >>>>> For more options, visit this group > >>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Thinking Sphinx" group. > >>>> To post to this group, send email to [email protected]. > >>>> To unsubscribe from this group, send email to > >>>> [email protected]. > >>>> For more options, visit this group > >>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > >>> -- > >>> You received this message because you are subscribed to the Google Groups > >>> "Thinking Sphinx" group. > >>> To post to this group, send email to [email protected]. > >>> To unsubscribe from this group, send email to > >>> [email protected]. > >>> For more options, visit this group > >>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Thinking Sphinx" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]. > >> For more options, visit this group > >> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Thinking Sphinx" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/thinking-sphinx?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/thinking-sphinx?hl=en. -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
