Hi guys Seems there's a few topics being covered here, so it's taking a bit to grok it all, but with regards to MVA attribute updating: if it's something that Sphinx supports, then I'd like TS to support it natively as well. So Nick, if you want to wrap your changes into a patch with specs, that'd be fantastic.
Obviously, you'll want to check what version of Sphinx is being used, but that's do-able via configuration.version. Aaron, what you're wanting to is a little bit more complex, but I like your approaches... maybe the more general hook is better, allowing for complex situations. A patch for that would be fantastic, too. You can invoke updates for Sphinx from any machine - all it actually does is update the attributes within searchd's memory, not the index files - and it's all over the socket anyway, so even if it did change the file, it can be remote. Hope this helps clear a couple of things up - would love to see what you guys come up with. And if there's other questions I've not answered, let me know :) -- Pat On 23/04/2010, at 2:16 AM, agibralter wrote: > Hmm I'm not sure I follow... As far as integration with the DSL I was > thinking of something like: http://gist.github.com/375411 > > But in general, delta indexes aside, if I wanted to do live updates on > attributes I could create some ActiveRecord before_save callbacks that > check for changed attributes and update the Sphinx indexes like so: > http://gist.github.com/375427 ? > > Now, my question is, does ThinkingSphinx.update_index need to be > called on a server running searchd? Or can it be called remotely? It > seems like it just uses the Riddle client and can be called from > anywhere... no need to have access to the actual index core files. > > Thanks again for the help! > > Best, > Aaron > > On Apr 22, 6:10 am, Nick Sellen <[email protected]> wrote: >> My extension is only a very small addition to reveal the attribute >> update functionality. It would need quite a bit more work to >> incorporate it into sphinx proper (someone was talking about doing it >> somewhere I read). >> >> It suits my needs well enough for now but it's a fairly manual process >> to run it though. I added it to get around the process of doing >> updates to a lot of records and avoiding this kind of process: >> . add a delayed job for each record to remove the it from the main >> index (well, set the "sphinx_deleted" attribute using client.update) >> . sql to update each single record to delta = 1 >> . and then the job to run the delta index >> >> I was using Model.suspended_delta { # do stuff with Model records } >> but that doesn't set the "sphinx_deleted" attribute so there was >> problems having duplicate entries in the main and delta indexes. >> >> Currently I use Model.update_all calls (which doesn't instantiate the >> model therefore no callbacks) and manually calculated >> ThinkingSphinx.update_index calls. But it's not ideal still because I >> still need to read quite a lot of individual rows from the db to >> update the MVA attributes (as you can't make a call to add or remove a >> value from the array you need to set the whole thing which requires >> fetching it first). >> >> So I think I'm going to go back to running the delta sql for mass >> updates going forwards and using the attribute update methods for >> smaller updates. e.g.: >> Model.update_all("some things I want to update",update conditions) >> Model.update_all('delta = 1',update conditions) >> Model.update_delta_index_and_wait # another addition I've made to >> perform the delta update synchonously >> >> On Apr 21, 10:52 pm, agibralter <[email protected]> wrote: >> >> >> >> >> >>> Hi Nick, >> >>> This looks awesome! I'm using Sphinx 0.9.9 -- so how exactly would I >>> use it? Set up a callback for my models that call the method in your >>> pastie when their attributes of interest get updated? Or is there a >>> way to hook into ThinkingSphinx's DSL for define_index such that `has >>> attribute` could take an option like :update => true? Does anyone know >>> how Pat feels about incorporating this into TS proper? >> >>> Also, does that update have to happen on the app server with searchd >>> running? >> >>> Thanks a ton! >> >>> -Aaron >> >>> On Apr 20, 4:14 am, Nick Sellen <[email protected]> wrote: >> >>>>> multiple app servers but only one Sphinx/searchd >> >>>> the caveat is "it will only work for a single searchd instance" - so >>>> you should be fine to use the delayed delta with multiple app servers >>>> (but it looks like you're not using delayed job anyway). >> >>>> I use sphinx in a simliar way to you and the most useful information >>>> in my index are the attributes (not fields) and you can actually live >>>> update the index for these. I've stopped doing delta updates where >>>> possible to do the live update instead. I've added a small extension >>>> to thinking sphinx which makes it easier runhttp://pastebin.com/KgwWREnj. >> >>>> (You need sphinx 0.9.9 if you want to live update MVA attributes >>>> though) >> >>>> Hope this is helpful. >> >>>> Nick >> >>>> On Apr 19, 5:12 pm, agibralter <[email protected]> wrote: >> >>>>> I use Sphinx for displaying filtered, sortable, and searchable lists >>>>> of items in my web app. I use Sphinx even when there is no searching >>>>> involved because ThinkingSphinx's sphinx_scopes make it very easy to >>>>> chain attribute constraints together and because Sphinx is very fast. >>>>> There is an obvious disadvantage though: lists are not displayed in >>>>> realtime. This means I have to leverage delta indexing as much as >>>>> possible... >> >>>>> Because I have multiple app servers but only one Sphinx/searchd >>>>> server, I cannot use the standard delta indexer. I have seen the >>>>> caveat at the bottom ofhttp://freelancing-god.github.com/ts/en/deltas.html >>>>> but I can't figure out a good way to send asynchronous delta update >>>>> jobs to my application server with searchd on it. Has anyone figured >>>>> out a good way to do this? Also, I'm not using delayed_job in my >>>>> app... I'm using Workling/Starling, but thinking of moving to >>>>> Resque... has anyone seen a delayed delta that uses Resque? If so, is >>>>> there a way to send the "async_update_deltas" jobs to my app server >>>>> with searchd? >> >>>>> Anyway, as an alternative, I've been using ts-datetime-delta to keep >>>>> the lists as up-to-date as possible, running the rake ts:index:delta >>>>> task every 2 minutes with cron on my server with searchd. I'm thinking >>>>> of moving to 1 minute to be more realtime... This seems much more >>>>> frequent than any examples of ts-datetime-delta I've seen... does this >>>>> sound crazy? Should I try and figure out a way to use the delayed >>>>> delta instead? >> >>>>> Sorry for all the questions! I appreciate any advice! >> >>>>> Best, >>>>> Aaron >> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Thinking Sphinx" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group >>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Thinking Sphinx" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group >>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Thinking Sphinx" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group >>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Thinking Sphinx" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group >> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/thinking-sphinx?hl=en. > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
