> updates just take place in searchd's memory, does that mean if searchd > restarts, the attributes will be stale until the next time the indexer is run?
thats true for MVA attributes but normal single value attributes will be persisted to disk on a clean shutdown of searchd: "They are very fast because they're working fully in RAM, but they can also be made persistent: updates are saved on disk on clean searchd shutdown initiated by SIGTERM signal." (from http://www.sphinxsearch.com/docs/current.html#api-func-updateatttributes) On Apr 28, 10:00 am, Pat Allan <[email protected]> wrote: > On 27/04/2010, at 2:58 AM, agibralter wrote: > > > Hi Pat, thank you for the response. I'll try to take a stab at a > > patch. Just to be clear, when you say updates just take place in > > searchd's memory, does that mean if searchd restarts, the attributes > > will be stale until the next time the indexer is run? > > That's correct. It's not a perfect implementation... > > > Also, on a separate note: do you have any thoughts on whether delayed- > > delta or datetime-delta would be more efficient in system that updates > > very often? I.e I'd like to have index updates reflected in a minute > > if possible. Are there any tradeoffs between > > > "set_property :delta => true" with delayed-delta > > > and > > > "set_property :delta => :datetime, :threshold => 2.minutes" with > > datetime-delta? > > Delayed Delta won't repeat a delta index job if there's already one there for > that index, so it could be a bit more reliable for getting changes in there > quickly without overloading the system. Also, maybe using > Model.suspended_delta is worthwhile if you're doing bulk updates? > > Beyond that, either should do the job, really. > > Cheers > > -- > Pat > > > > > > > Thanks again! > > > -ajg- > > > On Apr 22, 11:07 pm, Pat Allan <[email protected]> wrote: > >> Hi guys > > >> Seems there's a few topics being covered here, so it's taking a bit to > >> grok it all, but with regards to MVA attribute updating: if it's something > >> that Sphinx supports, then I'd like TS to support it natively as well. So > >> Nick, if you want to wrap your changes into a patch with specs, that'd be > >> fantastic. > > >> Obviously, you'll want to check what version of Sphinx is being used, but > >> that's do-able via configuration.version. > > >> Aaron, what you're wanting to is a little bit more complex, but I like > >> your approaches... maybe the more general hook is better, allowing for > >> complex situations. A patch for that would be fantastic, too. You can > >> invoke updates for Sphinx from any machine - all it actually does is > >> update the attributes within searchd's memory, not the index files - and > >> it's all over the socket anyway, so even if it did change the file, it can > >> be remote. > > >> Hope this helps clear a couple of things up - would love to see what you > >> guys come up with. And if there's other questions I've not answered, let > >> me know :) > > >> -- > >> Pat > > >> On 23/04/2010, at 2:16 AM, agibralter wrote: > > >>> Hmm I'm not sure I follow... As far as integration with the DSL I was > >>> thinking of something like:http://gist.github.com/375411 > > >>> But in general, delta indexes aside, if I wanted to do live updates on > >>> attributes I could create some ActiveRecord before_save callbacks that > >>> check for changed attributes and update the Sphinx indexes like so: > >>>http://gist.github.com/375427? > > >>> Now, my question is, does ThinkingSphinx.update_index need to be > >>> called on a server running searchd? Or can it be called remotely? It > >>> seems like it just uses the Riddle client and can be called from > >>> anywhere... no need to have access to the actual index core files. > > >>> Thanks again for the help! > > >>> Best, > >>> Aaron > > >>> On Apr 22, 6:10 am, Nick Sellen <[email protected]> wrote: > >>>> My extension is only a very small addition to reveal the attribute > >>>> update functionality. It would need quite a bit more work to > >>>> incorporate it into sphinx proper (someone was talking about doing it > >>>> somewhere I read). > > >>>> It suits my needs well enough for now but it's a fairly manual process > >>>> to run it though. I added it to get around the process of doing > >>>> updates to a lot of records and avoiding this kind of process: > >>>> . add a delayed job for each record to remove the it from the main > >>>> index (well, set the "sphinx_deleted" attribute using client.update) > >>>> . sql to update each single record to delta = 1 > >>>> . and then the job to run the delta index > > >>>> I was using Model.suspended_delta { # do stuff with Model records } > >>>> but that doesn't set the "sphinx_deleted" attribute so there was > >>>> problems having duplicate entries in the main and delta indexes. > > >>>> Currently I use Model.update_all calls (which doesn't instantiate the > >>>> model therefore no callbacks) and manually calculated > >>>> ThinkingSphinx.update_index calls. But it's not ideal still because I > >>>> still need to read quite a lot of individual rows from the db to > >>>> update the MVA attributes (as you can't make a call to add or remove a > >>>> value from the array you need to set the whole thing which requires > >>>> fetching it first). > > >>>> So I think I'm going to go back to running the delta sql for mass > >>>> updates going forwards and using the attribute update methods for > >>>> smaller updates. e.g.: > >>>> Model.update_all("some things I want to update",update conditions) > >>>> Model.update_all('delta = 1',update conditions) > >>>> Model.update_delta_index_and_wait # another addition I've made to > >>>> perform the delta update synchonously > > >>>> On Apr 21, 10:52 pm, agibralter <[email protected]> wrote: > > >>>>> Hi Nick, > > >>>>> This looks awesome! I'm using Sphinx 0.9.9 -- so how exactly would I > >>>>> use it? Set up a callback for my models that call the method in your > >>>>> pastie when their attributes of interest get updated? Or is there a > >>>>> way to hook into ThinkingSphinx's DSL for define_index such that `has > >>>>> attribute` could take an option like :update => true? Does anyone know > >>>>> how Pat feels about incorporating this into TS proper? > > >>>>> Also, does that update have to happen on the app server with searchd > >>>>> running? > > >>>>> Thanks a ton! > > >>>>> -Aaron > > >>>>> On Apr 20, 4:14 am, Nick Sellen <[email protected]> wrote: > > >>>>>>> multiple app servers but only one Sphinx/searchd > > >>>>>> the caveat is "it will only work for a single searchd instance" - so > >>>>>> you should be fine to use the delayed delta with multiple app servers > >>>>>> (but it looks like you're not using delayed job anyway). > > >>>>>> I use sphinx in a simliar way to you and the most useful information > >>>>>> in my index are the attributes (not fields) and you can actually live > >>>>>> update the index for these. I've stopped doing delta updates where > >>>>>> possible to do the live update instead. I've added a small extension > >>>>>> to thinking sphinx which makes it easier > >>>>>> runhttp://pastebin.com/KgwWREnj. > > >>>>>> (You need sphinx 0.9.9 if you want to live update MVA attributes > >>>>>> though) > > >>>>>> Hope this is helpful. > > >>>>>> Nick > > >>>>>> On Apr 19, 5:12 pm, agibralter <[email protected]> wrote: > > >>>>>>> I use Sphinx for displaying filtered, sortable, and searchable lists > >>>>>>> of items in my web app. I use Sphinx even when there is no searching > >>>>>>> involved because ThinkingSphinx's sphinx_scopes make it very easy to > >>>>>>> chain attribute constraints together and because Sphinx is very fast. > >>>>>>> There is an obvious disadvantage though: lists are not displayed in > >>>>>>> realtime. This means I have to leverage delta indexing as much as > >>>>>>> possible... > > >>>>>>> Because I have multiple app servers but only one Sphinx/searchd > >>>>>>> server, I cannot use the standard delta indexer. I have seen the > >>>>>>> caveat at the bottom > >>>>>>> ofhttp://freelancing-god.github.com/ts/en/deltas.html > >>>>>>> but I can't figure out a good way to send asynchronous delta update > >>>>>>> jobs to my application server with searchd on it. Has anyone figured > >>>>>>> out a good way to do this? Also, I'm not using delayed_job in my > >>>>>>> app... I'm using Workling/Starling, but thinking of moving to > >>>>>>> Resque... has anyone seen a delayed delta that uses Resque? If so, is > >>>>>>> there a way to send the "async_update_deltas" jobs to my app server > >>>>>>> with searchd? > > >>>>>>> Anyway, as an alternative, I've been using ts-datetime-delta to keep > >>>>>>> the lists as up-to-date as possible, running the rake ts:index:delta > >>>>>>> task every 2 minutes with cron on my server with searchd. I'm thinking > >>>>>>> of moving to 1 minute to be more realtime... This seems much more > >>>>>>> frequent than any examples of ts-datetime-delta I've seen... does this > >>>>>>> sound crazy? Should I try and figure out a way to use the delayed > >>>>>>> delta instead? > > >>>>>>> Sorry for all the questions! I appreciate any advice! > > >>>>>>> Best, > >>>>>>> Aaron > > >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Groups "Thinking Sphinx" group. > >>>>>>> To post to this group, send email to [email protected]. > >>>>>>> To unsubscribe from this group, send email to > >>>>>>> [email protected]. > >>>>>>> For more options, visit this group > >>>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google > >>>>>> Groups "Thinking Sphinx" group. > >>>>>> To post to this group, send email to [email protected]. > >>>>>> To unsubscribe from this group, send email to > >>>>>> [email protected]. > >>>>>> For more options, visit this group > >>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups "Thinking Sphinx" group. > >>>>> To post to this group, send email to [email protected]. > >>>>> To unsubscribe from this group, send email to > >>>>> [email protected]. > >>>>> For more options, visit this group > >>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. > > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Thinking Sphinx" group. > >>>> To post to this group, send email to [email protected]. > >>>> To unsubscribe from this group, send > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
