Ah, well how about that! Thanks for sharing Nick, I didn't know that. -- Pat
On 28/04/2010, at 8:51 PM, Nick Sellen wrote: >> updates just take place in searchd's memory, does that mean if searchd >> restarts, the attributes will be stale until the next time the indexer is >> run? > > thats true for MVA attributes but normal single value attributes will > be persisted to disk on a clean shutdown of searchd: > > "They are very fast because they're working fully in RAM, but they can > also be made persistent: updates are saved on disk on clean searchd > shutdown initiated by SIGTERM signal." (from > http://www.sphinxsearch.com/docs/current.html#api-func-updateatttributes) > > On Apr 28, 10:00 am, Pat Allan <[email protected]> wrote: >> On 27/04/2010, at 2:58 AM, agibralter wrote: >> >>> Hi Pat, thank you for the response. I'll try to take a stab at a >>> patch. Just to be clear, when you say updates just take place in >>> searchd's memory, does that mean if searchd restarts, the attributes >>> will be stale until the next time the indexer is run? >> >> That's correct. It's not a perfect implementation... >> >>> Also, on a separate note: do you have any thoughts on whether delayed- >>> delta or datetime-delta would be more efficient in system that updates >>> very often? I.e I'd like to have index updates reflected in a minute >>> if possible. Are there any tradeoffs between >> >>> "set_property :delta => true" with delayed-delta >> >>> and >> >>> "set_property :delta => :datetime, :threshold => 2.minutes" with >>> datetime-delta? >> >> Delayed Delta won't repeat a delta index job if there's already one there >> for that index, so it could be a bit more reliable for getting changes in >> there quickly without overloading the system. Also, maybe using >> Model.suspended_delta is worthwhile if you're doing bulk updates? >> >> Beyond that, either should do the job, really. >> >> Cheers >> >> -- >> Pat >> >> >> >> >> >>> Thanks again! >> >>> -ajg- >> >>> On Apr 22, 11:07 pm, Pat Allan <[email protected]> wrote: >>>> Hi guys >> >>>> Seems there's a few topics being covered here, so it's taking a bit to >>>> grok it all, but with regards to MVA attribute updating: if it's something >>>> that Sphinx supports, then I'd like TS to support it natively as well. So >>>> Nick, if you want to wrap your changes into a patch with specs, that'd be >>>> fantastic. >> >>>> Obviously, you'll want to check what version of Sphinx is being used, but >>>> that's do-able via configuration.version. >> >>>> Aaron, what you're wanting to is a little bit more complex, but I like >>>> your approaches... maybe the more general hook is better, allowing for >>>> complex situations. A patch for that would be fantastic, too. You can >>>> invoke updates for Sphinx from any machine - all it actually does is >>>> update the attributes within searchd's memory, not the index files - and >>>> it's all over the socket anyway, so even if it did change the file, it can >>>> be remote. >> >>>> Hope this helps clear a couple of things up - would love to see what you >>>> guys come up with. And if there's other questions I've not answered, let >>>> me know :) >> >>>> -- >>>> Pat >> >>>> On 23/04/2010, at 2:16 AM, agibralter wrote: >> >>>>> Hmm I'm not sure I follow... As far as integration with the DSL I was >>>>> thinking of something like:http://gist.github.com/375411 >> >>>>> But in general, delta indexes aside, if I wanted to do live updates on >>>>> attributes I could create some ActiveRecord before_save callbacks that >>>>> check for changed attributes and update the Sphinx indexes like so: >>>>> http://gist.github.com/375427? >> >>>>> Now, my question is, does ThinkingSphinx.update_index need to be >>>>> called on a server running searchd? Or can it be called remotely? It >>>>> seems like it just uses the Riddle client and can be called from >>>>> anywhere... no need to have access to the actual index core files. >> >>>>> Thanks again for the help! >> >>>>> Best, >>>>> Aaron >> >>>>> On Apr 22, 6:10 am, Nick Sellen <[email protected]> wrote: >>>>>> My extension is only a very small addition to reveal the attribute >>>>>> update functionality. It would need quite a bit more work to >>>>>> incorporate it into sphinx proper (someone was talking about doing it >>>>>> somewhere I read). >> >>>>>> It suits my needs well enough for now but it's a fairly manual process >>>>>> to run it though. I added it to get around the process of doing >>>>>> updates to a lot of records and avoiding this kind of process: >>>>>> . add a delayed job for each record to remove the it from the main >>>>>> index (well, set the "sphinx_deleted" attribute using client.update) >>>>>> . sql to update each single record to delta = 1 >>>>>> . and then the job to run the delta index >> >>>>>> I was using Model.suspended_delta { # do stuff with Model records } >>>>>> but that doesn't set the "sphinx_deleted" attribute so there was >>>>>> problems having duplicate entries in the main and delta indexes. >> >>>>>> Currently I use Model.update_all calls (which doesn't instantiate the >>>>>> model therefore no callbacks) and manually calculated >>>>>> ThinkingSphinx.update_index calls. But it's not ideal still because I >>>>>> still need to read quite a lot of individual rows from the db to >>>>>> update the MVA attributes (as you can't make a call to add or remove a >>>>>> value from the array you need to set the whole thing which requires >>>>>> fetching it first). >> >>>>>> So I think I'm going to go back to running the delta sql for mass >>>>>> updates going forwards and using the attribute update methods for >>>>>> smaller updates. e.g.: >>>>>> Model.update_all("some things I want to update",update conditions) >>>>>> Model.update_all('delta = 1',update conditions) >>>>>> Model.update_delta_index_and_wait # another addition I've made to >>>>>> perform the delta update synchonously >> >>>>>> On Apr 21, 10:52 pm, agibralter <[email protected]> wrote: >> >>>>>>> Hi Nick, >> >>>>>>> This looks awesome! I'm using Sphinx 0.9.9 -- so how exactly would I >>>>>>> use it? Set up a callback for my models that call the method in your >>>>>>> pastie when their attributes of interest get updated? Or is there a >>>>>>> way to hook into ThinkingSphinx's DSL for define_index such that `has >>>>>>> attribute` could take an option like :update => true? Does anyone know >>>>>>> how Pat feels about incorporating this into TS proper? >> >>>>>>> Also, does that update have to happen on the app server with searchd >>>>>>> running? >> >>>>>>> Thanks a ton! >> >>>>>>> -Aaron >> >>>>>>> On Apr 20, 4:14 am, Nick Sellen <[email protected]> wrote: >> >>>>>>>>> multiple app servers but only one Sphinx/searchd >> >>>>>>>> the caveat is "it will only work for a single searchd instance" - so >>>>>>>> you should be fine to use the delayed delta with multiple app servers >>>>>>>> (but it looks like you're not using delayed job anyway). >> >>>>>>>> I use sphinx in a simliar way to you and the most useful information >>>>>>>> in my index are the attributes (not fields) and you can actually live >>>>>>>> update the index for these. I've stopped doing delta updates where >>>>>>>> possible to do the live update instead. I've added a small extension >>>>>>>> to thinking sphinx which makes it easier >>>>>>>> runhttp://pastebin.com/KgwWREnj. >> >>>>>>>> (You need sphinx 0.9.9 if you want to live update MVA attributes >>>>>>>> though) >> >>>>>>>> Hope this is helpful. >> >>>>>>>> Nick >> >>>>>>>> On Apr 19, 5:12 pm, agibralter <[email protected]> wrote: >> >>>>>>>>> I use Sphinx for displaying filtered, sortable, and searchable lists >>>>>>>>> of items in my web app. I use Sphinx even when there is no searching >>>>>>>>> involved because ThinkingSphinx's sphinx_scopes make it very easy to >>>>>>>>> chain attribute constraints together and because Sphinx is very fast. >>>>>>>>> There is an obvious disadvantage though: lists are not displayed in >>>>>>>>> realtime. This means I have to leverage delta indexing as much as >>>>>>>>> possible... >> >>>>>>>>> Because I have multiple app servers but only one Sphinx/searchd >>>>>>>>> server, I cannot use the standard delta indexer. I have seen the >>>>>>>>> caveat at the bottom >>>>>>>>> ofhttp://freelancing-god.github.com/ts/en/deltas.html >>>>>>>>> but I can't figure out a good way to send asynchronous delta update >>>>>>>>> jobs to my application server with searchd on it. Has anyone figured >>>>>>>>> out a good way to do this? Also, I'm not using delayed_job in my >>>>>>>>> app... I'm using Workling/Starling, but thinking of moving to >>>>>>>>> Resque... has anyone seen a delayed delta that uses Resque? If so, is >>>>>>>>> there a way to send the "async_update_deltas" jobs to my app server >>>>>>>>> with searchd? >> >>>>>>>>> Anyway, as an alternative, I've been using ts-datetime-delta to keep >>>>>>>>> the lists as up-to-date as possible, running the rake ts:index:delta >>>>>>>>> task every 2 minutes with cron on my server with searchd. I'm thinking >>>>>>>>> of moving to 1 minute to be more realtime... This seems much more >>>>>>>>> frequent than any examples of ts-datetime-delta I've seen... does this >>>>>>>>> sound crazy? Should I try and figure out a way to use the delayed >>>>>>>>> delta instead? >> >>>>>>>>> Sorry for all the questions! I appreciate any advice! >> >>>>>>>>> Best, >>>>>>>>> Aaron >> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Thinking Sphinx" group. >>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>> [email protected]. >>>>>>>>> For more options, visit this group >>>>>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Thinking Sphinx" group. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit this group >>>>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Thinking Sphinx" group. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected]. >>>>>>> For more options, visit this group >>>>>>> athttp://groups.google.com/group/thinking-sphinx?hl=en. >> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Thinking Sphinx" group. >>>>>> To post to this group, send email to [email protected]. >>>>>> To unsubscribe from this group, send >> >> ... >> >> read more ยป > > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/thinking-sphinx?hl=en. > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/thinking-sphinx?hl=en.
