On Sun, Apr 5, 2009 at 10:31 AM, Adam Akhtar <rails-mailing-l...@andreas-s.net> wrote: > > Hi thanks for your replies. > > My main concern is performance. The data is not scraped beforehand in > advance, its scraped on demand by my users. They submit a search query > whch i then perform on several site, scrape their results and aggregate > them for the user. My site is basically a meta search engine. > > Storing results in a db > > pros: i get to use msql find conditions when the user wants to filter > the results even more. > > cons: ill only be temporarily storng these results. As soon as the user > does a new search there gone forever. I dont know the peformance hit of > storing a 1000 results in a db consisting of several fields. Is a db > still a wise choice? > > Using YAML: > > pros: not sure, but hey, i like using it! > cons: no msql conditions so id have to create my own methods > > does the above change anything? > > > > -- > Posted via http://www.ruby-forum.com/. > > > >
The benefit of YAML is that once you've scraped the data, you probably already have a structure in place which can easily be saved and restored. You could combine the two by storing the YAML in the database. >From a performance perspective, consider caching the results of the scraping for at least some period of time so that you don't have to scrape on every search (unless the source websites change VERY frequently) Andrew Timberlake http://ramblingsonrails.com http://www.linkedin.com/in/andrewtimberlake "I have never let my schooling interfere with my education" - Mark Twain --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk@googlegroups.com To unsubscribe from this group, send email to rubyonrails-talk+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
