Here's your problem in rails: Your web server is "single" threaded, so while you scrapping, its not doing anything else, so you will need more mongrels to take care of the users.
Generally you scale by having more threads, and cpu working on the problem. The database is probably not going to be your bottleneck for a while, its more the style. Why dont I train you a bit. We can do a screen share/skype session. On Apr 6, 9:27 pm, Adam Akhtar <rails-mailing-l...@andreas-s.net> wrote: > Thanks glennswest, im relatively new to rails. Whilst i think i > understood what you said can you (or anyone else) elaborate furhter on > the points below? I really appreciated your help. > > > Just thinking, your scrape should probably be in a worker, > > when you say a worker i take it you mean some temporary database? > > >Depending on what your using, you configure it to be a temp > > table even. > > Then in your search window you can do ajax based updated from the > > scrape. > > From the above do you mean whilst im scraping results from sites, when > one sites results get added to the db and i go off scraping another > sites results, i can simultaneously show the results that were just > added to the screen? > > > With the ability to then clear up the cache. > > after i get all the results and display them to the screen i can then > clear the table? > > You get more concurrency, > > Wasnt too sure what you meant by this but thats because im fresh to > rails and cant gather from the context. > > > and with > > the right javascript you could cancel the scrape in process. > > ahh so if whilst im scraping and simultaneously presenting already > scraped data from the db, the user decides to cancel the request, via > some javascript call i can terminate the outstanding scrape tasks and > move on? > > > > > Think this would scale and be more responsive > > In general how fast/slow is it to update a table with around 1000 > results? is it fast enough to handle this situation? Id prefer to stick > the objects in a temporary db because then id get to use existing > activerecord methods and mysql statements. Im just worrying about the > performance. > > -- > Posted viahttp://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk@googlegroups.com To unsubscribe from this group, send email to rubyonrails-talk+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
