On Fri, Sep 1, 2017 at 12:02 PM, Ben Edwards <[email protected]> wrote:
> Now a bit of context is probably needed. I will have quite a lot of scripts > (possibly 40, each one is a web scraper). They do very gentle scraping > (i.e. a http request every 15 seconds) so currently I run them concurrently > using cron. Out of curiousity -- what OS are you using that gives you sub-minute granularity for cron? > Doing this all within rails seems a great idea but I am worried > about the (resource) cost of doing this (initializing 40 rails environments) > will be quite high. Did some googeling and could not find out what the cost > is. Don't want to put undue stress on the server so although I like the > ideal it may be better to keep them as separate stand alone scripts outside > rails. Like I said I would rather run them inside rails. What so you > think? Depends on the server, but yes, starting 40 full rails environments every 15 seconds seems like a bad idea. At this point, you should probably be thinking in terms of background jobs. Or leaving them as standalone scripts. A lot depends on the details -- how close to exact that 15 seconds needs to be, how to handle timeouts, what you're doing with the data they're collecting, etc. But set it up, try it, see how it goes :-) -- Hassan Schroeder ------------------------ [email protected] twitter: @hassan Consulting Availability : Silicon Valley or remote -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CACmC4yC63VDO7fxYHigT9oNziHA2%2BdnvCLK11Vm8ead7pBpWrA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
