Thanks, for the record the scripts are just run once a day, then only get a web page every 15 seconds, once up and running they do very little.
In Linux, so could not run a script every 15 seconds via cron. On Friday, September 1, 2017 at 9:33:19 PM UTC+1, Hassan Schroeder wrote: > > On Fri, Sep 1, 2017 at 12:02 PM, Ben Edwards <[email protected] > <javascript:>> wrote: > > > Now a bit of context is probably needed. I will have quite a lot of > scripts > > (possibly 40, each one is a web scraper). They do very gentle scraping > > (i.e. a http request every 15 seconds) so currently I run them > concurrently > > using cron. > > Out of curiousity -- what OS are you using that gives you sub-minute > granularity for cron? > > > Doing this all within rails seems a great idea but I am worried > > about the (resource) cost of doing this (initializing 40 rails > environments) > > will be quite high. Did some googeling and could not find out what the > cost > > is. Don't want to put undue stress on the server so although I like the > > ideal it may be better to keep them as separate stand alone scripts > outside > > rails. Like I said I would rather run them inside rails. What so you > > think? > > Depends on the server, but yes, starting 40 full rails environments > every 15 seconds seems like a bad idea. At this point, you should > probably be thinking in terms of background jobs. Or leaving them > as standalone scripts. > > A lot depends on the details -- how close to exact that 15 seconds > needs to be, how to handle timeouts, what you're doing with the data > they're collecting, etc. > > But set it up, try it, see how it goes :-) > > -- > Hassan Schroeder ------------------------ [email protected] > <javascript:> > twitter: @hassan > Consulting Availability : Silicon Valley or remote > -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/7677ee10-0f3d-4f53-8f8e-bcb39506dc8f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
