Thanks Niphold for the answer... Do you see other usage of the queing stack proposed by Bruno other then emails shippement... I have a couple of CRON job that I would eventually include in web2py to make thing more managable when server update for instance or new deployment. But at the date of creation of these cronjob, the web2py cronjob were just created so I stick with unix cron job... Then I start read about Scheduler and now this new approach... So, just want to figure where I will go and what will be the "better" approach if there is one...
Thanks Richard On Fri, Apr 19, 2013 at 9:37 AM, Niphlod <niph...@gmail.com> wrote: > python-rq is more "engineered" towards speedier "I queued it and .. voilà > is executed" kind of environments. > > Scheduler needs to "balance" between that and "please run this every hour" > scenarios (periodic) ... additionally leaving your db alone as much as > possible ^_^ > > python-rq has an external "addon" to be able to schedule periodic > function, that needs to be active to do the work (move periodic to the > actual "please execute me now" queue) > > In any case, for a job that needs to be executed once or twice a week (and > probably takes less than 3 minutes), I'd go for a standard cron: having an > external process to monitor continously your queue for doing nothing for > the most part of the week (plus the eventual worker processes standing by > for python-rq) is frankly a waste of resources, with one or the other > system ^_^ > > > On Friday, April 19, 2013 3:27:48 PM UTC+2, Richard wrote: > >> Hello, >> >> First, thanks Bruno to share your experiments with python-rq, redis and >> queing stuff... >> >> I would like to know if this is the way to go over Scheduler for emails >> shippement (around 200 emails each time once or twice a weeks)? >> >> Thanks >> >> Richard >> >> >> On Fri, Apr 19, 2013 at 6:23 AM, Ricardo Pedroso <rmdpe...@gmail.com>wrote: >> >>> On Thu, Apr 18, 2013 at 10:02 PM, Niphlod <nip...@gmail.com> wrote: >>> > PS: on the web2py process side, it seems that at most 10 connections >>> are >>> > istantiated with Redis: is there a connection pool? >>> >>> I guess, because you are running the web server with 10 threads. >>> python redis by default use a connection pool... see below >>> >>> > On the worker side, instead, I noticed that each job generates a new >>> fresh >>> > connection to Redis: is this expected ? >>> >>> I guess because, before starts a work, RQ forks a process. >>> >>> >>> >>> This is how understand it, I may be wrong: >>> >>> If a request for a connection is in a different process a new empty >>> connection pool is created. >>> >>> If the request for a connection is in the same process but none >>> connections >>> is available python redis starts another one. >>> >>> If someone knows more details, please share. >>> >>> This is the relevant bits I guess, on how python redis manage the >>> connections: >>> >>> def _checkpid(self): >>> if self.pid != os.getpid(): >>> self.disconnect() >>> self.__init__(self.connection_**class, self.max_connections, >>> **self.connection_kwargs) >>> >>> def get_connection(self, command_name, *keys, **options): >>> "Get a connection from the pool" >>> self._checkpid() >>> try: >>> connection = self._available_connections.**pop() >>> except IndexError: >>> connection = self.make_connection() >>> self._in_use_connections.add(**connection) >>> return connection >>> >>> def disconnect(self): >>> "Disconnects all connections in the pool" >>> all_conns = chain(self._available_**connections, >>> self._in_use_connections) >>> for connection in all_conns: >>> connection.disconnect() >>> >>> Ricardo >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "web2py-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to web2py+un...@**googlegroups.com. >>> >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >>> >>> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "web2py-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to web2py+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
