TL;DR : Do you guys know or has experience with an alternative scheduler to web2py's one that you would recommend? we use features such as scheduling tasks at a specific time, repetitions, timeouts, and check the task current state. Any further comments, advice and experience is really appreciated.
Details: We've beeng experiencing several issues with web2py's scheduler limits, our scenario currently looks something like: - 30,000+ tasks queued daily. - Around 40+ workers distributed into four different servers (somewhere between 10 workers per server): We do this because of the nature of the tasks we are distributing, we need them to be run on a specific server located somewhere in the world that has access to do what we need. - A centralized database that holds tasks+runs (PostgreSQL) - A centralized REDIS that manages the scheduler (with redis_scheduler.py module wihtin gluon/contrib/redis_scheduler.py) We previously used web2py's (default) non-redis scheduler but we found the total workers that the database supported was around ~13-15 (too few for our usage scenario, above that number of workers started to get deadlocks from the database and ticker would fail to assign tasks) so we tried using redis_scheduler which worked (with one really weird bug we found a way to overcome but have found no fix yet) for some time until we needed more workers, now we are having issues, again with the relational database that holds the tasks (scheduler_task and scheduler_run tables, database connection gets closed after sometime when we have somewhere between 50+ workers); So I can only see that we need to replace web2py's scheduler on our application to support more workers and bring back stability to our project's hunger for workers nature... Do anyone of you guys know or has experience with an alternative scheduler? I've seen several options (rabbit-mq, python-rq, mrq, so on); but I'm not sure about the limitations of those schedulers... any further comments are really appreciated. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
