On Tuesday, October 30, 2018 at 7:41:09 AM UTC-7, Daniel R. wrote: > > Hello, > > I've been working on an application in Web2py for a while now and, for the > most part, I've been able to find answers to my questions online. However, > I am having trouble now trying to figure out if I can use Python > multiprocessing along with the Scheduler in Web2py in the way that I need > to do so. > > Essentially, I have an application that, upon some specified user input, > the application needs to run one or more resource-intensive calculations > (which can take upwards of an hour to complete). That being said, all the > calculations need to run in the background and the results need to be > reported back to the user. To deal with running the calculations in the > background, I have decided that I will use the Web2py built-in scheduler > and to deal with the resource-intensive aspect of the problem I am using > the Python multiprocessing Pool class so that the application can take > advantage of the multiple cores in the server that the application will be > running on. > > Starting the calculations using the Scheduler and multiprocessing has been > fairly painless. However, I lock a bunch of UI stuff so that the user can't > make changes while the tests are running, and, since the tests take so > long, another requirement is that the user has the ability to cancel all > the calculations (if they've made a mistake or have changed their mind > about waiting more time for the calculations to complete). The problem that > I'm having is that I cannot figure out how to use Web2py to cancel these > tests. > > I understand that to end a task is not complicated, and that to terminate > the worker processes using the Python multiprocessing module all I need to > do is call Pool.terminate(). The reason I'm having trouble doing this is > that I can't find any way to persistently store these Pool instances to > access them from another request. > > I have been considering that a potential solution may be using cache.ram, > if that will allow me to store a dictionary of these Pool instances. > However, I am open to any other potential solutions, like if there's > another way to get back to the correct Pool instance, if there's another > way to create a persistent global (like a Queue) that I can use to > communicate with the workers, or any other potential solutions. > > If I seem to have any misunderstandings or if I need to clarify anything > more please let me know. >
I would have looked at keeping it all in the Scheduler, but I admit that I haven't delved deeply into the multiprocessing aspects of the Scheduler, since I rarely need more than 1 worker at a time. Perhaps worker 1 can schedule worker 2 for you, etc. I can tell you that the controller that schedules your first worker knows the task number for that worker, and report it to the user to be used in the cancel URL. If you try the cascade of workers technique, then worker 1 would know the task number for worker 2, and can log that somehow for the cancel controller. The crudest way of logging would be to print it ... stdout for a worker is collected into task_run table, but you could also put it in an ordinary db table (which I suppose isn't really different from your thought of cache.ram). I think there was also a few posts a while ago about doing a progress bar on the client based on worker status. Oh, it was actually in the book, using a special string fed to print() and available in task_run, so monitoring has an example. <URL:http://web2py.com/books/default/chapter/29/04/the-core#Reporting-progress-percentages> (I don't have any additional examples of this; I do fire-and-leave, with the result sent via email.) /dps -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
