Re: Prefork vs threaded when I want to spawn my own threads
On Wed, Sep 4, 2013 at 2:36 PM, Javier Guerra Giraldez wrote: > above that, Redis is a clean, fast and very scalable platform for > queues. if you don't have heavy buzzword dependency, i wouldn't > bother checking AMQP systems and stay with Redis. > A thousand times this. You don't need AMQP systems until you need them. -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
Re: Prefork vs threaded when I want to spawn my own threads
On Wed, Sep 4, 2013 at 12:28 PM, wrote: > What were people using back in PHP days, cron all the way? MQs have existed for a long time; but weren't so popular. I think the Twitter problems were the wake-up call for most people. in part because ghetto queues do work, and work well up to a point. when you grow past its capacity it gets ugly, but below that it's a real solution. As first estimate, on anything that gets less than a hundred tasks a minute, I wouldn't feel bad about using them. above that, Redis is a clean, fast and very scalable platform for queues. if you don't have heavy buzzword dependency, i wouldn't bother checking AMQP systems and stay with Redis. -- Javier -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
Re: Prefork vs threaded when I want to spawn my own threads
On 3 september 2013 15:25:42 UTC+2 Russell Keith-Magee wrote: > > > On Tue, Sep 3, 2013 at 9:50 PM, > wrote: > >> In my Django application I need to schedule some tasks to run in future. >> Celery is not an option here because some crucial options - mainly, the >> ability to revoke() a task - are only available with fully-featured MQ >> backend and I only have a limited shared hosting where I can't install this >> kind of software. After some reading, I've found that supposedly "right" >> way to do that in Python is to spawn a subprocess and call sleep(). >> However, spawning a process does have an overhead, especially considering a >> rather simple nature of tasks I want to defer (retrieval and conditional >> deletion of one object). >> >> Then I decided to use threads instead. They have little to no overhead, >> share memory and Python has a nice little threading.Timer class which is >> exactly what I need. My code looks like this: >> >> http://pastebin.com/K6RukAX9 >> >> Here's when my questions come in. I admit my knowledge of threading is >> somewhat low and Python operates on high-level too, so nuances are not >> easily noticeable. From what I've learned, threads (can?) share memory with >> main thread/process. My primary concern here is that, as far as I >> understand, method=prefork (the default) creates a pool of processes >> waiting for work. It seems possible that it may impact my scheduler >> somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown >> ends. >> >> On the other hand, threaded mode is supposedly more efficient because, >> being free from overhead of process forking, it can create child threads at >> will. In my mind this raises a similar concern to above: what if framework >> decides to destroy a thread that happens to be the one where Timer has been >> previously spawned? >> >> It is entirely possible that my concerns are completely baseless and both >> methods have nothing to do with my scheduler. I'm asking for input from >> smarter people than me :) >> > > Lets back up a bit here. > > Web servers aren't designed to do the sort of thing you're describing. Web > requests are not designed to be long lived, and web servers are tuned with > this in mind. If you try to make them long lived, you're going to hit all > sorts of problems with thread/process lifecycle management. > > The solution here isn't to try and make a web server do something it > wasn't designed to do. The solution is to work out how to get a task queue > working on your hardware setup. > > You say you can't install a fully-fledged MQ service. Ok - that's fine -- > but don't throw out the baby with the bathwater. There are other ways to > back Celery -- if need be, you can use a database as your data store. > > Or, you could consider using an entirely different task queue. If your > service provider will give you access to Redis, look into RQ. It's > lightweight, but > > Or, if neither of those are viable, you can do a "ghetto queue" -- a > combination of cron and a database can be used to implement task queue-like > behaviour, without any need for celery. > > It's difficult to give more specific advice without knowing the specifics > of your problem domain and your deployment options -- but I implore you -- > don't keep down the path you're on. That way leads to madness :-) > > Yours, > Russ Magee %-) > I get the overall message. I would use threads because that method looks simple enough, but if it isn't - I won't insist on it. Regarding Celery, I was specifically referring to revoke() method, it's absolutely crucial for me to be able to revoke a scheduled task. Internet says it's only supported with AMQP and Redis ( http://stackoverflow.com/questions/7415657/celery-tasks-dont-get-revoked ) though now that I look at it, docs also claim compatibility with MongoDB. However, I didn't managed to get it to work on such a setup and complete lack of error messages didn't help (the call just fails silently). I don't have Redis on my host, but I guess a free 20 MB instance on a cloud will be enough for my needs, so I'll settle with one of these small-ish queuers. On a side note, I am a bit surprised that every sane method to do scheduled task execution depends on MQs or Redis. Those are fairly new tools while the problem definitely didn't appear yesterday. What were people using back in PHP days, cron all the way? Thank you very much for your remarks :) -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
Re: Prefork vs threaded when I want to spawn my own threads
On Tue, Sep 3, 2013 at 9:28 AM, Dig wrote: > My django application is host in uwsgi, which is support timer(interval). So > I insert my task parameters into database and execute tasks in timer > handler. note that uwsgi also has a spooler facility... -- Javier -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
Re: Prefork vs threaded when I want to spawn my own threads
I have a similar need. And I implement this feature with uwsgi. My django application is host in uwsgi, which is support timer(interval). So I insert my task parameters into database and execute tasks in timer handler. Regards, Dig On Sep 3, 2013 9:26 PM, "Russell Keith-Magee" wrote: > > On Tue, Sep 3, 2013 at 9:50 PM, wrote: > >> In my Django application I need to schedule some tasks to run in future. >> Celery is not an option here because some crucial options - mainly, the >> ability to revoke() a task - are only available with fully-featured MQ >> backend and I only have a limited shared hosting where I can't install this >> kind of software. After some reading, I've found that supposedly "right" >> way to do that in Python is to spawn a subprocess and call sleep(). >> However, spawning a process does have an overhead, especially considering a >> rather simple nature of tasks I want to defer (retrieval and conditional >> deletion of one object). >> >> Then I decided to use threads instead. They have little to no overhead, >> share memory and Python has a nice little threading.Timer class which is >> exactly what I need. My code looks like this: >> >> http://pastebin.com/K6RukAX9 >> >> Here's when my questions come in. I admit my knowledge of threading is >> somewhat low and Python operates on high-level too, so nuances are not >> easily noticeable. From what I've learned, threads (can?) share memory with >> main thread/process. My primary concern here is that, as far as I >> understand, method=prefork (the default) creates a pool of processes >> waiting for work. It seems possible that it may impact my scheduler >> somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown >> ends. >> >> On the other hand, threaded mode is supposedly more efficient because, >> being free from overhead of process forking, it can create child threads at >> will. In my mind this raises a similar concern to above: what if framework >> decides to destroy a thread that happens to be the one where Timer has been >> previously spawned? >> >> It is entirely possible that my concerns are completely baseless and both >> methods have nothing to do with my scheduler. I'm asking for input from >> smarter people than me :) >> > > Lets back up a bit here. > > Web servers aren't designed to do the sort of thing you're describing. Web > requests are not designed to be long lived, and web servers are tuned with > this in mind. If you try to make them long lived, you're going to hit all > sorts of problems with thread/process lifecycle management. > > The solution here isn't to try and make a web server do something it > wasn't designed to do. The solution is to work out how to get a task queue > working on your hardware setup. > > You say you can't install a fully-fledged MQ service. Ok - that's fine -- > but don't throw out the baby with the bathwater. There are other ways to > back Celery -- if need be, you can use a database as your data store. > > Or, you could consider using an entirely different task queue. If your > service provider will give you access to Redis, look into RQ. It's > lightweight, but > > Or, if neither of those are viable, you can do a "ghetto queue" -- a > combination of cron and a database can be used to implement task queue-like > behaviour, without any need for celery. > > It's difficult to give more specific advice without knowing the specifics > of your problem domain and your deployment options -- but I implore you -- > don't keep down the path you're on. That way leads to madness :-) > > Yours, > Russ Magee %-) > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-users+unsubscr...@googlegroups.com. > To post to this group, send email to django-users@googlegroups.com. > Visit this group at http://groups.google.com/group/django-users. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
Re: Prefork vs threaded when I want to spawn my own threads
On Tue, Sep 3, 2013 at 9:50 PM, wrote: > In my Django application I need to schedule some tasks to run in future. > Celery is not an option here because some crucial options - mainly, the > ability to revoke() a task - are only available with fully-featured MQ > backend and I only have a limited shared hosting where I can't install this > kind of software. After some reading, I've found that supposedly "right" > way to do that in Python is to spawn a subprocess and call sleep(). > However, spawning a process does have an overhead, especially considering a > rather simple nature of tasks I want to defer (retrieval and conditional > deletion of one object). > > Then I decided to use threads instead. They have little to no overhead, > share memory and Python has a nice little threading.Timer class which is > exactly what I need. My code looks like this: > > http://pastebin.com/K6RukAX9 > > Here's when my questions come in. I admit my knowledge of threading is > somewhat low and Python operates on high-level too, so nuances are not > easily noticeable. From what I've learned, threads (can?) share memory with > main thread/process. My primary concern here is that, as far as I > understand, method=prefork (the default) creates a pool of processes > waiting for work. It seems possible that it may impact my scheduler > somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown > ends. > > On the other hand, threaded mode is supposedly more efficient because, > being free from overhead of process forking, it can create child threads at > will. In my mind this raises a similar concern to above: what if framework > decides to destroy a thread that happens to be the one where Timer has been > previously spawned? > > It is entirely possible that my concerns are completely baseless and both > methods have nothing to do with my scheduler. I'm asking for input from > smarter people than me :) > Lets back up a bit here. Web servers aren't designed to do the sort of thing you're describing. Web requests are not designed to be long lived, and web servers are tuned with this in mind. If you try to make them long lived, you're going to hit all sorts of problems with thread/process lifecycle management. The solution here isn't to try and make a web server do something it wasn't designed to do. The solution is to work out how to get a task queue working on your hardware setup. You say you can't install a fully-fledged MQ service. Ok - that's fine -- but don't throw out the baby with the bathwater. There are other ways to back Celery -- if need be, you can use a database as your data store. Or, you could consider using an entirely different task queue. If your service provider will give you access to Redis, look into RQ. It's lightweight, but Or, if neither of those are viable, you can do a "ghetto queue" -- a combination of cron and a database can be used to implement task queue-like behaviour, without any need for celery. It's difficult to give more specific advice without knowing the specifics of your problem domain and your deployment options -- but I implore you -- don't keep down the path you're on. That way leads to madness :-) Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
Prefork vs threaded when I want to spawn my own threads
In my Django application I need to schedule some tasks to run in future. Celery is not an option here because some crucial options - mainly, the ability to revoke() a task - are only available with fully-featured MQ backend and I only have a limited shared hosting where I can't install this kind of software. After some reading, I've found that supposedly "right" way to do that in Python is to spawn a subprocess and call sleep(). However, spawning a process does have an overhead, especially considering a rather simple nature of tasks I want to defer (retrieval and conditional deletion of one object). Then I decided to use threads instead. They have little to no overhead, share memory and Python has a nice little threading.Timer class which is exactly what I need. My code looks like this: http://pastebin.com/K6RukAX9 Here's when my questions come in. I admit my knowledge of threading is somewhat low and Python operates on high-level too, so nuances are not easily noticeable. From what I've learned, threads (can?) share memory with main thread/process. My primary concern here is that, as far as I understand, method=prefork (the default) creates a pool of processes waiting for work. It seems possible that it may impact my scheduler somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown ends. On the other hand, threaded mode is supposedly more efficient because, being free from overhead of process forking, it can create child threads at will. In my mind this raises a similar concern to above: what if framework decides to destroy a thread that happens to be the one where Timer has been previously spawned? It is entirely possible that my concerns are completely baseless and both methods have nothing to do with my scheduler. I'm asking for input from smarter people than me :) Regards, Peter -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
