Re: Python Multiprocessing With Django
My program trigger some regression file in a remote host for that I used paramiko module. But I want to trigger same kinda file to different machines parallel-ly. I am using django so that people can use this as web app. I am using url-view-template model. How can use I multi threading in the views function? If you come up with a solution please let me know. It requires the( if "__name__"="main": ).How can I do that? On Friday, 6 March 2009 20:37:33 UTC+5:30, Vitaly Babiy wrote: > > I have a management command that that starts up a few process to process a > request: http://dpaste.com/7925/. > > But every so offen I get a exception say mysql went a way, but it didn't. > Process PoolWorker-3: > Traceback (most recent call last): > File "/usr/lib/python2.5/site- > packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/process.py", > > line 236, in _bootstrap >self.run() > File > "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/process.py", > > line 93, in run >self._target(*self._args, **self._kwargs) > File > "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py", > > line 71, in worker >put((job, i, result)) > File > "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/queues.py", > > line 353, in put >return send(obj) > PicklingError: Can't pickle : > attribute lookup http_tracker.models.DoesNotExist failed > Traceback (most recent call last): > File "/home/vbabiy/projects/git-projects/howsthedotcom/manage.py", line > 11, in >execute_manager(settings) > File > "/usr/lib/python2.5/site-packages/django/core/management/__init__.py", line > 350, in execute_manager >utility.execute() > File > "/usr/lib/python2.5/site-packages/django/core/management/__init__.py", line > 295, in execute >self.fetch_command(subcommand).run_from_argv(self.argv) > File "/usr/lib/python2.5/site-packages/django/core/management/base.py", > line 195, in run_from_argv >self.execute(*args, **options.__dict__) > File "/usr/lib/python2.5/site-packages/django/core/management/base.py", > line 222, in execute >output = self.handle(*args, **options) > File > "/home/vbabiy/projects/git-projects/howsthedotcom/tracker/management/commands/tracker.py", > > line 37, in handle >res = p.map(check_site, sites) > File > "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py", > > line 148, in map >return self.map_async(func, iterable, chunksize).get() > File > "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py", > > line 422, in get >raise self._value > _mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL > server during query') > > Any one got any idea why this is happening? > > Vitaly Babiy > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/238229f3-bf8c-41d4-a964-e2f4c1e0b517%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Python Multiprocessing With Django
Just in case someone else stumbles across this old thread. TLDR: The recipe may be able to be improved - I only needed to close the connection in the parent Python process, not in the child process. -- Best guess hypothesis as to what's going on is when you fork a process, the OS (e.g. OSX Mountain Lion for me) will simply copy the memory, including any current DB connection details. Then the child process in Django sometimes tries to reuse it. I say sometimes because Django's backend.mysql.base.py tries: self.connection.ping() If that fails, it closes the connection, then the Django magic of creating a new connection to the default DB will happen later when it is needed. Reusing a connection then sometimes leads to one or more whacky things in the child processes: - DatabaseError: (2013, 'Lost connection to MySQL server during query') - DatabaseError: (2006, 'MySQL server has gone away') - MultipleObjectsReturned: get() returned more than one matching query does not exist. Perhaps someone will be motivated one day to patch something like this into Django itself, but it feels like something that comes with the territory of handling multiprocessing without using a meaningful subset of concurrency such as RabbitMQ/Celery for a task queue. In other words, there's a reason Python's multiprocessing documentation is filled with many notes and warnings - concurrency is one of the more difficult to use constructs in the developer toolbox and as developers we should use it only when it is justified by specific requirements for extreme performance (or perhaps for the personal challenge). On Saturday, March 7, 2009 2:30:47 AM UTC+11, Jirka Vejrazka wrote: > > > I have a management command that that starts up a few process to process > a > > request: http://dpaste.com/7925/. > > Hi, I was recently debugging similar issue and came to a conclusion > (which may be wrong of course :) that multiprocessing and Django DB > connections don't play well together. I ended up closing Django DB > connection first thing in the new process. It'll recreate a new > connection when it needs one, but that one will have no references to > the connection used by the parent. > > So, my Process.start() calls a function which starts with: > >>> from django.db import connection > >>> connection.close() > > This solved my problem. > >Cheers > > Jirka > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/-u-g-YzdJTQJ. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: Python Multiprocessing With Django
> Thanks Jirka, that fixed the problem. But I think I will file a bug report > that seem to be very hacky :) Great, I'm glad it solved your problem. Personally, I don't think it is a bug, as we both pushed Django well beyond its "area of expertise". I really believe that Django is great for serving web content and everyone (including) me trying to implement it in different areas must be ready for problems and gotchas here and there. But YMMV. Cheers Jirka --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Python Multiprocessing With Django
Thanks Jirka, that fixed the problem. But I think I will file a bug report that seem to be very hacky :) Vitaly Babiy On Fri, Mar 6, 2009 at 11:15 AM, Vitaly Babiy wrote: > Alright I will have to give this a try > > Vitaly Babiy > > > > On Fri, Mar 6, 2009 at 10:30 AM, Jirka Vejrazka > wrote: > >> >> > I have a management command that that starts up a few process to process >> a >> > request: http://dpaste.com/7925/. >> >> Hi, I was recently debugging similar issue and came to a conclusion >> (which may be wrong of course :) that multiprocessing and Django DB >> connections don't play well together. I ended up closing Django DB >> connection first thing in the new process. It'll recreate a new >> connection when it needs one, but that one will have no references to >> the connection used by the parent. >> >> So, my Process.start() calls a function which starts with: >> >>> from django.db import connection >> >>> connection.close() >> >> This solved my problem. >> >> Cheers >> >> Jirka >> >> >> >> > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Python Multiprocessing With Django
Alright I will have to give this a try Vitaly Babiy On Fri, Mar 6, 2009 at 10:30 AM, Jirka Vejrazka wrote: > > > I have a management command that that starts up a few process to process > a > > request: http://dpaste.com/7925/. > > Hi, I was recently debugging similar issue and came to a conclusion > (which may be wrong of course :) that multiprocessing and Django DB > connections don't play well together. I ended up closing Django DB > connection first thing in the new process. It'll recreate a new > connection when it needs one, but that one will have no references to > the connection used by the parent. > > So, my Process.start() calls a function which starts with: > >>> from django.db import connection > >>> connection.close() > > This solved my problem. > > Cheers > > Jirka > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Python Multiprocessing With Django
> I have a management command that that starts up a few process to process a > request: http://dpaste.com/7925/. Hi, I was recently debugging similar issue and came to a conclusion (which may be wrong of course :) that multiprocessing and Django DB connections don't play well together. I ended up closing Django DB connection first thing in the new process. It'll recreate a new connection when it needs one, but that one will have no references to the connection used by the parent. So, my Process.start() calls a function which starts with: >>> from django.db import connection >>> connection.close() This solved my problem. Cheers Jirka --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
