On Wed, 2008-04-02 at 01:21 -0700, Valery wrote: > Hi > > did anyone here use Django in parallelized computations? > > I use for about a year a great parallelization approach based on > 'pmap' function from 'pprocess' module: > http://www.boddie.org.uk/python/pprocess.html > > My own code is designed to be strongly side-effect-free, however I am > experiencing strange problems after switching to Django. > Perhaps because I have wrong understanding of Django's DB access > model. > > Perhaps Django's class django.db.models.Model should be used with more > care under parallelized access? > > What about lockings there? > > What about memory read/write conflicts there?
You'll probably need to provide more details for us to be able to help here. "Experience strange problems" isn't a particularly specific problem description. What part of the request/response pipeline are you trying to do in parallel? If you're using separate processes, things should pretty much just work, I would think. After all, multiple processes running Django code at once is *precisely* how a web server works when it's handling Django. It's not like Django can only serve one request per machine at once. If you're trying to do multi-threaded operations within the same process within the same request/response handling, all bets are off. For example, some of the database backends we support cannot handle sharing cursors between multiple threads, or even sharing connections between threads. Each thread must therefore have its own connection and we're careful to do that as part of starting up a new request each time. But the module you point to (which I've never used) says it uses forking to create new processes, rather than just creating threads, so that shouldn't be an issue. There's nothing that would necessarily interfere with anything at the Python level, since they're in entirely different process spaces. You will get the standard transaction-based interaction if you're using a database that has any kind of transaction support (e.g. even with SQLite, updates in one process won't be visible to another process until the first process commits the result), but that's just normal parallel-access database behaviour. Do remember, though, that a fork() duplicates all your open file descriptors and things like that, so anything could happen if multiple things are trying to write back to the response path, for example. Again, it all depends on your code and that's where a better description of the "strange problems" will no doubt help. My general hunch, though, is that even if there are problems with things like the duped file descriptors, it's not something we should worry about. The "shared nothing" style of design is pretty useful for request-triggered applications and you're actually trying to do "shared something" work here within a single request path. Regards, Malcolm -- Experience is something you don't get until just after you need it. http://www.pointy-stick.com/blog/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
