Hi Malcolm,
many thanks for your reply. Let me give some more details.
I use parallelized fetching of database objects like this:
my_dictionaries = pprocess.pmap( fetch_object, ["serchterm1",
"searchterm2", "searchterm3"])
where fetch_object function is not read-only, but smth like this:
from mytsite.myapp.models import DBModel
def fetch_object(search_term):
lst = DBModel.objects.filter(myfield=search_term)
## now, a bit oversimplified, I have the following:
lst[0].access_counter = lst[0].access_counter + 1
lst[0].save()
return lst[0].__dict__.copy()
during the execution of pmap call meant above I obtain different
sporadic errors, like:
"Broken pipe"
"no results to fetch"
which seems to be the errors that come from pickle's module during
object serializations via pipe used in pprocess.
By the way, maybe Django's DB Model isn't much compatible with
serialization à lá pickle module?
Regards
Valery
On Apr 2, 10:46 am, Malcolm Tredinnick <[EMAIL PROTECTED]>
wrote:
> On Wed, 2008-04-02 at 01:21 -0700, Valery wrote:
> > Hi
>
> > did anyone here use Django in parallelized computations?
>
> > I use for about a year a great parallelization approach based on
> > 'pmap' function from 'pprocess' module:
> >http://www.boddie.org.uk/python/pprocess.html
>
> > My own code is designed to be strongly side-effect-free, however I am
> > experiencing strange problems after switching to Django.
> > Perhaps because I have wrong understanding of Django's DB access
> > model.
>
> > Perhaps Django's class django.db.models.Model should be used with more
> > care under parallelized access?
>
> > What about lockings there?
>
> > What about memory read/write conflicts there?
>
> You'll probably need to provide more details for us to be able to help
> here. "Experience strange problems" isn't a particularly specific
> problem description.
>
> What part of the request/response pipeline are you trying to do in
> parallel? If you're using separate processes, things should pretty much
> just work, I would think. After all, multiple processes running Django
> code at once is *precisely* how a web server works when it's handling
> Django. It's not like Django can only serve one request per machine at
> once. If you're trying to do multi-threaded operations within the same
> process within the same request/response handling, all bets are off. For
> example, some of the database backends we support cannot handle sharing
> cursors between multiple threads, or even sharing connections between
> threads. Each thread must therefore have its own connection and we're
> careful to do that as part of starting up a new request each time. But
> the module you point to (which I've never used) says it uses forking to
> create new processes, rather than just creating threads, so that
> shouldn't be an issue.
>
> There's nothing that would necessarily interfere with anything at the
> Python level, since they're in entirely different process spaces. You
> will get the standard transaction-based interaction if you're using a
> database that has any kind of transaction support (e.g. even with
> SQLite, updates in one process won't be visible to another process until
> the first process commits the result), but that's just normal
> parallel-access database behaviour.
>
> Do remember, though, that a fork() duplicates all your open file
> descriptors and things like that, so anything could happen if multiple
> things are trying to write back to the response path, for example.
> Again, it all depends on your code and that's where a better description
> of the "strange problems" will no doubt help.
>
> My general hunch, though, is that even if there are problems with things
> like the duped file descriptors, it's not something we should worry
> about. The "shared nothing" style of design is pretty useful for
> request-triggered applications and you're actually trying to do "shared
> something" work here within a single request path.
>
> Regards,
> Malcolm
>
> --
> Experience is something you don't get until just after you need
> it.http://www.pointy-stick.com/blog/
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
