I'm trying to parallelize some code which is essentially a network of
cells with an index. I can make the cells immutable, if I route all
access to them via the index, AND if I can update the index. The index
would not need to have items deleted, but updates would cause it to
point to different addresses as new versions of the cells replaced old
versions. Also the index is LARGE. I can't size it exactly, but it
would have over 100,000 entries. To be safe I've said it's indexed by a
ulong, though I don't have enough RAM to allow that to be reasonable.
So now I'm trying to parallelize it. I don't want a per thread copy for
multiple reasons (it's large, it gets updates, etc.). But if I read
TDPL correctly, shared won't allow me to update it. If I can't update
it, I need to restart the program to resize it AND the cells can't be
immutable.
It would appear that making the cells immutable is silly. All I need is
to protect a few small places where I adjust various weights. But TDPL
says that if I adopt that approach, I also need write locks around every
read. So that looks infeasible.
I need the index mainly to sequentially step through the cells, so I
could implement an equivalent functionality by having prior and next
pointers in each cell...but if I use pointers then making one cell
immutable makes the whole network immutable, and that's no good at all.
Is there a decent answer in D? This is basically one class replicated a
huge number of times. Ideally each instance would be in a separate
thread, but that would overwhelm the OS, and I don't have more than
around 6 cores on my current system. But if each instance were in a
separate thread, then I could do this with message passing....except for
stepping through all the cells sequentially. (Well, I could do that by
implementing prior thread and next thread variables, I guess, but that
seems contrary to the purpose of threads. Still, that's a mandatory
part of the problem.)
- Parallelizing code -- design problem in networkish code Charles Hixson
-
