It seems to me there are two extremes. One is that you replicate the data for each process. This has the disadvantage of consuming lots of memory "unnecessarily." Another extreme is that shared data is distributed over all processes. This has the disadvantage of making at least some of the data less accessible, whether in programming complexity and/or run-time performance. I'm not familiar with Global Arrays. I was somewhat familiar with HPF. I think the natural thing to do with those programming models is to distribute data over all processes, which may relieve the excessive memory consumption you're trying to address but which may also just put you at a different "extreme" of this spectrum. The middle ground I think might make most sense would be to share data only within a node, but to replicate the data for each node. There are probably multiple ways of doing this -- possibly even GA, I don't know. One way might be to use one MPI process per node, with OMP multithreading within each process|node. Or (and I thought this was the solution you were looking for), have some idea which processes are collocal. Have one process per node create and initialize some shared memory -- mmap, perhaps, or SysV shared memory. Then, have its peers map the same shared memory into their address spaces. You asked what source code changes would be required. It depends. If you're going to mmap shared memory in on each node, you need to know which processes are collocal. If you're willing to constrain how processes are mapped to nodes, this could be easy. (E.g., "every 4 processes are collocal".) If you want to discover dynamically at run time which are collocal, it would be harder. The mmap stuff could be in a stand-alone function of about a dozen lines. If the shared area is allocated as one piece, substituting the single malloc() call with a call to your mmap function should be simple. If you have many malloc()s you're trying to replace, it's harder. Andrei Fokau wrote:
|
- [OMPI users] Shared memory Andrei Fokau
- Re: [OMPI users] Shared memory Durga Choudhury
- Re: [OMPI users] Shared memory John Hearns
- Re: [OMPI users] Shared memory Reuti
- Re: [OMPI users] Shared memory Andrei Fokau
- Re: [OMPI users] Shared memory Eugene Loh
- Re: [OMPI users] Shared memo... Durga Choudhury
- Re: [OMPI users] Shared ... Andrei Fokau
- Re: [OMPI users] Sha... Andrei Fokau
- Re: [OMPI users] Sha... Jeff Squyres (jsquyres)
- Re: [OMPI users] Sha... Andrei Fokau
- Re: [OMPI users] Sha... Richard Treumann