Justin, knem allows a process to write into the address space of an other process, to do zero copy. in the case of osv, threads can simply do a memcpy(), and I doubt knew is even available. so a new btl that uses memcpy would be optimal on osv.
one option is to starts from the vader btl, and replace knem invocation with memcpy() an other option could be to extend the self btl but once again, this is for performance only, using tcp btl only should be enough to get things work. Cheers, Gilles On Wednesday, December 16, 2015, Justin Cinkelj <justin.cink...@xlab.si> wrote: > Vader is for intra-node communication only, right? So for inter-node > communication some other mechanism will be used anyway. > Why would be even better to write a new btl? To avoid memcpy (knem would > use it, if I understand you correctly; I guess code assumes that multiple > processes on same node have isolated address spaces). > > Fork + execve was one of first problems, yes. I replaced that with OSv > specific calls (ignore fork, and instead of execve start given binary in > new thread). The global variables required OSv modification - the guys from > http://osv.io/ took care of that (I was surprised that at the end, the > patches were really small and elegant). So while there are no real > processes, new binary / ELF file is loaded at different address then the > rest of OS - so it has separate global variables, and separate environ too. > Other resources like file descriptors are still shared. > > BR Justin > > On 15. 12. 2015 14:55, Gilles Gouaillardet wrote: > > Justin, > > at first glance, vader should be symmetric (e.g. > call opal_shmem_segment_dettach() instead of munmap() > Nathan, can you please comment ? > > using tid instead of pid should also do the trick > > that being said, a more elegant approach would be to create a new module > in the shmem framework > basically, create = malloc, attach = return the malloc'ed address, detach > = noop, destroy = free > > and an even better approach would be to write your own btl that replaces > vader. > basically, vader can use the knem module to write into an other process > address space. > since your os is thread only, knem invocation would become a simple memcpy. > > makes sense ? > > > as a side note, > ompi uses global variables, and orted forks and exec MPI tasks after > setting some environment variables. it seems porting ompi to this new os > was not so painful, and I would have expected some issues with the global > variables, and some race conditions with the environment. > did you already solve these issues ? > > Cheers, > > Gilles > > On Tuesday, December 15, 2015, Justin Cinkelj < > <javascript:_e(%7B%7D,'cvml','justin.cink...@xlab.si');> > justin.cink...@xlab.si > <javascript:_e(%7B%7D,'cvml','justin.cink...@xlab.si');>> wrote: > >> I'm trying to port Open MPI to OS with threads instead of processes. >> Currently, during MPI_Finalize, I get attempt to call munmap first with >> address of 0x200000c00000 and later 0x200000c00008. >> >> mca_btl_vader_component_close(): >> munmap (mca_btl_vader_component.my_segment, >> mca_btl_vader_component.segment_size) >> >> mca_btl_vader_component_init(): >> if(MCA_BTL_VADER_XPMEM != mca_btl_vader_component.single_copy_mechanism) { >> opal_shmem_segment_create (&component->seg_ds, sm_file, >> component->segment_size); >> component->my_segment = opal_shmem_segment_attach (&component->seg_ds); >> } else { >> mmap (NULL, component->segment_size, PROT_READ | PROT_WRITE, >> MAP_ANONYMOUS | MAP_SHARED, -1, 0); >> } >> >> But opal_shmem_segment_attach (from mmap module) ends with: >> /* update returned base pointer with an offset that hides our stuff */ >> return (ds_buf->seg_base_addr + sizeof(opal_shmem_seg_hdr_t)); >> >> So mca_btl_vader_component_close() should in that case call >> opal_shmem_segment_dettach() instead of munmap. >> Or actually, as at that point shmem_mmap module cleanup code is already >> done, vader could/should just skip cleanup part? >> >> Maybe I should ask first how does that setup/cleanup work on normal Linux >> system? >> Is mmap called twice, and vader and shmem_mmap module each uses different >> address (so vader munmap is indeed required in that case)? >> >> Second question. >> With two threads in one process, I got attempt to >> opal_shmem_segment_dettach() and munmap() on same mmap-ed address, from >> both threads. I 'fixed' that by replacing "ds_buf->seg_cpid = getpid()" >> with gettid(), and then each thread munmap-s only address allocated by >> itself. Is that correct? Or is it possible, that the second thread might >> still try to access data at that address? >> >> BR Justin >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/12/18417.php >> > > > _______________________________________________ > devel mailing listde...@open-mpi.org > <javascript:_e(%7B%7D,'cvml','de...@open-mpi.org');> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/12/18418.php > > >
