Hi Mark The purpose of orte_max_vm_size is to subdivide the allocation - i.e., for a given mpirun execution, you can specify to only use a certain number of the allocated nodes. If you want to further limit the VM to specific nodes in the allocation, then you would use -host option.
It’s a little more complicated for your use-case as orte-dvm defines the VM, not orte-submit. The latter simply tells orte-dvm to launch an application - the daemons have already been established by orte-dvm and cannot change. So if you want to setup orte-dvm and then submit to only some of the nodes, you would have to use the -host option. Note that -host supports an extended syntax for this purpose - you can ask for a specific number of “empty” nodes, you can tell it to use only so many slots on a node, etc. I’m confused by your examples because the max_vm_size values don’t seem right. If you have a VM of size 1 or 2, then max_vm_size can only be 1 or 2. You can’t have a max_vm_size larger than the number of available nodes. This is probably the source of the problem you are seeing - I can add some protection to ensure this doesn’t happen. We don’t appear to support either -host or -np as MCA params. I’m not sure -np would make sense, but we could add a param for -host. We do have a param for the default hostfile, but that probably wouldn’t help here. We can certainly extend the orte-dvm and orte-submit cmd lines. I only brought over a minimal set at first in order to get things running quickly, but no problem with increasing capability. Just a question of finding a little time. For ompi_info, try doing “ompi_info -l 9” to get the full output of params. > On Sep 3, 2015, at 5:08 AM, Mark Santcroos <mark.santcr...@rutgers.edu> wrote: > > Hi, > > I've been running into some funny issue with using orte-dvm (Hi Ralph ;-) and > trying to define the size of the created vm and for that I use "--mca > orte_max_vm_size" which in general seems to work. > > In this example I have a PBS job of 4 nodes and want to run the DVM on < 4 > nodes. > If I create the VM with size 3 or 4 (max_vm_size 1 and 0 respectively) > everything works as expected. > However, when I create a VM of size 1 or 2 (max_vm_size 3 and 2 respectively) > I get the stack trace below once I use orte-submit to start something within > the VM. > > [nid01280:02498] [[39239,0],0] orted:comm:process_commands() Processing > Command: ORTE_DAEMON_SPAWN_JOB_CMD > orte-dvm: ../../../../../src/ompi/opal/class/opal_list.h:547: > _opal_list_append: Assertion `0 == item->opal_list_item_refcount' failed. > [nid01280:02498] *** Process received signal *** > [nid01280:02498] Signal: Aborted (6) > [nid01280:02498] Signal code: (-6) > [nid01280:02498] [ 0] /lib64/libpthread.so.0(+0xf810)[0x2ba3e274a810] > [nid01280:02498] [ 1] /lib64/libc.so.6(gsignal+0x35)[0x2ba3e298b885] > [nid01280:02498] [ 2] /lib64/libc.so.6(abort+0x181)[0x2ba3e298ce61] > [nid01280:02498] [ 3] /lib64/libc.so.6(__assert_fail+0xf0)[0x2ba3e2984740] > [nid01280:02498] [ 4] > /global/homes/m/marksant/openmpi/edison/installed/HEAD/lib/libopen-rte.so.0(+0x83f16)[0x2ba3e1687f16] > [nid01280:02498] [ 5] > /global/homes/m/marksant/openmpi/edison/installed/HEAD/lib/libopen-rte.so.0(orte_plm_base_setup_virtual_machine+0x473)[0x2ba3e16907fe] > [nid01280:02498] [ 6] > /global/homes/m/marksant/openmpi/edison/installed/HEAD/lib/openmpi/mca_plm_alps.so(+0x274d)[0x2ba3e666574d] > [nid01280:02498] [ 7] > /global/homes/m/marksant/openmpi/edison/installed/HEAD/lib/libopen-pal.so.0(opal_libevent2022_event_base_loop+0xd81)[0x2ba3e198cee1] > [nid01280:02498] [ 8] > /global/homes/m/marksant/openmpi/edison/installed/HEAD/bin/orte-dvm[0x402e20] > [nid01280:02498] [ 9] /lib64/libc.so.6(__libc_start_main+0xe6)[0x2ba3e2977c36] > [nid01280:02498] [10] > /global/homes/m/marksant/openmpi/edison/installed/HEAD/bin/orte-dvm[0x401d19] > [nid01280:02498] *** End of error message *** > [nid05888:25419] > [[39239,0],1]:../../../../../../src/ompi/orte/mca/errmgr/default_orted/errmgr_default_orted.c(251) > updating exit status to 1 > > > Some questions: > - Am I understanding the purpose of orte_max_vm_size correctly? > - If so, then it seems some refcounting if off. Not sure where to start > looking though ... > - I would rather have a bit more flexible way of specifying the size of the > VM, but currently the orte-dvm command line parameters are limited. Would it > be a matter of copying over some of the "-host" parameter stuff from orte-run > or is it more involved? > - Can I configure the -host, -np, etc parameters also via MCA settings? > - Whats the magic combination of parameters to get all information about of > ompi_info? As I can't find a way to even "find" the orte_max_vm_size > parameter out of it, while I know it exists. > > Thanks! > > Mark > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/09/17930.php
