Got it, I have the input from the user and am testing it out.
It probably has less todo with torque and more cpuset's, I'm working on producing it myself also. Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion bro...@umich.edu (734)936-1985 On Jun 20, 2014, at 12:18 PM, Ralph Castain <r...@open-mpi.org> wrote: > Thanks - I'm just trying to reproduce one problem case so I can look at it. > Given that I don't have access to a Torque machine, I need to "fake" it. > > > On Jun 20, 2014, at 9:15 AM, Brock Palen <bro...@umich.edu> wrote: > >> In this case they are a single socket, but as you can see they could be >> ether/or depending on the job. >> >> Brock Palen >> www.umich.edu/~brockp >> CAEN Advanced Computing >> XSEDE Campus Champion >> bro...@umich.edu >> (734)936-1985 >> >> >> >> On Jun 19, 2014, at 2:44 PM, Ralph Castain <r...@open-mpi.org> wrote: >> >>> Sorry, I should have been clearer - I was asking if cores 8-11 are all on >>> one socket, or span multiple sockets >>> >>> >>> On Jun 19, 2014, at 11:36 AM, Brock Palen <bro...@umich.edu> wrote: >>> >>>> Ralph, >>>> >>>> It was a large job spread across. Our system allows users to ask for >>>> 'procs' which are laid out in any format. >>>> >>>> The list: >>>> >>>>> [nyx5406:2][nyx5427:2][nyx5506:2][nyx5311:3] >>>>> [nyx5329:4][nyx5398:4][nyx5396:11][nyx5397:11] >>>>> [nyx5409:11][nyx5411:11][nyx5412:3] >>>> >>>> Shows that nyx5406 had 2 cores, nyx5427 also 2, nyx5411 had 11. >>>> >>>> They could be spread across any number of sockets configuration. We start >>>> very lax "user requests X procs" and then the user can request more strict >>>> requirements from there. We support mostly serial users, and users can >>>> colocate on nodes. >>>> >>>> That is good to know, I think we would want to turn our default to 'bind >>>> to core' except for our few users who use hybrid mode. >>>> >>>> Our CPU set tells you what cores the job is assigned. So in the problem >>>> case provided, the cpuset/cgroup shows only cores 8-11 are available to >>>> this job on this node. >>>> >>>> Brock Palen >>>> www.umich.edu/~brockp >>>> CAEN Advanced Computing >>>> XSEDE Campus Champion >>>> bro...@umich.edu >>>> (734)936-1985 >>>> >>>> >>>> >>>> On Jun 18, 2014, at 11:10 PM, Ralph Castain <r...@open-mpi.org> wrote: >>>> >>>>> The default binding option depends on the number of procs - it is bind-to >>>>> core for np=2, and bind-to socket for np > 2. You never said, but should >>>>> I assume you ran 4 ranks? If so, then we should be trying to bind-to >>>>> socket. >>>>> >>>>> I'm not sure what your cpuset is telling us - are you binding us to a >>>>> socket? Are some cpus in one socket, and some in another? >>>>> >>>>> It could be that the cpuset + bind-to socket is resulting in some odd >>>>> behavior, but I'd need a little more info to narrow it down. >>>>> >>>>> >>>>> On Jun 18, 2014, at 7:48 PM, Brock Palen <bro...@umich.edu> wrote: >>>>> >>>>>> I have started using 1.8.1 for some codes (meep in this case) and it >>>>>> sometimes works fine, but in a few cases I am seeing ranks being given >>>>>> overlapping CPU assignments, not always though. >>>>>> >>>>>> Example job, default binding options (so by-core right?): >>>>>> >>>>>> Assigned nodes, the one in question is nyx5398, we use torque CPU sets, >>>>>> and use TM to spawn. >>>>>> >>>>>> [nyx5406:2][nyx5427:2][nyx5506:2][nyx5311:3] >>>>>> [nyx5329:4][nyx5398:4][nyx5396:11][nyx5397:11] >>>>>> [nyx5409:11][nyx5411:11][nyx5412:3] >>>>>> >>>>>> [root@nyx5398 ~]# hwloc-bind --get --pid 16065 >>>>>> 0x00000200 >>>>>> [root@nyx5398 ~]# hwloc-bind --get --pid 16066 >>>>>> 0x00000800 >>>>>> [root@nyx5398 ~]# hwloc-bind --get --pid 16067 >>>>>> 0x00000200 >>>>>> [root@nyx5398 ~]# hwloc-bind --get --pid 16068 >>>>>> 0x00000800 >>>>>> >>>>>> [root@nyx5398 ~]# cat >>>>>> /dev/cpuset/torque/12703230.nyx.engin.umich.edu/cpus >>>>>> 8-11 >>>>>> >>>>>> So torque claims the CPU set setup for the job has 4 cores, but as you >>>>>> can see the ranks were giving identical binding. >>>>>> >>>>>> I checked the pids they were part of the correct CPU set, I also >>>>>> checked, orted: >>>>>> >>>>>> [root@nyx5398 ~]# hwloc-bind --get --pid 16064 >>>>>> 0x00000f00 >>>>>> [root@nyx5398 ~]# hwloc-calc --intersect PU 16064 >>>>>> ignored unrecognized argument 16064 >>>>>> >>>>>> [root@nyx5398 ~]# hwloc-calc --intersect PU 0x00000f00 >>>>>> 8,9,10,11 >>>>>> >>>>>> Which is exactly what I would expect. >>>>>> >>>>>> So ummm, i'm lost why this might happen? What else should I check? >>>>>> Like I said not all jobs show this behavior. >>>>>> >>>>>> Brock Palen >>>>>> www.umich.edu/~brockp >>>>>> CAEN Advanced Computing >>>>>> XSEDE Campus Champion >>>>>> bro...@umich.edu >>>>>> (734)936-1985 >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2014/06/24672.php >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2014/06/24673.php >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2014/06/24675.php >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/06/24676.php >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/06/24677.php > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/06/24678.php
signature.asc
Description: Message signed with OpenPGP using GPGMail
