I guess the question might be over the integration point. If we are setting CPU_MAX at the opal/paffinity level, how does hwloc pickup that value? Or does hwloc just set its own bitmask sizes and loop limits, and your integration handles any disagreement?
On Aug 29, 2011, at 10:04 AM, Jeff Squyres wrote: > Or, if there's a specific problem in hwloc (i.e., hwloc proper -- not the > component in OMPI), post to hwloc-de...@open-mpi.org. > > I *think* that hwloc handles CPU sets of any size. I bumped the version of > hwloc to 1.2.1 (the latest stable release) in both the trunk and v1.5. v1.4 > doesn't have hwloc. > > > On Aug 29, 2011, at 11:57 AM, Ralph Castain wrote: > >> >> On Aug 29, 2011, at 8:35 AM, nadia.der...@bull.net wrote: >> >>> >>> devel-boun...@open-mpi.org wrote on 08/29/2011 04:20:30 PM: >>> >>>> De : Ralph Castain <r...@open-mpi.org> >>>> A : Open MPI Developers <de...@open-mpi.org> >>>> Date : 08/29/2011 04:26 PM >>>> Objet : Re: [OMPI devel] known limitation or bug in hwloc? >>>> Envoyé par : devel-boun...@open-mpi.org >>>> >>>> Actually, I'll eat those words. I was looking at the wrong place. >>>> >>>> Yes, that is a bug in hwloc. It needs to loop over CPU_MAX for those >>>> cases where the bit mask extends over multiple words. >>> >>> But I'm afraid the fix won't be trivial at all: hwloc in itself is >>> coherent: it loops overs NUM_BITS, but it uses masks that are NUM_BITS wide >>> (hwloc_bitmap_t set)... >> >> I guess I'm missing that - I just did a search and cannot find any reference >> to OPAL_PAFFINITY_BITMASK_T_NUM_BITS anywhere in paffinity/hwloc after the >> last change. >> >> Can you point me to where you believe a problem exists? Or feel free to >> submit a patch to fix it :-) We can push it upstream to the hwloc folks for >> their consideration. >> >> >>> >>> Regards, >>> Nadia >>>> >>>> >>>> On Aug 29, 2011, at 7:16 AM, Ralph Castain wrote: >>>> >>>>> Actually, if you look closely at the definition of those two >>>> values, you'll see that it really doesn't matter which one we loop >>>> over. The NUM_BITS value defines the actual total number of bits in >>>> the mask. The CPU_MAX is the total number of cpus we can support, >>>> which was set to a value such that the two are equal (i.e., it's a >>>> power of two that happens to be an integer multiple of 64). >>>>> >>>>> I believe the original intent was to allow CPU_MAX to be >>>> independent of address-alignment questions, so NUM_BITS could >>>> technically be greater than CPU_MAX. Even if this happens, though, >>>> all that would do is cause the loop to run across more bits than required. >>>>> >>>>> So it doesn't introduce a limitation at all. In hindsight, we >>>> could simplify things by eliminating one of those values and just >>>> putting a requirement on the number that it be a multiple of 64 so >>>> it aligns with a memory address. >>>>> >>>>> >>>>> On Aug 29, 2011, at 7:05 AM, Kenneth Lloyd wrote: >>>>> >>>>>> Nadia, >>>>>> >>>>>> Interesting. I haven't tried pushing this to levels above 8 on a >>>>>> particular >>>>>> machine. Do you think that the cpuset / paffinity / hwloc only applies at >>>>>> the machine level, at which time you need to employ a graph with carto? >>>>>> >>>>>> Regards, >>>>>> >>>>>> Ken >>>>>> >>>>>> -----Original Message----- >>>>>> From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] On >>>>>> Behalf Of nadia.derbey >>>>>> Sent: Monday, August 29, 2011 5:45 AM >>>>>> To: Open MPI Developers >>>>>> Subject: [OMPI devel] known limitation or bug in hwloc? >>>>>> >>>>>> Hi list, >>>>>> >>>>>> I'm hitting a limitation with paffinity/hwloc with cpu numbers >= 64. >>>>>> >>>>>> In opal/mca/paffinity/hwloc/paffinity_hwloc_module.c, module_set() is >>>>>> the routine that sets the calling process affinity to the mask given as >>>>>> parameter. Note that "mask" is a opal_paffinity_base_cpu_set_t (so we >>>>>> allow the cpus to be potentially numbered up to >>>>>> OPAL_PAFFINITY_BITMASK_CPU_MAX - 1). >>>>>> >>>>>> The problem with module_set() is that is loops over >>>>>> OPAL_PAFFINITY_BITMASK_T_NUM_BITS bits to check if these bits are set in >>>>>> the mask: >>>>>> >>>>>> for (i = 0; ((unsigned int) i) < OPAL_PAFFINITY_BITMASK_T_NUM_BITS; ++i) >>>>>> { >>>>>> if (OPAL_PAFFINITY_CPU_ISSET(i, mask)) { >>>>>> hwloc_bitmap_set(set, i); >>>>>> } >>>>>> } >>>>>> >>>>>> Given "mask"'s type, I think module_set() should instead loop over >>>>>> OPAL_PAFFINITY_BITMASK_CPU_MAX bits. >>>>>> >>>>>> Note that module_set() uses a type for its internal mask that is >>>>>> coherent with OPAL_PAFFINITY_BITMASK_T_NUM_BITS (hwloc_bitmap_t). >>>>>> >>>>>> So I'm wondering whether this is a known limitation I've never heard of >>>>>> or an actual bug? >>>>>> >>>>>> Regards, >>>>>> Nadia >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>> ----- >>>>>> No virus found in this message. >>>>>> Checked by AVG - www.avg.com >>>>>> Version: 10.0.1392 / Virus Database: 1520/3864 - Release Date: 08/28/11 >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>> >>>> >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
