Actually that did turn out to help. The nvml# devices appear to be numbered in the way that CUDA_VISABLE_DEVICES sees them, while the cuda# devices are in the order that PBS and nvidia-smi see them.
PCIBridge PCIBridge PCIBridge PCI 10de:1021 CoProc L#2 "cuda0" GPU L#3 "nvml2" PCIBridge PCI 10de:1021 CoProc L#4 "cuda1" GPU L#5 "nvml3" PCIBridge PCIBridge PCIBridge PCI 10de:1021 CoProc L#6 "cuda2" GPU L#7 "nvml0" PCIBridge PCI 10de:1021 CoProc L#8 "cuda3" GPU L#9 "nvml1" Right now I am trying to create a python script that will take the XML output of lstopo and give me just the cuda and nvml devices in order. I dont' know if some value are deterministic though. Could I ignore the CoProc line and just use the: GPU L#3 "nvml2" GPU L#5 "nvml3" GPU L#7 "nvml0" GPU L#9 "nvml1" Is the L# always going to be in the oder I would expect? Because then I already have my map then. Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion bro...@umich.edu (734)936-1985 On Feb 5, 2014, at 1:19 AM, Brice Goglin <brice.gog...@inria.fr> wrote: > Hello Brock, > > Some people reported the same issue in the past and that's why we added the > "nvml" objects. CUDA reorders devices by "performance". Batch-schedulers are > somehow supposed to use "nvml" for managing GPUs without actually using them > with CUDA directly. And the "nvml" order is the "normal" order. > > You need "tdk" (https://developer.nvidia.com/tesla-deployment-kit) to get > nvml library and development headers installed. Then hwloc can build its > "nvml" backend. Once ready, you'll see a hwloc "cudaX" and a hwloc "nvmlY" > object in each NVIDIA PCI devices, and you can get their locality as usual. > > Does this help? > > Brice > > > > Le 05/02/2014 05:25, Brock Palen a écrit : >> We are trying to build a system to mask users to the GPU's they were >> assigned by our batch system (torque). >> >> The batch system sets the GPU's into thread exclusive mode when assigned to >> a job, so we want the GPU that the batch system assigns to be the one set in >> CUDA_VISIBLE_DEVICES, >> >> Problem is on our nodes what the batch system sees as gpu 0 is not the same >> GPU that CUDA_VISIBLE_DEVICES sees as 0. Actually 0 is 2. >> >> You can see this behavior is you run >> >> nvidia-smi and look at the PCI ID's of the devices. You can then look at >> the PCI ID's outputed by deviceQuery from the SDK examples and see they are >> in a different order. >> >> The ID's you would set in CUDA_VISIBLE_DEVICES matches the order that >> deviceQuery sees, not the order that nvida-smi sees. >> >> Example (All values turned to decimal to match deviceQuery): >> >> nvidia-smi order: 9, 10, 13, 14, 40, 43, 48, 51 >> dviceQuery order: 13, 14, 9, 10, 40, 43, 48, 51 >> >> >> Can hwloc help me with this? Right now I am hacking a script based on the >> output of the two commands, and making a map, between the two and then set >> CUDA_VISIBLE_DEVICES >> >> Any ideas would be great. Later as we currently also use CPU sets, we want >> to pass GPU locality information to the scheduler to make decisions to match >> GPU-> CPU Socket information, as performance of threads across QPI domains >> is very poor. >> >> Thanks >> >> Machine (64GB) >> NUMANode L#0 (P#0 32GB) >> Socket L#0 + L3 L#0 (20MB) >> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 >> (P#0) >> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 >> (P#1) >> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 >> (P#2) >> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 >> (P#3) >> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 >> (P#4) >> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 >> (P#5) >> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 >> (P#6) >> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 >> (P#7) >> HostBridge L#0 >> PCIBridge >> PCI 1000:0087 >> Block L#0 "sda" >> Block L#1 "sdb" >> PCIBridge >> PCIBridge >> PCIBridge >> PCI 10de:1021 >> CoProc L#2 "cuda0" >> PCIBridge >> PCI 10de:1021 >> CoProc L#3 "cuda1" >> PCIBridge >> PCIBridge >> PCIBridge >> PCI 10de:1021 >> CoProc L#4 "cuda2" >> PCIBridge >> PCI 10de:1021 >> CoProc L#5 "cuda3" >> PCIBridge >> PCI 8086:1521 >> Net L#6 "eth0" >> PCI 8086:1521 >> Net L#7 "eth1" >> PCIBridge >> PCI 102b:0533 >> PCI 8086:1d02 >> NUMANode L#1 (P#1 32GB) >> Socket L#1 + L3 L#1 (20MB) >> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 >> (P#8) >> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 >> (P#9) >> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU >> L#10 (P#10) >> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU >> L#11 (P#11) >> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU >> L#12 (P#12) >> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU >> L#13 (P#13) >> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU >> L#14 (P#14) >> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU >> L#15 (P#15) >> HostBridge L#12 >> PCIBridge >> PCIBridge >> PCIBridge >> PCI 15b3:1003 >> Net L#8 "eth2" >> Net L#9 "ib0" >> Net L#10 "eoib0" >> OpenFabrics L#11 "mlx4_0" >> PCIBridge >> PCIBridge >> PCIBridge >> PCI 10de:1021 >> CoProc L#12 "cuda4" >> PCIBridge >> PCI 10de:1021 >> CoProc L#13 "cuda5" >> PCIBridge >> PCIBridge >> PCIBridge >> PCI 10de:1021 >> CoProc L#14 "cuda6" >> PCIBridge >> PCI 10de:1021 >> CoProc L#15 "cuda7" >> >> >> Brock Palen >> >> www.umich.edu/~brockp >> >> CAEN Advanced Computing >> XSEDE Campus Champion >> >> bro...@umich.edu >> >> (734)936-1985 >> >> >> >> >> >> >> _______________________________________________ >> hwloc-users mailing list >> >> hwloc-us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > > _______________________________________________ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
signature.asc
Description: Message signed with OpenPGP using GPGMail