Re: [OMPI devel] Any plans to support Intel MIC (Xeon Phi) in Open-MPI?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Ralph, very quick reply as I've got an SGI engineer waiting for me.. ;-) On 03/05/13 12:21, Ralph Castain wrote: > So the first problem is: how to know the Phi's are present, how > many you have on each node, etc? We could push that into something > like the hostfile, but that requires that someone build the file. > Still, it would only have to be built once, so maybe that's not too > bad - could have a "wildcard" entry if every node is the same, > etc. We're using Slurm, and it supports them already apparently, so I'm not sure if that helps? > Next, we have to launch processes across the PCI bus. We had to do > an "rsh" launch of the MPI procs onto RR's cell processors as they > appeared to be separate "hosts", though only visible on the local > node (i.e., there was a stripped-down OS running on the cell) - > Paul's cmd line implies this may also be the case here. If the same > method works here, then we have most of that code still available > (needs some updating). We would probably want to look at whether or > not binding could be supported on the Phi local OS. I believe that is the case - you can login via SSH to them is my understanding. We've not got that far with ours yet.. > Finally, we have to wire everything up. This is where RR got a > little tricky, and we may encounter the same thing here. On RR, the > cell's didn't have direct access to the interconnects - any > messaging had to be relayed by a process running on the main cpu. > So we had to create the ability to "route" MPI messages from > processes running on the cells to processes residing on other > nodes. Gotcha. > Solving the first two is relatively straightforward. In my mind, > the primary issue is the last one - does anyone know if a process > on the Phi's can "see" interconnects like a TCP NIC or an > Infiniband adaptor? I'm not sure, but I can tell you that the Intel RPMs include an OFED install that looks like it's used on the Phi (if my reading is correct). cheers, Chris - -- Christopher SamuelSenior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlGDOoAACgkQO2KABBYQAh/ZrQCgjwf5PDZWF7LYYcujxfLgiYP4 lLYAn1tMt4AQ0/Jz0o+gJMvudfEGjf99 =vQ5j -END PGP SIGNATURE-
Re: [OMPI devel] Any plans to support Intel MIC (Xeon Phi) in Open-MPI?
On May 2, 2013, at 9:18 PM, Christopher Samuel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hi Ralph, very quick reply as I've got an SGI engineer waiting for > me.. ;-) > > On 03/05/13 12:21, Ralph Castain wrote: > >> So the first problem is: how to know the Phi's are present, how >> many you have on each node, etc? We could push that into something >> like the hostfile, but that requires that someone build the file. >> Still, it would only have to be built once, so maybe that's not too >> bad - could have a "wildcard" entry if every node is the same, >> etc. > > We're using Slurm, and it supports them already apparently, so I'm not > sure if that helps? It does - but to be clear: your saying that you can directly launch processes onto the Phi's via srun? If so, then this may not be a problem, assuming you can get confirmation that the Phi's have direct access to the interconnects. If the answer to both is "yes", then just srun the MPI procs directly - we support direct launch and use PMI to wireup. Problem solved :-) And yes - that support is indeed in the 1.6 series...just configure --with-pmi. You may need to provide the path to where pmi.h is located under the slurm install, but probably not. > >> Next, we have to launch processes across the PCI bus. We had to do >> an "rsh" launch of the MPI procs onto RR's cell processors as they >> appeared to be separate "hosts", though only visible on the local >> node (i.e., there was a stripped-down OS running on the cell) - >> Paul's cmd line implies this may also be the case here. If the same >> method works here, then we have most of that code still available >> (needs some updating). We would probably want to look at whether or >> not binding could be supported on the Phi local OS. > > I believe that is the case - you can login via SSH to them is my > understanding. We've not got that far with ours yet.. > >> Finally, we have to wire everything up. This is where RR got a >> little tricky, and we may encounter the same thing here. On RR, the >> cell's didn't have direct access to the interconnects - any >> messaging had to be relayed by a process running on the main cpu. >> So we had to create the ability to "route" MPI messages from >> processes running on the cells to processes residing on other >> nodes. > > Gotcha. > >> Solving the first two is relatively straightforward. In my mind, >> the primary issue is the last one - does anyone know if a process >> on the Phi's can "see" interconnects like a TCP NIC or an >> Infiniband adaptor? > > I'm not sure, but I can tell you that the Intel RPMs include an OFED > install that looks like it's used on the Phi (if my reading is correct). > > cheers, > Chris > - -- > Christopher SamuelSenior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.org.au/ http://twitter.com/vlsci > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEARECAAYFAlGDOoAACgkQO2KABBYQAh/ZrQCgjwf5PDZWF7LYYcujxfLgiYP4 > lLYAn1tMt4AQ0/Jz0o+gJMvudfEGjf99 > =vQ5j > -END PGP SIGNATURE- > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] Any plans to support Intel MIC (Xeon Phi) in Open-MPI?
Le 03/05/2013 02:47, Ralph Castain a écrit : > Brice: do the Phis appear in the hwloc topology object? Yes, on Linux, you will see something like this in lstopo v1.7: HostBridge L#0 PCIBridge PCI 8086:225c CoProc L#2 "mic0" And these contain some attributes saying how many cores and how much memory there is in the Phi. lstopo from the host doesn't currently create cache/core/thread objects inside the Phi but you can also run lstopo inside the Phi if needed. Brice
Re: [OMPI devel] Any plans to support Intel MIC (Xeon Phi) in Open-MPI?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/05/13 14:30, Ralph Castain wrote: > On May 2, 2013, at 9:18 PM, Christopher Samuel > wrote: > >> We're using Slurm, and it supports them already apparently, so I'm >> not sure if that helps? > > It does - but to be clear: your saying that you can directly launch > processes onto the Phi's via srun? Ah no, Slurm 2.5 supports them as coprocessors, allocated as GPUs are. I've been told Slurm 2.6 (under development) may support them as nodes in their own right, but that's not something I've had time to look into myself (yet). > If so, then this may not be a problem, assuming you can get > confirmation that the Phi's have direct access to the interconnects. I'll see what I can do. There is a long README which will be my light reading on the train home tonight here: http://registrationcenter.intel.com/irc_nas/3047/readme-en.txt This seems to indicate how that works, but other parts imply that it *may* require Intel True Scale InfiniBand adapters: 3.4 Starting Intel(R) MPSS with OFED Support 1) Start the Intel(R) MPSS service. Section 2.3, "Starting Intel(R) MPSS Services" explains how. Do not proceed any further if Intel(R) MPSS is not started. 2) Start IB and HCA services. user_prompt> sudo service openibd start user_prompt> sudo service opensmd start 3) Start The Intel(R) Xeon Phi(TM) coprocessor specific OFED service. user_prompt> sudo service ofed-mic start 4) To start the experimental ccl-proxy service (see /etc/mpxyd.conf) user_prompt> sudo service mpxyd start 3.5 Stopping Intel(R) MPSS with OFED Support o If the installed version is earlier than 2.x.28xx unload the driver using: user_prompt> sudo modprobe -r mic o If the installed version is 2.x.28xx or later, unload the driver using: user_prompt> sudo service ofed-mic stop user_prompt> sudo service mpss stop user_prompt> sudo service mpss unload user_prompt> sudo service opensmd stop user_prompt> sudo service openibd stop o If the experimental ccl-proxy driver was started, unload the driver using: user_prompt> sudo service mpxyd stop > If the answer to both is "yes", then just srun the MPI procs > directly - we support direct launch and use PMI to wireup. Problem > solved :-) That would be ideal, I'll do more digging into Slurm 2.6 (we had planned on starting off with that, but as coprocessors, but this may be enough for us to change). > And yes - that support is indeed in the 1.6 series...just configure > --with-pmi. You may need to provide the path to where pmi.h is > located under the slurm install, but probably not. Brilliant, thanks! All the best, Chris - -- Christopher SamuelSenior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlGDUOMACgkQO2KABBYQAh9lcQCeIp5KjX2PJ/2Cia6fc51hSjFW 26UAn1eKqTqjZil7S8xwJrDDL5wkGof/ =2A67 -END PGP SIGNATURE-
