Re: [openib-general] MVAPICH2 SRPM and install file patches
Vladimir Sokolovsky wrote: On Wed, 2007-01-31 at 20:32 -0500, Shaun Rowland wrote: I've placed the MVAPICH2 SRPM on the OFA server in ~rowland/ofed_1_2, and it is linked to here: http://www.openfabrics.org/~rowland/ofed_1_2/ Hi Shaun, Please change mvapich2.spec to avoid using of %build macro. It removes RPM_BUILD_ROOT on SuSE distros: Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.9418 + umask 022 + cd /var/tmp/OFEDRPM/BUILD + /bin/rm -rf /var/tmp/OFED ++ dirname /var/tmp/OFED + /bin/mkdir -p /var/tmp + /bin/mkdir /var/tmp/OFED + cd mvapich2-0.9.8 + export OPEN_IB_HOME=/var/tmp/OFED/usr/local/ofed + OPEN_IB_HOME=/var/tmp/OFED/usr/local/ofed Thank you for pointing out this issue on SuSE. I've made the change and placed a new SRPM in my directory (mvapich2-0.9.8-2.src.rpm) and updated my latest.txt file. -- Shaun Rowland [EMAIL PROTECTED] http://www.cse.ohio-state.edu/~rowland/ ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] MVAPICH2 SRPM and install file patches
Shaun, Thanks for doing this. I see things like romio and shlibs configurable in the patch, what about other MVAPICH2 features like fault tolerance, multi rail, threads, and MPD? How can configure them when I use install.sh to compile and install OFED? I also didn't quite understand the ib-vs-iwarp configuration, I thought OFED 1.2 would support both. Scott Weitzenkamp SQA and Release Manager Server Virtualization Business Unit Cisco Systems -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Shaun Rowland Sent: Wednesday, January 31, 2007 5:33 PM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; openib-general@openib.org Subject: [openib-general] MVAPICH2 SRPM and install file patches I've placed the MVAPICH2 SRPM on the OFA server in ~rowland/ofed_1_2, and it is linked to here: http://www.openfabrics.org/~rowland/ofed_1_2/ Additionally, I am including a patch in this email that updates the ofed_1_2_scripts files from the GIT repository we were given to handle the MVAPICH2 SRPM file. Basically, installing MVAPICH2 is similar to the other MPI packages, except that I have added a choice option to build with iWARP support or not. The default is IB only. If the user has selected the librdmacm packages and the mvapich2 package, this choice is presented. This is also saved in the ofed.conf file using an MVAPICH2_IMPL variable, and the librdmacm packages are added as dependencies if the iWARP version of MVAPICH2 is desired and they are not already in the ofed.conf file, which seems like standard behavior in the scripts. The resulting binary RPM uses the name convention mvapich2_compiler as normal in either case. There are various ways this could be implemented, perhaps in a better manner. This is what I was able to come up with by today. Since the installation scripts given were very similar to the original OFED 1.1 scripts, I was able to test the installation procedure using OFED 1.1 files. Everything worked for me, including building the mpitests package against the mvapich2 package. There are some comments about this in what I have done. I hope that it is helpful in getting our SRPM integrated into the installation scripts. Additionally, I put a README file in my ofed_1_2 directory that contains information about the macros that can be used with our SRPM file. The SRPM can be used to install against an existing OFED installation, and those macros control various aspects of the result. There is one special macro I use for when the SRPM is being built along with the OFED source, and its use should be clear in the patched build.sh script and associated comment. -- Shaun Rowland [EMAIL PROTECTED] http://www.cse.ohio-state.edu/~rowland/ ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] MVAPICH2 SRPM and install file patches
Scott Weitzenkamp (sweitzen) wrote: Shaun, Thanks for doing this. I see things like romio and shlibs configurable in the patch, what about other MVAPICH2 features like fault tolerance, multi rail, threads, and MPD? How can configure them when I use install.sh to compile and install OFED? Hi Scott. I had thought about this a little when I was testing with the install/build scripts Vlad gave us. I would appreciate his input if I get anything wrong here as well. From the perspective of the user running the install.sh script, the MPI packages are essentially built one way. You do get to pick the compiler(s) to use, but as for other options - you would have to edit the build.sh function associated with the desired package. I created a hack for the iwarp vs ib configuration for MVAPICH2 because I needed to distinguish between the two (for reasons I will outline at the end of this message). Theoretically, you should be able to export the proper variables from our make.mvapich2.* scripts before running the install.sh script, and the features would be enabled. For instance, you could do: export MULTI_THREAD=yes ./install.sh This is not a good solution for installing OFED, but should work due to not conflicting with anything else - at least that I am aware. I see that I need to update the make.mvapich2.iwarp script to have the multithreading option anyway as well, so it would not quite work 100% right now. As far as each feature you asked about: * fault tolerance - this is controlled during the build process with $ENABLE_CKPT and requires $BLCR_HOME pointing to a BLCR installation. This only works for single threaded builds without rdmacm support (the ib case only, essentially). * multi rail - this is controlled by runtime environment variables after installation. * threads - This is controlled by $MULTI_THREAD during the build process. As noted above, there's a restriction with fault tolerance. * MPD - MPD is used by MVAPICH2 as it is based on MPICH2. There are actually a number of options that could be chosen. I believe from our side, it will be good for me to go ahead and put these in our SRPM now. Our SRPM can be used outside of the OFED installation system of course, and these should really be there. There are even other devices, like uDAPL. I did the SRPM in the install/build script patches the way I did because that seemed like a good set of options for how the OFED installation system works. There's no framework or examples of asking about features to build in an MPI package. I just quickly tacked on the iwarp question and made up a new configuration variable for the ofed.conf file, but it's not necessarily a good way to do it. One possibility would be to create a shell function that sets various build options for MPI packages. Variables could be set in this function using some name convention, in our case perhaps MVAPICH2_OPT_whatever. In such a function (probably one for each package, that seems to be the convention), it would be easier to code all the exceptions for features - if there are any. There are some in our case, as I've mentioned. This configuration function could be called when the user is choosing to install MVAPICH2. This leads to a number of problems. Can the user select different options for each of the compiler versions of the MPI package? I think clearly the answer should be no. Even as implemented now, you cannot install the iwarp and ib version of MVAPICH2 at the same time during the install process. You must choose one or the other. Being able to do either would require one of two changes: 1. Having another level of installer system configuration where I could selected the devices desired, and options for each device (by device here, I mean uDAPL, IB, iWARP). - or - 2. Make multiple RPM packages to fit into how the installer currently interacts with SRPMs, prompts, etc. I've only had a limitted time to investigate this, so what I have done so far mostly fits with how the OFED install system does things with the other packages - except for my iwarp vs ib question prompt. I think there's potential for a lot of compilication here. A configuration function for each package would be one possible way to contain that, however I'd have to go back and check out how things work again to see how something like that would fit in. So, I will add these new feature options to our SRPM because they could be used outside of the OFED installation system anyway, and we would like that to be possible and give the ability to set these options. However, I cannot say what would be best for the OFED installation system. It might be better to just go with what we have now - more mainstream builds, and let the user do their own build if they want to highly customize or something. Otherwise, I've given one possible idea from the perspective of someone who is new to the install system. Vlad, do you have any opinion here? Do you see where I am coming from
Re: [openib-general] MVAPICH2 SRPM and install file patches
I also didn't quite understand the ib-vs-iwarp configuration, I thought OFED 1.2 would support both. There are 2 reasons our SRPM has to be told whether it is being built for iWARP or IB: 1. We need to use -DRDMA_CM_RNIC during the build for iWARP (this is actually done by invoking our make.mvapich2.iwarp script in the RPM build). I believe the iWARP build will work over IB too. The difference, I think, is that the iWARP build uses the RDMA-CM and the IB build uses the IB-CM. Shaun, is this correct? If so, I suggest you define these options differently. Perhaps IBCM vs RDMACM? Right now it implies that you cannot run the same mvapich build over both transports. My 2 cents. Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] MVAPICH2 SRPM and install file patches
On Wed, 2007-01-31 at 20:32 -0500, Shaun Rowland wrote: I've placed the MVAPICH2 SRPM on the OFA server in ~rowland/ofed_1_2, and it is linked to here: http://www.openfabrics.org/~rowland/ofed_1_2/ ofed_1_2_scripts.patch applied. Thanks, -- Vladimir Sokolovsky [EMAIL PROTECTED] Mellanox Technologies Ltd. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] MVAPICH2 SRPM and install file patches
I've placed the MVAPICH2 SRPM on the OFA server in ~rowland/ofed_1_2, and it is linked to here: http://www.openfabrics.org/~rowland/ofed_1_2/ Additionally, I am including a patch in this email that updates the ofed_1_2_scripts files from the GIT repository we were given to handle the MVAPICH2 SRPM file. Basically, installing MVAPICH2 is similar to the other MPI packages, except that I have added a choice option to build with iWARP support or not. The default is IB only. If the user has selected the librdmacm packages and the mvapich2 package, this choice is presented. This is also saved in the ofed.conf file using an MVAPICH2_IMPL variable, and the librdmacm packages are added as dependencies if the iWARP version of MVAPICH2 is desired and they are not already in the ofed.conf file, which seems like standard behavior in the scripts. The resulting binary RPM uses the name convention mvapich2_compiler as normal in either case. There are various ways this could be implemented, perhaps in a better manner. This is what I was able to come up with by today. Since the installation scripts given were very similar to the original OFED 1.1 scripts, I was able to test the installation procedure using OFED 1.1 files. Everything worked for me, including building the mpitests package against the mvapich2 package. There are some comments about this in what I have done. I hope that it is helpful in getting our SRPM integrated into the installation scripts. Additionally, I put a README file in my ofed_1_2 directory that contains information about the macros that can be used with our SRPM file. The SRPM can be used to install against an existing OFED installation, and those macros control various aspects of the result. There is one special macro I use for when the SRPM is being built along with the OFED source, and its use should be clear in the patched build.sh script and associated comment. -- Shaun Rowland [EMAIL PROTECTED] http://www.cse.ohio-state.edu/~rowland/ diff --git a/build.sh b/build.sh index c4fe469..380bd98 100755 --- a/build.sh +++ b/build.sh @@ -428,6 +428,130 @@ mpi_osu() return 0 } +mvapich2() +{ +local iwarp=0 + +if [ $MVAPICH2_IMPL = iwarp ]; then +iwarp=1 +fi + +echo + +if [ $iwarp -eq 0 ]; then +echo Building the MVAPICH2 RPM with IB support. Please wait... +else +echo Building the MVAPICH2 RPM with iWARP support. Please wait... +fi + +echo + +for mpi_comp in ${MPI_COMPILER_mvapich2} +do + MVAPICH2_RPM=${MVAPICH2_NAME}_${mpi_comp}-${MVAPICH2_VER}-${MVAPICH2_REL}.${build_arch}.rpm + MVAPICH2_PREFIX=${STACK_PREFIX}/mpi/${mpi_comp}/${MVAPICH2_NAME}-${MVAPICH2_VERSION} + +case ${mpi_comp} in +gcc) +MVAPICH2_COMP_ENV=CC=gcc CXX=g++ + +if [ $is_gfortran -eq 1 ]; then +MVAPICH2_COMP_ENV=$MVAPICH2_COMP_ENV F77=gfortran +elif [ $is_gcc_g77 -eq 1 ]; then +MVAPICH2_COMP_ENV=$MVAPICH2_COMP_ENV F77=g77 +fi +;; +pathscale) +MVAPICH2_COMP_ENV=CC=pathcc CXX=pathCC F77=pathf90 F90=pathf90 + +# On i686 the PathScale compiler requires -g optimization +# for MVAPICH2 in the shared library configuration. +if [ $ARCH = i686 ]; then +MVAPICH2_COMP_ENV=$MVAPICH2_COMP_ENV OPT_FLAG=-g +fi +;; +pgi) +MVAPICH2_COMP_ENV=CC=pgcc CXX=pgCC F77=pgf77 F90=pgf90 +;; +intel) +# The -i-dynamic flag is required for MVAPICH2 in the shared +# library configuration. +MVAPICH2_COMP_ENV='CC=icc -i-dynamic CXX=icpc -i-dynamic F77=ifort -i-dynamic F90=ifort -i-dynamic' +;; +esac + +ex rpmbuild --rebuild \ + --define \'_topdir ${RPM_DIR}\' \ + --define \'_name ${MVAPICH2_NAME}_${mpi_comp}\' \ + --define \'_prefix ${MVAPICH2_PREFIX}\' \ + --define \'build_root ${BUILD_ROOT}\' \ + --define \'open_ib_home ${STACK_PREFIX}\' \ + --define \'ofed_build_root ${BUILD_ROOT}\' \ + --define \'comp_env ${MVAPICH2_COMP_ENV}\' \ + --define \'iwarp ${iwarp}\' \ + --define \'romio 1\' \ + --define \'shared_libs 1\' \ + --define \'auto_req 1\' \ + $MVAPICH2_SRC_RPM +ex $MV -f ${RPM_DIR}/RPMS/$build_arch/${MVAPICH2_RPM} $RPMS +let BUILD_COUNTER++ + +if [ $mpitests == y ]; then +echo +echo Building the mpitests RPM over MVAPICH2 using the ${mpi_comp} compiler. Please wait... +echo + MPITESTS_RPM=${MPITESTS_NAME}_${MVAPICH2_NAME}_${mpi_comp}-${MPITESTS_VERSION}.${build_arch}.rpm + +# rowland: The MVAPICH2 SRPM was built above by specifying +# ofed_build_root (set to the same