Re: [openib-general] MVAPICH2 SRPM and install file patches

[Shaun Rowland]

Vladimir Sokolovsky wrote:
 On Wed, 2007-01-31 at 20:32 -0500, Shaun Rowland wrote:
 I've placed the MVAPICH2 SRPM on the OFA server in ~rowland/ofed_1_2,
 and it is linked to here:

 http://www.openfabrics.org/~rowland/ofed_1_2/

 
 Hi Shaun,
 Please change mvapich2.spec to avoid using of %build macro.
 It removes RPM_BUILD_ROOT on SuSE distros:
 
 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.9418
 + umask 022
 + cd /var/tmp/OFEDRPM/BUILD
 + /bin/rm -rf /var/tmp/OFED
 ++ dirname /var/tmp/OFED
 + /bin/mkdir -p /var/tmp
 + /bin/mkdir /var/tmp/OFED
 + cd mvapich2-0.9.8
 + export OPEN_IB_HOME=/var/tmp/OFED/usr/local/ofed
 + OPEN_IB_HOME=/var/tmp/OFED/usr/local/ofed
 

Thank you for pointing out this issue on SuSE. I've made the change and
placed a new SRPM in my directory (mvapich2-0.9.8-2.src.rpm) and updated
my latest.txt file.
-- 
Shaun Rowland   [EMAIL PROTECTED]
http://www.cse.ohio-state.edu/~rowland/

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] MVAPICH2 SRPM and install file patches

[Scott Weitzenkamp (sweitzen)]

Shaun,

Thanks for doing this.

I see things like romio and shlibs configurable in the patch, what about
other MVAPICH2 features like fault tolerance, multi rail, threads, and
MPD?  How can configure them when I use install.sh to compile and
install OFED?

I also didn't quite understand the ib-vs-iwarp configuration, I thought
OFED 1.2 would support both.

Scott Weitzenkamp
SQA and Release Manager
Server Virtualization Business Unit
Cisco Systems
 

 -Original Message-
 From: [EMAIL PROTECTED] 
 [mailto:[EMAIL PROTECTED] On Behalf Of Shaun Rowland
 Sent: Wednesday, January 31, 2007 5:33 PM
 To: [EMAIL PROTECTED]
 Cc: [EMAIL PROTECTED]; openib-general@openib.org
 Subject: [openib-general] MVAPICH2 SRPM and install file patches
 
 I've placed the MVAPICH2 SRPM on the OFA server in ~rowland/ofed_1_2,
 and it is linked to here:
 
 http://www.openfabrics.org/~rowland/ofed_1_2/
 
 Additionally, I am including a patch in this email that updates the
 ofed_1_2_scripts files from the GIT repository we were given to
 handle the MVAPICH2 SRPM file. Basically, installing MVAPICH2 
 is similar
 to the other MPI packages, except that I have added a choice option to
 build with iWARP support or not. The default is IB only. If 
 the user has
 selected the librdmacm packages and the mvapich2 package, 
 this choice is
 presented. This is also saved in the ofed.conf file using an
 MVAPICH2_IMPL variable, and the librdmacm packages are added as
 dependencies if the iWARP version of MVAPICH2 is desired and they are
 not already in the ofed.conf file, which seems like standard 
 behavior in
 the scripts. The resulting binary RPM uses the name convention
 mvapich2_compiler as normal in either case. There are various ways
 this could be implemented, perhaps in a better manner. This is what I
 was able to come up with by today. Since the installation 
 scripts given
 were very similar to the original OFED 1.1 scripts, I was able to test
 the installation procedure using OFED 1.1 files. Everything worked for
 me, including building the mpitests package against the mvapich2
 package. There are some comments about this in what I have 
 done. I hope
 that it is helpful in getting our SRPM integrated into the 
 installation
 scripts.
 
 Additionally, I put a README file in my ofed_1_2 directory 
 that contains
 information about the macros that can be used with our SRPM file. The
 SRPM can be used to install against an existing OFED installation, and
 those macros control various aspects of the result. There is 
 one special
 macro I use for when the SRPM is being built along with the 
 OFED source,
 and its use should be clear in the patched build.sh script and
 associated comment.
 -- 
 Shaun Rowland [EMAIL PROTECTED]
 http://www.cse.ohio-state.edu/~rowland/
 

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] MVAPICH2 SRPM and install file patches

[Shaun Rowland]

Scott Weitzenkamp (sweitzen) wrote:
 Shaun,
 
 Thanks for doing this.
 
 I see things like romio and shlibs configurable in the patch, what about
 other MVAPICH2 features like fault tolerance, multi rail, threads, and
 MPD?  How can configure them when I use install.sh to compile and
 install OFED?

Hi Scott. I had thought about this a little when I was testing with the
install/build scripts Vlad gave us. I would appreciate his input if I
get anything wrong here as well. From the perspective of the user
running the install.sh script, the MPI packages are essentially built
one way. You do get to pick the compiler(s) to use, but as for other
options - you would have to edit the build.sh function associated with
the desired package. I created a hack for the iwarp vs ib configuration
for MVAPICH2 because I needed to distinguish between the two (for
reasons I will outline at the end of this message).

Theoretically, you should be able to export the proper variables from
our make.mvapich2.* scripts before running the install.sh script, and
the features would be enabled. For instance, you could do:

export MULTI_THREAD=yes
./install.sh

This is not a good solution for installing OFED, but should work due to
not conflicting with anything else - at least that I am aware. I see
that I need to update the make.mvapich2.iwarp script to have the
multithreading option anyway as well, so it would not quite work 100%
right now.

As far as each feature you asked about:

* fault tolerance
  - this is controlled during the build process with $ENABLE_CKPT 
and requires $BLCR_HOME pointing to a BLCR installation.  This only 
works for single threaded builds without rdmacm support (the ib case 
only, essentially).

* multi rail
  - this is controlled by runtime environment variables after 
installation.

* threads
  - This is controlled by $MULTI_THREAD during the build process. 
As noted above, there's a restriction with fault tolerance.

* MPD
  - MPD is used by MVAPICH2 as it is based on MPICH2.

There are actually a number of options that could be chosen. I believe
from our side, it will be good for me to go ahead and put these in our
SRPM now. Our SRPM can be used outside of the OFED installation system
of course, and these should really be there. There are even other
devices, like uDAPL.

I did the SRPM in the install/build script patches the way I did
because that seemed like a good set of options for how the OFED
installation system works. There's no framework or examples of asking
about features to build in an MPI package. I just quickly tacked on the
iwarp question and made up a new configuration variable for the
ofed.conf file, but it's not necessarily a good way to do it.

One possibility would be to create a shell function that sets various
build options for MPI packages. Variables could be set in this function
using some name convention, in our case perhaps MVAPICH2_OPT_whatever.
In such a function (probably one for each package, that seems to be the
convention), it would be easier to code all the exceptions for features
- if there are any. There are some in our case, as I've mentioned. This
configuration function could be called when the user is choosing to
install MVAPICH2.

This leads to a number of problems. Can the user select different
options for each of the compiler versions of the MPI package? I think
clearly the answer should be no. Even as implemented now, you cannot
install the iwarp and ib version of MVAPICH2 at the same time during the
install process. You must choose one or the other. Being able to do
either would require one of two changes:

1. Having another level of installer system configuration where I could
selected the devices desired, and options for each device (by device
here, I mean uDAPL, IB, iWARP).

- or -

2. Make multiple RPM packages to fit into how the installer currently
interacts with SRPMs, prompts, etc.

I've only had a limitted time to investigate this, so what I have done
so far mostly fits with how the OFED install system does things with the
other packages - except for my iwarp vs ib question prompt. I think
there's potential for a lot of compilication here. A configuration
function for each package would be one possible way to contain that,
however I'd have to go back and check out how things work again to see
how something like that would fit in.

So, I will add these new feature options to our SRPM because they could
be used outside of the OFED installation system anyway, and we would
like that to be possible and give the ability to set these options.
However, I cannot say what would be best for the OFED installation
system. It might be better to just go with what we have now - more
mainstream builds, and let the user do their own build if they want to
highly customize or something. Otherwise, I've given one possible idea
from the perspective of someone who is new to the install system. Vlad,
do you have any opinion here? Do you see where I am coming from 

Re: [openib-general] MVAPICH2 SRPM and install file patches

[Steve Wise]

  I also didn't quite understand the ib-vs-iwarp configuration, I thought
  OFED 1.2 would support both.
 
 There are 2 reasons our SRPM has to be told whether it is being built
 for iWARP or IB:
 
 1. We need to use -DRDMA_CM_RNIC during the build for iWARP (this is
 actually done by invoking our make.mvapich2.iwarp script in the RPM build).

I believe the iWARP build will work over IB too.  The difference, I
think, is that the iWARP build uses the RDMA-CM and the IB build uses
the IB-CM.  

Shaun, is this correct?  

If so, I suggest you define these options differently.  Perhaps IBCM vs
RDMACM? Right now it implies that you cannot run the same mvapich build
over both transports.  

My 2 cents.


Steve.



___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [openib-general] MVAPICH2 SRPM and install file patches

[Vladimir Sokolovsky]

On Wed, 2007-01-31 at 20:32 -0500, Shaun Rowland wrote:
 I've placed the MVAPICH2 SRPM on the OFA server in ~rowland/ofed_1_2,
 and it is linked to here:
 
 http://www.openfabrics.org/~rowland/ofed_1_2/

ofed_1_2_scripts.patch applied.

Thanks,



-- 
Vladimir Sokolovsky [EMAIL PROTECTED]
Mellanox Technologies Ltd.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

[openib-general] MVAPICH2 SRPM and install file patches

[Shaun Rowland]

I've placed the MVAPICH2 SRPM on the OFA server in ~rowland/ofed_1_2,
and it is linked to here:

http://www.openfabrics.org/~rowland/ofed_1_2/

Additionally, I am including a patch in this email that updates the
ofed_1_2_scripts files from the GIT repository we were given to
handle the MVAPICH2 SRPM file. Basically, installing MVAPICH2 is similar
to the other MPI packages, except that I have added a choice option to
build with iWARP support or not. The default is IB only. If the user has
selected the librdmacm packages and the mvapich2 package, this choice is
presented. This is also saved in the ofed.conf file using an
MVAPICH2_IMPL variable, and the librdmacm packages are added as
dependencies if the iWARP version of MVAPICH2 is desired and they are
not already in the ofed.conf file, which seems like standard behavior in
the scripts. The resulting binary RPM uses the name convention
mvapich2_compiler as normal in either case. There are various ways
this could be implemented, perhaps in a better manner. This is what I
was able to come up with by today. Since the installation scripts given
were very similar to the original OFED 1.1 scripts, I was able to test
the installation procedure using OFED 1.1 files. Everything worked for
me, including building the mpitests package against the mvapich2
package. There are some comments about this in what I have done. I hope
that it is helpful in getting our SRPM integrated into the installation
scripts.

Additionally, I put a README file in my ofed_1_2 directory that contains
information about the macros that can be used with our SRPM file. The
SRPM can be used to install against an existing OFED installation, and
those macros control various aspects of the result. There is one special
macro I use for when the SRPM is being built along with the OFED source,
and its use should be clear in the patched build.sh script and
associated comment.
--
Shaun Rowland   [EMAIL PROTECTED]
http://www.cse.ohio-state.edu/~rowland/
diff --git a/build.sh b/build.sh
index c4fe469..380bd98 100755
--- a/build.sh
+++ b/build.sh
@@ -428,6 +428,130 @@ mpi_osu()
 return 0
 }
 
+mvapich2()
+{
+local iwarp=0
+
+if [ $MVAPICH2_IMPL = iwarp ]; then
+iwarp=1
+fi
+
+echo
+
+if [ $iwarp -eq 0 ]; then
+echo Building the MVAPICH2 RPM with IB support. Please wait...
+else
+echo Building the MVAPICH2 RPM with iWARP support. Please wait...
+fi
+
+echo
+
+for mpi_comp in ${MPI_COMPILER_mvapich2}
+do
+
MVAPICH2_RPM=${MVAPICH2_NAME}_${mpi_comp}-${MVAPICH2_VER}-${MVAPICH2_REL}.${build_arch}.rpm
+
MVAPICH2_PREFIX=${STACK_PREFIX}/mpi/${mpi_comp}/${MVAPICH2_NAME}-${MVAPICH2_VERSION}
+
+case ${mpi_comp} in
+gcc)
+MVAPICH2_COMP_ENV=CC=gcc CXX=g++
+
+if [ $is_gfortran -eq 1 ]; then
+MVAPICH2_COMP_ENV=$MVAPICH2_COMP_ENV F77=gfortran
+elif [ $is_gcc_g77 -eq 1 ]; then
+MVAPICH2_COMP_ENV=$MVAPICH2_COMP_ENV F77=g77
+fi
+;;
+pathscale)
+MVAPICH2_COMP_ENV=CC=pathcc CXX=pathCC F77=pathf90 
F90=pathf90
+
+# On i686 the PathScale compiler requires -g optimization
+# for MVAPICH2 in the shared library configuration.
+if [ $ARCH = i686 ]; then
+MVAPICH2_COMP_ENV=$MVAPICH2_COMP_ENV OPT_FLAG=-g
+fi
+;;
+pgi)
+MVAPICH2_COMP_ENV=CC=pgcc CXX=pgCC F77=pgf77 F90=pgf90
+;;
+intel)
+# The -i-dynamic flag is required for MVAPICH2 in the shared
+# library configuration.
+MVAPICH2_COMP_ENV='CC=icc -i-dynamic CXX=icpc -i-dynamic 
F77=ifort -i-dynamic F90=ifort -i-dynamic'
+;;
+esac
+
+ex rpmbuild --rebuild \
+   --define \'_topdir ${RPM_DIR}\' \
+   --define \'_name ${MVAPICH2_NAME}_${mpi_comp}\' \
+   --define \'_prefix ${MVAPICH2_PREFIX}\' \
+   --define \'build_root ${BUILD_ROOT}\' \
+   --define \'open_ib_home ${STACK_PREFIX}\' \
+   --define \'ofed_build_root ${BUILD_ROOT}\' \
+   --define \'comp_env ${MVAPICH2_COMP_ENV}\' \
+   --define \'iwarp ${iwarp}\' \
+   --define \'romio 1\' \
+   --define \'shared_libs 1\' \
+   --define \'auto_req 1\' \
+   $MVAPICH2_SRC_RPM
+ex $MV -f ${RPM_DIR}/RPMS/$build_arch/${MVAPICH2_RPM} $RPMS
+let BUILD_COUNTER++
+
+if [ $mpitests == y ]; then
+echo
+echo Building the mpitests RPM over MVAPICH2 using the 
${mpi_comp} compiler. Please wait...
+echo
+
MPITESTS_RPM=${MPITESTS_NAME}_${MVAPICH2_NAME}_${mpi_comp}-${MPITESTS_VERSION}.${build_arch}.rpm
+
+# rowland: The MVAPICH2 SRPM was built above by specifying
+# ofed_build_root (set to the same