RE: [easybuild] MODULEPATH issue when supporting intel and intelcuda toolchain concurrently

[Alan O'Cais]

We have used a toolchain-based naming scheme in the past and moved away
from it because it was too complicated for users (too many levels in the
hierarchy and the naming was very non-intuitive.

On 3 Mar 2017 6:18 pm, "Siddiqui, Shahzeb" <shahzeb.siddi...@pfizer.com>
wrote:

> Hello,
>
>
>
> Specifically for Intel I think for HMNS treat icc and ifort as packages
> and only toolchain have MODULEPATH
>
>
>
> The solution I think could be.
>
> Remove from icc and ifort module the  modulepath for Compiler/intel
>
> Rather have iccifort module have a modulePath for iccifort
>
>
>
> When loading icc or ifort separately you wouldn’t see the modules built
> with intel or iccifortcuda, but rather you would see packages when loading
> the toolchain: iccifort or intel
>
>
>
> impi module would reside in iccifort or iccifortcuda dependening on
> toolchain. The module would be in separate trees.
>
>
>
> When loading either iccifort or iccifortcuda you should see the impi from
> its toolchain tree. The same goes with imkl, it should be in a tree for
> iimpi or iimpic since that is the toolchain used. Both intel and intelcuda
> should be treated as separate module trees. When loading intel it should
> have MODULEPATH for Compiler/intel and intelcuda should have MODULEPATH for
> Compiler/intel-CUDA.
>
>
>
> Intel should load  -> iccifort, iimpi, imkl
>
> Intel Cuda -> iccifortcuda, iimpic, imkl
>
>
>
> No need to have explicit load for icc and ifort in the intel module since
> the iccifort toolchain will take care of this. Keep in mind both iccifort
> and iccifortcuda will load icc and ifort that will come from the same
> module. They would reside under Core/
>
>
>
> If this is done this way we can see the intelcuda tree while having icc
> and ifort loaded since they wouldn’t be loading the intel. In my
> environment, I treat all toolchains as family so that I can’t have an
> instance where intel and intelcuda are loaded at the same time.
>
>
>
>
>
>
>
> *From:* easybuild-requ...@lists.ugent.be [mailto:easybuild-request@
> lists.ugent.be] *On Behalf Of *Alan O'Cais
> *Sent:* Thursday, March 2, 2017 10:32 AM
> *To:* easybuild@lists.ugent.be
> *Subject:* Re: [easybuild] MODULEPATH issue when supporting intel and
> intelcuda toolchain concurrently
>
>
>
> Dear Shahzeb,
>
>
>
> I think this is probably the same (or at least related to the) issue that
> is being discussed in https://github.com/hpcugent/easybuild-framework/
> pull/2135
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hpcugent_easybuild-2Dframework_pull_2135=DwMFaQ=UE1eNsedaKncO0Yl_u8bfw=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg=CsWNr_6qdkJE_RoT54a_jmrskakWb47pfZcmQpcT4-Q=rtu9QLIWL9XOUXetPh9eal99n64mPjldU22rDdTP9GE=>
>
>
>
> It also exposes one of the problems of a HMNS, the potential
> non-uniqueness of module names. The problem with not building software with
> minimal toolchains is that you can have multiple copies at various levels
> of your toolchain hierarchy. What module you end up loading is then
> dependent on the order that you load things (perhaps not in Lmod because it
> is hierarchy-aware but definitely for other module tools). This can clearly
> lead to issues.
>
>
>
> Alan
>
>
>
> On 2 March 2017 at 15:32, Siddiqui, Shahzeb <shahzeb.siddi...@pfizer.com>
> wrote:
>
> Hello,
>
>
>
> I seem to notice an issue when building modules using
> HierarchicalNamingScheme when building out the intel and intelcuda
> toolchains.
>
>
>
> I notice that MODULEPATH is set for icc and ifort for intel directory.
> This is correct when setting up intel toolchain.
>
>
>
> hpcswadm@hpcv18$grep -iR MODULEPATH
>
> icc/2017.1.132-GCC-5.2.0.lua:prepend_path("MODULEPATH",
> "/nfs/grid/software/testing/RHEL7/easybuild/modules/all/
> Compiler/intel/2017.1.132-GCC-5.2.0")
>
> ifort/2017.1.132-GCC-5.2.0.lua:prepend_path("MODULEPATH",
> "/nfs/grid/software/testing/RHEL7/easybuild/modules/all/
> Compiler/intel/2017.1.132-GCC-5.2.0")
>
>
>
> impi  gets installed in the path for intel as expected.
>
>
>
> hpcswadm@hpcv18$ls -R /nfs/grid/software/testing/
> RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/
>
> /nfs/grid/software/testing/RHEL7/easybuild/modules/all/
> Compiler/intel/2017.1.132-GCC-5.2.0/:
>
> impi
>
>
>
> /nfs/grid/software/testing/RHEL7/easybuild/modules/all/
> Compiler/intel/2017.1.132-GCC-5.2.0/impi:
>
> 2017.1.132.lua
>
>
>
> As for impi built with iccifortcuda it gets installed in
>
>
>
> hpcswadm@hpcv18$ls -R

RE: [easybuild] MODULEPATH issue when supporting intel and intelcuda toolchain concurrently

[Siddiqui, Shahzeb]

Hello,

Specifically for Intel I think for HMNS treat icc and ifort as packages and 
only toolchain have MODULEPATH

The solution I think could be.
Remove from icc and ifort module the  modulepath for Compiler/intel
Rather have iccifort module have a modulePath for iccifort

When loading icc or ifort separately you wouldn’t see the modules built with 
intel or iccifortcuda, but rather you would see packages when loading the 
toolchain: iccifort or intel

impi module would reside in iccifort or iccifortcuda dependening on toolchain. 
The module would be in separate trees.

When loading either iccifort or iccifortcuda you should see the impi from its 
toolchain tree. The same goes with imkl, it should be in a tree for iimpi or 
iimpic since that is the toolchain used. Both intel and intelcuda should be 
treated as separate module trees. When loading intel it should have MODULEPATH 
for Compiler/intel and intelcuda should have MODULEPATH for Compiler/intel-CUDA.

Intel should load  -> iccifort, iimpi, imkl
Intel Cuda -> iccifortcuda, iimpic, imkl

No need to have explicit load for icc and ifort in the intel module since the 
iccifort toolchain will take care of this. Keep in mind both iccifort and 
iccifortcuda will load icc and ifort that will come from the same module. They 
would reside under Core/

If this is done this way we can see the intelcuda tree while having icc and 
ifort loaded since they wouldn’t be loading the intel. In my environment, I 
treat all toolchains as family so that I can’t have an instance where intel and 
intelcuda are loaded at the same time.



From: easybuild-requ...@lists.ugent.be 
[mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Alan O'Cais
Sent: Thursday, March 2, 2017 10:32 AM
To: easybuild@lists.ugent.be
Subject: Re: [easybuild] MODULEPATH issue when supporting intel and intelcuda 
toolchain concurrently

Dear Shahzeb,

I think this is probably the same (or at least related to the) issue that is 
being discussed in 
https://github.com/hpcugent/easybuild-framework/pull/2135<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hpcugent_easybuild-2Dframework_pull_2135=DwMFaQ=UE1eNsedaKncO0Yl_u8bfw=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg=CsWNr_6qdkJE_RoT54a_jmrskakWb47pfZcmQpcT4-Q=rtu9QLIWL9XOUXetPh9eal99n64mPjldU22rDdTP9GE=>

It also exposes one of the problems of a HMNS, the potential non-uniqueness of 
module names. The problem with not building software with minimal toolchains is 
that you can have multiple copies at various levels of your toolchain 
hierarchy. What module you end up loading is then dependent on the order that 
you load things (perhaps not in Lmod because it is hierarchy-aware but 
definitely for other module tools). This can clearly lead to issues.

Alan

On 2 March 2017 at 15:32, Siddiqui, Shahzeb 
<shahzeb.siddi...@pfizer.com<mailto:shahzeb.siddi...@pfizer.com>> wrote:
Hello,

I seem to notice an issue when building modules using HierarchicalNamingScheme 
when building out the intel and intelcuda toolchains.

I notice that MODULEPATH is set for icc and ifort for intel directory. This is 
correct when setting up intel toolchain.

hpcswadm@hpcv18$grep -iR MODULEPATH
icc/2017.1.132-GCC-5.2.0.lua:prepend_path("MODULEPATH", 
"/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0")
ifort/2017.1.132-GCC-5.2.0.lua:prepend_path("MODULEPATH", 
"/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0")

impi  gets installed in the path for intel as expected.

hpcswadm@hpcv18$ls -R 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/:
impi

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/impi:
2017.1.132.lua

As for impi built with iccifortcuda it gets installed in

hpcswadm@hpcv18$ls -R 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/:
2017.1.132-GCC-5.2.0-7.5.18

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/2017.1.132-GCC-5.2.0-7.5.18:
impi

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/2017.1.132-GCC-5.2.0-7.5.18/impi:
2017.1.132.lua

The problem is when loading iimpic toolchain it loads up icc and ifort modules 
along with impi that belongs to the module tree from intel and not intel-CUDA. 
I am not sure if this is a problem, but it seems like the impi is not being 
picked up correctly.

hpcswadm@hpcv18$ml av iimpi
- 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Core 
-
   iimpi/2017.01-GCC-5.2.0 (TC)iimpic/2017.01-GCC-5.2.0 (TC)

hpcswadm@hpcv18$ml

Currently Loaded Modules:
  1) EasyBuil

Re: [easybuild] MODULEPATH issue when supporting intel and intelcuda toolchain concurrently

[Alan O'Cais]

At JSC we handle this issue by treating CUDA as a simple dependency of packages 
built at the compiler level, we only incorporate it into a toolchain when we 
use a CUDA-aware MPI (which means that the MODULEPATH expansion only happens 
once rather than twice, once for CUDA and once for MPI). Since our MPI 
implementations are in a "family" this is very safe. It also has very little 
side-effects because how CUDA is included is very heterogeneous across packages 
and typically needs to be implemented by hand anyway.

On 2 March 2017 at 16:31, Alan O'Cais 
> wrote:
Dear Shahzeb,

I think this is probably the same (or at least related to the) issue that is 
being discussed in https://github.com/hpcugent/easybuild-framework/pull/2135

It also exposes one of the problems of a HMNS, the potential non-uniqueness of 
module names. The problem with not building software with minimal toolchains is 
that you can have multiple copies at various levels of your toolchain 
hierarchy. What module you end up loading is then dependent on the order that 
you load things (perhaps not in Lmod because it is hierarchy-aware but 
definitely for other module tools). This can clearly lead to issues.

Alan

On 2 March 2017 at 15:32, Siddiqui, Shahzeb 
> wrote:
Hello,

I seem to notice an issue when building modules using HierarchicalNamingScheme 
when building out the intel and intelcuda toolchains.

I notice that MODULEPATH is set for icc and ifort for intel directory. This is 
correct when setting up intel toolchain.

hpcswadm@hpcv18$grep -iR MODULEPATH
icc/2017.1.132-GCC-5.2.0.lua:prepend_path("MODULEPATH", 
"/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0")
ifort/2017.1.132-GCC-5.2.0.lua:prepend_path("MODULEPATH", 
"/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0")

impi  gets installed in the path for intel as expected.

hpcswadm@hpcv18$ls -R 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/:
impi

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/impi:
2017.1.132.lua

As for impi built with iccifortcuda it gets installed in

hpcswadm@hpcv18$ls -R 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/:
2017.1.132-GCC-5.2.0-7.5.18

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/2017.1.132-GCC-5.2.0-7.5.18:
impi

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/2017.1.132-GCC-5.2.0-7.5.18/impi:
2017.1.132.lua

The problem is when loading iimpic toolchain it loads up icc and ifort modules 
along with impi that belongs to the module tree from intel and not intel-CUDA. 
I am not sure if this is a problem, but it seems like the impi is not being 
picked up correctly.

hpcswadm@hpcv18$ml av iimpi
- 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Core 
-
   iimpi/2017.01-GCC-5.2.0 (TC)iimpic/2017.01-GCC-5.2.0 (TC)

hpcswadm@hpcv18$ml

Currently Loaded Modules:
  1) EasyBuild/3.1.0

hpcswadm@hpcv18$ml iimpic
Currently Loaded Modules:
  1) EasyBuild/3.1.0   3) icc/2017.1.132-GCC-5.2.0   (I)   5) impi/2017.1.132 
(I)   7) iimpic/2017.01-GCC-5.2.0 (
TC)
  2) GCC/5.2.0 4) ifort/2017.1.132-GCC-5.2.0 (I)   6) CUDA/7.5.18

hpcswadm@hpcv18$which mpicc
/nfs/grid/software/testing/RHEL7/easybuild/software/Compiler/intel/2017.1.132-GCC-5.2.0/impi/2017.1.132/bin64/mpicc

The one that should be loaded is from
/nfs/grid/software/testing/RHEL7/easybuild/software/Compiler/intel-CUDA/2017.1.132-GCC-5.2.0-7.5.18/impi/2017.1.132/bin64/mpicc

I think the impi module should not sit inside intel directory or somehow icc 
and ifort MODULEPATH need to be changed to intel-CUDA when loading iimpic
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0
 --
   impi/2017.1.132 (I,L)

Anyone else come across this issue.


Shahzeb Siddiqui
HPC Linux Engineer
B2220-447.2
Groton, CT




--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN



--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN

Re: [easybuild] MODULEPATH issue when supporting intel and intelcuda toolchain concurrently

[Alan O'Cais]

Dear Shahzeb,

I think this is probably the same (or at least related to the) issue that is 
being discussed in https://github.com/hpcugent/easybuild-framework/pull/2135

It also exposes one of the problems of a HMNS, the potential non-uniqueness of 
module names. The problem with not building software with minimal toolchains is 
that you can have multiple copies at various levels of your toolchain 
hierarchy. What module you end up loading is then dependent on the order that 
you load things (perhaps not in Lmod because it is hierarchy-aware but 
definitely for other module tools). This can clearly lead to issues.

Alan

On 2 March 2017 at 15:32, Siddiqui, Shahzeb 
> wrote:
Hello,

I seem to notice an issue when building modules using HierarchicalNamingScheme 
when building out the intel and intelcuda toolchains.

I notice that MODULEPATH is set for icc and ifort for intel directory. This is 
correct when setting up intel toolchain.

hpcswadm@hpcv18$grep -iR MODULEPATH
icc/2017.1.132-GCC-5.2.0.lua:prepend_path("MODULEPATH", 
"/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0")
ifort/2017.1.132-GCC-5.2.0.lua:prepend_path("MODULEPATH", 
"/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0")

impi  gets installed in the path for intel as expected.

hpcswadm@hpcv18$ls -R 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/:
impi

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0/impi:
2017.1.132.lua

As for impi built with iccifortcuda it gets installed in

hpcswadm@hpcv18$ls -R 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/:
2017.1.132-GCC-5.2.0-7.5.18

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/2017.1.132-GCC-5.2.0-7.5.18:
impi

/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel-CUDA/2017.1.132-GCC-5.2.0-7.5.18/impi:
2017.1.132.lua

The problem is when loading iimpic toolchain it loads up icc and ifort modules 
along with impi that belongs to the module tree from intel and not intel-CUDA. 
I am not sure if this is a problem, but it seems like the impi is not being 
picked up correctly.

hpcswadm@hpcv18$ml av iimpi
- 
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Core 
-
   iimpi/2017.01-GCC-5.2.0 (TC)iimpic/2017.01-GCC-5.2.0 (TC)

hpcswadm@hpcv18$ml

Currently Loaded Modules:
  1) EasyBuild/3.1.0

hpcswadm@hpcv18$ml iimpic
Currently Loaded Modules:
  1) EasyBuild/3.1.0   3) icc/2017.1.132-GCC-5.2.0   (I)   5) impi/2017.1.132 
(I)   7) iimpic/2017.01-GCC-5.2.0 (
TC)
  2) GCC/5.2.0 4) ifort/2017.1.132-GCC-5.2.0 (I)   6) CUDA/7.5.18

hpcswadm@hpcv18$which mpicc
/nfs/grid/software/testing/RHEL7/easybuild/software/Compiler/intel/2017.1.132-GCC-5.2.0/impi/2017.1.132/bin64/mpicc

The one that should be loaded is from
/nfs/grid/software/testing/RHEL7/easybuild/software/Compiler/intel-CUDA/2017.1.132-GCC-5.2.0-7.5.18/impi/2017.1.132/bin64/mpicc

I think the impi module should not sit inside intel directory or somehow icc 
and ifort MODULEPATH need to be changed to intel-CUDA when loading iimpic
/nfs/grid/software/testing/RHEL7/easybuild/modules/all/Compiler/intel/2017.1.132-GCC-5.2.0
 --
   impi/2017.1.132 (I,L)

Anyone else come across this issue.


Shahzeb Siddiqui
HPC Linux Engineer
B2220-447.2
Groton, CT




--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt