> On 20 Nov 2018, at 14:27, Kenneth Hoste <[email protected]> wrote:
>
> Dear Jakob,
>
> On 20/11/2018 13:57, Jakob Schiøtz wrote:
>> Hi,
>> I very often have problems with the lmod cache, as if it suddenly stop
>> updating and new modules are no longer listed. In the end, I have been
>> setting LMOD_IGNORE_CACHE=1 in the .bashrc
>
> This is a bit off-topic for the EasyBuild mailing list, but: how do you have
> this set up exactly? Do you have a cron job responsible for updating the Lmod
> cache?
>
> That has been working great for us, I honestly can't remember the last time
> we have problems with our setup.
>
> Maybe our run_lmod_cache.py script can be helpful to you?
> It's available from https://github.com/hpcugent/Lmod-UGent .
I think the problem is that I do something unusual. On top of the “system”
modules that are maintained centrally for our cluster, I also build my own
modules as a user. On top of that, I have three different paths for these
homebuilt modules, based on which CPU architecture I am logged into. I don’t
have any cron jobs running, but normally when I build a package it appears
immediately in “module avail”. Until it stops doing so, for reasons I cannot
work out. But disabling the cache works for me.
FYI, this is my setup ($CPU_ARCH is set by the system to the name of the Intel
architecture of the CPU):
export MY_EB_SUFFIX=_experimental
if [[ -n "$SLURM_JOB_NODELIST" && -n "$__Init_My_Modules" ]]; then
if [[ "$__Init_My_Modules" != "$CPU_ARCH" ]]; then
# Modules are messed up, we inherited stuff from the submit node.
echo "CLEANING UP ENVIRONMENT."
module purge
module unuse $HOME/easybuild$MY_EB_SUFFIX/$__Init_My_Modules/modules/all
unset __Init_My_Modules
fi
fi
# EasyBuild setup
if [ -z "$__Init_My_Modules" ]; then
export __Init_My_Modules=$CPU_ARCH
export EASYBUILD_PREFIX=$HOME/easybuild$MY_EB_SUFFIX/$CPU_ARCH
module use $EASYBUILD_PREFIX/modules/all
export
EASYBUILD_ROBOT_PATHS=$HOME/development/easybuild-easyconfigs/easybuild/easyconfigs:
export EASYBUILD_GITHUB_USER=schiotz
export EASYBUILD_DETECT_LOADED_MODULES=error
export EASYBUILD_GIT_WORKING_DIRS_PATH=$HOME/development
# EXPERIMENTAL: Build in ram-disk
export EASYBUILD_BUILDPATH=/run/user/$UID/eb_build
module load EasyBuild
export PYTHONPATH=$HOME/development/easybuild-easyblocks:$PYTHONPATH
export LMOD_IGNORE_CACHE=1
else
module refresh
fi
Best regards
Jakob
>
>> It gives a one second delay or so when running modules avail, but rather
>> that than getting incorrect results.
>
> 1 second is definitely acceptable, but almost instant answers from 'module
> avail' is easy to get used to as well. ;-)
>
>
> regards,
>
> Kenneth
>
>> Jakob
>>> On 19 Nov 2018, at 16:04, Alan O'Cais <[email protected]> wrote:
>>>
>>> Hi Loris,
>>>
>>> Probably your lmod cache is just out of date. Have you tried to actually
>>> load the module?
>>>
>>> Try
>>> module --ignore_cache avail
>>>
>>> On Mon, 19 Nov 2018 at 15:47, Loris Bennett <[email protected]>
>>> wrote:
>>> Hi,
>>>
>>> This maybe more of an Lmod problem. I have built two toolchains:
>>>
>>> [loris@admin ~]$ ll ~/shared/modules/toolchain/foss/
>>> total 0
>>> lrwxrwxrwx 1 loris loris 53 Nov 15 11:44 2018a.lua ->
>>> /home/loris/shared/modules/all/foss/2018a.lua
>>> lrwxrwxrwx 1 loris loris 53 Nov 19 12:40 2018b.lua ->
>>> /home/loris/shared/modules/all/foss/2018b.lua
>>>
>>> but Lmod only shows one:
>>>
>>> [loris@admin ~]$ module av foss/
>>>
>>> --------------------- /home/loris/shared/modules/all ---------------------
>>> foss/2018a
>>>
>>> Use "module spider" to find all possible modules.
>>> Use "module keyword key1 key2 ..." to search for all possible modules
>>> matching
>>> any of the "keys".
>>>
>>> The foss-2018b has been installed as dependency for
>>> R-3.5.1-foss-2018b.eb, but the installation of R itself is failing due
>>> to a checksum problem. Is there some EasyBuild mechanism which prevents
>>> the dependencies from being displayed if the installation of the
>>> top-level packages has not been completed?
>>>
>>> Cheers,
>>>
>>> Loris
>>>
>>> --
>>> Dr. Loris Bennett (Mr.)
>>> ZEDAT, Freie Universität Berlin Email [email protected]
>>>
>>>
>>> --
>>> Dr. Alan O'Cais
>>> E-CAM Software Manager
>>> Juelich Supercomputing Centre
>>> Forschungszentrum Juelich GmbH
>>> 52425 Juelich, Germany
>>>
>>> Phone: +49 2461 61 5213
>>> Fax: +49 2461 61 6656
>>> E-mail: [email protected]
>>> WWW: http://www.fz-juelich.de/ias/jsc/EN
>>>
>>>
>>> ------------------------------------------------------------------------------------------------
>>> ------------------------------------------------------------------------------------------------
>>> Forschungszentrum Juelich GmbH
>>> 52425 Juelich
>>> Sitz der Gesellschaft: Juelich
>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>> Prof. Dr. Sebastian M. Schmidt
>>> ------------------------------------------------------------------------------------------------
>>> ------------------------------------------------------------------------------------------------
>>>
>> --
>> Jakob Schiøtz, professor, Ph.D.
>> Department of Physics
>> Technical University of Denmark
>> DK-2800 Kongens Lyngby, Denmark
>> http://www.fysik.dtu.dk/~schiotz/
--
Jakob Schiøtz, professor, Ph.D.
Department of Physics
Technical University of Denmark
DK-2800 Kongens Lyngby, Denmark
http://www.fysik.dtu.dk/~schiotz/
