Re: [easybuild] REPOSITORYPATH broken in 4.5.0
Hi Arnau, We had the same problem. Check this issue in github: https://github.com/easybuilders/easybuild-framework/issues/3892 Regards, Pablo. On Tue, Nov 9, 2021 at 12:35 PM Arnau wrote: > Hi all, > > we have been using the EB/git integration for while so, after a successful > compilation, the easyconfig is automagically pushed to a git repo. > > I usually define this: > > EASYBUILD_REPOSITORY=GitRepository > EASYBUILD_REPOSITORYPATH=https://bitbucket[...]/phaser/easyconfigs.git > > but now, while testing 4.5.0 the git integration is broken. Logs show this: > > == 2021-11-09 12:25:12,008 gitrepo.py:107 WARNING Git local repo > initialization failed, it might already exist: 'git clone > /home/x2phasr1/easyconfigs/https:/bitbucket.[...]phaser/easyconfigs.git' > returned with exit code 128 > stderr: 'Cloning into 'easyconfigs'... > fatal: > '/home/x2phasr1/easyconfigs/https:/bitbucket[...]/phaser/easyconfigs.git' > does not appear to be a git repository > fatal: Could not read from remote repository. > > > So, the repo path is not well build and EB is appending the easyconfig > path and the giturl ... > > I've read the release notes for 4.5 and did not find any reference to a > modification on this parameter, so I'm wondering if I'm missing something > or it's just new bug. > > Anyone on 4.5 successfully using the git feature? > > TIA, > Arnau >
[easybuild] easyconfig for gromacs-2020.2 + plumed 2.6 ?
Hi Anyone in the list has a working easyconfig for Gromacs 2020 + plumed 2.6 ? crappy, hacky, dirty easyconfigs which would not be accepted upstream and would make Kenneth scream are welcome. In fact are preferred ;) thanks! Pablo. -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] icc, --modules-tool, --module-syntax
On Mon, Nov 25, 2019 at 7:08 PM Marcelo Carignano wrote: > Hello there, > today I installed EasyBuild on a Cray XC50, installation went ok. > Then I tried to install CP2K by doing: > > > eb CP2K-6.1-intel-2018a.eb --modules-tool EnvironmentModulesC > --module-syntax Tcl --robot > > > and eb started fetching and installing all the pieces. > > > I understand this is a complex process and I am not trying to understand > everything in a couple of days, but there are things I cannot figure out > and it will make my day to just have an idea of what is going on. > > > First thing is related to --modules-tool EnvironmentModulesC > --module-syntax Tcl > > > I couldn't figure out how to define the corresponding env variables > to avoid the need of using --modules-tool and --module-syntax. Is there a > way to define these var in .bashrc? > > My .bashrc has the following lines: > you can configure easybuild using cli arguments, env vars or a config file https://easybuild.readthedocs.io/en/latest/Configuration.html#supported-configuration-types a possible approach is to create an example config file and adapt it to your needs https://easybuild.readthedocs.io/en/latest/Configuration.html#generating-a-template-configuration-file no matter if you use cli arguments, env var or config file it's always good practice to review what config is being applied https://easybuild.readthedocs.io/en/latest/Configuration.html#overview-of-current-configuration-show-config-show-full-config > > EASYBUILD_PREFIX=$HOME/.local/easybuild > > MODULEPATH=$MODULEPATH:$EASYBUILD_PREFIX/modules/all > > > The second point is about icc/2018.1.163-GCC-6.4.0-2.28, which is in the > dependency list of the CP2K eb file I started with. How is it that eb can > download the sources from intel? Are they really open? I have the feeling > that the process has stopped since it has been trying to fetch the files > for a long time.. Please let me know if I am missing something here. When I > try to access the intel address from the icc.eb file, I get the request for > an email and serial number... which I can't believe eb knows to be able to > pass it on... > > Again, I am a bit confused. Any help is welcome. > You are right, easybuild cannot download the intel tarballs. If you want to install intel you should download the required files yourself (check the easyconfigs or the failed build logs to find out the proper tarball names that easybuild expects) and you have to manually copy those tarballs to the folder where easybuild will look for them. https://easybuild.readthedocs.io/en/latest/Configuration.html#sourcepath If you are starting to learn how easybuild works I would recommend you to start using a foss toolchain instead of intel toolchain. foss toolchains are easier for the first try because all the tarballs can be automatically donwloaded. Once you get the feeling on how easybuild works with foss toolchains you can move forward to intel toolchains. e.g. I would start by trying "eb CP2K-6.1-foss-2019a.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot" regards, Pablo. > > Thank you very much, > > > Marcelo Carignano. > > > > > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] Singularity 3?
On Tue, Sep 10, 2019 at 11:03 AM Loris Bennett wrote: > > Thanks for the information. I appreciate the security concerns, > although it seems to me that an EasyBuild version could be the best > solution. That way I can build a new version as soon as it comes out > and deploy it immediately, without having to wait for the package to > appear in a given repository. > take a look here in case you want to build your own Singularity rpms https://github.com/pescobar/build-singularity-rpms-in-singularity regards, Pablo. > > The downside is that I have to make sure I am aware that a new version > has appeared and have to disable the old version when I install a new > one. For the former I would just subscribe to the Singularity mailing > list. What would be the best approach for the latter? Would it be an > idea to have a constant, dummy version so that the existing version > would simply be replaced by the new version? > > Cheers, > > Loris > > -- > Dr. Loris Bennett (Mr.) > ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de > > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] Singularity 3?
Hi Loris In singularity 3.x the developers did a new implementation using golang instead of the python/C used in 2.x https://archive.sylabs.io/2018/02/singularity-golang/ This changed the build procedure for 3.x so the existing easyconfigs no longer work with 3.x . Someone would need to create new easyconfigs for 3.x. Also some users prefer to deploy singularity as RPM in their systems. regards, Pablo. On Mon, Sep 9, 2019 at 4:06 PM Loris Bennett wrote: > Hi, > > I see the most recent version of Singularity available is > > Singularity-2.4.2-GCC-5.4.0-2.26.eb > > Is there any particular reason why there are no easyconfigs for version > 3? > > Regards > > Loris > > -- > Dr. Loris Bennett (Mr.) > ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] Generic easyblock for Unpack+Cmd
try to add this line to your easyconfig: unpack_options = '--strip-components=1' On Thu, Aug 15, 2019 at 4:29 PM Maik Schmidt wrote: > Hi Kenneth, > > I just tried that, but the problem is that I get the subdirectory from > the tar in my installdir when using "buildininstalldir". Just using > PackedBinary without install_cmd on the other hand puts the contents of > the subdirectory in the tar into the installdir. Is this > working-as-intended? Can I influence that behaviour somehow? > > Thanks, > Maik > > Am 15.08.19 um 15:40 schrieb Kenneth Hoste: > > On 15/08/2019 15:23, Maik Schmidt wrote: > >> Hi all, > >> > >> short question: which easyblock is best used for the case: > >> > >> - unpack to installdir first > >> > >> - run a command afterwards > >> > >> ? > >> > >> I have tried the following: > >> > >> easyblock = "PackedBinary" > >> > >> install_cmd = "..." > >> > >> However, as soon as "install_cmd" is defined, the files are not > >> copied to the installdir automatically anymore. > >> > >> Of course, I could add a "cp" to my install_cmd or use the CmdCp > >> easyblock, but that feels a little clumsy. > >> > >> Maybe a paramter could be added to PackedBinary to simple execute a > >> command after the unpacking? That would be helpful in a few cases, I > >> believe. > > > > How about setting "buildininstalldir = True" (together with using > > PackedBinary and install_cmd)? > > > > > > regards, > > > > Kenneth > > > -- > Maik Schmidt > HPC Services > > Technische Universität Dresden > Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH) > Andreas-Pfitzmann-Bau E049 > D-01187 Dresden > Telefon: +49 351 463-32836 > > > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] Generic easyblock for Unpack+Cmd
you could also use the Tarball easyblock + postinstallcmds . e.g. https://gist.github.com/pescobar/5412f1fd889563f3600741a861047d3a On Thu, Aug 15, 2019 at 3:26 PM Maik Schmidt wrote: > Hi all, > > short question: which easyblock is best used for the case: > > - unpack to installdir first > > - run a command afterwards > > ? > > I have tried the following: > > easyblock = "PackedBinary" > > install_cmd = "..." > > However, as soon as "install_cmd" is defined, the files are not copied > to the installdir automatically anymore. > > Of course, I could add a "cp" to my install_cmd or use the CmdCp > easyblock, but that feels a little clumsy. > > Maybe a paramter could be added to PackedBinary to simple execute a > command after the unpacking? That would be helpful in a few cases, I > believe. > > Thanks, > > Maik > > > -- > Maik Schmidt > HPC Services > > Technische Universität Dresden > Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH) > Andreas-Pfitzmann-Bau E049 > D-01187 Dresden > Telefon: +49 351 463-32836 > > > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] Openblas(foss) matrix issue
Hi Ole, You only need to rebuild "OpenBLAS/0.3.1-GCC-7.3.0-2.30". These are the commands: $> wget https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/develop/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb $> wget https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/develop/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.3.1_disable-special-handling-of-OpenMP-thread-memory-allocation.patch $> eb ./OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb --force module "ScaLAPACK/2.0.2-gompi-2018b-OpenBLAS-0.3.1" only provides a static library "libscalapack.a" so you should also rebuild it after you have update your OpenBLAS installation. In principle every other module that does dynamic linking with OpenBLAS should be fixed and use the new library. The problem could be if any of your existing modules did static linking with openblas and the binary includes the old buggy openblas library. e.g. in my system I had to rebuild this one https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/j/JAGS/JAGS-4.3.0-foss-2018b.eb I hope it helps. regards, Pablo. On Wed, May 29, 2019 at 9:47 AM Ole Holm Nielsen wrote: > > Hi Pablo, > > Thanks for the good news! For those of us who are not experts, could > you kindly describe the commands required to update our currently > installed OpenBLAS modules correctly? Are there any caveats with > updating an existing module in a toolchain (foss)? > > We currently have these OpenBLAS modules installed: > > $ ml av OpenBLAS/ > > -- /home/modules/modules/all > --- > OpenBLAS/0.2.20-GCC-6.4.0-2.28OpenBLAS/0.3.5-GCC-8.2.0-2.31.1 (D) > OpenBLAS/0.3.1-GCC-7.3.0-2.30 > > So is the update command simply this one? > > eb --rebuild OpenBLAS-0.3.5-GCC-8.2.0-2.31.1.eb > OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb OpenBLAS-0.2.20-GCC-6.4.0-2.28.eb > > Is the --force flag also required? > > I suppose that we do not have to rebuild the ScaLAPACK modules as well? > > Thanks a lot, > Ole > > > > On 5/28/19 5:14 PM, Pablo Escobar Lopez wrote: > > thank you Carlos! You did a great job figuring out this fix :) > > > > I can confirm that after applying this patch in our cluster the issue > > seems to be solved for us. Now we pass these tests with > > "OpenBLAS/0.3.1-GCC-7.3.0-2.30": > > https://github.com/eylenth/Openblas_matrix_issue > > https://github.com/xianyi/BLAS-Tester > > > > I also got a confirmation from a colleague in our user support team that > > a problem he was trying to debug with some R code is solved after this > > fix was applied. > > > > I have sent a PR with the fix upstream: > > https://github.com/easybuilders/easybuild-easyconfigs/pull/8396 > > > > In case anyone else test the workaround it would be nice if you report > > in the mailing list or in the pull request in github if it's working > > fine for you too. > > > > regards, > > Pablo > > > > > > On Tue, May 28, 2019 at 2:32 PM Carlos Fenoy > <mailto:fenoym...@gmail.com>> wrote: > > > > Hi, > > > > After fighting a long time with this, we managed to get a solution > > that passes both the "Openblas_matrix_issue" and "BLAS_tester" test > > suites. > > > > To solve the issue we had to apply a patch and add a new build > > parameter (USE_SIMPLE_THREADED_LEVEL3=1) to OpenBLAS to make it work > > with multiple openmp threads. > > > > This is how the buildopts line looks like for us: > > > > buildopts = ' USE_SIMPLE_THREADED_LEVEL3=1 BINARY=64 USE_THREAD=1 > > USE_OPENMP=1 CC="$CC" FC="$F77" DYNAMIC_ARCH=1' > > > > And the patch, we got it from this commit on the OpenBLAS repo: > > > > https://github.com/xianyi/OpenBLAS/commit/b14f44d2adbe1ec8ede0cdf06fb8b09f3c4b6e43 > > (you > > can get the patch by adding .patch at the end of the URL) > > > > Regards, > > Carlos > > > > On Mon, May 27, 2019 at 6:15 PM Pablo Escobar Lopez > > mailto:pablo.escobarlo...@unibas.ch>> > > wrote: > > > > Hi, > > > > did anyone found a working patch or workaround for the matrix > > issue when using OpenBLAS-0.3.1 ? > > > > After a lot of try&error I couldn't pass the tests in > > https://github.com/eylenth/Openblas_matrix_issue when using > > > > https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybui
Re: [easybuild] Openblas(foss) matrix issue
thank you Carlos! You did a great job figuring out this fix :) I can confirm that after applying this patch in our cluster the issue seems to be solved for us. Now we pass these tests with "OpenBLAS/0.3.1-GCC-7.3.0-2.30": https://github.com/eylenth/Openblas_matrix_issue https://github.com/xianyi/BLAS-Tester I also got a confirmation from a colleague in our user support team that a problem he was trying to debug with some R code is solved after this fix was applied. I have sent a PR with the fix upstream: https://github.com/easybuilders/easybuild-easyconfigs/pull/8396 In case anyone else test the workaround it would be nice if you report in the mailing list or in the pull request in github if it's working fine for you too. regards, Pablo On Tue, May 28, 2019 at 2:32 PM Carlos Fenoy wrote: > Hi, > > After fighting a long time with this, we managed to get a solution that > passes both the "Openblas_matrix_issue" and "BLAS_tester" test suites. > > To solve the issue we had to apply a patch and add a new build parameter > (USE_SIMPLE_THREADED_LEVEL3=1) to OpenBLAS to make it work with multiple > openmp threads. > > This is how the buildopts line looks like for us: > > buildopts = ' USE_SIMPLE_THREADED_LEVEL3=1 BINARY=64 USE_THREAD=1 > USE_OPENMP=1 CC="$CC" FC="$F77" DYNAMIC_ARCH=1' > > And the patch, we got it from this commit on the OpenBLAS repo: > https://github.com/xianyi/OpenBLAS/commit/b14f44d2adbe1ec8ede0cdf06fb8b09f3c4b6e43 > (you > can get the patch by adding .patch at the end of the URL) > > Regards, > Carlos > > On Mon, May 27, 2019 at 6:15 PM Pablo Escobar Lopez < > pablo.escobarlo...@unibas.ch> wrote: > >> Hi, >> >> did anyone found a working patch or workaround for the matrix issue when >> using OpenBLAS-0.3.1 ? >> >> After a lot of try&error I couldn't pass the tests in >> https://github.com/eylenth/Openblas_matrix_issue when using >> https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb >> . >> No matter what patches, toolchainopts or buildopts I use (and I have tried >> few different combinations) . Is anyone able to pass the tests using >> openblas-0.3.1 ? >> >> I could pass the tests using openblas-0.3.5 but upgrading my foss/2018b >> toolchain would be quite messy because I use RPATH. The less intrusive >> solution for my users would be to be able to patch openblas-0.3.1 somehow >> but I couldn't find a working solution. Any suggestions? >> >> regards, >> Pablo. >> >> p.s. in a related topic, IMHO unless there is a proper workaround I would >> suggest to stop providing openblas-0.3.1 with easybuild. Right now we are >> distributing a broken library >> >> >> On Tue, May 7, 2019 at 6:34 PM Mikael Öhman wrote: >> >>> Hi Thomas, >>> >>> I can also confirm these issues. I tried rebuilding OpenBLAS+R after the >>> fix in #7180, but I still saw the same problems. >>> Very large matrix-matrix multiplications randomly gave the wrong result. >>> Very large errors. The larger the matrix, the more frequent the errors. >>> >>> In the end, I compiled an intel-version (but I had to remove a few >>> extensions that didn't build) and removed my Foss version from our >>> installations. >>> >>> Perhaps it's related to hardware; I saw this on happen skylake servers. >>> I haven't had time to check if this >>> https://github.com/easybuilders/easybuild-easyconfigs/issues/8197 >>> also affects 0.3.1 >>> >>> Best regards, Mikael >>> >>> >>> On Tue, May 7, 2019 at 6:12 PM Thomas Eylenbosch < >>> thomas.eylenbosch@agro.basf-se.com> wrote: >>> >>>> Hello >>>> >>>> >>>> >>>> Some of our end users reported a calculation issue with matrices when >>>> they are working with a foss/2018b module >>>> >>>> >>>> >>>> I reproduced this error with Python and R that are compiled with the >>>> foss/2018b toolchain, the output returns unexcepted results. >>>> >>>> Then I reproduced this error with Python and R that are compiled with >>>> the foss/2016b toolchain , then it gives me the expected behavior. >>>> >>>> >>>> >>>> You can reproduce this error with the following github repository: >>>> >>>> https://github.com/eylenth/Openblas_matrix_issue >>>>
Re: [easybuild] Openblas(foss) matrix issue
Hi,
did anyone found a working patch or workaround for the matrix issue when
using OpenBLAS-0.3.1 ?
After a lot of try&error I couldn't pass the tests in
https://github.com/eylenth/Openblas_matrix_issue when using
https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb
.
No matter what patches, toolchainopts or buildopts I use (and I have tried
few different combinations) . Is anyone able to pass the tests using
openblas-0.3.1 ?
I could pass the tests using openblas-0.3.5 but upgrading my foss/2018b
toolchain would be quite messy because I use RPATH. The less intrusive
solution for my users would be to be able to patch openblas-0.3.1 somehow
but I couldn't find a working solution. Any suggestions?
regards,
Pablo.
p.s. in a related topic, IMHO unless there is a proper workaround I would
suggest to stop providing openblas-0.3.1 with easybuild. Right now we are
distributing a broken library
On Tue, May 7, 2019 at 6:34 PM Mikael Öhman wrote:
> Hi Thomas,
>
> I can also confirm these issues. I tried rebuilding OpenBLAS+R after the
> fix in #7180, but I still saw the same problems.
> Very large matrix-matrix multiplications randomly gave the wrong result.
> Very large errors. The larger the matrix, the more frequent the errors.
>
> In the end, I compiled an intel-version (but I had to remove a few
> extensions that didn't build) and removed my Foss version from our
> installations.
>
> Perhaps it's related to hardware; I saw this on happen skylake servers. I
> haven't had time to check if this
> https://github.com/easybuilders/easybuild-easyconfigs/issues/8197
> also affects 0.3.1
>
> Best regards, Mikael
>
>
> On Tue, May 7, 2019 at 6:12 PM Thomas Eylenbosch <
> thomas.eylenbosch@agro.basf-se.com> wrote:
>
>> Hello
>>
>>
>>
>> Some of our end users reported a calculation issue with matrices when
>> they are working with a foss/2018b module
>>
>>
>>
>> I reproduced this error with Python and R that are compiled with the
>> foss/2018b toolchain, the output returns unexcepted results.
>>
>> Then I reproduced this error with Python and R that are compiled with the
>> foss/2016b toolchain , then it gives me the expected behavior.
>>
>>
>>
>> You can reproduce this error with the following github repository:
>>
>> https://github.com/eylenth/Openblas_matrix_issue
>>
>>
>>
>> I have also tried to recompile the OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb
>> easyconfig file with “toolchainopts = {'vectorize': False}” ( cfr.
>> https://github.com/easybuilders/easybuild-easyconfigs/issues/7180)
>>
>> But is still giving me unexpected behavior
>>
>>
>>
>> Can someone try to reproduce the error with the R/Python(foss/2018b)
>> modules. Or can someone give me feedback on this?
>>
>>
>>
>> Thank you in advance.
>>
>>
>>
>> Met vriendelijke groet / Kind regards / Beste Grüße
>>
>> *Thomas Eylenbosch*
>>
>> *Ext: Gluo N.V.*
>>
>>
>>
>> BASF Agricultural Solutions Belgium NV
>>
>> Technologiepark 101
>>
>> B-9052 Ghent (Zwijnaarde)
>>
>> BELGIUM
>>
>> E-mail: *thomas.eylenbosch@agro.basf-se.com
>> *
>>
>> [image: cid:image001.jpg@01D42B1E.FC3C98F0]
>>
>> BASF Agricultural Solutions Belgium NV, Registered Office: 9052 Gent,
>> Belgium
>>
>> Registration: RPR Gent: 0685.756.742
>>
>>
>>
>
--
Pablo Escobar López
Linux/HPC systems engineer
sciCORE, University of Basel
SIB Swiss Institute of Bioinformatics
Re: [easybuild] Building foss-2019a fails in binutils-2.31.1.eb (Skylake node)
On Tue, Feb 26, 2019 at 11:24 AM Kenneth Hoste wrote: > > One easy thing we could do is make the binutils easyblock check whether > both 'gcc' and 'g++' are present, and emit a clear warning if they're > missing? > That would help significantly, since pinpointing the underlying problem > is clearly not trivial. > +1 > > Not sure we should make that a hard failure though, as there may be > situation where not having gcc/g++ is actually fine (e.g. when 'cc' and > 'c++' compilers are available, and can be used to compile binutils). > > One other option could be to detect that the build failed because g++ is > not there (by recognizing the pattern in the configure output or in > config.log). > > Same applies for GCC(core), where you also need a system C++ compiler > with sufficiently recent versions... > > > regards, > > Kenneth > > > > > > > /Ole > > > >> From: easybuild-requ...@lists.ugent.be > >> on behalf of Ole Holm Nielsen > >> > >> Sent: Friday, February 22, 2019 09:11 > >> To: easybuild@lists.ugent.be > >> Subject: Re: [easybuild] Building foss-2019a fails in > >> binutils-2.31.1.eb (Skylake node) > >> > >> Hi Olivier, > >> > >> On 2/20/19 10:18 PM, Olivier Mattelaer wrote: > >>> I actually face the same issue. > >>> > >>> The actual error message is this one: > >>> > >>> configure: error: in > >>> > `/usr/local/Software/build/lm3-w091/binutils/2.31.1/dummy-/binutils-2.31.1/gold': > > >>> > >>> configure: error: C++ preprocessor "/lib/cpp" fails sanity check > >>> See `config.log' for more details > >>> yes > >>> checking whether compiling a cross-assembler... no > >>> checking for size_t... checking locale.h usability... make[1]: *** > >>> [configure-gold] Error 1 > >>> make[1]: *** Waiting for unfinished jobs > >>> > >>> > >>> I guess that doing: "yum install gcc-c++" would solve the issue. > >>> But I have not tested yet. But does it make sense to do that? > >> > >> Your suggestion fixed the problem with binutils: yum install gcc-c++ > >> Now it build correctly, and the foss-2019a build process is continuing. > >> > >> What's the root cause of this issue? I'm guessing that the EB file > >> > .../EasyBuild/3.8.1/lib/python2.7/site-packages/easybuild_easyconfigs-3.8.1-py2.7.egg/easybuild/easyconfigs/b/binutils/binutils-2.31.1.eb > > >> > >> > >> must include gcc-c++ as an osdependencies package. > >> > >> If this guess is correct, I could open an issue. > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] deprecating the 'goolf' and 'ictce' toolchains
goolf/1.7.20 is my default toolchain. most of my installations use it I would like to move to foss as my default toolchain "soon" but I have more urgent tasks and I won't be able to do it in coming months. In any case I think I would survive if you deprecate it On Fri, Nov 16, 2018 at 12:06 AM Lev Lafayette wrote: > On Thu, 2018-11-15 at 08:29 +0100, Kenneth Hoste wrote: > > > > Any particular reason for that, beyond sticking to what's familiar? > > Not really; it's what we started with and we've stayed with it. > > > Which versions are you still using? > > Just... > > goolf/2015a > goolfc/2017.01 > > > Both goolf & ictce are different from foss & intel in the sense that > > goolf & ictce do not include an EasyBuild-installed binutils along with > > the GCC that is used as a base. > > Yes, that's about it. > > All the best, > > -- > Lev Lafayette, BA (Hons), GradCertTerAdEd (Murdoch), GradCertPM, MBA > (Tech Mngmnt) (Chifley) > Senior HPC Support and Training Officer +61383444193 +61432255208 > Department of Infrastructure Services, University of Melbourne > > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] installing lm-sensors
you are overriding "installopts" so only the last variable definition is
being used. That's why your custom prefix is not applied. You should do it
like this
installopts = 'PREFIX=%(installdir)s ETCDIR=%(installdir)s/etc/'
On Fri, May 4, 2018 at 10:58 AM, Mr. Joseph John <
jose...@rajagiritech.edu.in> wrote:
> Hi,
>
> This is the eb file I used, following your instructions. Is there
> something wrong with this?
>
> easyblock = 'ConfigureMake'
>
> name = "lm-sensors"
> version = "3.4.0"
>
> homepage = "https://github.com/groeck/lm-sensors";
> description = """The lm-sensors package, provides user-space support for
> the
> hardware monitoring drivers in Linux. """
>
> buildopts = 'all'
>
> toolchain = {'name': 'dummy', 'version': ''}
>
> source_urls = ['https://github.com/groeck/lm-sensors.git']
> sources = [SOURCE_TAR_GZ]
> checksums = ['e79c58404f8d792c153207a957b1ed59ed5df109b25482a39526106a4a4a
> cbd3']
>
> dependencies = [('GCC', '4.9.2'), ('Bison', '3.0.4'), ('flex', '2.6.0')]
>
> skipsteps = ['configure']
> installopts = 'PREFIX=%(installdir)s/'
> installopts = 'ETCDIR=%(installdir)s/etc/'
>
> moduleclass = "lang"
>
> Yours sincerely,
>
>
>
>
> *Joseph John *
>
> *Assistant Professor *
>
> *Department of Computer Science & Engineering *
> *Rajagiri School of Engineering & Technology*
> *https://josephjohnjj.github.io/ <https://josephjohnjj.github.io/>*
>
> On Fri, May 4, 2018 at 2:19 PM, Pablo Escobar Lopez <
> pablo.escobarlo...@unibas.ch> wrote:
>
>> it looks like your not properly passing the install prefix to the
>> makefile. you have an example in the easyconfig I pointed as reference
>>
>> https://github.com/easybuilders/easybuild-easyconfigs/blob/6
>> 8176cd383d3bc3569894bced5a3fda001d0e4fb/easybuild/easyconfig
>> s/l/LuaJIT/LuaJIT-2.0.2-GCC-4.9.2.eb#L20
>>
>>
>>
>> On Fri, May 4, 2018 at 10:35 AM, Mr. Joseph John <
>> jose...@rajagiritech.edu.in> wrote:
>>
>>> Hi,
>>>
>>> I tried ETCDIR=%(installdir)s/etc/, but now I am getting error in
>>> another directory (/usr/). Is there a way to solve this for all the
>>> directories?
>>>
>>> == 2018-05-04 04:29:53,745 easyblock.py:870 DEBUG Creating the
>>> installation directory /sNow/easybuild/centos/7.3.161
>>> 1/Broadwell/software/lm-sensors/3.4.0 (cleanup: True)
>>> == 2018-05-04 04:29:53,745 easyblock.py:881 INFO Found old directory
>>> /sNow/easybuild/centos/7.3.1611/Broadwell/software/lm-sensors/3.4.0
>>> == 2018-05-04 04:29:53,746 filetools.py:1203 INFO Path
>>> /sNow/easybuild/centos/7.3.1611/Broadwell/software/lm-sensors/3.4.0
>>> successfully removed.
>>> == 2018-05-04 04:29:53,746 easyblock.py:888 INFO Removed old directory
>>> /sNow/easybuild/centos/7.3.1611/Broadwell/software/lm-sensors/3.4.0
>>> == 2018-05-04 04:29:53,746 filetools.py:1107 INFO Creating directory
>>> /sNow/easybuild/centos/7.3.1611/Broadwell/software/lm-sensors/3.4.0
>>> (parents: True, set_gid: False, sticky: False)
>>> == 2018-05-04 04:29:53,747 easyblock.py:2452 INFO Running method
>>> install_step part of step install
>>> == 2018-05-04 04:29:53,747 run.py:163 DEBUG run_cmd: running cmd make
>>> install
>>> ETCDIR=/sNow/easybuild/centos/7.3.1611/Broadwell/software/lm-sensors/3.4.0/etc/
>>> (in /sNow/easybuild/centos/7.3.1611/Broadwell/build/lmsensors/3.
>>> 4.0/dummy-/lm-sensors-master)
>>> == 2018-05-04 04:29:53,747 run.py:183 INFO running cmd: make install
>>> ETCDIR=/sNow/easybuild/centos/7.3.1611/Broadwell/software/lm
>>> -sensors/3.4.0/etc/
>>> == 2018-05-04 04:29:53,825 build_log.py:158 ERROR EasyBuild crashed with
>>> an error (at easybuild/centos/7.3.1611/Broa
>>> dwell/software/EasyBuild/3.5.0/lib/python2.7/site-packages/v
>>> sc_base-2.5.8-py2.7.egg/vsc/utils/exceptions.py:124 in __init__): cmd "
>>> make install ETCDIR=/sNow/easybuild/centos/
>>> 7.3.1611/Broadwell/software/lm-sensors/3.4.0/etc/" exited with exit
>>> code 2 and output:
>>> mkdir -p /usr/local/lib /usr/local/include/sensors /usr/local/man/man3
>>> /usr/local/man/man5
>>> mkdir: cannot create directory '/usr/local/include/sensors': Read-only
>>> file system
>>> mkdir: cannot create dire
Re: [easybuild] installing lm-sensors
chain = {'name': 'dummy', 'version': ''}
>>>
>>> source_urls = ['https://github.com/groeck/lm-sensors.git']
>>> sources = [SOURCE_TAR_GZ]
>>> checksums = ['e79c58404f8d792c153207a957b1ed59ed5df109b25482a39526106a4a
>>> 4acbd3']
>>>
>>> dependencies = [('GCC', '4.9.2'), ('Bison', '3.0.4'), ('flex', '2.6.0')]
>>>
>>> skipsteps = ['configure']
>>> installopts = 'PREFIX=%(installdir)s'
>>>
>>> moduleclass = "lang"
>>>
>>>
>>> But i keep getting this error:
>>>
>>> mkdir -p /etc /etc/sensors.d
>>> mkdir: cannot create directory '/etc/sensors.d': Read-only file system
>>> make: *** [install-etc] Error 1
>>> (at easybuild/centos/7.3.1611/Broadwell/software/EasyBuild/3.5.0
>>> /lib/python2.7/site-packages/easybuild_framework-3.5.0-py2.7
>>> .egg/easybuild/tools/run.py:481 in parse_cmd_output)
>>> == 2018-05-02 01:21:36,082 easyblock.py:2685 WARNING build failed (first
>>> 300 chars): cmd " make install PREFIX=/sNow/easybuild/centos/
>>> 7.3.1611/Broadwell/software/lm-sensors/3.4.0" exited with exit code 2
>>> and output:
>>> mkdir -p
>>> /sNow/easybuild/centos/7.3.1611/Broadwell/software/lm-sensors/3.4.0/lib
>>> /sNow/easybuild/centos/7.3.1611/Broadwell/software/lm-sensors/3.4.0/include/sensors
>>> /sNo
>>> == 2018-05-02 01:21:36,082 easyblock.py:279 INFO Closing log for
>>> application name lm-sensors version 3.4.0
>>>
>>> Is there anyway to set these permissions through easybuild or should I
>>> change the permission separately?
>>>
>>> Yours sincerely,
>>>
>>>
>>>
>>> *Joseph John *
>>>
>>> *Assistant Professor *
>>>
>>> *Department of Computer Science & Engineering *
>>> *Rajagiri School of Engineering & Technology*
>>> *https://josephjohnjj.github.io/ <https://josephjohnjj.github.io/>*
>>>
>>> On Sun, Mar 25, 2018 at 1:10 AM, Pablo Escobar Lopez <
>>> pablo.escobarlo...@unibas.ch> wrote:
>>>
>>>> Hi Joseph,
>>>>
>>>> The installation procedure is "make all" and "make install" so it
>>>> shouldn't be difficult to install it using easybuild. Providing the
>>>> dependencies with easybuild shouldn't be difficult neither.
>>>> https://github.com/groeck/lm-sensors/blob/master/INSTALL
>>>>
>>>> You can use this easyconfig as reference
>>>> https://github.com/easybuilders/easybuild-easyconfigs/blob/6
>>>> 8176cd383d3bc3569894bced5a3fda001d0e4fb/easybuild/easyconfig
>>>> s/l/LuaJIT/LuaJIT-2.0.2-GCC-4.9.2.eb
>>>>
>>>> I think the only missing detail in that easyconfig is
>>>> buildopts = 'all'
>>>>
>>>> You still need proper support from your kernel but I guess this should
>>>> be provided by most modern distributions out of the box.
>>>>
>>>> regards,
>>>> Pablo.
>>>>
>>>>
>>>> On Sat, Mar 24, 2018 at 6:52 AM, Mr. Joseph John <
>>>> jose...@rajagiritech.edu.in> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I there anyay to install lm-sensors using easybuild?
>>>>>
>>>>> yours sincerely,
>>>>>
>>>>>
>>>>> *Joseph John *
>>>>>
>>>>> *Assistant Professor *
>>>>>
>>>>> *Department of Computer Science & Engineering *
>>>>> *Rajagiri School of Engineering & Technology*
>>>>> *https://josephjohnjj.github.io/ <https://josephjohnjj.github.io/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Pablo Escobar López
>>>> Linux/HPC systems engineer
>>>> sciCORE, University of Basel
>>>> SIB Swiss Institute of Bioinformatics
>>>> http://scicore.unibas.ch
>>>>
>>>
>>>
>>
>
--
Pablo Escobar López
Linux/HPC systems engineer
sciCORE, University of Basel
SIB Swiss Institute of Bioinformatics
[easybuild] Re: iomkl-2018.02 toolchain. What OpenMPI version?
btw, I forgot to mention that iomk-2018.02 would be based in https://github.com/easybuilders/easybuild-easyconfigs/pull/6077 On Tue, Mar 27, 2018 at 6:52 PM, Pablo Escobar Lopez < pablo.escobarlo...@unibas.ch> wrote: > Hi, > > I plan to send a PR for iomkl-2018.02 and I wanted to ask for advice about > what OpenMPI version should be included so it's accepted upstream. > > Latest OpenMPI version is 3.0.0 and I personally prefer to avoid new major > releases ending in x.0.0 unless there is a clear reason like a new feature > I need (which is not the case) so IMHO I would create iomkl-2018.02 using > latest OpenMPI release in branch 2.x > > opinions? > > regards, > Pablo. > > -- > Pablo Escobar López > Linux/HPC systems engineer > sciCORE, University of Basel > SIB Swiss Institute of Bioinformatics > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
[easybuild] iomkl-2018.02 toolchain. What OpenMPI version?
Hi, I plan to send a PR for iomkl-2018.02 and I wanted to ask for advice about what OpenMPI version should be included so it's accepted upstream. Latest OpenMPI version is 3.0.0 and I personally prefer to avoid new major releases ending in x.0.0 unless there is a clear reason like a new feature I need (which is not the case) so IMHO I would create iomkl-2018.02 using latest OpenMPI release in branch 2.x opinions? regards, Pablo. -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics
Re: [easybuild] installing lm-sensors
Hi Joseph, The installation procedure is "make all" and "make install" so it shouldn't be difficult to install it using easybuild. Providing the dependencies with easybuild shouldn't be difficult neither. https://github.com/groeck/lm-sensors/blob/master/INSTALL You can use this easyconfig as reference https://github.com/easybuilders/easybuild-easyconfigs/blob/68176cd383d3bc3569894bced5a3fda001d0e4fb/easybuild/easyconfigs/l/LuaJIT/LuaJIT-2.0.2-GCC-4.9.2.eb I think the only missing detail in that easyconfig is buildopts = 'all' You still need proper support from your kernel but I guess this should be provided by most modern distributions out of the box. regards, Pablo. On Sat, Mar 24, 2018 at 6:52 AM, Mr. Joseph John < jose...@rajagiritech.edu.in> wrote: > Hi, > > I there anyay to install lm-sensors using easybuild? > > yours sincerely, > > > *Joseph John * > > *Assistant Professor * > > *Department of Computer Science & Engineering * > *Rajagiri School of Engineering & Technology* > *https://josephjohnjj.github.io/ <https://josephjohnjj.github.io/>* > -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Compilation problem - R-3.4.3
Hi, I have seen similar errors when trying to build in a nfs share. I would suggest you try to do "eb --buildpath=/path/to/some/local/disk" and check if this workarounds the problem. regards, Pablo. 2018-02-21 9:33 GMT+01:00 Philippart Raphaël : > Hello, > > I have a problem with the compilation of R-3.4.3 with the > eb R-3.4.3-foss-2017b-X11-20171023.eb > > I have this error with the command eb --modules-tool EnvironmentModulesC > --module-syntax Tcl --ignore-checksums R-3.4.3-foss-2017b-X11-20171023.eb > --debug --robot > > == FAILED: Installation ended unsuccessfully (build > directory: > /cm/shared/apps/easyb/.local/easybuild/build/Perl/5.26.0/GCCcore-6.4.0): > build failed (first 300 chars): Failed to remove path > /cm/shared/apps/easyb/.local/easybuild/build/Perl/5.26.0/GCCcore-6.4.0 > with shutil.rmtree, even after 3 attempts. > == Results of the build can be found in the log file(s) > /tmp/eb-BUjouG/easybuild-Perl-5.26.0-20180221.091909.usKKL.log > ERROR: Build of /usr/lib/python2.7/site-packages/easybuild_ > easyconfigs-3.5.1-py2.7.egg/easybuild/easyconfigs/p/Perl/Perl-5.26.0-GCCcore-6.4.0.eb > failed (err: 'build failed (first 300 chars): Failed to remove path > /cm/shared/apps/easyb/.local/easybuild/build/Perl/5.26.0/GCCcore-6.4.0 > with shutil.rmtree, even after 3 attempts.’) > > Here is the folder with the error. > > easyb@genetic.master01 ~ $ ll -d /cm/shared/apps/easyb/.local/ > easybuild/build/Perl/5.26.0/GCCcore-6.4.0 > drwxr-xr-x 229 easyb wheel 8192 Feb 20 15:51 /cm/shared/apps/easyb/. > local/easybuild/build/Perl/5.26.0/GCCcore-6.4.0 > > I tried to remove the folder and retried the installation without success, > same error. > > I saw for this error, this information in the GitHub > > https://github.com/easybuilders/easybuild-framework/pull/1722/commits/ > d13062bca79f994aeadba05b6a8b96ac0f3a2ef8 > > But It’s an information of 2016 and the modification in the file tools.py > is well done. > > easyb@genetic.master01 ~ $ eb --version > This is EasyBuild 3.5.1 (framework: 3.5.1, easyblocks: 3.5.1) on host > master01. > > What can I do for it pls ? > > Best regards > > > Raphaël Philippart > Opérations : Systèmes > SeGI - ULiège > Allée de la Découverte, 8 > Quartier Polytech 1 - Bâtiment B26 > 4000 Liège > Phone +32-4-366 4981 <+32%204%20366%2049%2081> > Fax +32-4-366 2920 <+32%204%20366%2029%2020> > Email rphilipp...@uliege.be > > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] problem building custom python3 module
HI Alan, Thanks for the tip! I didn't remember the "modaltsoftname" option and using it worked fine and I could build the module. I guess you are right and my easyconfig modifications were breaking the EBROOTPYTHON env var which probably is used by the numpy easyblock. Thanks! Pablo. 2018-01-12 9:27 GMT+01:00 Alan O'Cais : > Hmm, I wonder if you would be better off using modaltsoftname rather than > changing the name of the software, your approach could kick off some > problems with EBROOTPYTHON perhaps? > > Alan > > On 11 January 2018 at 13:12, Pablo Escobar Lopez < > pablo.escobarlo...@unibas.ch> wrote: > >> Hi, >> >> One of my users needs to load both python2 and python3 modules together >> so I tried to install a custom python3 module named Python3. The main >> modification I did in the easyconfig are: >> >> easyblock = 'EB_Python' >> name = 'Python3' >> sources = ['Python-%(version)s.tgz'] >> >> The main installation of python3 works ok but in the extensions part I >> have a weird problem when building numpy >> >> "gcc: error: unrecognized command line option ‘-fstack-protector-strong’" >> "cmd "/bin/python2 setup.py config" exited with exitcode 1 and output" >> >> This is the full log: >> https://gist.github.com/anonymous/8e4b0f91b6d4a264c8628f6c63f25114 >> >> any explanation about why the system python2 binary is being used? Any >> suggestion about how to solve it? >> >> Do you recommend a different approach to be able to load both python2 and >> python3 modules together? >> >> regards, >> Pablo. >> >> >> -- >> Pablo Escobar López >> HPC systems engineer >> sciCORE, University of Basel >> SIB Swiss Institute of Bioinformatics >> http://scicore.unibas.ch >> > > > > -- > Dr. Alan O'Cais > E-CAM Software Manager > Juelich Supercomputing Centre > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49 2461 61 5213 <+49%202461%20615213> > Fax: +49 2461 61 6656 <+49%202461%20616656> > E-mail: a.oc...@fz-juelich.de > WWW:http://www.fz-juelich.de/ias/jsc/EN > > > > > > > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > > > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
[easybuild] problem building custom python3 module
Hi, One of my users needs to load both python2 and python3 modules together so I tried to install a custom python3 module named Python3. The main modification I did in the easyconfig are: easyblock = 'EB_Python' name = 'Python3' sources = ['Python-%(version)s.tgz'] The main installation of python3 works ok but in the extensions part I have a weird problem when building numpy "gcc: error: unrecognized command line option ‘-fstack-protector-strong’" "cmd "/bin/python2 setup.py config" exited with exitcode 1 and output" This is the full log: https://gist.github.com/anonymous/8e4b0f91b6d4a264c8628f6c63f25114 any explanation about why the system python2 binary is being used? Any suggestion about how to solve it? Do you recommend a different approach to be able to load both python2 and python3 modules together? regards, Pablo. -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] TensorFlow with GPU support.
Hi Jakob, I installed Tensorflow in my cluster few days ago modifying your easyconfigs. I have just sent two PR with the two easyconfigs I installed: https://github.com/easybuilders/easybuild-easyconfigs/pull/5590 https://github.com/easybuilders/easybuild-easyconfigs/pull/5591 I used cuDDN 6.0 as dependency instead of cuDDN 7.x because the provided .whl is linked with 6.0. If you try 7.x you will get a ".so lib not found" error regards, Pablo. 2018-01-04 10:23 GMT+01:00 Jakob Schiøtz : > Hi, > > I made a TensorFlow easyconfig a while ago depending on Python with the > foss toolchain; and including a variant with GPU support (PR 4904). The > latter has not yet been merged, probably because it is annoying to have > something that can only build on a machine with a GPU (it fails the sanity > check otherwise, as TensorFlow with GPU support cannot load on a machine > without it). > > Since I made that PR, two newer releases of TensorFlow have appeared (1.3 > and 1.4). There are easyconfigs for 1.3 with the Intel tool chain. I am > considering making easyconfigs for TensorFlow 1.4 with > Python-3.6.3-foss-2017b (both with and without GPU support), but first I > would like to know if anybody else is doing this - it is my impression that > somebody who actually know what they are doing may be working on > TensorFlow. :-) > > Best regards > > Jakob > > -- > Jakob Schiøtz, professor, Ph.D. > Department of Physics > Technical University of Denmark > DK-2800 Kongens Lyngby, Denmark > http://www.fysik.dtu.dk/~schiotz/ > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Configuration variables in easyconfig
I don't think you can define this in a easyconfig file. easyconfigs are not meant to be used to define global easybuild options. The only hack I can think of is to do something like this in your easyconfig: preconfigopts = ' export EASYBUILD_INSTALLPATH=/your/custom/path && ' prebuildopts = ' export EASYBUILD_INSTALLPATH=/your/custom/path && ' preinstallopts = ' export EASYBUILD_INSTALLPATH=/your/custom/path && ' I haven't tried it so I am not sure if it would work but in case it works this is a non recommended quick&dirty hack. regards, Pablo. 2017-11-27 14:56 GMT+01:00 nan...@luis.uni-hannover.de < nan...@luis.uni-hannover.de>: > > I guess the question was a bit misleading. I meant if one could have those > config variables in *.eb file > > Best regards, > Gizo > > > On Monday, November 27, 2017 12:59 CET, nan...@luis.uni-hannover.de < > nan...@luis.uni-hannover.de> wrote: > > > > > Hello, > > > > is there any way to set easybuild env. variables, for example the > installation path, from within an > > easybuild config file ? > > > > Thank you. > > > > best regards, > > Gizo > > > > > > -- > > -- > Dr. Gizo Nanava > Leibniz Universitaet IT Services > Leibniz Universitaet Hannover > Schlosswender Str. 5 > D-30159 Hannover > Tel +49 511 762 7919085 > http://www.luis.uni-hannover.de > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Configuration variables in easyconfig
https://easybuild.readthedocs.io/en/latest/Configuration.html#configuration-file-s 2017-11-27 12:59 GMT+01:00 nan...@luis.uni-hannover.de < nan...@luis.uni-hannover.de>: > > Hello, > > is there any way to set easybuild env. variables, for example the > installation path, from within an > easybuild config file ? > > Thank you. > > best regards, > Gizo > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Installing pygraph
Hi Joseph, If there is no available easyconfig in the official easybuild release the first thing you should check is if there is any pending pull request for this application which still hasn't been merged upstream: https://github.com/easybuilders/easybuild-easyconfigs/pulls If there is no pull request then you will have to write the easyconfig yourself. You can find the docs here: http://easybuild.readthedocs.io/en/latest/Writing_easyconfig_files.html You can see some examples of easyconfigs installing python packages here: https://github.com/easybuilders/easybuild-easyconfigs/blob/68176cd383d3bc3569894bced5a3fda001d0e4fb/easybuild/easyconfigs/w/wheel/wheel-0.29.0-foss-2016a-Python-2.7.11.eb https://github.com/easybuilders/easybuild-easyconfigs/blob/68176cd383d3bc3569894bced5a3fda001d0e4fb/easybuild/easyconfigs/m/matplotlib/matplotlib-1.5.1-foss-2016a-Python-2.7.11.eb regards, Pablo. 2017-11-17 12:54 GMT+01:00 Mr. Joseph John : > Hi, > Is there anyway to install pygraph through easybuild. I couldnt not > find any .eb file for this. > > > *Joseph John * > > *Assistant Professor * > > *Department of Computer Science & Engineering * > *Rajagiri School of Engineering & Technology* > *https://josephjohnjj.github.io/ <https://josephjohnjj.github.io/>* > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] MANPATH issue
Hi Kenneth,
Thanks for your quick reply. I see that I hit this problem because I am not
using the default profile file provided by Lmod. I think I will update my
profile file to apply the workaround provided by lmod
#
# If MANPATH is empty, Lmod is adding a trailing ":" so that
# the system MANPATH will be found
if [ -z "${MANPATH:-}" ]; then
export MANPATH=:
fi
thanks!
Pablo.
2017-09-07 12:12 GMT+02:00 Kenneth Hoste :
> Hi Pablo,
>
> That's actually something for Lmod to worry about.
>
> man considers the default location for manpages if there's an empty entry
> in $MANPATH, for example ":/scicore/soft/apps/GCC/4.8.4/share/man".
>
> It's Lmod's job to make sure the empty entry stays there...
>
> See also https://github.com/TACC/Lmod/issues/191
>
>
> regards,
>
> Kenneth
>
>
> On 07/09/2017 12:05, Pablo Escobar Lopez wrote:
>
> Hi,
>
> I have noticed that when I load a module defining MANPATH in my centos7
> cluster then the default man pages are no longer available. E.g.
>
> $> ml purge
> $> man ls
> This works fine and MANPATH is not defined
>
> $> ml purge
> $> ml GCC/4.8.4
> $> man ls
> No manual entry for ls
> $> echo $MANPATH
> /scicore/soft/apps/GCC/4.8.4/share/man:/scicore/soft/apps/
> binutils/2.25.1/share/man
>
> I can workaround it doing this:
>
> $> ml purge
> $> export MANPATH=/usr/share/man
> $> ml GCC/4.8.4
> $> man ls
> This works fine
> $> echo $MANPATH
> /scicore/soft/apps/GCC/4.8.4/share/man:/scicore/soft/apps/
> binutils/2.25.1/share/man:/usr/share/man
>
> Before I was not hitting this problem because I had a file
> /etc/profile.d/sge.sh which defines MANPATH on login but now that I no
> longer define MANPATH on login I noticed this problem.
>
> Did anyone else experience this problem? I can workaround it by adding a
> profile file /etc/profile.d/man.sh which defines "MANPATH=/usr/share/man"
> on login but before doing it I wanted to ask in case someone suggest a
> different/better approach.
>
> Maybe EasyBuild should make sure that MANPATH always contains
> /usr/share/man ?
>
> Pablo.
>
> --
> Pablo Escobar López
> HPC systems engineer
> sciCORE, University of Basel
> SIB Swiss Institute of Bioinformatics
> http://scicore.unibas.ch
>
>
>
--
Pablo Escobar López
HPC systems engineer
sciCORE, University of Basel
SIB Swiss Institute of Bioinformatics
http://scicore.unibas.ch
[easybuild] MANPATH issue
Hi, I have noticed that when I load a module defining MANPATH in my centos7 cluster then the default man pages are no longer available. E.g. $> ml purge $> man ls This works fine and MANPATH is not defined $> ml purge $> ml GCC/4.8.4 $> man ls No manual entry for ls $> echo $MANPATH /scicore/soft/apps/GCC/4.8.4/share/man:/scicore/soft/apps/binutils/2.25.1/share/man I can workaround it doing this: $> ml purge $> export MANPATH=/usr/share/man $> ml GCC/4.8.4 $> man ls This works fine $> echo $MANPATH /scicore/soft/apps/GCC/4.8.4/share/man:/scicore/soft/apps/binutils/2.25.1/share/man:/usr/share/man Before I was not hitting this problem because I had a file /etc/profile.d/sge.sh which defines MANPATH on login but now that I no longer define MANPATH on login I noticed this problem. Did anyone else experience this problem? I can workaround it by adding a profile file /etc/profile.d/man.sh which defines "MANPATH=/usr/share/man" on login but before doing it I wanted to ask in case someone suggest a different/better approach. Maybe EasyBuild should make sure that MANPATH always contains /usr/share/man ? Pablo. -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] need help to write easyconfig file for fall3d
I missed that probably you also want preconfigopts = 'export SCRIPTDIR=%(installdir)s && ' 2017-07-25 11:13 GMT+02:00 Pablo Escobar Lopez : > Hi Yann, > > Try doing this: > > prebuildopts = 'export SCRIPTDIR=%(installdir)s && ' > > preinstallopts = 'export SCRIPTDIR=%(installdir)s && ' > > Pablo. > > 2017-07-25 10:38 GMT+02:00 Yann Sagon : > >> Dear users, >> >> I'm trying to create an easyconfig file for fall3d. >> >> The first problem with this software is that during the configure step >> (./configure) it relies on an env variable $SCRIPTDIR to define where the >> scripts are to be copied (it doesn't use --prefix for this). >> >> The second problem is that this same variable is used as well by the >> install step (make install) to generate dynamically the scripts. >> >> I have tried to set this variable like this: >> >> SCRIPTDIR='%(installdir)s' >> >> but it seems it's not correctly defined (at least at this moment) as it's >> using the build dir instead. Or it's not the correct way to set an env >> variable with EB? >> >> Any clue? >> >> >> -- >> Yann SAGON >> Ingénieur système HPC >> 24 Rue du Général-Dufour >> 1211 Genève 4 - Suisse >> Tél. : +41 (0)22 379 7737 <+41%2022%20379%2077%2037> >> yann.sa...@unige.ch - www.unige.ch >> > > > > -- > Pablo Escobar López > HPC systems engineer > sciCORE, University of Basel > SIB Swiss Institute of Bioinformatics > http://scicore.unibas.ch > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] need help to write easyconfig file for fall3d
Hi Yann, Try doing this: prebuildopts = 'export SCRIPTDIR=%(installdir)s && ' preinstallopts = 'export SCRIPTDIR=%(installdir)s && ' Pablo. 2017-07-25 10:38 GMT+02:00 Yann Sagon : > Dear users, > > I'm trying to create an easyconfig file for fall3d. > > The first problem with this software is that during the configure step > (./configure) it relies on an env variable $SCRIPTDIR to define where the > scripts are to be copied (it doesn't use --prefix for this). > > The second problem is that this same variable is used as well by the > install step (make install) to generate dynamically the scripts. > > I have tried to set this variable like this: > > SCRIPTDIR='%(installdir)s' > > but it seems it's not correctly defined (at least at this moment) as it's > using the build dir instead. Or it's not the correct way to set an env > variable with EB? > > Any clue? > > > -- > Yann SAGON > Ingénieur système HPC > 24 Rue du Général-Dufour > 1211 Genève 4 - Suisse > Tél. : +41 (0)22 379 7737 <+41%2022%20379%2077%2037> > yann.sa...@unige.ch - www.unige.ch > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] RE: support for tmolex and tcoffee
Hi Shahzeb, here you have ;) https://github.com/hpcugent/easybuild-easyconfigs/pull/4709/files 2017-06-12 17:25 GMT+02:00 Kenneth Hoste : > Hi Shahzeb, > > On 12/06/2017 17:16, Siddiqui, Shahzeb wrote: > > I haven’t heard any response. Does anyone have easyconfig for these > packages that they mind sharing or submitting a pull request. > > > I'm not aware of easyconfigs for these software packages yet. > > Maybe this is a good opportunity to try and compose one yourself? > See the existing examples for already supported software, and the > documentation at [1] and [2]. > > > regards, > > Kenneth > > PS: See you at the HPCKP meeting in San Sebastian later this week! > > [1] http://easybuild.readthedocs.io/en/latest/Writing_ > easyconfig_files.html > [2] http://easybuild.readthedocs.io/en/latest/version-specific/ > generic_easyblocks.html > > > > Thanks > > > > > > > > *From:* easybuild-requ...@lists.ugent.be [mailto:easybuild-request@ > lists.ugent.be ] *On Behalf Of *Siddiqui, > Shahzeb > *Sent:* Tuesday, June 6, 2017 2:20 PM > *To:* easybuild@lists.ugent.be > *Subject:* [easybuild] support for tmolex and tcoffee > > > > Hi, > > > > I was wondering if there is support for tmolex and tcoffee. They are > distributed as binary files (.bin). > > > > http://www.cosmologic.de/turbomole/tmolex.html > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cosmologic.de_turbomole_tmolex.html&d=DwMFAg&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=3ynDLL8CShk-Cklp_0lUNKh0HLXKIYwhz0Smv32cwCg&s=a6TSADf792-q130eRRu1TsK5zAh1g7oc8nBODwMRPiA&e=> > > http://www.cosmologic.de/support-download/downloads/tmolex-client.html > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cosmologic.de_support-2Ddownload_downloads_tmolex-2Dclient.html&d=DwMFAg&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=3ynDLL8CShk-Cklp_0lUNKh0HLXKIYwhz0Smv32cwCg&s=mp0ZSdwC_pKT4QxYac6IdO2XBVZH1mUI3HPcQgOh678&e=> > > http://www.tcoffee.org/Projects/tcoffee/index.html#DOWNLOAD > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tcoffee.org_Projects_tcoffee_index.html-23DOWNLOAD&d=DwMFAg&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=3ynDLL8CShk-Cklp_0lUNKh0HLXKIYwhz0Smv32cwCg&s=A3gkPkbHZ_sdAmR5yQNFvCGAQTtbx6H_QPRHREOqldo&e=> > > > > Anyone have any easyconfig files for these? > > > > > > > > Shahzeb Siddiqui > > HPC Linux Engineer > > B2220-447.2 > > Groton, CT > > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Easybuild R.3.4.0 is available
2017-04-27 14:25 GMT+02:00 Benjamin Evans : > > I noticed you have qiime in the list of installed modules. I know that a > full qiime from source easyconfig has been on the wishlist for a while. > > This one works for me even it's not "clean" to merge https://github.com/hpcugent/easybuild-easyconfigs/pull/3069 This one it's much simpler to install and it's merged in the master branch https://github.com/hpcugent/easybuild-easyconfigs/pull/3868 > Cheers, > Ben > > On Wed, Apr 26, 2017 at 1:03 PM, Dey, John F wrote: > >> The latest R is available from the FredHutch: >> https://github.com/FredHutch/easybuild-life-sciences >> >> R-3.4.0-foss-2016b-fh1.eb >> <https://github.com/FredHutch/easybuild-life-sciences/blob/master/easybuild/easyconfigs/R/R-3.4.0-foss-2016b-fh1.eb> >> >> >> This build has over 500 modules which have all been upgraded for this >> release. >> Model documentation: https://fredhutch.github.io/easybuild-life-sciences/ >> >> >> >> >> >> *John Dey* >> >> *HPC Operations* >> >> SciComp Computing >> *O* *206.667.4308 <(206)%20667-4308>* >> *M* *360.649.2731 <(360)%20649-2731>* >> >> jf...@fredhutch.org >> >> >> >> SciComp Office Hours >> >> Wed. 10-noon M4-B102 >> >> >> >> [image: >> ttp://www.fredhutch.org/content/dam/public/email-signatures/3/fred_hutch_logo.png] >> <http://www.fredhutch.org/> >> Fred Hutchinson Cancer Research Center >> 1100 Fairview Ave. N., Mail Stop J4-402 >> Seattle, WA 98109 >> *fredhutch.org <http://www.fredhutch.org/>* >> > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Bootstrap specific EasyBuild version
p.s. I haven't specified it and probably it's not obvious for those not
familiar with pip but during the "pip install" step you can choose which
easybuild version you want to install E.g.
$> pip install easybuild==3.0.0
2017-04-05 20:24 GMT+02:00 Pablo Escobar Lopez :
> some time ago I did some testing to see how easybuild could be
> bootstrapped using conda. This should work in any machine even if you don't
> have a supported python version or a modules tool available
>
> $> curl -o /tmp/Miniconda2-latest-Linux-x86_64.sh
> https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
> $> bash /tmp/Miniconda2-latest-Linux-x86_64.sh -b -f
> $> export PATH=~/miniconda2/bin:$PATH
> $> conda install -c pescobar lmod=6.5.12 -y
> $> pip install easybuild
> $> source ~/miniconda2/lmod/lmod/init/bash
> $> eb --modules-tool=Lmod foss-2016a.eb -r --dry-run
>
> Pablo.
>
>
> 2017-04-05 19:20 GMT+02:00 Alvarez, Damian :
>
>> Alternatively, for people that regularly contribute back, you could clone
>> the repo[s] somewhere in your home, write a module for it, and use it to
>> install your system wide EB. We use that every time we deploy a new stack.
>> It is close to trivial. This is the TCL module we have for it (and yes, it
>> has tabs, I think it is the only module we wrote that has them):
>>
>> #%Module
>> proc ModulesHelp { } {
>> puts stderr { EasyBuild is a software build and installation framework
>> written in Python that allows you to install software in a structured,
>> repeatable and robust way. - Homepage: http://hpcugent.github.com/eas
>> ybuild/
>> }
>> }
>>
>> module-whatis {Description: EasyBuild is a software build and
>> installation framework
>> written in Python that allows you to install software in a structured,
>> repeatable and robust way. - Homepage: http://hpcugent.github.com/eas
>> ybuild/}
>>
>> set root /path/to/easybuild-framework
>>
>> conflict EasyBuild
>>
>> prepend-pathPATH$root
>> setenv EBROOTEASYBUILD "$root"
>> setenv EBVERSIONEASYBUILD "dev"
>>
>> prepend-pathPYTHONPATH $root
>> prepend-pathPYTHONPATH $root/../easybuild-easyblocks
>> prepend-pathPYTHONPATH $root/../easybuild-easyconfigs
>> prepend-pathPYTHONPATH $root/../vsc-base/lib
>>
>> Damian
>>
>> On 05/04/17 19:07, "easybuild-requ...@lists.ugent.be on behalf of
>> Kenneth Hoste" > kenneth.ho...@ugent.be> wrote:
>>
>> On 05/04/2017 07:59, Åke Sandgren wrote:
>> > Install the bootstrap in a temp dir, mod load it, install the
>> version
>> > you want in the proper place, drop the temp.
>>
>> This is indeed the only way for now, the bootstrap script doesn't
>> support specifying a particular version to install.
>>
>> If there's enough demand for it, we could, but since you can just
>> temporarily use the latest EasyBuild version to install an older
>> version
>> as Åke suggested, I don't see much value in it to be honest...
>>
>> Without the bootstrap script, you could also just use 'easy_install'
>> or
>> 'pip install', both of which support specifying specific versions, but
>> you wouldn't get an EasyBuild module to load then of course.
>>
>>
>> regards,
>>
>> Kenneth
>> >
>> > On 04/05/2017 12:47 AM, Pieter Neerincx wrote:
>> >> Hi all,
>> >>
>> >> I'm familiar with the bootstrap script for EasyBuild and it does a
>> very nice job when I want to get the latest and greatest EasyBuild deployed
>> on a bare / plain vanilla server, but I cannot find an option to tell the
>> bootstrap script to install a specific version... and sometimes this comes
>> in handy when you want to reproduce an environment with the risk of running
>> into new bugs or backwards incompatible features ;).
>> >>
>> >> When deploying a tool with EasyBuild itself it's easy to request a
>> specific version, but how do I do that for EasyBuild (when there is not
>> already another EasyBuild on the system)?
>> >>
>> >> Cheers,
>> >>
>> >> Pieter
>> >>
>> >>
>> >> -
>> >> phone:
Re: [easybuild] Bootstrap specific EasyBuild version
some time ago I did some testing to see how easybuild could be bootstrapped
using conda. This should work in any machine even if you don't have a
supported python version or a modules tool available
$> curl -o /tmp/Miniconda2-latest-Linux-x86_64.sh
https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
$> bash /tmp/Miniconda2-latest-Linux-x86_64.sh -b -f
$> export PATH=~/miniconda2/bin:$PATH
$> conda install -c pescobar lmod=6.5.12 -y
$> pip install easybuild
$> source ~/miniconda2/lmod/lmod/init/bash
$> eb --modules-tool=Lmod foss-2016a.eb -r --dry-run
Pablo.
2017-04-05 19:20 GMT+02:00 Alvarez, Damian :
> Alternatively, for people that regularly contribute back, you could clone
> the repo[s] somewhere in your home, write a module for it, and use it to
> install your system wide EB. We use that every time we deploy a new stack.
> It is close to trivial. This is the TCL module we have for it (and yes, it
> has tabs, I think it is the only module we wrote that has them):
>
> #%Module
> proc ModulesHelp { } {
> puts stderr { EasyBuild is a software build and installation framework
> written in Python that allows you to install software in a structured,
> repeatable and robust way. - Homepage: http://hpcugent.github.com/
> easybuild/
> }
> }
>
> module-whatis {Description: EasyBuild is a software build and installation
> framework
> written in Python that allows you to install software in a structured,
> repeatable and robust way. - Homepage: http://hpcugent.github.com/
> easybuild/}
>
> set root /path/to/easybuild-framework
>
> conflict EasyBuild
>
> prepend-pathPATH$root
> setenv EBROOTEASYBUILD "$root"
> setenv EBVERSIONEASYBUILD "dev"
>
> prepend-pathPYTHONPATH $root
> prepend-pathPYTHONPATH $root/../easybuild-easyblocks
> prepend-pathPYTHONPATH $root/../easybuild-easyconfigs
> prepend-pathPYTHONPATH $root/../vsc-base/lib
>
> Damian
>
> On 05/04/17 19:07, "easybuild-requ...@lists.ugent.be on behalf of Kenneth
> Hoste" kenneth.ho...@ugent.be> wrote:
>
> On 05/04/2017 07:59, Åke Sandgren wrote:
> > Install the bootstrap in a temp dir, mod load it, install the version
> > you want in the proper place, drop the temp.
>
> This is indeed the only way for now, the bootstrap script doesn't
> support specifying a particular version to install.
>
> If there's enough demand for it, we could, but since you can just
> temporarily use the latest EasyBuild version to install an older
> version
> as Åke suggested, I don't see much value in it to be honest...
>
> Without the bootstrap script, you could also just use 'easy_install' or
> 'pip install', both of which support specifying specific versions, but
> you wouldn't get an EasyBuild module to load then of course.
>
>
> regards,
>
> Kenneth
> >
> > On 04/05/2017 12:47 AM, Pieter Neerincx wrote:
> >> Hi all,
> >>
> >> I'm familiar with the bootstrap script for EasyBuild and it does a
> very nice job when I want to get the latest and greatest EasyBuild deployed
> on a bare / plain vanilla server, but I cannot find an option to tell the
> bootstrap script to install a specific version... and sometimes this comes
> in handy when you want to reproduce an environment with the risk of running
> into new bugs or backwards incompatible features ;).
> >>
> >> When deploying a tool with EasyBuild itself it's easy to request a
> specific version, but how do I do that for EasyBuild (when there is not
> already another EasyBuild on the system)?
> >>
> >> Cheers,
> >>
> >> Pieter
> >>
> >>
> >> -
> >> phone: +31 6 143 66 783
> >> e-mail: pieter.neeri...@gmail.com
> >> skype: pieter.online
> >> -
> >>
>
>
>
>
>
>
>
>
>
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
>
>
>
>
>
>
--
Pablo Escobar López
HPC systems engineer
sciCORE, University of Basel
SIB Swiss Institute of Bioinformatics
http://scicore.unibas.ch
Re: [easybuild] R foss threads question
Hi Jure, I have experienced similar issues in the past. I think this depends on the R library you use for parallelization but I don't remember the details. I suggest that you also define OPENBLAS_NUM_THREADS with the same value as OMP_NUM_THREADS. Defining both env vars with the same value shouldn't hurt and In theory openblas won't respect the value of OMP_NUM_THREADS unless it has been compiled with USE_OPENMP=1 which I think it's not the case in foss/2016b (see https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.2.18-GCC-5.4.0-2.26-LAPACK-3.6.1.eb ) Also see here https://github.com/xianyi/OpenBLAS#usages regards, Pablo. 2017-02-02 13:20 GMT+01:00 Jure Pečar : > > Hi all, > > I have here R-3.3.1 built with foss-2016b. Ganglia showed me lots of red > on the nodes where this R is running. I found out that by default this R > will spawn number of threads equal to number of cpus on the node (does this > come from openblas?), irregardless of what slurm is limiting the job to. > So I prepared some prolog scripts that set OMP_NUM_THREADS based on > --cpus-per-task and did some testing with http://r.research.att.com/ > benchmarks/R-benchmark-25.R > These are the numbers: > cores > 1 37.77user 0.93system 0:39.49elapsed 98%CPU > 2 39.70user 5.83system 0:36.90elapsed 123%CPU > 4 42.12user 12.13system 0:34.71elapsed 156%CPU > 8 46.62user 25.82system 0:33.80elapsed 214%CPU > 16 52.83user 53.06system 0:33.29elapsed 318%CPU > Trend is obvious ... > > I want to know is this is expected R behaviour, since my experience with > it is limited. Also my google-fu for single letter things is limited, so I > would appreciate any pointers to articles I have to read or experience you > can share. Thanks. > > > -- > > Jurij Pečar > HPC Engineer, IT Operations, IT Services > EMBL Heidelberg, Meyerhofstraße 1, 69117, Heidelberg, Germany > Room 13-401 > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Fwd: [Open MPI Announce] Open MPI v2.0.2 released
x a problem when using MPI dynamic memory windows. Thanks to > Christoph Niethammer for reporting. > - Fix a problem with Open MPI's pkgconfig files. Thanks to Alastair > McKinstry > for reporting. > - Fix a problem with MPI_IREDUCE when the same buffer is supplied for the > send and recv buffer arguments. Thanks to Valentin Petrov for reporting. > - Fix a problem with atomic operations on PowerPC. Thanks to Paul > Hargrove for reporting. > > Known issues (to be addressed in v2.0.3): > > - See the list of fixes slated for v2.0.3 here: > https://github.com/open-mpi/ompi/milestone/23 > > -- > Jeff Squyres > jsquy...@cisco.com > > ___ > announce mailing list > annou...@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/announce > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] R extra packages
Hi Shahzeb, yes, that's the workflow. Besides doing "R CMD INSTALL" you can also use the functions provided by R to install R packages from the CRAN or Bioconductor repositories. regards, Pablo. 2016-12-20 16:10 GMT+01:00 Siddiqui, Shahzeb : > Hi Pablo, > > > > This seems interesting. Have you used this tool to build your R collection. > > > > If I understand correctly the steps I would need to do is > > > > 1. Install R via Easybuild > > 2. Load R module > > 3. Run R CMD INSTALL on each R package > > 4. Run the script generateEasyConfig.R (Any specific parameters you > used) > > 5. Verify easyconfig for R packages and rebuild them with Easybuild > > > > *From:* easybuild-requ...@lists.ugent.be [mailto:easybuild-request@list > s.ugent.be] *On Behalf Of *Pablo Escobar Lopez > *Sent:* Saturday, December 17, 2016 4:51 AM > > *To:* easybuild@lists.ugent.be > *Subject:* Re: [easybuild] R extra packages > > > > Personally I would not recommend the approach of adding a dedicated > easyconfig for each R package. I think it would be hard to maintain > specially when defining the dependency chain. > > > > Have you look at this? https://github.com/fgeor > gatos/easybuild.experimental/tree/master/users/pneerincx > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_fgeorgatos_easybuild.experimental_tree_master_users_pneerincx&d=DgMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=vgm6I2WP7uYsV5FpE0KcJ2rVGCXcmVTIfAvGQpxzxGE&s=mcDECtGvEobIMOXjUy3jD_rN80CRpAjFIP3gIgQ5Eyo&e=> > > > > Using that script you can install R to you home folder using easybuild, > then load R from your home, install all the required R packages using the > standard R tools which will take care of dependencies resolution and then > generate a R easyconfig which would include all the R packages in the " > exts_list" > > > > regards, > > Pablo. > > > > 2016-12-17 4:54 GMT+01:00 Jack Perdue : > > On 12/16/2016 04:37 PM, Jens Timmerman wrote: > > Hi, > > > > > > On 16/12/2016 20:44, Siddiqui, Shahzeb wrote: > >> It comes down to whether we want simplicity in R, Python, or Perl > >> installation by having packages installed inside the easyconfig file. > >> It works well for most packages assuming there is no issue during > >> build. There are cases when some package don't build properly and > >> having to troubleshoot them with an entire list of packages become > >> difficult. > >> > >> I would like to decouple the R packages into individual easyconfig > >> file, because I can resolve any dependencies and work on installing > >> the particular package. Even if the process might be repetitive for > >> most builds, at least the R collection will be built correctly. > >> Another benefit is that I would like to package into RPMs, and it > >> would be beneficial to create a RPM for each R package rather than > >> just the main R. Just like how RHEL provides Python RPMs for its base > >> and Python related packages, I think we can achieve this with Easybuild. > >> > >> I also need packages from Bioconductor and would like this in Easybuild. > >> > >> > > You can already do this, just use the RPackage easyblock: > > https://github.com/hpcugent/easybuild-easyblocks/blob/master > /easybuild/easyblocks/generic/rpackage.py > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hpcugent_easybuild-2Deasyblocks_blob_master_easybuild_easyblocks_generic_rpackage.py&d=DgMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=vgm6I2WP7uYsV5FpE0KcJ2rVGCXcmVTIfAvGQpxzxGE&s=A7s1ffsl4M02TJrS8Evvto9ONl3pjwCAwRmiTvMkNV0&e=> > > see for example all these easyconfigs: > > https://github.com/hpcugent/easybuild-easyconfigs/search?utf > 8=%E2%9C%93&q=rpackage > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hpcugent_easybuild-2Deasyconfigs_search-3Futf8-3D-25E2-259C-2593-26q-3Drpackage&d=DgMFaQ&c=UE1eNsedaKncO0Yl_u8bfw&r=RMJdCm7m5fiPWhajwKUnEW5yn4eK2YdUWW-MLVShghg&m=vgm6I2WP7uYsV5FpE0KcJ2rVGCXcmVTIfAvGQpxzxGE&s=lCrQFQc5hVfIZkcr5qv3YR3uH1RmOmetArOh3LMzeV8&e=> > > > > > > As you will see these already use packages from Bioconductor etc. > > > > In fact this was the first way to install R packages in EasyBuild, the > > ext_lists option was only added later. > > This Rpackage EasyBlock will install the R package like a user would > > normally do, and generate a module wi
Re: [easybuild] HPL without OFED
Hi David, You won't find many easyconfigs using the "-no-OFED" toolchains and goalf-1.1.0 is quite old. I don't think it's a good choice specially if you are starting a new installation. I would recommend that you use one of the foss toolchains ( https://github.com/hpcugent/easybuild-easyconfigs/tree/master/easybuild/easyconfigs/f/foss) . Select the foss toolchain which better fits your preferences or requirements and tweak the OpenMPI easyconfig included in the toolchain to disable the infiniband support. This way you can reuse any of the available easyconfigs for the foss toolchain that you choose. regards, Pablo. 2016-12-20 15:29 GMT+01:00 David Ramírez : > Hi everybody > > I'm looking for HPL + GCC witout OFED, to make test. But I don't see it on > EasyConfigs. > > I'm making a demo but now I can't use > > eb HPL-2.0-goalf-1.1.0-no-OFED.eb --robot > > > Este correo y sus archivos asociados son privados y confidenciales y va > dirigido exclusivamente a su destinatario. Si recibe este correo sin ser el > destinatario del mismo, le rogamos proceda a su eliminación y lo ponga en > conocimiento del emisor. La difusión por cualquier medio del contenido de > este correo podría ser sancionada conforme a lo previsto en las leyes > españolas. No se autoriza la utilización con fines comerciales o para su > incorporación a ficheros automatizados de las direcciones del emisor o del > destinatario . > > This mail and its attached files are confidential and are exclusively > intended to their addressee. In case you may receive this mail not being > its addressee, we beg you to let us know the error by reply and to proceed > to delete it. The circulation by any mean of this mail could be penalised > in accordance with the Spanish legislation. The use of both the transmitter > and the addressee’s address with a commercial aim, or in order to be > incorporated to automated files, is not authorised. > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] R extra packages
real popular here but we tend to make things easy for users > >> (to not confuse them). > >> Making it easy for ourselves seems to be another argument (hard to > >> disagree with the ease of "pip install"). > >> > >> But if I was doing performance analysis like I've done in the past > >> I'd want to nail down every version I'm using and make sure that some > >> random update didn't warp my data. > >> > >> jack > >> > >> > >>> -Original Message- > >>> From: easybuild-requ...@lists.ugent.be > >>> [mailto:easybuild-requ...@lists.ugent.be] On Behalf Of Jack Perdue > >>> Sent: Friday, December 16, 2016 11:39 AM > >>> To: easybuild@lists.ugent.be > >>> Subject: Re: [easybuild] R extra packages > >>> > >>> On 12/16/2016 10:22 AM, Siddiqui, Shahzeb wrote: > >>>> Hello, > >>>> > >>>> I would like to create a easyconfig for each R package, I find that > >>>> to be best suitable for our environment at Pfizer. I know we can pass > >>>> along the R packages in the R easybuild using exts_list . The only > >>>> issue is we have our own set of R packages that we need to install, > >>>> and I am not sure in which order they need to be installed. > >>>> > >>>> It's important to separate the R installation from the R packages > >>>> because in the case of a build error in R package it would fail to > >>>> build R if they are both together. > >>>> > >>>> I would like to build R packages as modules which will be > >>>> particularly important for some special R packages that require a > >>>> special dependency. For instance we need R2Jags and that has a > >>>> dependency for JAGS. I have JAGS in a module environment so I would > >>>> like R2Jags module file to load Jags before a user can use R2Jags. > >>>> > >>>> When loading base R module it will allow user to have access to base > >>>> R packages that is quite small. The special R packages that are build > >>>> with Easybuild will have their R module only visible when loading R. > >>>> This can be done via MODULEPATH variable. > >>>> > >>>> Similar to how you have Python package like numpy, I would like this > >>>> framework for R packages. > >>>> > >>> If you want to look over my experiment for automatically building > >>> easyconfigs based on CRAN data, see: > >>> > >>> http://www.siliconslick.com/easybuild/R_madness/ > >>> > >>> There are the scripts (which will need work), the easyconfigs > >>> created, and the modules built from those. > >>> > >>> I thought it was a great idea. Others on our staff, not so much > >>> (they are content to use R and Python's built in tools for > >>> installing to a directory out of the EasyBuild tree). > >>> > >>> Personally, as someone who likes to keep track of what versions he's > >>> using (and not have some "pip" update something under my feet), I'd > >>> spend time doing the same for Python... but I got plenty other stuff > >>> to keep me busy [I'm having a good old time with --minimal-toolchains > >>> and my new 2016D toolchains based on GCCcore/6.2.0] > >>> > >>> jack > >>> > >>>> Shahzeb Siddiqui > >>>> > >>>> HPC Linux Engineer > >>>> > >>>> B2220-447.2 > >>>> > >>>> Groton, CT > >>>> > >>>> > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] How to hide modules at the Lmod level?
2016-11-19 15:44 GMT+01:00 Ole Holm Nielsen : > > > IMHO, the --hidden option ought to be the EB default for all "system" > modules which are irrelevant to end users. > > I think that deciding what modules are irrelevant for users is something subjective which depends on the site/admin/users preferences or policies. IMHO, one of the big advantages of easybuild is that it allows customization to fit the site preferences. With the current options easybuild allows you to choose what to hide and what not and I think it's fine like that so it meet the requirements for any site. Just my opinion ;) -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] How to hide modules at the Lmod level?
Hi Ole, I think Lmod 6.x doesn't support using a modulerc file. This feature was added in latest Lmod 7.0. See the changelog here https://github.com/TACC/Lmod What Lmod 6.x supports is hidding a module if the version in the module name starts by dot. e.g. a module like "zlib/.1.2.8" would be hidden by Lmod. To list hidden and non-hidden modules you should do "module av --show-hidden" You can generate hidden modules with Easybuild (including the required dot) using the --hide-deps option http://easybuild.readthedocs.io/en/latest/Manipulating_ dependencies.html?highlight=hidden#installing-dependencies-as-hidden- modules-using-hide-deps You can also use the --hidden option (see build parameters) http://easybuild.readthedocs.io/en/latest/version-specific/ easyconfig_parameters.html?highlight=hidden If you are using Lmod 6.x and you want to hide modules you will have to regenerate the module files to add the required dot. For this the --module-only option in easybuild can be useful. If you upgrade to Lmod 7.x you can use the modulerc file and avoid renaming module names. regards, Pablo. 2016-11-19 12:57 GMT+01:00 Ole Holm Nielsen : > I think that EB dependencies showing up in .../modules/all are confusing > to end users. We should avoid exposing dependency modules such as > Bison, GCCcore, M4, zlib, etc.etc. to end users. > > Lmod and Tmod apparently allow you to hide modules in a modulerc file. > For Lmod 7.0 the syntax is "hide-module foo/3.2.1". > > However, I can't get this to work. If I create a file > .../modules/all/.modulerc with the line "hide-module zlib/1.2.8", the > zlib still shows up with "module avail". I haven't been able to find > any documentation of the function of MODULERCFILE or .modulerc in Lmod > or Tmod. We're using Lmod 6.5.1 on CentOS 7.2. > > Question: Has anyone succeeded in hiding modules with Lmod in this way? > > Thanks, > Ole > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Has anyone created EB files for the Intel 2017 compilers?
2016-11-03 15:12 GMT+01:00 Vanzo, Davide : > Pablo, > > I am trying to install the 2017.1 version and I will be happy to share as > soon as I fix a little issue with the MKL. > thanks! > > -- > Davide Vanzo, PhD > Application Developer > Adjunct Assistant Professor of Chemical and Biomolecular Engineering > Advanced Computing Center for Research and Education (ACCRE) > Vanderbilt University - Hill Center 201 > (615)-875-9137 > www.accre.vanderbilt.edu > > On Nov 3 2016, at 6:23 am, Pablo Escobar Lopez < > pablo.escobarlo...@unibas.ch> wrote: > >> Hi Ole, >> >> Could you share your iomkl easyconfig using the latest intel2017 compiler? >> >> regards, >> Pablo. >> >> 2016-11-03 12:07 GMT+01:00 Fotis Georgatos : >> >> Hi Ole, all, >> >> On Nov 2, 2016, at 2:33 PM, Ole Holm Nielsen >> wrote: >> > On 11/02/2016 12:52 PM, Åke Sandgren wrote: >> >> env INTEL_LICENSE_FILE=port@host eb intel-2016a.eb ... >> > >> > Fantastic, this worked for me! I wonder where the usage of >> license_file and/or INTEL_LICENSE_FILE within EB might be documented? >> >> It SHOULD ;-) >> >> There’s one caveat with the port@host approach: >> >> One day, Intel’s practices will force you to setup a second license server >> (either because you need to setup a new flexlm server version or, >> because Intel has the nasty habit of silently deprecating old components >> in new licenses >> and you need to point to a test instance without touching your production >> instance yet) >> >> So, it’s more beneficial to point to a file, so that you can easily >> redirect clients >> to a new server instance, from a *single* point of control - without >> touching the modulefiles at all! >> If you have 100s of Intel modulefiles in module buildset trees, you know >> what I’m talking about! >> >> Also one more potential caveat: although it IS possible to drop in that >> one just the first 2 lines >> of the license file (ie. license server coordinates) don’t do that either >> and use the whole thing: >> I’ve met situations where certain Intel components (not all) have bugs >> and don’t cooperate well. >> I no longer have the Intel ticket numbers but you can take my word for >> it: I’ve spend endless hours around it. >> >> Last but not least, something I’ve not tried yet: you could be pointing >> instead to a licensefile >> sitting on /dev/shm, presumably populated at node boot time, so that you >> can have more granularity >> of control - at node level rather than cluster. This would permit you fi. >> to modify node behaviour >> in a rolling update fashion. I haven’t yet been hit by the absence of >> this but it’s a nice to have. >> >> Fotis >> >> -- >> echo "sysadmin know better bash than english" | sed s/min/mins/ \ >> | sed 's/better bash/bash better/' # signal detected in a CERN forum >> >> >> >> >> >> >> >> >> >> >> -- >> Pablo Escobar López >> HPC systems engineer >> sciCORE, University of Basel >> SIB Swiss Institute of Bioinformatics >> http://scicore.unibas.ch >> > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Has anyone created EB files for the Intel 2017 compilers?
Hi Ole, Could you share your iomkl easyconfig using the latest intel2017 compiler? regards, Pablo. 2016-11-03 12:07 GMT+01:00 Fotis Georgatos : > Hi Ole, all, > > On Nov 2, 2016, at 2:33 PM, Ole Holm Nielsen > wrote: > > On 11/02/2016 12:52 PM, Åke Sandgren wrote: > >> env INTEL_LICENSE_FILE=port@host eb intel-2016a.eb ... > > > > Fantastic, this worked for me! I wonder where the usage of license_file > and/or INTEL_LICENSE_FILE within EB might be documented? > > It SHOULD ;-) > > There’s one caveat with the port@host approach: > > One day, Intel’s practices will force you to setup a second license server > (either because you need to setup a new flexlm server version or, > because Intel has the nasty habit of silently deprecating old components > in new licenses > and you need to point to a test instance without touching your production > instance yet) > > So, it’s more beneficial to point to a file, so that you can easily > redirect clients > to a new server instance, from a *single* point of control - without > touching the modulefiles at all! > If you have 100s of Intel modulefiles in module buildset trees, you know > what I’m talking about! > > Also one more potential caveat: although it IS possible to drop in that > one just the first 2 lines > of the license file (ie. license server coordinates) don’t do that either > and use the whole thing: > I’ve met situations where certain Intel components (not all) have bugs and > don’t cooperate well. > I no longer have the Intel ticket numbers but you can take my word for it: > I’ve spend endless hours around it. > > Last but not least, something I’ve not tried yet: you could be pointing > instead to a licensefile > sitting on /dev/shm, presumably populated at node boot time, so that you > can have more granularity > of control - at node level rather than cluster. This would permit you fi. > to modify node behaviour > in a rolling update fashion. I haven’t yet been hit by the absence of this > but it’s a nice to have. > > Fotis > > -- > echo "sysadmin know better bash than english" | sed s/min/mins/ \ > | sed 's/better bash/bash better/' # signal detected in a CERN forum > > > > > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] easybuild -devel files for users?
wouldn't make sense if compiler modules provide $CC $CXX and Co. by default? 2016-10-22 23:59 GMT+02:00 Alan O'Cais : > Maybe I'm misunderstanding but that is pretty much exactly why I 'created' > the trivial buildenv easyblock. The users just load the module that gets > created and they can simply use $CC and friends in their build processes, > making switching toolchains very straightforward. > > On 22 Oct 2016 4:09 p.m., "Vanzo, Davide" > wrote: > >> From a user perspective it would be more interesting to have this option >> under Lmod than EB. The end user is most likely to interact only with the >> environment modules manager than EB directly. The idea would be that the >> user loads all the modules he/she needs according to a specific >> toolchain and can get all the linking flags by calling something like: >> >> $module -L >> >> In this way the user can use command substitution directly into >> Makefiles. More importantly, this should also take into account ordering >> the lib flags accordingly since sometimes order is essential for >> successful linking. >> That is something that we discussed internally since this functionality >> is provided by our current modules manager (although in a very primitive >> way) and would be lost when transitioning to Lmod. >> >> -- >> Davide Vanzo, PhD >> Application Developer >> Adjunct Assistant Professor of Chemical and Biomolecular Engineering >> Advanced Computing Center for Research and Education (ACCRE) >> Vanderbilt University - Hill Center 201 >> (615)-875-9137 >> www.accre.vanderbilt.edu >> >> On Oct 22 2016, at 8:12 am, Jack Perdue wrote: >> >>> On 10/22/2016 02:47 AM, Kenneth Hoste wrote: >>> > Hi Jack, >>> > >>> > On 21/10/16 02:46, Jack Perdue wrote: >>> >> Howdy EB gods (Ken and crew), >>> >> >>> >> So EB, puts some useful environment settings in >>> >> //easybuild/*-devel >>> >> >>> >> Is there a way to utilize those variables when not >>> >> using EB? >>> >> >>> >> For example, I assume that if a user loads icc that they >>> >> more than likely would like CC=icc (CFLAGS is a different >>> >> matter). >>> >> >>> >> But EB doesn't do that now (probably for good reason). >>> > >>> > It doesn't right now, no, but would it be interesting to add an option >>> > to include statements to define the build environment as used by >>> > EasyBuild in the toolchain module? >>> > >>> > You can argue for it both ways, it's just a matter of what EasyBuild >>> > should do by default (and we currently have a window of opportunity to >>> > change the default behavior with EasyBuild 3.0 being the next release). >>> >>> I'd definitely like to see it as an option. For testing, >>> --dump-env-script on >>> ImageMagick-7.0.3-2-intel-2016b.eb would be a good start. I'd like to >>> see _everything_ set in that .env instead set in the modules first... as >>> just a >>> start. >>> >>> Beyond that, I'd also like to see a line with -lX11 -lXabc -lX123 so I >>> don't need to >>> remember the name of all the X11 libs. >>> >>> But it should be a site (or user) selectable option (I could see some >>> people not wanting such things). >>> >>> >>> jack (who won't be making it to Utah but does have new cluster coming >>> online >>> and has the ability to provide beer through his agents) >>> >> > > -------- > > > > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > > > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] FFTW dynamic libraries?
Hi Kenneth, I needed this because while compiling an internal application I noticed that it requires "libfftw3f_threads.so". As my compute nodes have fftw-devel rpm installed the build system was using this library from the system. After recompiling my FFTW module with --enable-shared the application uses the libfftw3f_threads.so lib provided by my FFTW module instead of the system one. These are the contents of my $EBROOTFFTW/lib/ folder when compiling with --enable-shared. It includes both the static and dynamic libraries: https://gist.github.com/pescobar/90cf77b0bcb5c1494b18365ce50a78cb IMHO, it makes sense that the FFTW installation in easybuild provides both the dynamic and static libraries so applications depending on the fftw dynamic libs can use the easbuild FFTW module. Does this have any drawback I am missing? regards, Pablo. 2016-10-13 12:47 GMT+02:00 Kenneth Hoste : > Hi Pablo, > > On 13/10/16 11:48, Pablo Escobar Lopez wrote: > > Hi, > > I have noticed that FFTW-3.3.3-gompi-1.4.10.eb is not building the > dynamic FFTW libraries. For this I think the fix would be to change > configopts from this: > > common_configopts = "--enable-threads --enable-openmp --with-pic" > > to > > common_configopts = "--enable-threads --enable-openmp --with-pic > *--enable-shared*" > > Is there any reason why the provided FFTW easyconfig is not building the > dynamic libraries. Would it be safe to apply this change so the dynamic > libs are built? or is there any issue which could be triggered by doing > this change which I am missing? > > > It's probably just because nobody has had a strict need for the dynamic > libraries yet? What's your use case for needing them? > > The only thing I can think of is that by using --enable-shared the static > libraries may not be built anymore, but you'll have to check that (and the > sanity check should fail anyway in that case). > > > regards, > > Kenneth > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
[easybuild] FFTW dynamic libraries?
Hi, I have noticed that FFTW-3.3.3-gompi-1.4.10.eb is not building the dynamic FFTW libraries. For this I think the fix would be to change configopts from this: common_configopts = "--enable-threads --enable-openmp --with-pic" to common_configopts = "--enable-threads --enable-openmp --with-pic *--enable-shared*" Is there any reason why the provided FFTW easyconfig is not building the dynamic libraries. Would it be safe to apply this change so the dynamic libs are built? or is there any issue which could be triggered by doing this change which I am missing? regards, Pablo. -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] How to build modules compiled for different hardware architectures?
2016-10-04 11:49 GMT+02:00 Martin : > > > * Within R users are loading the module and doing install.packages() > on a random node, later they want to use that package on a script and it > gives "random" errors with illegal instruction depending on where it ran. > -- User feedback is that this "never happens" with an RPM based R install > when they put packages in their own R user libs. I know this is not > directly EB related still this is the user feedback I get. > If you install your R module with easybuild using --optarch=GENERIC if the users link a R library in their home folder the same compiler flags would be used (no optimization). This should solve this problem (it works for me at least). The only case where I can imagine you could hit this issue is in case the build script provided with the R library hardcodes the optimization options (see the openblas example below) > > * We have 3 geographically distributed sites and would like to be able to > compile once and then rsync to the others. We are still running into > problems if we aren't carefully selecting the host with the least features. > Where after doing the rsync software simply bails out. We are in the > process of investigating this. It seems OpenBLAS related which looksl ike > its doing some kind of optimization on its own. I hope to be able to look > at what EPEL or Debian do to get rid of this. > OpenBLAS is a special case. See "Build environment vs hardcoding in build scripts" here http://easybuild.readthedocs.io/en/latest/Controlling_compiler_optimization_flags.html . For my generic software stack I use buildopts = 'TARGET=PRESCOTT BINARY=64 USE_THREAD=1 CC="$CC" FC="$F77" NO_AFFINITY=1' regards, Pablo. -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: Discontinuation of packages (Re: [easybuild] next EasyBuild conf call: *Thu* Sept 29th 2016, 5pm CET)
roducing an “attic” type concept might be an >> >> idea. Depreciated packages get to the attic. A typical “eb -S” >> >> would not show them, but e.g. an extra “attic” or “antique” >> >> option would. >> >> >> >> Best wishes >> >> Joachim >> >> >> >> >> >> >> >>> On 29 Sep 2016, at 18:36, Kenneth Hoste > >>> <mailto:kenneth.ho...@ugent.be>> wrote: >> >>> >> >>> notes are available at >> >>> https://github.com/hpcugent/easybuild/wiki/Conference-call- >> notes-20160929 >> >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ >> github.com_hpcugent_easybuild_wiki_Conference-2Dcall- >> 2Dnotes-2D20160929&d=CwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=yuyoB >> kmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m=QycjT3AMp8mFa5J-RDc >> s2kGxbKac7JJX5bpmHAd9efw&s=N2C0HQpIsDmyOZxHfPYSEoQ0RWTM6m-qFdrDeiDdQLM&e= >> > >> >>> >> >>> Next conf call is planned for Wed Oct 12th 5pm CET, cfr. >> >>> https://plus.google.com/events/cir0grjke0kbfutgmgp2v9nfm3s >> >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus. >> google.com_events_cir0grjke0kbfutgmgp2v9nfm3s&d=CwMGaQ&c= >> ODFT-G5SujMiGrKuoJJjVg&r=yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWc >> QG9gmjbA&m=QycjT3AMp8mFa5J-RDcs2kGxbKac7JJX5bpmHAd9efw&s=lqH >> DMMXzR2b_QOl3MWdeGZ6-K4fq1p9r90SzNOaen7k&e=> >> >>> >> >>> On 27/09/16 21:03, Kenneth Hoste wrote: >> >>>> Dear EasyBuilders, >> >>>> >> >>>> The next EasyBuild conf call is planned for *Thursday* >> >>>> September 29th 2016, 5pm CET; >> >>>> see also >> >>>> https://plus.google.com/events/c8tqca652anjfqblifjam5q9s6k >> >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus. >> google.com_events_c8tqca652anjfqblifjam5q9s6k&d=CwMGaQ&c= >> ODFT-G5SujMiGrKuoJJjVg&r=yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWc >> QG9gmjbA&m=QycjT3AMp8mFa5J-RDcs2kGxbKac7JJX5bpmHAd9efw&s=n2y >> NmrgzxmalXYD5jMGcOalL_vvGx4FsUxv3DSvnIOw&e=> >> >>>> . >> >>>> >> >>>> (just this once, I've planned it on a Thursday rather than a >> >>>> Wednesday due to an agenda conflict) >> >>>> >> >>>> >> >>>> Topics I have in mind include: >> >>>> >> >>>>* (very early) outlook to EasyBuild v3.0: default config >> >>>> changes, (minor) backwards-incompatible changes >> >>>> >> >>>>* deprecating toolchains: what & how >> >>>> * see also >> >>>> https://github.com/hpcugent/easybuild/wiki/Deprecated-toolchains >> >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ >> github.com_hpcugent_easybuild_wiki_Deprecated-2Dtoolchains& >> d=CwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=yuyoBkmTkIQPbv1BTF9U27ww >> 5Lm7GhsMmWcQG9gmjbA&m=QycjT3AMp8mFa5J-RDcs2kGxbKac7JJX5bpmHA >> d9efw&s=gSnvv8s_znJmFMPSBIPgAAsZAc8mMg6hAWiO03QUWck&e=> >> >>>> >> >>>>* update on support for RPATH (WIP) >> >>>>* >> >>>> https://github.com/hpcugent/easybuild-framework/compare/dev >> elop...boegel:rpath?expand=1 >> >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ >> github.com_hpcugent_easybuild-2Dframework_compare_develop... >> boegel-3Arpath-3Fexpand-3D1&d=CwMGaQ&c=ODFT-G5SujMiGrKuoJJjV >> g&r=yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m=QycjT3AMp8 >> mFa5J-RDcs2kGxbKac7JJX5bpmHAd9efw&s=6II9MfeFGUY-xpwQ-GUiv-n- >> m2x4p8oKd04yCGhzTr0&e=> >> >>>> >> >>>>* Q&A >> >>>> >> >>>> Suggestions for additional topics are welcome. >> >>>> >> >>>> Please let me know if you're planning to attend this conf call. >> >>>> >> >>>> More information about the EasyBuild conf calls is available at >> >>>> https://github.com/hpcugent/easybuild/wiki/Conference-calls >> >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__ >> github.com_hpcugent_easybuild_wiki_Conference-2Dcalls&d= >> CwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=yuyoBkmTkIQPbv1BTF9U27ww5L >> m7GhsMmWcQG9gmjbA&m=QycjT3AMp8mFa5J-RDcs2kGxbKac7JJX5bpmHAd9 >> efw&s=nABxLrDAlozucg_Duk9GoWHhALQ-rgs7w2oUqJPu7K0&e=> >> >>>> . >> >>>> >> >>>> >> >>>> regards, >> >>>> >> >>>> Kenneth >> >>> >> >> >> >> >> >> >> >> >> >> -- >> >> Pablo Escobar López >> >> HPC systems engineer >> >> sciCORE, University of Basel >> >> SIB Swiss Institute of Bioinformatics >> >> http://scicore.unibas.ch >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ >> scicore.unibas.ch_&d=CwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r=yuyoB >> kmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m=QycjT3AMp8mFa5J-RDc >> s2kGxbKac7JJX5bpmHAd9efw&s=nIOTUVJ_iBoI31ySMfWLAJEoa3AdRIR3AZfBUeklBZw&e= >> > >> > >> >> > > > -- > Pablo Escobar López > HPC systems engineer > sciCORE, University of Basel > SIB Swiss Institute of Bioinformatics > http://scicore.unibas.ch > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: Discontinuation of packages (Re: [easybuild] next EasyBuild conf call: *Thu* Sept 29th 2016, 5pm CET)
>> work around it. Introducing an “attic” type concept might be an > >> idea. Depreciated packages get to the attic. A typical “eb -S” > >> would not show them, but e.g. an extra “attic” or “antique” > >> option would. > >> > >> Best wishes > >> Joachim > >> > >> > >> > >>> On 29 Sep 2016, at 18:36, Kenneth Hoste >>> <mailto:kenneth.ho...@ugent.be>> wrote: > >>> > >>> notes are available at > >>> https://github.com/hpcugent/easybuild/wiki/Conference- > call-notes-20160929 > >>> <https://urldefense.proofpoint.com/v2/url?u=https- > 3A__github.com_hpcugent_easybuild_wiki_Conference- > 2Dcall-2Dnotes-2D20160929&d=CwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r= > yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m=QycjT3AMp8mFa5J- > RDcs2kGxbKac7JJX5bpmHAd9efw&s=N2C0HQpIsDmyOZxHfPYSEoQ0RWTM6m > -qFdrDeiDdQLM&e=> > >>> > >>> Next conf call is planned for Wed Oct 12th 5pm CET, cfr. > >>> https://plus.google.com/events/cir0grjke0kbfutgmgp2v9nfm3s > >>> <https://urldefense.proofpoint.com/v2/url?u=https- > 3A__plus.google.com_events_cir0grjke0kbfutgmgp2v9nfm3s&d=CwMGaQ&c=ODFT- > G5SujMiGrKuoJJjVg&r=yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m= > QycjT3AMp8mFa5J-RDcs2kGxbKac7JJX5bpmHAd9efw&s=lqHDMMXzR2b_QOl3MWdeGZ6- > K4fq1p9r90SzNOaen7k&e=> > >>> > >>> On 27/09/16 21:03, Kenneth Hoste wrote: > >>>> Dear EasyBuilders, > >>>> > >>>> The next EasyBuild conf call is planned for *Thursday* > >>>> September 29th 2016, 5pm CET; > >>>> see also > >>>> https://plus.google.com/events/c8tqca652anjfqblifjam5q9s6k > >>>> <https://urldefense.proofpoint.com/v2/url?u=https- > 3A__plus.google.com_events_c8tqca652anjfqblifjam5q9s6k&d=CwMGaQ&c=ODFT- > G5SujMiGrKuoJJjVg&r=yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m= > QycjT3AMp8mFa5J-RDcs2kGxbKac7JJX5bpmHAd9efw&s=n2yNmrgzxmalXYD5jMGcOalL_ > vvGx4FsUxv3DSvnIOw&e=> > >>>> . > >>>> > >>>> (just this once, I've planned it on a Thursday rather than a > >>>> Wednesday due to an agenda conflict) > >>>> > >>>> > >>>> Topics I have in mind include: > >>>> > >>>>* (very early) outlook to EasyBuild v3.0: default config > >>>> changes, (minor) backwards-incompatible changes > >>>> > >>>>* deprecating toolchains: what & how > >>>>* see also > >>>> https://github.com/hpcugent/easybuild/wiki/Deprecated-toolchains > >>>> <https://urldefense.proofpoint.com/v2/url?u=https- > 3A__github.com_hpcugent_easybuild_wiki_Deprecated- > 2Dtoolchains&d=CwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r= > yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m=QycjT3AMp8mFa5J- > RDcs2kGxbKac7JJX5bpmHAd9efw&s=gSnvv8s_znJmFMPSBIPgAAsZAc8mMg6hAWiO03 > QUWck&e=> > >>>> > >>>>* update on support for RPATH (WIP) > >>>>* > >>>> https://github.com/hpcugent/easybuild-framework/compare/ > develop...boegel:rpath?expand=1 > >>>> <https://urldefense.proofpoint.com/v2/url?u=https- > 3A__github.com_hpcugent_easybuild-2Dframework_compare_ > develop...boegel-3Arpath-3Fexpand-3D1&d=CwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r= > yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m=QycjT3AMp8mFa5J- > RDcs2kGxbKac7JJX5bpmHAd9efw&s=6II9MfeFGUY-xpwQ-GUiv-n- > m2x4p8oKd04yCGhzTr0&e=> > >>>> > >>>>* Q&A > >>>> > >>>> Suggestions for additional topics are welcome. > >>>> > >>>> Please let me know if you're planning to attend this conf call. > >>>> > >>>> More information about the EasyBuild conf calls is available at > >>>> https://github.com/hpcugent/easybuild/wiki/Conference-calls > >>>> <https://urldefense.proofpoint.com/v2/url?u=https- > 3A__github.com_hpcugent_easybuild_wiki_Conference-2Dcalls&d=CwMGaQ&c=ODFT- > G5SujMiGrKuoJJjVg&r=yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m= > QycjT3AMp8mFa5J-RDcs2kGxbKac7JJX5bpmHAd9efw&s=nABxLrDAlozucg_Duk9GoWHhALQ- > rgs7w2oUqJPu7K0&e=> > >>>> . > >>>> > >>>> > >>>> regards, > >>>> > >>>> Kenneth > >>> > >> > >> > >> > >> > >> -- > >> Pablo Escobar López > >> HPC systems engineer > >> sciCORE, University of Basel > >> SIB Swiss Institute of Bioinformatics > >> http://scicore.unibas.ch > >> <https://urldefense.proofpoint.com/v2/url?u=http- > 3A__scicore.unibas.ch_&d=CwMGaQ&c=ODFT-G5SujMiGrKuoJJjVg&r= > yuyoBkmTkIQPbv1BTF9U27ww5Lm7GhsMmWcQG9gmjbA&m=QycjT3AMp8mFa5J- > RDcs2kGxbKac7JJX5bpmHAd9efw&s=nIOTUVJ_iBoI31ySMfWLAJEoa3AdRIR3AZfBUe > klBZw&e=> > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: Discontinuation of packages (Re: [easybuild] next EasyBuild conf call: *Thu* Sept 29th 2016, 5pm CET)
I like what Joachim suggest about the "attic" concept. Old/deprecated easyconfigs provided "as-is" but not included in the regular tests. I think I suggested something similar in one of the conf calls. About how to search in the legacy easyconfigs I would prefer a different approach . What I would suggest is that instead of skipping those ones when doing "eb --search" I would still show them when doing a regular search with "eb --search" but in case the search result comes from the "attic" print a big disclaimer saying something like "I found this legacy easyconfig in the attic but we aware that this is not tested". Or maybe in case some legacy stuff is found print a url which links to the docs sections explaining what is the attic. My fear about not searching by default in legacy easyconfigs is that many people would skip the extra option needed to search in the legacy easyconfigs and they wouldn't notice that it's there. Many times even if the legacy easyconfig doesn't work it can be useful as a start point to write a new one so it's useful to find them easily. I know many of us are sysadmins and we like to reply RTFM but easybuild has many many different config flags and it's easy to skip this one ( even for a sysadmin ;) regards, Pablo. 2016-09-30 11:41 GMT+02:00 Joachim Hein : > Hi Kenneth, > > I had a look into the notes and think some extra discussion is required on > the “discontinuation business”. Not sure where to start this, so I am > replying here. > > I occasionally get requests for “historic” and “fossile” versions of > packages - not all of them totally unreasonable, e.g. API change in > application. Examples include OpenFOAM and Gromacs. Prior to EasyBuild I > would have said no, unless you give me a strong reason. > > With EasyBuild, I check whether a to the user acceptable version is in the > “back catalog” and try to easybuild it. If it builds without much issue I > roll it out. If it breaks I am not worse off than without EasyBuild. > Typically I build with the “old” machinery of the time (that is what it was > tested with) - installing a legacy GCC and OpenMPI in EB is typically easy. > > > I think here are two big selling points of EB, which we perhaps not even > push enough. > >- Better user experience - many requests for legacy versions can >easily be supported. It is typically easier to just build it instead of >discussing with the user. >- Building software with a tried and tested software stack gives >stability against bugs introduced in the latest version > > > So, how do you plan to do the discontinuation? Removing toolchain > versions will break everything that builds on that toolchain. Keeping them > as “unsupported” seems the way forward to me. That is, it is still > available in some form but no longer tested and fixed. If they still work, > good, if they are broken: work around it. Introducing an “attic” type > concept might be an idea. Depreciated packages get to the attic. A > typical “eb -S” would not show them, but e.g. an extra “attic” or “antique” > option would. > > Best wishes > Joachim > > > > On 29 Sep 2016, at 18:36, Kenneth Hoste wrote: > > notes are available at https://github.com/hpcugent/ > easybuild/wiki/Conference-call-notes-20160929 > > Next conf call is planned for Wed Oct 12th 5pm CET, cfr. > https://plus.google.com/events/cir0grjke0kbfutgmgp2v9nfm3s > > On 27/09/16 21:03, Kenneth Hoste wrote: > > Dear EasyBuilders, > > The next EasyBuild conf call is planned for *Thursday* September 29th > 2016, 5pm CET; > see also https://plus.google.com/events/c8tqca652anjfqblifjam5q9s6k . > > (just this once, I've planned it on a Thursday rather than a Wednesday due > to an agenda conflict) > > > Topics I have in mind include: > >* (very early) outlook to EasyBuild v3.0: default config changes, > (minor) backwards-incompatible changes > >* deprecating toolchains: what & how >* see also https://github.com/hpcugent/easybuild/wiki/Deprecated- > toolchains > >* update on support for RPATH (WIP) >* https://github.com/hpcugent/easybuild-framework/compare/ > develop...boegel:rpath?expand=1 > >* Q&A > > Suggestions for additional topics are welcome. > > Please let me know if you're planning to attend this conf call. > > More information about the EasyBuild conf calls is available at > https://github.com/hpcugent/easybuild/wiki/Conference-calls . > > > regards, > > Kenneth > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] EB 2.9.0: Irresolvable dependencies for Python 3.5.2
2016-09-29 11:14 GMT+02:00 Ole Holm Nielsen : > > I'm beginning to wonder if I should go back to an older foss toolchain? > > Personally I prefer to don't use the latest and greatest toolchain unless it's mandatory because the application I need to build requires it. Based on my experience this leads to hit new issues because many times you are using a compiler that the original developer never tried. Default toolchain in my cluster is goolf/1.7.20 (gcc/4.8) for historic reasons. If I had to build something from scratch right now probably I would go for foss-2015b.eb (latest foss using gcc 4.9.x). But that's a matter of taste and requirements. Regarding your question about internal documentation, what I do is to keep all my installed easyconfigs in a git repository ( http://easybuild.readthedocs.io/en/latest/Configuration.html?highlight=repositorypath#easyconfigs-repo). You can also use a local folder to store your installed easyconfigs instead of a git repo. This repo is my "documentation". If I need to reinstall something in the future I point the robot-path to my private repository of installed easyconfigs by doing "eb application-foo-bar.eb -r /path/to/my/local/repo/" regards, Pablo. > Thanks for any advice, > Ole > > > > On 09/29/2016 11:07 AM, Ole Holm Nielsen wrote: > > Pablo, Kenneth: Thanks a lot! The --try-toolchain works correctly. > > > > Question: Is --try-toolchain just a hack for testing out something new, > > or is it supposed to be used also for production modules? > > > > If used for production, how may we remember/document the requirement for > > --try-toolchain with some older easyconfig? Documentation would seem to > > become an issue. > > > > Thanks, > > Ole > > > > On 09/29/2016 10:48 AM, Kenneth Hoste wrote: > >> Hi Ole, > >> > >> On 29/09/16 10:37, Pablo Escobar Lopez wrote: > >>> have you tried "eb Python-3.5.2-foss-2016.04.eb > >>> --try-toolchain=foss,2016.09 -r" ? > >> > >> +1 on this, the problem is that there are no easyconfigs available for > >> the dependencies of Python 3.5.2 with foss/2016.09. > >> > >> If you combine --try-toolchain (or --try-toolchain-version) with > >> --robot, EasyBuild will first construct the full dependency graph using > >> the foss/2016.04, and then generate easyconfigs (in a temp dir) for each > >> of the components using foss/2016.09, and then proceed to building and > >> installing everything. > >> > >> > >> regards, > >> > >> Kenneth > >>> > >>> 2016-09-29 8:55 GMT+02:00 Ole Holm Nielsen >>> <mailto:ole.h.niel...@fysik.dtu.dk>>: > >>> > >>> I need to build Python 3.5 for the EB 2.9.0 toolchain foss-2016.09. > >>> I copied the file Python-3.5.2-foss-2016.04.eb and replaced 04 by > >>> 09. > >>> > >>> Unfortunately, irresolvable dependencies are found: > >>> > >>> # eb Python-3.5.2-foss-2016.09.eb -r > >>> == temporary log file in case of crash > >>> /tmp/eb-mvPeVH/easybuild-tixAOo.log > >>> == resolving dependencies ... > >>> ERROR: Irresolvable dependencies encountered: > >>> bzip2/1.0.6-foss-2016.09, > >>> zlib/1.2.8-foss-2016.09, libreadline/6.3-foss-2016.09, > >>> ncurses/6.0-foss-2016.09, SQLite/3.13.0-foss-2016.09, > >>> Tk/8.6.5-foss-2016.09, GMP/6.1.1-foss-2016.09, > >>> XZ/5.2.2-foss-2016.09, > >>> libffi/3.2.1-foss-2016.09 > >>> > >>> I suppose that these dependencies could be built independently of > >>> the > >>> foss toolchain. > >>> > >>> Question: Does anyone have a working EB file > >>> Python-3.5.2-foss-2016.09.eb? > >>> > >>> Thanks a lot, > >>> Ole > >>> > >>> > >>> > >>> > >>> -- > >>> Pablo Escobar López > >>> HPC systems engineer > >>> sciCORE, University of Basel > >>> SIB Swiss Institute of Bioinformatics > >>> http://scicore.unibas.ch > >> > > > > -- > Ole Holm Nielsen > PhD, Manager of IT services > Department of Physics, Technical University of Denmark, > Building 307, DK-2800 Kongens Lyngby, Denmark > E-mail: ole.h.niel...@fysik.dtu.dk > Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/ > Tel: (+45) 4525 3187 / Mobile (+45) 5180 1620 > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] EB 2.9.0: Irresolvable dependencies for Python 3.5.2
have you tried "eb Python-3.5.2-foss-2016.04.eb --try-toolchain=foss,2016.09 -r" ? 2016-09-29 8:55 GMT+02:00 Ole Holm Nielsen : > I need to build Python 3.5 for the EB 2.9.0 toolchain foss-2016.09. > I copied the file Python-3.5.2-foss-2016.04.eb and replaced 04 by 09. > > Unfortunately, irresolvable dependencies are found: > > # eb Python-3.5.2-foss-2016.09.eb -r > == temporary log file in case of crash /tmp/eb-mvPeVH/easybuild-tixAOo.log > == resolving dependencies ... > ERROR: Irresolvable dependencies encountered: bzip2/1.0.6-foss-2016.09, > zlib/1.2.8-foss-2016.09, libreadline/6.3-foss-2016.09, > ncurses/6.0-foss-2016.09, SQLite/3.13.0-foss-2016.09, > Tk/8.6.5-foss-2016.09, GMP/6.1.1-foss-2016.09, XZ/5.2.2-foss-2016.09, > libffi/3.2.1-foss-2016.09 > > I suppose that these dependencies could be built independently of the > foss toolchain. > > Question: Does anyone have a working EB file Python-3.5.2-foss-2016.09.eb? > > Thanks a lot, > Ole > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] How to build modules compiled for different hardware architectures?
I use a similar solution. I have /soft/apps/arch1 /soft/apps/arch2/soft/apps/archN in a nfs server and I use autofs to mount the right folder in /soft/apps in each compute node. This way the path to access the software stack is the same in every machine (/soft/apps) but each machines uses the right one for its arch. To do this I build each new application in different machines with the different cpu types which already have the /soft/apps folder mounted. I use a flat naming scheme. I also have /soft/apps/generic which is a "special" software stack built with "eb --optarch=GENERIC" so when I need to make the software stack available in a non compute node (e.g a web server) I use the generic software stack and I don't need to worry about which specific cpu I have in this machine. Another trick I do is that my easybuild module (which is in the arch-independent folder so it's the same everywhere) includes some alias. E.g. eb-ivybridge="eb --configfiles=/soft/apps/easybuild-config-files/ivybridge.cfg". I keep the different easybuild config files in the arch specific folders (e.g. ivibridge.cfg is only accesible in /soft/apps/ivybrige/easybuild-config-files/ivybridge.cfg) this way if by mistake I do "eb-nehalem" when I am logged in a ivybridge machine I get an error "file nehalem.cfg not found" about the way to figure out the arch in the current machine I am not aware of any easy way. What I would do is adding a file /etc/profile.d/cpu_arch.sh which defines a environment variable like "cpu_arch=ivybridge". You can define a variable in your config management tool which can be used to generate this profile file. You can also use this variable to define the right software stack to mount in the machine. regards, Pablo. 2016-09-29 9:14 GMT+02:00 Åke Sandgren : > What we are aiming for is a combination. > > The achitecture independent code (like EasyBuild itself, intel and > portland compilers and similar) are installed under /eb/common/ > The arch dependent packages are under /eb/opt/... which is mounted per > client architecture/OS distro version, from the server file system. > > Actually /eb/common is also mounted depending on OS distribution. > > One then needs two module use lines, one for /eb/common/modules and one > for /eb/opt/modules. > > We keep the downloaded source dir (Easybuild "sourcepath") common for > all to minimize repeating downloads. > > However, I'm not sure this will work with a hierarchial module layout. > Haven't tried it yet. > > On 09/29/2016 08:38 AM, Ole Holm Nielsen wrote: > > We're installing EasyBuild 2.9.0 on our new cluster, and user > > application codes should be built on top of the latest foss-2016 > toolchain. > > > > Since our cluster has 4 different generations of Intel Xeon hardware > > (Nehalem, Sandy/Ivy Bridge, Haswell, Broadwell), users want to compile > > their application codes optimized for the compute node hardware (gcc > > -march=native). > > > > Question: Is there a best practices method for providing EasyBuild > > modules which are compiled for different hardware architectures? > > > > I can see some possible solutions: > > > > 1. Build totally separate and complete EB module trees for each of the > > architectures. Mount the correct EB module tree by NFS on the same > > mount point (/home/modules, say) on compute nodes based upon its > > architecture. > > > > 2. Within a single EB hierarchy, build multiple versions of just the > > application code modules requiring optimized code. This has the > > advantage of a shared module tree for all non-application-code modules. > > Users must then identify the compute node's architecture at run-time and > > load the correct module for that architecture (this sounds complicated > > for users). > > > > > > Bonus question: What's the most portable, reliable and lightweight way > > to determine which Intel Xeon architecture you're working on? Googling > > the question suggests using /proc/cpuinfo: > > > > # grep "model name" /proc/cpuinfo | head -1 > > model name: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz > > > > but then you must parse this string to discover whether you're on Xeon, > > Xeon v2, v3, v4, or v5 (from next year). > > > > Thanks for sharing your insights, > > Ole > > -- > Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden > Internet: a...@hpc2n.umu.se Phone: +46 90 7866134 Fax: +46 90-580 14 > Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Python 2.6 usage poll
Hi, Since some time ago I wanted to look at how difficult is to get non python software installed inside an anaconda environment and this looked like a good excuse to test it so I did a quick test to get Easybuil+Lmod inside conda. In case anyone want to test it, you can try this: $> wget -O /tmp/Miniconda2-latest-Linux-x86_64.sh https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh $> bash /tmp/Miniconda2-latest-Linux-x86_64.sh -b -f $> export PATH=~/miniconda2/bin:$PATH $> conda install -c pescobar lmod=6.5.12 $> pip install easybuild $> source ~/miniconda2/lmod/lmod/init/bash $> eb --modules-tool=Lmod foss-2016a.eb -r --dry-run Following these steps I could get EasyBuild+Lmod running in a vanilla centos7 installation quite easily. I think it should also work in any other linux distribution, no matter which are the installed packages from the OS repos. Personally I like this concept of running easybuild in a self-contained environment with fixed python version and python libraries and even extra non-python dependencies. IMHO, this would simplify the development and testing of easybuild and would make easier to plan the migration to python3. Also developers wouldn't need to worry if the user are going to deploy easybuild in centos6 (python2.6), centos7 (python2.7) or any other distro because easybuild would provide its own well tested python version + dependencies. Still this allows anyone to use a different approach if they prefer like using system's python or a centrally installed Lmod or environment modules (e.g. for a production deployment I would still use an independent Lmod installation). In any case, as I said, this is just an idea/suggestion so the main developers evaluate it. How easybuild is distributed now works fine for me and I also have the choice to use conda or not. regards, Pablo. 2016-09-19 2:44 GMT+02:00 Christopher Samuel : > On 16/09/16 20:15, Pablo Escobar Lopez wrote: > > > I remember we have commented in some hackathon (while having some beers > > I think) about the possibility of distributing easybuild as a > > self-contained app. Using something like mini-conda > > (http://conda.pydata.org/miniconda.html) with easybuild already > > installed on top of hit could do the trick. With this approach > > installing easybuild would be as simple as uncompressing a tarball, > > modifying PATH and use it (supposing a modules tool is already > > available). Has this been discussed again or it got lost with the beers? > > ;) > > The downside to that is it limits support to Intel hardware, it'd be > nicer to have something that could be more widely available. > > Plus, that's yet another version of Python to manage.. :-( > > All the best, > Chris (you are lost in a maze of conflicting Python dependencies, all > alike) > > -- > Christopher SamuelSenior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 > http://www.vlsci.org.au/ http://twitter.com/vlsci > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Python 2.6 usage poll
I remember we have commented in some hackathon (while having some beers I think) about the possibility of distributing easybuild as a self-contained app. Using something like mini-conda (http://conda.pydata.org/miniconda.html) with easybuild already installed on top of hit could do the trick. With this approach installing easybuild would be as simple as uncompressing a tarball, modifying PATH and use it (supposing a modules tool is already available). Has this been discussed again or it got lost with the beers? ;) Maybe easybuild 3.x could be a good moment to do this change. IMHO, it would simplify the development of easybuild as only one python environment needs to be supported and tested, it would make easybuild more portable and would give the main developers more freedom to choose the python version to use. The provided mini-conda could also provide all the required dependencies so the --dep-graph option or the git integration works out of the box (right now you need to add extra dependencies to get those options working). Maybe we could also include a working Lmod installation inside the mini-conda tarball so users can choose to use Lmod from the tarball or not. Maybe it would also help for installations reproducibility. Probably I am missing some drawbacks about this approach but I don't see them now. regards, Pablo. 2016-09-16 11:33 GMT+02:00 Jens Timmerman : > > > On 09/16/2016 10:34 AM, Riccardo Murri wrote: > > Hi all, > > > >> If you would be so kind to fill this in it would give me a sense of > what is going on in the > >> python 2.x/3.x support landscape. > > > > It might be worth to note that `pip` is dropping support for Python 2.6 > > in release 9.0, scheduled beginning of next year: see > https://github.com/pypa/pip/issues/3955 > > > on that note, here is a list of scientific python applications that have > pledged to drop python 2.X support in 2020: > https://python3statement.github.io/ > > EB can install these and provide a python3 module, so I expect no issues > on using these on HPC systems, > but it is a statement alright. > > > Ciao, > > R > > > > -- > > Regards, > Jens > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] COMSOL, Feko, ..
here you have the one I use for Schrodinger https://github.com/hpcugent/easybuild-easyconfigs/pull/762 I suggest you to look for in the list of open pull requests. Maybe you can find some working or semi-working easyconfigs there which hasn't been merged yet https://github.com/hpcugent/easybuild-easyconfigs/pulls regards, Pablo. 2016-08-17 16:47 GMT+02:00 nan...@luis.uni-hannover.de < nan...@luis.uni-hannover.de>: > > Hi, > > I can not find easyconfig&easyblocks of the following packages: COMSOL, > Feko, Gaussian, Amber(ambermd.org), > > Crystal ( https://www.crystalsolutions.eu), STAR-CCM+ ( > http://www.cd-adapco.com) and Schroedinger ( https://www.schrodinger.com). > > Could you please share them if you have any. > > Thank you. > > Gizo > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Lua modules and Lmod
Hi Maxime,
maybe this simple example showing how to add a new cli flag to easybuild
can help you as reference
https://github.com/hpcugent/easybuild-framework/pull/1250/files
regards,
Pablo.
2016-08-11 15:51 GMT+02:00 Maxime Boissonneault <
maxime.boissonnea...@calculquebec.ca>:
> Hi Kenneth,
> I just used the default GCC-4.9.2.eb file.
>
> Indeed, I would expect this to be a config parameter.
>
> If I can get a bit of guidance on how to define a new config parameter
> and then how to use it in the module generation, I don't mind doing a
> pull request for it.
>
> Regards,
>
> Maxime
>
>
> On 16-08-11 09:37, Kenneth Hoste wrote:
> > Hi Maxime,
> >
> > On 11/08/16 15:15, Maxime Boissonneault wrote:
> >> Hi,
> >>
> >> I am looking at modules generated with EasyBuild. I have the
> >> following config parameters:
> >>
> >> modules-tool: Lmod
> >> module-naming-scheme=HierarchicalMNS
> >> module-syntax=Lua
> >>
> >>
> >> For some reason, the GCC module contains a conflict over GCC:
> >> [mboisson@colosse1 easybuild]$ cat modules/all/Core/GCC/4.9.2.lua
> >>
> >> conflict("GCC")
> >> prepend_path("MODULEPATH",
> >> "/mnt/sda1/maxime/easybuild/modules/all/Compiler/GCC/4.9.2")
> >> ...
> >>
> >>
> >> Conflicts on module names are unnecessary when using Lmod. I was
> >> looking to disable this, and I find that in "module_generator.py",
> >> there is already the code which seems aware of this:
> >> 475 if self.app.cfg['moduleloadnoconflict']:
> >> 476 self.log.info("Nothing to do to ensure no conflicts
> >> can occur on load when using Lua modules files/Lmod")
> >>
> >> However, it does not seem to work in practice. Is this a bug ?
> >
> > You should be able to bypass the 'conflict' line from being included
> > by adding 'moduleloadnoconflict' in the easyconfig file you're using.
> > Are you saying it doesn't?
> >
> > Why the desire to kick out this conflict statement, it doesn't really
> > hurt to have it there, or does it?
> >
> > Obviously, this is not ideal when you want to avoid the conflict
> > statement in each and every module that is being generated.
> > For that, a configuration option should be added as well so you can
> > specify this once and for all...
> >
> >
> > regards,
> >
> > Kenneth
>
>
> --
> -
> Maxime Boissonneault
> Analyste de calcul - Calcul Québec, Université Laval
> Président - Comité de coordination du soutien à la recherche de Calcul
> Québec
> Team lead - Research Support National Team, Compute Canada
> Instructeur Software Carpentry
> Ph. D. en physique
>
>
--
Pablo Escobar López
HPC systems engineer
sciCORE, University of Basel
SIB Swiss Institute of Bioinformatics
http://scicore.unibas.ch
Re: [easybuild] best easyblock for simple package installation (wget, untar, install)
if you are downloading a tarball I think you want the "PackedBinary" easyblock 2016-07-19 13:56 GMT+02:00 Bernd Mohr : > Hi EasyBuilders, > > what would be the best EasyBlock to use for the installation of PACKAGE? > > Installation is as follows: > > a) wget /PACKAGE_.tar.gz > b) tar zxvf PACKAGE_.tar.gz > c) cd PACKAGE > d) ./install > > I guess it would be 'Binary' with install_cmd=./install ... > right? > > Thanks > Bernd > > -- > Dr.-Ing. Bernd Mohr > Juelich Supercomputing Centre > Institute for Advanced Simulation > > E-Mail: b.m...@fz-juelich.de > WWW: > http://www.fz-juelich.de/SharedDocs/Personen/IAS/JSC/EN/staff/mohr_b.html > > > > > > > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > > -------- > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Question about builddependencies and pkg-config
2016-06-22 22:57 GMT+02:00 Jack Perdue : > On 06/22/2016 03:31 PM, McGough, Benjamin E wrote: > > I have been struggling with creating a clean environment in which to > build with EasyBuild and test easyconfigs. > > I wish I could be pardoned if I just left a: > > 8^) > > and walked away... > > You are not alone. > > On our present cluster, I have done the best to strip > RH/CentOS -devel packages to force use of EB modules > on GPFS. In terms of X11 I'm already convinced I'm going > to try to get us to use the system X11 libs and not fight that > battle with EB. I also use the same approach. I install everything which is X11 related from system packages. I find too complex to install X11 libs using easybuild and I don't see any special advantage. As example, I install cairo-devel rpm in our machines. > EB's use of X11 _is_ very inconsistent. And > a royal pain in the arse. > > I'm not complaining... I'm just saying that there is some tweaking > that can be done. > > For our next cluster (RH/CentOS 6 or 7... dunno yet)... I plan to strip > everything > out of EB that I can get from CentOS or EPEL I'm sick of dealing > with X11 in EB. > > Our new approach will basically be "if a customer requires an app that > needs a newer version, build it with EB... else use the upstream version". > > In terms of EB, being a RH person for now, it would be nice to see two > fully decked out easyconfig trees... one with all the -devel packages > installed and one with none of them. > > I have followed the same road in different direction :) First I was using system packages for most dependencies which were no scientific software like zlib, bzip2, ncurses, libpng and similars (that's why I implemented "eb --filter-deps" option) but now I prefer to provide all those deps using easybuild as hidden modules (eb --hide-deps) and rely on system only for X11 related stuff. The main advantage I see for this approach is that this way the software stack is more self-contained. Now I use centos6 and when we upgrade to centos7 all scientific apps in my software stack will still use same dependencies and in theory this should help for scientific results reproducibility. There are also many scientific apps which require a different dependencies that the ones provided by system packages Pretty sure I know who will have to do the work to see that happen. > > jack > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Where is the LAPACK dependency defined for the foss toolchains
Hi Pieter,
This is the R-3.3.0 easyconfig I am using . I think the biggest change I
had to do was adding the XZ and PCRE deps
https://gist.github.com/anonymous/5380f4ed8ca853c4d1f41cca7fbafedb
And this is what I get in the configure output:
https://gist.github.com/pescobar/dd8430e9eda17aec3eaa8c05483e89d1
I think I get "LAPACK(generic)" because I have "lapack-devel" rpm installed
in my system.
Be aware that you should use a recent PCRE easyconfig
https://github.com/hpcugent/easybuild-easyconfigs/pull/2561
And you should recompile you bzip2 module using latest easybuild/2.8. If
you had no problem with bzip2 maybe you have an rpm providing libbz2.so and
you are using this one instead of the bzip2 version provided by easybuild
https://github.com/hpcugent/easybuild-easyblocks/pull/910
regards,
Pablo.
2016-05-29 17:32 GMT+02:00 Pieter Neerincx :
> Hi Kenneth and Pablo,
>
> Thanks for the intel'! This is indeed related. When I drop
> LAPACK_LIBS="$LIBLAPACK" from the preconfigopts, I get "... BLAS(OpenBLAS),
> LAPACK(in blas), ..." and the build no longer fails on compiling
> libRlapack.so.
> For the record this is for R 3.3.0.
>
> Unfortunately it now fails on libR.so itself :(. Looks like for some of
> the dependencies there is no shared lib available.
> This may be related to
>
> DEPRECATED AND DEFUNCT
> • The previously included versions of zlib, bzip2, xz and PCRE
> have been removed, so suitable external (usually system) versions are
> required (see the ‘R Installation and Administration’ manual).
>
> To make a standard R 3.3.0 compile (on Centos 6.7) I now have:
> * added some additional dependencies to the easyconfig that I previously
> did not need to specify
> * removed --enable-R-shlib from configure options
> * Used this workaround:
>
> IFS=':' read -r -a list_of_dirs <<< "${CPATH}"
> for path in ${list_of_dirs[@]}; do export CPPFLAGS="${CPPFLAGS:-}
> -I${path}"; done
>
> IFS=':' read -r -a list_of_dirs <<< "${LIBRARY_PATH}"
> for path in ${list_of_dirs[@]}; do export LDFLAGS="${LDFLAGS:-}
> -L${path}"; done
>
> The good news is that the code to detect the presence of libpng has been
> fixed. Previously the configure script would try to detect libpng twice:
> * once using pkg-config as dependency for Cairo to enable the 'cairo'
> device.
> * once without using pkg-config to enable the 'PNG' device.
> The latter would fail miserably when libpng was not available from default
> paths like in the case when deployed with EasyBuild elsewhere.
>
> So far I now have:
>
> Interfaces supported: tcltk
> External libraries:readline, BLAS(OpenBLAS), LAPACK(in blas),
> curl
> Additional capabilities: PNG, JPEG, TIFF, NLS, cairo, ICU
> Options enabled: R profiling
>
> Capabilities skipped:
> Options not enabled: shared BLAS, memory profiling
>
> Now lets see if the large list of additional (BioConductor and CRAN)
> packages we had in a previous version of R also will play along and
> compile...
>
> Cheers,
>
> Pi
>
>
>
> > On 28 May 2016, at 22:22, Pablo Escobar Lopez <
> pablo.escobarlo...@unibas.ch> wrote:
> >
> > Hi Pieter,
> >
> > maybe related?
> https://github.com/hpcugent/easybuild-easyconfigs/issues/1435
> >
> > Pablo
> >
> > 2016-05-28 16:23 GMT+02:00 Kenneth Hoste :
> > Hi Pieter,
> >
> > On 28/05/16 15:04, Pieter Neerincx wrote:
> > > Hi all,
> > >
> > > I'm trying to build a new R with a recent foss toolchain. I've tried
> 2015b and 2016a, but so far no luck. The build fails on compiling
> libRlapack.so. After some digging I found out that I have for example as
> part of foss/2015b:
> > >
> > > ScaLAPACK/2.0.2-gompi-2015b-OpenBLAS-0.2.14-LAPACK-3.5.0
> > >
> > > with a dependency on
> > >
> > > OpenBLAS/0.2.14-GNU-4.9.3-2.25-LAPACK-3.5.0
> > >
> > > The name suggests that LAPACK/3.5.0 is a dependency and will be loaded
> too, but LAPACK is absent according to "module list", which makes sense as
> none of the deployed module files specify LAPACK.
> > > I also cannot find LAPACK back as OS dependency in the corresponding
> *.eb easyconfigs... What am I missing here?
> >
> > The LAPACK sources are provided to OpenBLAS, which picks it up during
> > the build and embeds the LAPACK symbols in the OpenBLAS library:
> >
> > $ nm $EBROOTOPENBLAS/lib/libopenblas.a | grep dgeev
> > dgeev.o:
> > 0
Re: [easybuild] Where is the LAPACK dependency defined for the foss toolchains
Hi Pieter, maybe related? https://github.com/hpcugent/easybuild-easyconfigs/issues/1435 Pablo 2016-05-28 16:23 GMT+02:00 Kenneth Hoste : > Hi Pieter, > > On 28/05/16 15:04, Pieter Neerincx wrote: > > Hi all, > > > > I'm trying to build a new R with a recent foss toolchain. I've tried > 2015b and 2016a, but so far no luck. The build fails on compiling > libRlapack.so. After some digging I found out that I have for example as > part of foss/2015b: > > > > ScaLAPACK/2.0.2-gompi-2015b-OpenBLAS-0.2.14-LAPACK-3.5.0 > > > > with a dependency on > > > > OpenBLAS/0.2.14-GNU-4.9.3-2.25-LAPACK-3.5.0 > > > > The name suggests that LAPACK/3.5.0 is a dependency and will be loaded > too, but LAPACK is absent according to "module list", which makes sense as > none of the deployed module files specify LAPACK. > > I also cannot find LAPACK back as OS dependency in the corresponding > *.eb easyconfigs... What am I missing here? > > The LAPACK sources are provided to OpenBLAS, which picks it up during > the build and embeds the LAPACK symbols in the OpenBLAS library: > > $ nm $EBROOTOPENBLAS/lib/libopenblas.a | grep dgeev > dgeev.o: > T dgeev_ > dgeevx.o: > T dgeevx_ > lapacke_dgeev.o: > T LAPACKE_dgeev > U LAPACKE_dgeev_work > lapacke_dgeev_work.o: > T LAPACKE_dgeev_work > U dgeev_ > lapacke_dgeevx.o: > T LAPACKE_dgeevx > U LAPACKE_dgeevx_work > lapacke_dgeevx_work.o: > T LAPACKE_dgeevx_work > U dgeevx_ > > > I was confused by this too initially, see > https://github.com/xianyi/OpenBLAS/issues/203 > > > regards, > > Kenneth > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] SPAdes-3.7.1-goolf-1.4.10.eb FAILED
I see this in your logs Could NOT find BZip2 (missing: BZIP2_LIBRARIES BZIP2_INCLUDE_DIR) try adding bzip2 as a dependency in the spades easyconfig. 2016-04-22 19:29 GMT+02:00 Klima, Robert (NIH/NIAID) [C] < robert.kl...@nih.gov>: > Hi, > > My installation SPAdes-3.7.1-goolf-1.4.10.eb failed: > > == FAILED: Installation ended unsuccessfully (build directory: > /sysapps/cluster/build/SPAdes/3.7.1/goolf-1.4.10): build failed (first 300 > chars): cmd " cmake > /sysapps/cluster/build/SPAdes/3.7.1/goolf-1.4.10/SPAdes-3.7.1/src > -DCMAKE_INSTALL_PREFIX=/sysapps/cluster/software/SPAdes/3.7.1-goolf-1.4.10 > -DCMAKE_C_COMPILER='gcc' -DCMAKE_Fortran_FLAGS='-O2 -march=native' > -DCMAKE_CXX_FLAGS='-O2 -march=native' -DCMAKE_CXX_COMPILER='g++' > -DCMAKE_Fortran > == Results of the build can be found in the log file > /tmp/eb-KF29IH/easybuild-SPAdes-3.7.1-20160422.131913.RMypm.log > ERROR: Build of > /sysapps/cluster/software/EasyBuild/2.7.0/lib/python2.7/site-packages/easybuild_easyconfigs-2.7.0-py2.7.egg/easybuild/easyconfigs/s/SPAdes/SPAdes-3.7.1-goolf-1.4.10.eb > failed (err: 'build failed (first 300 chars): cmd " cmake > /sysapps/cluster/build/SPAdes/3.7.1/goolf-1.4.10/SPAdes-3.7.1/src > -DCMAKE_INSTALL_PREFIX=/sysapps/cluster/software/SPAdes/3.7.1-goolf-1.4.10 > -DCMAKE_C_COMPILER=\'gcc\' -DCMAKE_Fortran_FLAGS=\'-O2 -march=native\' > -DCMAKE_CXX_FLAGS=\'-O2 -march=native\' -DCMAKE_CXX_COMPILER=\'g++\' > -DCMAKE_Fortran') > > I am attaching log files. > > I hope you will help me to build SPAdes 3.7.1, > > Thank you, > Robert > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Using easybuild to manage different Python toolstacks
2016-03-18 15:11 GMT+01:00 Fotis Georgatos : > > ie. we might be able to stop most of the network activity in user-space, > w/out requiring blockage at root level (there are some caveats, agreed), > by rewiring certain calls. thoughts? > > Not sure if adding and external dependency to easybuild and using LD_PRELOAD is a good idea. I think Kenneth was looking at a solution using python but I don't remember the details. I think he should decide about this. > F. > > -- > > echo "sysadmin know better bash than english" | sed s/min/mins/ \ > | sed 's/better bash/bash better/' # Yelling in a CERN forum > > -- > *From:* easybuild-requ...@lists.ugent.be [easybuild-requ...@lists.ugent.be] > on behalf of Pablo Escobar Lopez [pablo.escobarlo...@unibas.ch] > *Sent:* Friday, March 18, 2016 14:50 > *To:* easybuild@lists.ugent.be > *Subject:* Re: [easybuild] Using easybuild to manage different Python > toolstacks > > as Alan suggests I think python bundle is the way to go for what you want. > > I would also suggest that if you want to assure reproducibility you should > test your bundles in a machine without internet access. Some python > libraries download extra dependencies during installation so if you run > your easyconfig in a machine with internet access you won't notice which > extra deps have been downloaded during the installation and if you run the > same easyconfig in the future maybe different versions of the dependencies > can be downloaded. If you try the easyconfig without internet access you > would get an error in case any extra dependency needs to be download so you > can check the error log and add the missing dependency to your bundle. The > only procedure I know now to address this issue is by try&error in a > machine without internet access. Some development is being done to address > this in easybuild in a cleaner way but nothing available in the main > easybuild release yet as far as I know. > > regards, > Pablo. > > 2016-03-18 14:42 GMT+01:00 hilb...@uni-bremen.de : > >> Thanks, Alan, >> >> I'll look into this following your numba example. >> >> I'm just wondering, shouldn't for example the numpy version be part of >> the toolchain definition? Libraries like pandas and scipy use the numpy >> C stuff, if I remember correctly, so that a pandas build with numpy 1.9 >> is different from a pandas build with numpy 1.10. Not sure if numba >> from your example should also suffer from this or not. >> >> Cheers, >> Andreas. >> >> >> >> Alan O'Cais writes: >> >> > Dear Andreas, >> > >> > I was actually going to look into this soon. What you are describing (I >> think) is a Python package bundle for the SciPy Stack< >> http://www.scipy.org/stackspec.html>. An example of what that might look >> like is >> > >> https://github.com/hpcugent/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/n/numba/numba-0.22.1-intel-2015b-Python-2.7.11.eb >> > The Python dependency is given by the version suffix and the blas >> dependency is provided by the toolchain itself (a toolchain is >> compiler+mpi+blas/lapack/fftw) >> > >> > You could work on this yourself based on that example and I'd be happy >> to help if you run into issues. >> > >> > Alan >> > >> > On 18 March 2016 at 14:20, > hilb...@uni-bremen.de>> wrote: >> > I recently stumbled across easybuild, it's an amazing piece of >> > software -- thanks for making this available! >> > >> > I'm wondering if there is any documentation / hints available for >> > managing different version of the Python scientific toolstack with >> > easybuild. I'm thinking of toolchains like >> > "py27-np19-sp016-pd017-openblas" -- you get my point. So basically I'm >> > looking at using easybuild as a replacement for virtualenvs / Anaconda >> > envs. >> > >> > Is there anything I need to consider for this use case? >> > >> > Cheers, >> > Andreas. >> > >> > -- >> > Dr. Andreas Hilboll >> > >> > Center for Marine Environmental Sciences (MARUM) >> > - AND - >> > Institute of Environmental Physics (IUP) >> > >> > University of Bremen >> > >> > U3145 >> > Otto-Hahn-Allee 1 >> > D-28359 Bremen >> > Germany >> > >> > +49(0)421 218 62133 (phone) >> > +49(0)421 218 98 62133 (fax) >> > http://www.iup.uni-bremen.de/~hilboll >> >> >> -- >> Dr. Andreas Hilboll >> >> Center for Marine Environmental Sciences (MARUM) >> - AND - >> Institute of Environmental Physics (IUP) >> >> University of Bremen >> >> U3145 >> Otto-Hahn-Allee 1 >> D-28359 Bremen >> Germany >> >> +49(0)421 218 62133 (phone) >> +49(0)421 218 98 62133 (fax) >> http://www.iup.uni-bremen.de/~hilboll >> > > > > -- > Pablo Escobar López > HPC systems engineer > sciCORE, University of Basel > SIB Swiss Institute of Bioinformatics > http://scicore.unibas.ch > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Using easybuild to manage different Python toolstacks
as Alan suggests I think python bundle is the way to go for what you want. I would also suggest that if you want to assure reproducibility you should test your bundles in a machine without internet access. Some python libraries download extra dependencies during installation so if you run your easyconfig in a machine with internet access you won't notice which extra deps have been downloaded during the installation and if you run the same easyconfig in the future maybe different versions of the dependencies can be downloaded. If you try the easyconfig without internet access you would get an error in case any extra dependency needs to be download so you can check the error log and add the missing dependency to your bundle. The only procedure I know now to address this issue is by try&error in a machine without internet access. Some development is being done to address this in easybuild in a cleaner way but nothing available in the main easybuild release yet as far as I know. regards, Pablo. 2016-03-18 14:42 GMT+01:00 hilb...@uni-bremen.de : > Thanks, Alan, > > I'll look into this following your numba example. > > I'm just wondering, shouldn't for example the numpy version be part of > the toolchain definition? Libraries like pandas and scipy use the numpy > C stuff, if I remember correctly, so that a pandas build with numpy 1.9 > is different from a pandas build with numpy 1.10. Not sure if numba > from your example should also suffer from this or not. > > Cheers, > Andreas. > > > > Alan O'Cais writes: > > > Dear Andreas, > > > > I was actually going to look into this soon. What you are describing (I > think) is a Python package bundle for the SciPy Stack< > http://www.scipy.org/stackspec.html>. An example of what that might look > like is > > > https://github.com/hpcugent/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/n/numba/numba-0.22.1-intel-2015b-Python-2.7.11.eb > > The Python dependency is given by the version suffix and the blas > dependency is provided by the toolchain itself (a toolchain is > compiler+mpi+blas/lapack/fftw) > > > > You could work on this yourself based on that example and I'd be happy > to help if you run into issues. > > > > Alan > > > > On 18 March 2016 at 14:20, hilb...@uni-bremen.de>> wrote: > > I recently stumbled across easybuild, it's an amazing piece of > > software -- thanks for making this available! > > > > I'm wondering if there is any documentation / hints available for > > managing different version of the Python scientific toolstack with > > easybuild. I'm thinking of toolchains like > > "py27-np19-sp016-pd017-openblas" -- you get my point. So basically I'm > > looking at using easybuild as a replacement for virtualenvs / Anaconda > > envs. > > > > Is there anything I need to consider for this use case? > > > > Cheers, > > Andreas. > > > > -- > > Dr. Andreas Hilboll > > > > Center for Marine Environmental Sciences (MARUM) > > - AND - > > Institute of Environmental Physics (IUP) > > > > University of Bremen > > > > U3145 > > Otto-Hahn-Allee 1 > > D-28359 Bremen > > Germany > > > > +49(0)421 218 62133 (phone) > > +49(0)421 218 98 62133 (fax) > > http://www.iup.uni-bremen.de/~hilboll > > > -- > Dr. Andreas Hilboll > > Center for Marine Environmental Sciences (MARUM) > - AND - > Institute of Environmental Physics (IUP) > > University of Bremen > > U3145 > Otto-Hahn-Allee 1 > D-28359 Bremen > Germany > > +49(0)421 218 62133 (phone) > +49(0)421 218 98 62133 (fax) > http://www.iup.uni-bremen.de/~hilboll > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] adding local variables to module files
I have this in some of my easyconfigs. It works for me with tcl modules, I
haven't tried with lua syntax. I think you could also use $HOME or $USER or
any other env var. It's not exactly what you ask of adding env vars which
are local to the module file but maybe it helps...
modextravars = {
'PYRO_LOOKUP_FILE': '\$EBROOTAMPLICONNOISE/Data/LookUp_E123.dat',
'SEQ_LOOKUP_FILE': '\$EBROOTAMPLICONNOISE/Data/Tran.dat',
}
2016-02-29 20:25 GMT+01:00 Glenn Johnson :
> Hi Kenneth,
>
> Yes, that helps. The example I gave was TCL but I did plan on using Lua
> for EB generated modules. I think the only environment variable that we
> ever need to grab is $USER, as in my example. Is there another (better?)
> way to accomplish this?
>
> Thanks.
>
>
> Glenn
>
> On Mon, Feb 29, 2016 at 1:12 PM, Kenneth Hoste
> wrote:
>
>> Hi Glenn,
>>
>> On 29/02/16 20:02, Glenn Johnson wrote:
>>
>> I am relatively new to Easybuild so I apologize if I overlooked
>> something. Is there a way to add extra variables that are local to
>> environment module files? That would be in addition to the 'root' variable
>> that is set. As an example, our (non-EB) Gaussian09 module uses the
>> following to set the user scratch directory:
>>
>> #%Module
>> ...
>> set user $env(USER)
>> ...
>> setenv GAUSS_SCRDIR /scratch/Users/$user
>>
>> I see the option to add extra environment variables:
>> modextravars
>>
>> How can I get the extra variables to use the value of a module local
>> variable?
>>
>>
>> The only way right now would be to use 'modtclfooter'.
>>
>> Since you get no control over the order, you'll probably need to jam the
>> 'setenv' in there too.
>>
>> modtclfooter = "set user $env(USER)\nsetenv GAUSS_SCRDIR
>> /scratch/Users/$user"
>>
>> Note that if you ever start generating module files in Lua syntax (to be
>> consumed by Lmod), this will simply be ignored; to be 'safe' you should add
>> the equivalent 'modluafooter' too.
>>
>> I guess we could add support for something like "modlocalvars", but since
>> you also want to grab values from the environment, it would have to be both
>> 'modlocalvarstcl' and 'modlocalvarslua', and I don't like where this is
>> going... :)
>>
>> Does this help?
>>
>>
>> regards,
>>
>> Kenneth
>>
>
>
--
Pablo Escobar López
HPC systems engineer
sciCORE, University of Basel
SIB Swiss Institute of Bioinformatics
http://scicore.unibas.ch
Re: [easybuild] Building a life sciences sub-community and citation practices
Hi Ben, To maintain different subsets of R and Python libraries I suggest you look at the "bundle" easyblock. This is how I am handling this now. Before I was using your approach of adding all the R libraries in the main R easyconfig but adding all the extensions in the main R easyconfig becomes hard to maintain. I had more than 400 R packages in my R easyconfig and upgrading this was a mess. Having bundles you can have smaller subsets of libraries and you can also add new bundles and keep the old ones. Here you have some examples https://github.com/hpcugent/easybuild-easyconfigs/blob/242a3e19308754796c7dd800d2f9f6f40f6c9e14/easybuild/easyconfigs/r/R-bundle-extra/R-bundle-extra-20150323-intel-2015a-R-3.1.3.eb https://github.com/hpcugent/easybuild-easyconfigs/blob/7787c2c75443ce222443489e4e8ff6b9dffd21d5/easybuild/easyconfigs/i/IPython/IPython-3.2.3-intel-2015b-Python-2.7.10.eb When managing R bundles which contain bioconductor packages there is a problem to distribute them because the bioconductor guys remove old versions from their repository so if you have a working bundle today it will fail downloading the required libraries few days later when the bioconductor mirror publish new versions and remove the old ones. Kenneth (EB release manager) tried to contact them and asked to keep the old versions online but they are not willing to do it. What galaxy developers have done to workaround this problem is building their own bioconductor mirror which also includes old versions ( https://bioarchive.galaxyproject.org/#/pkg/Biobase/). But this is not a complete mirror, it only includes the R packages used in galaxy ( https://galaxyproject.org/) . There has been discussion before in the EasyBuild community about maintaining a mirror with the source tarballs but nothing is in place yet. To be able to distribute R bundles I think something like this would be needed. If you are managing R installation with easybuild maybe you are interested in this https://github.com/fgeorgatos/easybuild.experimental/tree/master/users/pneerincx . It's not merged upstream but it works. The developer of this script ( https://github.com/pneerincx) is in this mailing list. What I do is to use this script to generate an R easyconfig and then I only use the extensions list section to create a R bundle. regards, Pablo. 2016-02-17 23:49 GMT+01:00 McGough, Benjamin E : > We have begun to use EasyBuild to replace our manual build process for > scientific software. As we have learned more and more about EasyBuild, a > couple of questions have come up. > > First, I am 95% positive I read a 'best practices' for citing > EasyBuild/EasyConfigs... but I cannot find it anywhere now. If such exists > in the documentation, please point me to it. > > I believe we will have need to cite the version of EasyBuild and the > specific easyconfig used to build software involved in published research. > Would it be wise to cite github URLs for EasyBuild and easyconfigs? We may > also have a need to cite easyconfigs that have not been accepted into the > easyconfigs repository, but would like to use a generic URL not associated > with our institution. Is there a common practice for this? > > Second, we have begun to build R and Python with an expanded set of > included extensions (libraries/modules). We intend to keep this up for new > versions of R and Python, including version updates for the extensions and > inclusion of new extensions by request and dependency. My initial feeling > is that this might be useful to others, and I had planned to submit these > easyconfigs with a versionsuffix of something like 'life-sciences' to > indicate the extensions selection is primarily life science-driven. Now I > am not sure that is the right way to proceed. Is there a common practice I > should read? We also have a python script using APIs from CRAN and PyPI to > update the easyconfigs. Is that something that would be a useful tool to > others? > > Thanks! > > Ben McGough > System Administrator > Center IT/Scientific Computing > O 206.667.7818 > bmcgo...@fredhutch.org > > 1100 Fairview Ave. N. > P.O. Box 19024 > Seattle, WA 98109 > > Fred Hutch / Cures Start Here > fredhutch.org > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Python packages?
HI Elizabeth, There are different options to handle python packages in EB. You can use the "PythonPackage" easyblock or the "Bundle" easyblock. Personally I use PythonPackage for python apps without extra deps and Bundle for apps with many deps but this is quite subjective. Some examples: https://github.com/hpcugent/easybuild-easyconfigs/blob/242a3e19308754796c7dd800d2f9f6f40f6c9e14/easybuild/easyconfigs/c/cutadapt/cutadapt-1.5-intel-2014b-Python-2.7.8.eb https://github.com/hpcugent/easybuild-easyconfigs/blob/242a3e19308754796c7dd800d2f9f6f40f6c9e14/easybuild/easyconfigs/h/HTSeq/HTSeq-0.5.4p5-goolf-1.4.10-Python-2.7.6.eb https://github.com/hpcugent/easybuild-easyconfigs/blob/242a3e19308754796c7dd800d2f9f6f40f6c9e14/easybuild/easyconfigs/g/GC3Pie/GC3Pie-2.4.2.eb https://github.com/hpcugent/easybuild-easyconfigs/pull/2457 If you want to be able to reproduce your build I recommend you to run EB in machine without internet access. Many python packages will download extra dependencies when running "python setup.py build" so to be sure you include all the deps in the easyconfigs you need to do some try&error in a machine without internet access so the automatic download fails, then you can check the error log and add the missing dependencies to the easyconfig btw, for next easybuild release we will have %(pyver)s and %(pyshorver)s templates available for the easyconfigs https://github.com/hpcugent/easybuild-framework/pull/1595 regards, Pablo. -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch 2016-02-08 22:50 GMT+01:00 Elizabeth Fischer : > Hello, > > Is there any documentation on how EasyBuild interacts with Python > packages? I see there's an EasyBuild for matplotlib, but not for basemap. > What is the recommended way to go about installing stuff? Pure EasyBuild? > EasyBuild + PIP? EasyBuild + Conda? > > I'm really confused... > > Thanks, > -- Elizabeth > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
[easybuild] EasyBuild Workshop, March 2, Basel
Can I send some spam to the mailing list? :) As part of the eSCT training program, sciCORE, the scientific computing center at University of Basel is organizing a workshop on EasyBuild. The aim of the workshop is to learn, share information about and practice the EasyBuild software management tool, as a key component in the management of scientific HPC infrastructures. This is an advanced workshop, aimed at system administrators, user support teams, or anyone managing scientific software installation in HPC clusters. Even though we will touch advanced topics, it is not necessary previous knowledge; an experienced attendant who has not tried EasyBuild before should be able to follow the course without problems. Location This workshop will take place on March 2 2016, 9:00-16:00 at Klingelbergstrasse 61, Floor 13th, 4056 Basel, Switzerland Registration The workshop has a small fee of CHF 50 to cover for logistics, and is open to anyone interested. There is a limit of 20 participants due to space availability. Attendants are encouraged to bring their own laptops. For registration please follow the link http://www.isb-sib.ch/edu/Registration/SIB_courses_app.php?id=352 Program The workshop will have a practical approach, with a short first part for introduction and theory, and a second one for the hands-on session where more advanced topics will be touched. The main topics will be: General introduction: managing the software stack, understanding and writing EasyConfigs Hands-on practice Enhancing/extending EasyBuild Integration with RPM system Please feel free to forward this invitation to anyone interested. -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] Warning from EasyBuild 2.6.0
this is what I get [escobar@login18 easybuild-easyconfigs]$ eb --help|head /usr/lib/python2.6/site-packages/keyring/backend.py:16: UserWarning: Module easybuild was already imported from /scicore/soft/apps/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_framework-2.6.0-py2.6.egg/easybuild/__init__.pyc, but /scicore/soft/apps/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_easyblocks-2.6.0-py2.6.egg is being added to sys.path import pkg_resources 2016-02-05 11:11 GMT+01:00 Kenneth Hoste : > Hi Pablo, > > Do you get both "Module vsc was already imported" and "Module easybuild > was already imported" warnings on this system? > > > K. > > > On 05/02/16 11:05, Pablo Escobar Lopez wrote: > > in case it helps, I have noticed that I only get these warnings in the > machine were i have python-keyring installed. > > this is my output for the command that Kenneth asks (identical is the > machine which triggers the warning and in the machine which doesn't trigger > the warning) > > [escobar@login18 easybuild-easyconfigs]$ python -c 'import sys;print > sys.version;import vsc;print vsc.__path__;import easybuild;print > easybuild.__path__' > 2.6.6 (r266:84292, Jan 22 2014, 09:42:36) > [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] > > ['/scicore/soft/apps/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_base-2.2.4-py2.6.egg/vsc'] > ['/scicore/soft/apps/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_framework-2.6.0-py2.6.egg/easybuild', > '/scicore/soft/apps/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_easyblocks-2.6.0-py2.6.egg/easybuild'] > > 2016-02-05 10:06 GMT+01:00 Kenneth Hoste : > >> Hi Judith, >> >> We're looking into a solution for these warnings, both in vsc-base and >> EasyBuild, see also >> https://github.com/hpcugent/easybuild-framework/issues/1588 >> >> Can you provide us the output of the following command? >> >> python -c 'import sys;print sys.version;import vsc;print >> vsc.__path__;import easybuild;print easybuild.__path__' >> >> >> regards, >> >> Kenneth >> >> >> On 03/02/16 19:13, Kenneth Hoste wrote: >> >> Hi Judith, >> >> On 28/01/16 22:37, Gardiner, Judith wrote: >> >> [ruby01]$ eb --version >> >> /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_base-2.4.16-py2.6.egg/vsc/__init__.py:29: >> UserWarning: Module vsc was already imported from >> /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_base-2.4.16-py2.6.egg/vsc/__init__.pyc, >> but >> /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_install-0.9.12-py2.6.egg >> is being added to sys.path >> >> import pkg_resources >> >> /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_base-2.4.16-py2.6.egg/vsc/__init__.py:29: >> UserWarning: Module easybuild was already imported from >> /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_framework-2.6.0-py2.6.egg/easybuild/__init__.pyc, >> but >> /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_easyblocks-2.6.0-py2.6.egg >> is being added to sys.path >> >> import pkg_resources >> >> This is EasyBuild 2.6.0 (framework: 2.6.0, easyblocks: 2.6.0) on host >> ruby01.osc.edu. >> >> >> >> First of all, these warnings are harmless in the sense that everything >> will work even though they are being spit out. >> I agree they are ugly though, and we should try and get rid of them. >> >> The problem is that vsc-base was split up recently, into what vsc-base is >> now and vsc-install, which provides support for installing/testing vsc-* >> packages. >> >> This implies that the 'vsc' namespace is spread across two different >> directories now, and that "import vsc" in Python will result in reading >> vsc/__init__.py from two different places. Python (or setuptools?) thinks >> this is weird, and throws the warnings. >> >> Same applies to the 'easybuild' namespace, which is imported from two >> locations (framework & easyblocks). >> >> This is a well known issue, and there are workaround to just silence the >> warning [1,2], but that's probably not the best way to 'fix' this issue. >> >> We're actually getting it too on our systems, but not consistently, it >> depends on how you install EasyBuild (an I *think* also on the version of >>
Re: [easybuild] Warning from EasyBuild 2.6.0
in case it helps, I have noticed that I only get these warnings in the machine were i have python-keyring installed. this is my output for the command that Kenneth asks (identical is the machine which triggers the warning and in the machine which doesn't trigger the warning) [escobar@login18 easybuild-easyconfigs]$ python -c 'import sys;print sys.version;import vsc;print vsc.__path__;import easybuild;print easybuild.__path__' 2.6.6 (r266:84292, Jan 22 2014, 09:42:36) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] ['/scicore/soft/apps/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_base-2.2.4-py2.6.egg/vsc'] ['/scicore/soft/apps/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_framework-2.6.0-py2.6.egg/easybuild', '/scicore/soft/apps/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_easyblocks-2.6.0-py2.6.egg/easybuild'] 2016-02-05 10:06 GMT+01:00 Kenneth Hoste : > Hi Judith, > > We're looking into a solution for these warnings, both in vsc-base and > EasyBuild, see also > https://github.com/hpcugent/easybuild-framework/issues/1588 > > Can you provide us the output of the following command? > > python -c 'import sys;print sys.version;import vsc;print > vsc.__path__;import easybuild;print easybuild.__path__' > > > regards, > > Kenneth > > > On 03/02/16 19:13, Kenneth Hoste wrote: > > Hi Judith, > > On 28/01/16 22:37, Gardiner, Judith wrote: > > [ruby01]$ eb --version > > /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_base-2.4.16-py2.6.egg/vsc/__init__.py:29: > UserWarning: Module vsc was already imported from > /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_base-2.4.16-py2.6.egg/vsc/__init__.pyc, > but > /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_install-0.9.12-py2.6.egg > is being added to sys.path > > import pkg_resources > > /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/vsc_base-2.4.16-py2.6.egg/vsc/__init__.py:29: > UserWarning: Module easybuild was already imported from > /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_framework-2.6.0-py2.6.egg/easybuild/__init__.pyc, > but > /usr/local/easybuild/2.6.0/software/EasyBuild/2.6.0/lib/python2.6/site-packages/easybuild_easyblocks-2.6.0-py2.6.egg > is being added to sys.path > > import pkg_resources > > This is EasyBuild 2.6.0 (framework: 2.6.0, easyblocks: 2.6.0) on host > ruby01.osc.edu. > > > > First of all, these warnings are harmless in the sense that everything > will work even though they are being spit out. > I agree they are ugly though, and we should try and get rid of them. > > The problem is that vsc-base was split up recently, into what vsc-base is > now and vsc-install, which provides support for installing/testing vsc-* > packages. > > This implies that the 'vsc' namespace is spread across two different > directories now, and that "import vsc" in Python will result in reading > vsc/__init__.py from two different places. Python (or setuptools?) thinks > this is weird, and throws the warnings. > > Same applies to the 'easybuild' namespace, which is imported from two > locations (framework & easyblocks). > > This is a well known issue, and there are workaround to just silence the > warning [1,2], but that's probably not the best way to 'fix' this issue. > > We're actually getting it too on our systems, but not consistently, it > depends on how you install EasyBuild (an I *think* also on the version of > setuptools, but I'm not sure about that). > > I've opened an issue on this in the EasyBuild framework repository on > GitHub [3], and tagged it to try and fix this by the next EasyBuild release. > > > regards, > > Kenneth > > [1] > http://stackoverflow.com/questions/7239518/module-pytz-was-already-imported > [2] > http://stackoverflow.com/questions/3861336/how-do-you-correct-module-already-loaded-userwarnings-in-python > [3] https://github.com/hpcugent/easybuild-framework/issues/1588 > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] ffmpeg versus FFmpeg
to workaround this "issue" I install Lmod with "./configure --with-caseIndependentSorting=yes" ( https://github.com/TACC/Lmod/blob/master/configure#L1317) and I always use the tab-completion in Lmod 2016-01-18 22:59 GMT+01:00 Riccardo Murri : > (Elizabeth Fischer, Mon, Jan 18, 2016 at 04:02:43PM -0500:) > > It is my opinion that all package names should be lower case. Or > > case-insensitive. Having mixed-case case-dependent names just adds > > possibility for error without any gain. > > +1 > > Ciao, > R > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] foss/2016a and intel/2016a common toolchains
in my opinion I would not include gcc5 in a toolchain which should be considered as "stable". Many developers haven't tried gcc5 yet so I think we would hit many new issues when trying to compile applications which have never been compiled with gcc5. For me in the bioinformatics field is a totally no go. I would stay by now with gcc4 in the foss toolchain and provide gcc5 as a different toolchain ( maybe foss/2016.02 ?) for those who need it or want to test it. Anyway I am even more conservative and I am now planning my upgrade from goolf/1.4.10 to goolf/1.7.20 so feel free to ignore my comments if you enjoy being in the bleeding edge :P my two cents. 2016-01-14 1:18 GMT+01:00 Fotis Georgatos : > > Hi, > > I would side closer to Damian and Kenneth, ie. play a bit conservative, as > regards the common gcc compiler version. > IMHO, the difference in community opinion among GCC/4* or GCC/5 relates > more to if you are *supporter* or *supported*. > Let’s remind here why easyconfigs are called “examples” as opposed to > “solutions” for HPC systems! > > When you deal with many users, you need to provide an environment suitable > for multiple domains, exploiting your limited/finite time. > Compatibility issues among multiple tools such as Intel, CUDA, Matlab, PGI > and open source components, inevitably imply > a lesser common denominator. Shop your arguments from the URLs here: > https://github.com/hpcugent/easybuild-easyconfigs/pull/1294 > > fi. > If you are into Life Sciences and you do narrow MD experiments, you have > good reasons to stretch versions to greatest and latest; > however, if you require a compatibility layer with the bioinformaticians > next door, that makes it a tad more constrained. > > > From my point of view, the following dep chain must be fully supported by > upstream paths or, the generic toolchains are failing: > (Python, Boost, Perl) -> Intel compilers -> GCC ; allowing CUDA in the mix > is a bonus, for Molecular Dynamics intentions. > There is loose affinity here with Common Deps list: > https://github.com/hpcugent/easybuild-easyconfigs/issues/1689 > > My own litmus test often is a variant of > HPCBIOS_LifeSciences-20130829-goolf-1.4.10.eb & a Perl that keeps other > builds happy; > an equivalent symmetric set of builds against some recent Intel toolchain, > would be convincing: it is centrally supportable and, > sufficiently advanced for further experimentation on the user-end (we > should never claim fitness, rather a working skeleton!). > > F. > > On Jan 13, 2016, at 10:17 AM, Damian Alvarez > wrote: > > Another thing to consider might be CUDA. CUDA 7.5 requires GCC >4.3.4 && > <4.9.2 (even though with 4.9.3 works fine). It seems you can make it work > (at your own risk) with 5.X as long as you don't use C++11. However, that > means that CUDA+C++11 developers will have troubles (i.e.: won't work) in > EB-managed environments if we go for GCC 5.X. > > > > PGI-based toolchains might also require GCC <5.X, but I am not sure > about that, the documentation is not clear on that regard. > > > > I'd suggest to at least keep GCCcore as 4.9.3. With just GCCcore being > 4.9.3, the foss toolchain (with GCC 5.X) will be problematic for CUDA > usage, but not so the Intel toolchain. > > -- > echo "sysadmin know better bash than english" | sed s/min/mins/ \ > | sed 's/better bash/bash better/' # signal detected in a CERN forum > > > > > > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] custom glibc
I am also curious to know if someone has tried to use a custom glibc. 2015-12-12 11:29 GMT+01:00 Kenneth Hoste : > Hi Olaf, > > On 13/11/15 10:32, Olaf Walter wrote: > > Dear easybuilders, > > > > has anybody ever attempted to do an easybuild with a custom glibc? > > > > The GCC HOWTO document has some basic hints, and I thought this could be > handled by a custom easyconfig for gcc. > > <http://www.tldp.org/HOWTO/Glibc2-HOWTO-6.html> > http://www.tldp.org/HOWTO/Glibc2-HOWTO-6.html > > > > I would expect major incompatibilites with the rest of the distro. Does > anybody have experiences with that? > > > Since there has been no answer on this, I guess the answer is no. > > Why would you need a custom glibc? > > It seems like a good way to cause lots of problems to me... > > > regards, > > Kenneth > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
Re: [easybuild] New to EasyBuild - problem installing
Hola Oscar, installing scientific software as root is a non recommended practice that's why easybuild triggers an error and asks you not to do it. For testing you can use your standard account. What many easybuild users do once they use easybuild in production is having a dedicated user account in the cluster which is used to manage the easybuild installations. The error about "Name or service not known" coming from the urllib2 library I guess it's a problem with your dns. The bootstrap script will try to connect to the internet and download all the required software to install easybuild. I guess the machine where you are executing easybuild cannot resolve some of the domains where the bootstrap script is trying to connect to. I would review your dns and network configuration. saludos, Pablo. 2015-10-21 16:03 GMT+02:00 o...@uo.edu.cu : > Please let me introduce myself: > > My name is Oscar Au-Alvarez. I am the admin of the recently installed HPC > at > the University of Oriente (eastern Cuba). I heard of EasyBuild from Dieter > Roefs who came to my university in a working visitit within VLIRUOS. With > Dieter I realized that EasyBuild is what I need as I have had many > headaches > installing, compiling some programs. So I tried to install EasyBuild but > have > had problems with the installation. > > I have 2 questions: > > 1- I tried to install EasyBuld bootstrapping it as root, and I got the > following message: " Don't run the EasyBuild bootstrap script as root, > since > stage 2 (installing EasyBuild with EasyBuild) will fail"l. > My question is: Why root cannot bootstrap EasyBuild? > > 2-I tried to install EasyBuild as non-root user with "python > bootstrap_eb.py > $HOME/.local", and it starts STAGE 0 but aborts with the following error > message: "urllib2.URLError: known>" > My question: How do I solve this? > > > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://scicore.unibas.ch
Re: [easybuild] difference between `--filter-deps` and `--hide-deps` ?
Filter-deps will skip the build (dependency should be provided by os packages). Hide-deps will build the package but the module will be hidden (module can be loaded but not seen when doing module available) More details here http://easybuild.readthedocs.org/en/latest/Manipulating_dependencies.html Pablo. Sent from my mobile. On Jul 31, 2015 14:35, "Riccardo Murri" wrote: > Hello, > > in trying to solve the issue with libreadline being recompiled in a way > that's incompatible with the system provided one, I noticed these two > options in EB:: > > --filter-deps=FILTER-DEPS > Comma separated list of dependencies that you DON'T > want to install with EasyBuild, because equivalent > OS > packages are installed. (e.g. --filter- > deps=zlib,ncurses) (type comma-separated list) > > --hide-deps=HIDE-DEPS > Comma separated list of dependencies that you want > automatically hidden, (e.g. > --hide-deps=zlib,ncurses) > (type comma-separated list) > > What's exactly the difference between the two? Does "filter" consider > the dependency satisfied (hence, skip building), whereas "hide" still > builds it but does not provide a module file? > > Thanks, > R > > -- > Riccardo Murri > http://www.s3it.uzh.ch/about/team/#Riccardo.Murri > > S3IT: Services and Support for Science IT > University of Zurich > Winterthurerstrasse 190, CH-8057 Zürich (Switzerland) > > Tel: +41 44 635 4222 > Fax: +41 44 635 6888 >
Re: [easybuild] ISC15???
2015-07-15 17:24 GMT+02:00 Kenneth Hoste : > > I don't care much for German beer though. ;-) > > > hahahahah Jack be aware you should never ever start the beer flame with a belgian guy. It's much worse than the vim vs emacs discussion..even if everybody knows that the combination of a really cold spanish beer and a good "tapa" is the best :P -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] Incompatibilities with different CPUs
Hi Martin, openblas will ignore the optarch option in easybuild but you can use something like this in the OpenBLAS easyconfig buildopts = 'TARGET=SANDYBRIDGE BINARY=64 ' + threading + ' CC="$CC" FC="$F77"' here you have the list of supported targets in openblas https://github.com/xianyi/OpenBLAS/blob/develop/TargetList.txt If you want to share the same openblas installation across all your different cpu types you will need to use the target which match your oldest cpu. Another approach is to maintain a different software stack by cpu type as described in the slides I attach (page 9) regards, Pablo. 2015-07-14 13:21 GMT+02:00 Martin : > Hi, > > I have a couple of different CPUs and in my situation EB is not so much > about a HPC site or repeatable build but rather about managing the stuff. > So here's the thing: > > When using R/3.2.0-foss-2015a-bare (just a simply --try-software-version > easyconfig) and subsequently installing a few R packages will work on one > host but fail on another. > > The error is a classical illegal instruction in the underlying OpenBLAS > library (I could reproduce this specific problem > > The problem is that OpenBLAS/0.2.13-GCC-4.9.2-LAPACK-3.5.0 seems to > using some magic so that is is processor dependent. The processors involved > are: > > * AMD Opteron(TM) Processor 6238 and > * Six-Core AMD Opteron(tm) Processor 8431 > > There are also a few Intel processors and a few other AMD CPUs here. > What I'm looking for is a setting for easybuild that optimizes for > portability between compatible CPU architectures. > > Does anyone have suggestions how to achieve the most compatibility and > care less about byte-level reproducability or performance? The target > platform I have in mind is "64bit x86 compatible". > > I have optarch="" in my config.cfg where I thought that would already > take care of it. > > Any thoughts would be appreciated. > > /Martin > -- > -- > http://www.xing.com/profile/Martin_Marcher > http://www.linkedin.com/in/martinmarcher > Mobil: +43 / 660 / 62 45 103 > UID: ATU68801424 > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch hpc-ch-slides.pdf Description: Adobe PDF document
Re: [easybuild] EasyBuild Newbie questions
Hola Armando, That command line will install only the framework but you need to install also the easyblocks and easyconfigs to have a fully working easybuild setup. These are the three things you need to get installed (plus a modules system like Lmod) https://github.com/hpcugent/easybuild-framework https://github.com/hpcugent/easybuild-easyblocks https://github.com/hpcugent/easybuild-easyconfigs The recommended installation method is described in the documentation. I suggest you to take a look at it https://easybuild.readthedocs.org/en/latest/Installation.html un saludo, Pablo. 2015-06-29 20:14 GMT+02:00 Armando Fandango : > Hi > > I have installed easybuild using the following command option: > > pip install > http://github.com/hpcugent/easybuild-framework/archive/master.tar.gz > > However, now I do not have eb command anywhere on my system. > > I searched but could not find anything on help file regarding this method > of installation. > > > Please advise. > > Regards > > -- armando > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] Easier way of installing R packages
ok, thanks for your effort Pieter 2015-06-29 16:25 GMT+02:00 Pieter Neerincx : > > On Jun 29, 2015, at 3:37 PM, Pablo Escobar Lopez < > pablo.escobarlo...@unibas.ch> wrote: > > > > > > > 2015-06-24 10:58 GMT+02:00 Pieter Neerincx : > > > > Yesterday I started working on an R-script that generates an *.eb from > an existing R installation. This is based on Stefano's code (Many thanks > for sharing that!) and adds some commandline switches, help and code to > figure out whether a package came from a CRAN repo or from BioConductor. > Hence this is slightly different compared to the above as it generates a > complete *.eb from pure R, but I can easily add an option to only generate > a pkglist file, which may then be used by an existing *.eb. > > > > Is it Ok if I add this with a README to the /r/R/ dir in the > EasyConfigs repo and create a pull request or would you prefer this kind of > supplementary code to go elsewhere? > > > > > > > > is this code published somewhere? > > As soon as I solved the last bugs I'll publish and report back to this > list... > > Cheers, > > Pi > > > thanks > > Pablo. > > > > > > > > -- > > Pablo Escobar López > > HPC systems engineer > > Biozentrum, University of Basel > > Swiss Institute of Bioinformatics SIB > > Email: pablo.escobarlo...@unibas.ch > > Phone: +41 61 267 21 80 > > http://www.biozentrum.unibas.ch > > ----- > phone: +31 6 143 66 783 > e-mail: pieter.neeri...@gmail.com > skype: pieter.online > - > > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] Easier way of installing R packages
2015-06-24 10:58 GMT+02:00 Pieter Neerincx : > > Yesterday I started working on an R-script that generates an *.eb from an > existing R installation. This is based on Stefano's code (Many thanks for > sharing that!) and adds some commandline switches, help and code to figure > out whether a package came from a CRAN repo or from BioConductor. Hence > this is slightly different compared to the above as it generates a complete > *.eb from pure R, but I can easily add an option to only generate a pkglist > file, which may then be used by an existing *.eb. > > Is it Ok if I add this with a README to the /r/R/ dir in the > EasyConfigs repo and create a pull request or would you prefer this kind of > supplementary code to go elsewhere? > > is this code published somewhere? thanks Pablo. -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] Binary of a python package
Hi Oliver,
I tried to install that software doing "python setup.py install --user" and
I cannot find anything in ~/.local/bin . I only get files
in ~/.local/lib/python2.7/site-packages/ which is what easybuild is doing
and then setting up PYTHONPATH in the modulefile, so when I load the misopy
module I can import misopy in the python interpreter
[soft@login18 eb-test-installs]$ module load misopy
[soft@login18 eb-test-installs]$ python
Python 2.7.5 (default, Jun 20 2015, 21:57:09)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import misopy
>>>
are you sure that misopy/0.5.3 is adding something to ~/.local/bin? At
least in my system that's not the case
I think what Niek suggests won't work for the PythonPackage easyblock. You
can check all the available options in the PythonPackage easyblock by
running "eb -a -e PythonPackage" and you can verify that those options are
not avaiable by doing "eb -a -e PythonPackage| egrep
'executables|files_to_copy'". The option "files_to_copy" is only available
in the MakeCp easyblock.
$> eb -a -e MakeCp| egrep "executables|files_to_copy"
files_to_copy* List of files or dirs to copy [default: []]
regards,
Pablo.
2015-06-23 19:11 GMT+02:00 Niek de Klein :
> HI Oliver,
>
> You can try the parameter executables and files_to_copy, like so:
>
> executables = ['binary_file_name']
> files_to_copy = [(executables, 'bin')]
>
> Cheers,
> Niek
>
>
> On Tue, Jun 23, 2015 at 5:29 PM, Stolpe, Oliver
> wrote:
> > Hello list,
> >
> > I wrote an easyconfig file to install a python package. The installation
> is
> > successful besides the fact that easybuild throws away the binary that is
> > generated (or main entrance point for the standalone). When I install the
> > software on my machine with python setup.py build/install --user, it puts
> > the binary to ~/.local/bin, but I don't know where easybuild puts this
> file.
> > When I look into the easybuild/software/MYSOFTWARE directory, there is no
> > bin folder.
> >
> > Please help me save the binary :-)
> >
> > Thanks,
> > Oliver
> >
> > The easyconfig file:
> >
> > easyblock = 'PythonPackage'
> >
> > name = 'misopy'
> > version = '0.5.3'
> >
> > homepage = 'http://genes.mit.edu/burgelab/miso/index.html'
> > description = """MISO (Mixture of Isoforms) is a probabilistic framework
> > that
> > quantitates the expression level of alternatively spliced genes from
> > RNA-Seq
> > data, and identifies differentially regulated isoforms or exons across
> > samples.
> > By modeling the generative process by which reads are produced from
> > isoforms in
> > RNA-Seq, the MISO model uses Bayesian inference to compute the
> probability
> > that
> > a read originated from a particular isoform."""
> >
> > toolchain = {'name': 'goolf', 'version': '1.4.10'}
> >
> > source_urls = ['http://pypi.python.org/packages/source/m/misopy']
> > sources = ['%(namelower)s-%(version)s.tar.gz']
> >
> > python = 'Python'
> > pythonver = '2.7.6'
> > versionsuffix = '-%s-%s' % (python, pythonver)
> >
> > dependencies = [
> > ('matplotlib', '1.3.1', versionsuffix),
> > (python, pythonver),
> > ]
> >
> > moduleclass = 'bio'
> >
>
--
Pablo Escobar López
HPC systems engineer
Biozentrum, University of Basel
Swiss Institute of Bioinformatics SIB
Email: pablo.escobarlo...@unibas.ch
Phone: +41 61 267 21 80
http://www.biozentrum.unibas.ch
Re: [easybuild] Easier way of installing R packages
Hi Niek, I think many people in this list has suffer the R dependencies nightmare. I wrote an small scirpt which if you already have a R easyconfig will try to give you which are the latest versions available in R repos. It's not perfect but it helps a little bit at least. Here you have it in case it's helpfful for you https://github.com/pescobar/random-scripts/blob/master/easybuild-update-R-libs.py regards Pablo. 2015-06-18 16:38 GMT+02:00 Jack Perdue : > Howdy Niek, > > We have a non-Easybuild module tree. The attachment > from before lives in /software/tamusc/modulefiles/R. > It loads the EB R and then sets the needed env vars > for building extensions. > > Note that I don't maintain the R extensions so don't > have details. Here are the scripts written by my colleague > Maikel Pennings for such things. You'd have to ask him to > explain them. > > Jack Perdue > Lead Systems Administrator > High Performance Research Computing > TAMU Division of Research > j-per...@tamu.eduhttp://sc.tamu.edu > SC Helpdesk: h...@sc.tamu.edu > > On 06/18/2015 09:27 AM, Niek de Klein wrote: > > Hi Jack, > > > > Thank you for your answer. I don't understand the attached file. How > > do you use that to install extensions, and how do you keep it so that > > if you want to migrate to another cluster you can install R with the > > exact same R libraries installed? Do you keep a separate file with all > > the R packages you installed extra? > > > > Thanks, > > Niek > > > > On Thu, Jun 18, 2015 at 4:20 PM, Jack Perdue wrote: > >> Howdy Niek, > >> > >> FWIW, here is our latest build of R (with lots of > >> dependencies added): > >> > >> > http://www.siliconslick.com/easybuild/ebfiles_repo_cleaned/ada/R/R-3.2.0-intel-2015B-default-mt.eb > >> > >> For extensions, we have another (non-EB) module (attached) > >> for installing extensions to the above by hand. > >> > >> Between the two, we've: > >> > >> a) tried to make sure we aren't using system libs (e.g. TCL) > >> b) are able to satisfy user requests for misc. R extensions > >> (automagically) > >> > >> Jack Perdue > >> Lead Systems Administrator > >> High Performance Research Computing > >> TAMU Division of Research > >> j-per...@tamu.eduhttp://sc.tamu.edu > >> SC Helpdesk: h...@sc.tamu.edu > >> > >> > >> On 06/18/2015 08:57 AM, Niek de Klein wrote: > >>> Hi mailinglist, > >>> > >>> I'm probably missing something here and unnecessarily complicating > >>> things. I am trying to get R installed with just a few packages we > >>> use. One of the packages, ggplot2, has a lot of dependencies. So when > >>> I try to install R with one of the existing EasyBuild files and have > >>> ggplot2 as the dependency it will throw an error due to the > >>> dependencies of ggplot2 not being installed, and I have to add the > >>> dependencies to the .eb file. And then add the dependencies of the > >>> dependencies to the .eb file, etc etc. > >>> > >>> When installing libraries from within R you can simply give the option > >>> "dependency=True" and it will automagically install all dependencies > >>> for you. Is there an option like that when installing R and some R > >>> libraries? > >>> > >>> Thanks, > >>> Niek > >> > > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] updating mathematica easyblock. how to generate the "sources" file?
Hi Fotis, that one looks a good solution. one question.why not adding p7zip as a build dependency? this way you could avoid extra workarounds like the start_dir = '..' isn't it? Pablo. 2015-04-14 10:35 GMT+02:00 Fotis Georgatos : > > Good morning Pablo, > > I’ve also been there; a “less pure” way to do this is to rely on p7zip, > such as: > > https://github.com/fgeorgatos/easybuild.experimental/blob/master/users/fgeorgatos/my_easyblocks/easybuild/easyblocks/MATLAB-2013a.eb > > The reason this is not “pure" is that we don’t yet have anything near > “extractdeps”; > ie. deps only become visible at build time rather than extract time. > The workaround is to consider p7zip as yet another source. > > To be precice, I am far more emphatic for removing the human hand from the > build processes > rather than looking for perfect purity, so for me the above has always > been the one and only way > to deliver anything that requires .iso extraction (as opposed to manually > creating a tarball). > > hope this helps in some way, > > F. > > On Apr 14, 2015, at 9:12 AM, Pablo Escobar Lopez < > pablo.escobarlo...@unibas.ch> wrote: > > I need to install Mathematica 10.0.2 which is not supported in easybuild > so I will try to update the available easyblock and I have a doubt. > > > > I have a Mathematica iso which contains the installer. As far as I know > Easybuild doesn't support reading .iso files and I am not sure which is the > recommended approach to deal with .iso files. > > > > which is the "easybuild way" to do this? should I create a .tar.gz > "manually" with all the content of the .iso and comment in the easyconfig > how the source file has been created? Or is there any other recommended > approach? I am asking before starting tweaking the current easyblock to be > sure I do it "the easybuild way" so it can be merged upstream in the > future..if I manage to get it working :) > > > > cheers, > Fotis > > > -- > echo "sysadmin know better bash than english" | sed s/min/mins/ \ > | sed 's/better bash/bash better/' # signal detected in a CERN forum > > > > > > > > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
[easybuild] updating mathematica easyblock. how to generate the "sources" file?
Hi, I need to install Mathematica 10.0.2 which is not supported in easybuild so I will try to update the available easyblock and I have a doubt. I have a Mathematica iso which contains the installer. As far as I know Easybuild doesn't support reading .iso files and I am not sure which is the recommended approach to deal with .iso files. which is the "easybuild way" to do this? should I create a .tar.gz "manually" with all the content of the .iso and comment in the easyconfig how the source file has been created? Or is there any other recommended approach? I am asking before starting tweaking the current easyblock to be sure I do it "the easybuild way" so it can be merged upstream in the future..if I manage to get it working :) regards, Pablo. -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] How to deal with dependencies for R
Hi Martin, R extensions change quite often and the url also changes so the normal approach is "manual fix". In case it's useful for you I wrote this script some days ago to deal with this, it's not perfect and still requires some manual work but it works for ~90% of the R libraries so it saves you some time https://github.com/pescobar/random-scripts/blob/master/easybuild-update-R-libs.py To install extra R libraries you can just add the new library to the exts_list and then rerun the easyconfig with "eb R-3.1.2.eb -k -f" This way only libraries not already installed will be compiled. This is commonly used to add extra libraries for R/Perl/Python. This is the easybuild help: -k, --skip Skip existing software (useful for installing additional packages) (def False) regards, pablo. 2015-03-31 17:41 GMT+02:00 Martin : > Hi, > > I'm trying to install R-3.1.2-goolf-1.5.14.eb. > > I'm running a simple: > > eb R-3.1.2-goolf-1.5.14.eb --robot -s fetch > > but it fails on most of the packages in exts_list. How are people > dealing with this? > > Are you just going thru the list and manually fix the versions, I keep > thinking there has to be a better way. > > Also how do you (later) install an additional package? Extend the list > in exts_list and "reinstall" or provide another module? Are there > established best practices for this? > > thanks, > Martin > > > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] ? HOWTO: install MACS, a python package.
Hi Malcom, some time ago I sent a pull request for MACS https://github.com/hpcugent/easybuild-easyconfigs/pull/1247 I suggest you to review the list of pending pull requests in github before you start writting new easyconfigs https://github.com/hpcugent/easybuild-easyconfigs/pulls regards, Pablo. 2015-03-31 4:30 GMT+02:00 Malcolm Cook : > pip install macs2 works fine to install MACS2 without easyBuild (c.f. > https://github.com/taoliu/MACS/blob/master/INSTALL.rst) > > I have only ever done this without virtualenv, thus, my macs2 installs > have always been direct into the system wide python libraries. > > “For shame”, you say. > > “Good thing you are now using EasyBuild and modules”, you say. > > “Alas, there is no easyconfig for MACS”, I say. > > So here is what I’m poised to try tomorrow or the next day: > >- create an easyconfig for macs2 >- based on easyblocks/generic/pythonpackage.py >- using a toolchain which provides a python. say: >Python/2.7.3-goolf-1.7.20 >- possibly(?) modified (somehow?) to provide a compatible NumPy (>= >1.6) > > I appreciate any feedback or pointers to examples or advice on best > practices in this regard. > > Now, I want to appreciate best practices for installing such applications . > > My next attempt will be to eb ImageMagick (built with --enable-shared=yes), > followed somehow by the perl “Image::Magick” module, and I welcome > similar pointers. > > Cheers, > > ~ Malcolm Cook > > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] will openmpi with infiniband support fall back to ethernet if infiniband not available? [bcc - adr][faked - adr]
Hi John thanks for your feedback. I just asked this in the openmpi-users mailing list. You are right that's the right place to ask. Pablo. 2015-03-16 11:25 GMT+01:00 John Hearns : > I am thinking about trying to compile openmpi with infiniband support > (I would compile openmpi in one of the machine connected to the infiniband > network) and then use this openmpi version also in the non infiniband > machines. I am "expecting" that when using this version of openmpi in a > non-infiniband machine openmpi will detect this and will fallback to use > ethernet. > > > > > > Pablo, it should do that. > > Perhaps you should be asking this question over on the Openmpi list? > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > -- > > Scanned by *MailMarshal* - M86 Security's comprehensive email content > security solution. > > -- > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
[easybuild] will openmpi with infiniband support fall back to ethernet if infiniband not available?
Hi all, This question is not strictly related to easybuild but I supposed someone here could give me some feedback on the topic. I am planning to upgrade my software stack (using easybuild of course :) and now I will have machines without infiniband and machines with infiniband, I would like to share the same software stack across all those machines. I am thinking about trying to compile openmpi with infiniband support (I would compile openmpi in one of the machine connected to the infiniband network) and then use this openmpi version also in the non infiniband machines. I am "expecting" that when using this version of openmpi in a non-infiniband machine openmpi will detect this and will fallback to use ethernet. Does anyone has experience with this? I am expecting correctly or this won't work? In case this is possible, is the any configuration/compilation tweaks required when building openmpi for this to work? Or am I thinking stupid stuff and I should look for a different approach? :) thanks in advance for any help. Pablo. -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] easybuild new user experience
2015-02-11 17:09 GMT+01:00 Cook, Malcolm :
>
>
> Fotis, I note your example name, "sandybridge", apparently encoded an
> intel processor microarchitecture, NOT the name of a linux distribution
> (such as c6 or c7 for releases of centOS, as proposed). I'm trying to
> understand the implications of possibly needing to support a heterogeneous
> environment having multiple CentOS versions (6.5 and 7.x) on multiple core
> types (sandybridge) and would appreciate any more clarity here. Are you
> possibly suggesting that buildsets for each combination of microprocessor
> and OS version might be appropriate
> (provoking visions of /opt/apps/{sandybridge,Nehalem}/centOS{6,7}/MMDD
> )
>
>
>
Hi Malcom,
In case it's helpful, my approach to support different cpu types is to have
a different soft stack by cpu type and then use automounter to access the
right software stack.
As example, I have all my soft stacks in this paths:
/mnt/soft/apps/{sandybridge,nehalem,bulldozer}
Then I use aumounter to "map" /soft/apps to the right folder in
/mnt/soft/apps depending on the cpu type of the machine. In all my compute
nodes the path where the software is accesible is /soft/apps (this is what
my users know about) but automounter is mapping /soft/apps to the right
folder in /mnt/soft/apps/{sandybridge,nehalem,bulldozer}
Of course, for this you need to rebuild all your applications for all the
cpu types you want to support and have a different soft stack for each cpu
but once you have an easyconfig building the application 1 time or 100
times is easy :)
I suppose that adapting that approach to also support multiple
distributions shouldn't be difficult.
regards,
Pablo.
--
Pablo Escobar López
HPC systems engineer
Biozentrum, University of Basel
Swiss Institute of Bioinformatics SIB
Email: pablo.escobarlo...@unibas.ch
Phone: +41 61 267 21 80
http://www.biozentrum.unibas.ch
Re: [easybuild] EasyBuild configuration
Hi again Valeriu, in reply to your question about using your existing compiler modules, it's technically possible but as you probably have noticed it's not straightforward My suggestion would be to start using the default easybuild toolchains at least while you start testing and learning easybuild and once you feel confident with how easybuild works you can try to tweak it. Basically I think you will need to write your own easybuild toolchains which include your desired compiler modules. In my opinion, if you try to do this tweak in your first approach to easybuild it's going to be "hard" regards, pablo. 2014-11-21 15:00 GMT+01:00 Fotis Georgatos : > > On Nov 21, 2014, at 2:55 PM, Fotis Georgatos wrote: > > If your previous modules really function well as drop-in replacement, > this should work. > > Clarification: this statement is correct only if the symlinked software > builds > are relocatable themselves and do not make heavy assumptions about paths. > In sort, it’s not a great solution unless you are certain how bin/lib etc > behave. > > F. > > -- > echo "sysadmin know better bash than english" | sed s/min/mins/ \ > | sed 's/better bash/bash better/' # signal detected in a CERN forum > > > > > > > > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] EasyBuild configuration
Hi Valeriu,
you can always add as many enviroment variables as you need using the
options "modextravars" and "modextrapaths" in your easyconfigs.
I paste here an example but if you grep for "modextra" in your easyconfigs
folder you will find more.
modextravars = {
'TRANSABYSS_PATH': '$root',
'TRANSABYSS_VERSION': version,
'ABYSSPATH': '$EBROOTABYSS/bin',
'PICARD_DIR': '$EBROOTPICARD'
}
modextrapaths = {
'PYTHONPATH': '$root'
}
regards,
Pablo
2014-11-21 14:36 GMT+01:00 Alan O'Cais :
> Hi Valeriu,
>
> The first question comes up a lot. Yes it is possible but tedious and
> non-trivial at present (and even more difficult in a hierarchical modules).
> If it really is compilers that you would like to keep it will be much
> easier to do a rebuild with EasyBuild than to do wrappers to import the
> existing modules.
>
> For the seocnd question there is actually work started already in there
> for supporting the PRACE CPE environment but it is not in there yet:
> https://github.com/hpcugent/easybuild-easyblocks/issues/82
>
> Alan
>
> On 21 November 2014 14:14, Valeriu Codreanu
> wrote:
>
>> Hello,
>>
>> I have succeeded in installing EasyBuild on one of our systems. Also, I
>> have built some modules, and am very enthusiastic about it because it makes
>> things a lot easier.
>>
>> However, in order to deploy it on our system I need to know a few extra
>> things:
>>
>>1. We currently have a lot of software modules already installed. We
>>would prefer if possible to not rebuild every module with EasyBuild, but
>>re-use some of the existing modules (e.g. Intel Compilers) when building
>>other libraries/applications through EasyBuild. Do you have a procedure
>> for
>>achieving this?
>>2. The generated module files set some classic env vars (e.g.
>>$LD_LIBRARY_PATH) and some EasyBuild specific ones (e.g.
>>$EBROOTGOMPI). Our current module system also sets some other env vars
>>(e.g. PRACE_CFLAGS). What would be the easiest way to instruct EasyBuild
>>to set these extra env vars?
>>
>> Thanks,
>> Vali
>>
>
>
>
> --
> Dr. Alan O'Cais
> Application Support
> Juelich Supercomputing Centre
> Forschungszentrum Juelich GmbH
> 52425 Juelich, Germany
>
> Phone: +49 2461 61 5213
> Fax: +49 2461 61 6656
> E-mail: a.oc...@fz-juelich.de
> WWW:http://www.fz-juelich.de/ias/jsc/EN
>
>
>
>
>
>
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
>
>
>
>
>
>
--
Pablo Escobar López
HPC systems engineer
Biozentrum, University of Basel
Swiss Institute of Bioinformatics SIB
Email: pablo.escobarlo...@unibas.ch
Phone: +41 61 267 21 80
http://www.biozentrum.unibas.ch
Re: [easybuild] Any easy way to get lots of CPAN modules for a Perl installation?
Hi Tin
I attach a quick and dirty script I use when I need to add extra perl
libraries to my Perl easyconfig in case it's useful for you.
The output is something like this:
$> ./get-perl-package.sh Digest::SHA1
found a package for Digest::SHA1 :
https://cpan.metacpan.org/authors/id/G/GA/GAAS/Digest-SHA1-2.13.tar.gz
===Please add the following lines to the extensions list in the Perl
easyconfig file
('Digest::SHA1', '2.13', {
'source_tmpl': 'Digest-SHA1-2.13.tar.gz',
'source_urls': ['https://cpan.metacpan.org/authors/id/G/GA/GAAS'],
}),
regards,
Pablo.
2014-11-20 2:04 GMT+01:00 tin h :
> Hello fellow easybuilder :)
>
>
> On a traditional, OS-based Perl installation, one would use CPAN to
> automatically fetch and install modules and libraries, including all the
> dependencies.
>
>
>
> What easy (as in "lazy") way is there to accomplish the same thing when
> using EasyBuild?
>
>
>
> I have a Perl.eb that looks like the one below. I understand that I
> can list all the CPAN modules by listing them in the exts_list, but this a
> very long and tedious process, as it does not take care of dependencies.
> Things like BioPerl has a gigantic list of dependent modules, and I rather
> not write out this list by hand. I did say I wanted to be lazy, yes? :p
>
>
> Thoughts? How the the Perl guru handling this?
>
>
> Much thanks in advance,
>
> Tin
>
>
>
>
>
>
> name = 'Perl'
>
> version = '5.20.0'
>
> toolchain = {'name': 'goolf', 'version': '1.5.14-NX'}
>
> source_urls = ['http://www.cpan.org/src/5.0']
>
> sources = [SOURCELOWER_TAR_GZ]
>
>
> exts_list = [
>
> ('DBI', '1.631', {
>
> 'source_urls': ['http://www.cpan.org/modules/by-module/DBI/TIMB/'
> ],
>
> }),
>
> ('Module::Build', '0.4205', {
>
> 'source_tmpl': 'Module-Build-0.4205.tar.gz',
>
> 'source_urls': ['
> http://www.cpan.org/modules/by-module/Module/LEONT/'],
>
> }),
>
> ('Devel::StackTrace', '1.32', {
>
> 'source_tmpl': 'Devel-StackTrace-1.32.tar.gz',
>
> 'source_urls': ['
> http://www.cpan.org/modules/by-module/Devel/DROLSKY'],
>
> }),
>
> ('Class::Data::Inheritable', '0.08', {
>
> 'source_tmpl': 'Class-Data-Inheritable-0.08.tar.gz',
>
> 'source_urls': ['http://www.cpan.org/modules/by-module/Class/TMTM'
> ],
>
> })
> ]
>
> moduleclass = 'lang'
>
>
--
Pablo Escobar López
HPC systems engineer
Biozentrum, University of Basel
Swiss Institute of Bioinformatics SIB
Email: pablo.escobarlo...@unibas.ch
Phone: +41 61 267 21 80
http://www.biozentrum.unibas.ch
get-perl-package.sh
Description: Bourne shell script
Re: [easybuild] Any easy way to get lots of CPAN modules for a Perl installation?
Hi Tin
I attach a quick and dirty script I use when I need to add extra perl
libraries to my Perl easyconfig in case it's useful for you.
The output is something like this:
$ ./get-perl-package.pl Digest::SHA1
found a package for Digest::SHA1 :
https://cpan.metacpan.org/authors/id/G/GA/GAAS/Digest-SHA1-2.13.tar.gz
===Please add the following lines to the easybuild config file
('Digest::SHA1', '2.13', {
'source_tmpl': 'Digest-SHA1-2.13.tar.gz',
'source_urls': ['https://cpan.metacpan.org/authors/id/G/GA/GAAS'],
}),
regards,
Pablo.
2014-11-20 2:04 GMT+01:00 tin h :
> Hello fellow easybuilder :)
>
>
> On a traditional, OS-based Perl installation, one would use CPAN to
> automatically fetch and install modules and libraries, including all the
> dependencies.
>
>
>
> What easy (as in "lazy") way is there to accomplish the same thing when
> using EasyBuild?
>
>
>
> I have a Perl.eb that looks like the one below. I understand that I
> can list all the CPAN modules by listing them in the exts_list, but this a
> very long and tedious process, as it does not take care of dependencies.
> Things like BioPerl has a gigantic list of dependent modules, and I rather
> not write out this list by hand. I did say I wanted to be lazy, yes? :p
>
>
> Thoughts? How the the Perl guru handling this?
>
>
> Much thanks in advance,
>
> Tin
>
>
>
>
>
>
> name = 'Perl'
>
> version = '5.20.0'
>
> toolchain = {'name': 'goolf', 'version': '1.5.14-NX'}
>
> source_urls = ['http://www.cpan.org/src/5.0']
>
> sources = [SOURCELOWER_TAR_GZ]
>
>
> exts_list = [
>
> ('DBI', '1.631', {
>
> 'source_urls': ['http://www.cpan.org/modules/by-module/DBI/TIMB/'
> ],
>
> }),
>
> ('Module::Build', '0.4205', {
>
> 'source_tmpl': 'Module-Build-0.4205.tar.gz',
>
> 'source_urls': ['
> http://www.cpan.org/modules/by-module/Module/LEONT/'],
>
> }),
>
> ('Devel::StackTrace', '1.32', {
>
> 'source_tmpl': 'Devel-StackTrace-1.32.tar.gz',
>
> 'source_urls': ['
> http://www.cpan.org/modules/by-module/Devel/DROLSKY'],
>
> }),
>
> ('Class::Data::Inheritable', '0.08', {
>
> 'source_tmpl': 'Class-Data-Inheritable-0.08.tar.gz',
>
> 'source_urls': ['http://www.cpan.org/modules/by-module/Class/TMTM'
> ],
>
> })
> ]
>
> moduleclass = 'lang'
>
>
get-perl-package.sh
Description: Bourne shell script
Re: [easybuild] easybuild build problem
ld_main()
> >>> File "/scratch/tmpIu2dbb/eb_stage1/lib/python2.6/site-packages/
> >>> easybuild_framework-1.15.2-py2.6.egg/easybuild/main.py", line 279, in
> >>> main
> >>> modlist = session_module_list(testing=testing)
> >>> File "/scratch/tmpIu2dbb/eb_stage1/lib/python2.6/site-packages/
> >>> easybuild_framework-1.15.2-py2.6.egg/easybuild/tools/testing.py", line
> >>> 163, in
> >>> session_module_list
> >>> return modtool.list()
> >>> File "/scratch/tmpIu2dbb/eb_stage1/lib/python2.6/site-packages/
> >>> easybuild_framework-1.15.2-py2.6.egg/easybuild/tools/modules.py", line
> >>> 551, in
> >>> list
> >>> return self.run_module('list')
> >>> File "/scratch/tmpIu2dbb/eb_stage1/lib/python2.6/site-packages/
> >>> easybuild_framework-1.15.2-py2.6.egg/easybuild/tools/modules.py", line
> >>> 541, in
> >>> run_module
> >>> self.log.error(line)
> >>> File "/scratch/tmpIu2dbb/eb_stage1/lib/python2.6/site-packages/
> >>> easybuild_framework-1.15.2-py2.6.egg/easybuild/tools/build_log.py",
> >>> line 105,
> >>> in error
> >>> raise EasyBuildError(newMsg)
> >>> easybuild.tools.build_log.EasyBuildError: 'EasyBuild crashed with an
> >>> error (at
> >>> easybuild/tools/modules.py:541 in run_module):
> >>> ModuleCmd_List.c(146):FATAL:996: The environment variables
> >>> LOADEDMODULES and
> >>> _LMFILES_ have inconsistent lengths.'
> >>>
> >>> After I set up _LMFILES_ to the same value as LOADEDMODULES (probably
> >>> not
> >>> correct), EasyBuild installed.
> >>>
> >>> I can do: module load EasyBuild, and eb --help, but when I try to use
> >>> EasyBuild to install a module I get:
> >>>
> >>> eb WRF-3.5.1-goolf-1.4.10-dmpar.eb -Dr
> >>> == temporary log file in case of crash
> >>> /scratch/easybuild-EZ0NgN/easybuild-
> >>> LLd1gH.log
> >>> ERROR: EasyBuild crashed with an error (at
> easybuild/software/EasyBuild/
> >>>
> 1.15.2/lib/python2.6/site-packages/easybuild_framework-1.15.2-py2.6.egg/
> >>> easybuild/tools/modules.py:541 in run_module): homkat(4):ERROR:154:
> >>> Version
> >>> symbol 'default' loops.
> >>>
> >>> I mention that I use the Tcl module environment, version 3.1.6.
> >>>
> >>> Could you please give me some hints on what could be wrong?
> >>>
> >>> Best regards,
> >>> Vali
>
>
--
Pablo Escobar López
HPC systems engineer
Biozentrum, University of Basel
Swiss Institute of Bioinformatics SIB
Email: pablo.escobarlo...@unibas.ch
Phone: +41 61 267 21 80
http://www.biozentrum.unibas.ch
Re: [easybuild] CMake-3.3.0-intel-2014b : problem building
You are right but that would also depend if in the makefile CC and CXX are being used, isn't it? In this specific example of freebayes the build system is a little bit weird. It includes different git submodules and make is calling cmake to build dependencies so I didn't want to spend too much time with itthe workaround worked at the first try for me so that was enough for my needs -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch 2014-10-28 12:06 GMT+01:00 Ward Poelmans : > On Tue, Oct 28, 2014 at 12:04 PM, Pablo Escobar Lopez > wrote: > > I verified it when I tried the workaround and it seems to work fine, at > > least in this case. > > That's because you are using a gcc toolchain and it is in the path. I > suspect this would fall back to system gcc with an ictce toolchain. > > Ward > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] CMake-3.3.0-intel-2014b : problem building
I verified it when I tried the workaround and it seems to work fine, at least in this case. Here is my build log == 2014-10-27 10:36:55,249 main.run INFO cmd "unset CC && unset CXX && make -j 1 " exited with exitcode 0 and output: cat: src/version_git.h: No such file or directory wget -q http://hypervolu.me/freebayes/build/ & cd src && make make[1]: Entering directory `/tmp/easybuild/freebayes/0.9.18-1-g4233a23-dirty/goolf-1.4.10/freebayes/src' DETECTED_VERSION = v0.9.18-1-g4233a23-dirty CURRENT_VERSION = Updating version file. cd ../bamtools && mkdir -p build && cd build && cmake .. && make -- The C compiler identification is GNU 4.7.2 -- The CXX compiler identification is GNU 4.7.2 -- Check for working C compiler: /import/bc2/apps/GCC/4.7.2/bin/cc -- Check for working C compiler: /import/bc2/apps/GCC/4.7.2/bin/cc -- works By the way, I just sent a pull request for this easyconfig in case anyone want to look at it https://github.com/hpcugent/easybuild-easyconfigs/pull/1165 2014-10-28 11:54 GMT+01:00 Ward Poelmans : > On Tue, Oct 28, 2014 at 11:10 AM, Pablo Escobar > wrote: > > > recently I wrote a FreeBayes easyconfig and I had to add this to the > > easyconfig to avoid that error > > > > # workaround to avoid the error "The C compiler identification is > unknown" > > prebuildopts = "unset CC && unset CXX && " > > Don't you fall back to system gcc than? > > Ward > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] CMake-3.3.0-intel-2014b : problem building
Hi Franky, recently I wrote a FreeBayes easyconfig and I had to add this to the easyconfig to avoid that error # workaround to avoid the error "The C compiler identification is unknown" prebuildopts = "unset CC && unset CXX && " maybe you can give it a try Pablo. 2014-10-28 10:56 GMT+01:00 Backeljauw Franky < franky.backelj...@uantwerpen.be>: > Hello all, > > We have great difficulty in installing CMake on RHEL 6.4. We have tried > both with CMake-2.8.12.1-intel-2014a.eb and CMake-3.3.0-intel-2014b.eb > which is included in EasyBuild 1.15.2. The same problem occurs with the > foss-toolchains as well as with (e.g.) CMake-2.8.12.1-GCC-4.8.2.eb. > > We get the following errors: > > -- The C compiler identification is unknown > CMake Error at Modules/CMakeDetermineCCompiler.cmake:170 (configure_file): > configure_file Problem configuring file > Call Stack (most recent call first): > CMakeLists.txt:16 (project) > > > -- The CXX compiler identification is unknown > CMake Error at Modules/CMakeDetermineCXXCompiler.cmake:168 > (configure_file): > configure_file Problem configuring file > Call Stack (most recent call first): > CMakeLists.txt:16 (project) > > > -- Check for working C compiler: > /apps/antwerpen/ivybridge/sl6/icc/2013.5.192-GCC-4.8.3/bin/intel64/icc > CMake Error at Modules/CMakeTestCCompiler.cmake:47 (try_compile): > Unknown extension ".c" for file > > > /apps/antwerpen/easybuild/build/CMake/3.0.0/intel-2014b/cmake-3.0.0/CMakeFiles/CMakeTmp/testCCompiler.c > >try_compile() works only for enabled languages. Currently these are: > > C CXX > > I have included the full log for the CMake-3.0.0 build (cmake.log). > > I hope someone can help us out here. > > -- Many thanks for your reply, > > Franky Backeljauw >
[easybuild] easybuild and centos 7 ?
Hi, Has anyone tried easybuild in centos 7 ? any feedback before I start some testing with it? thanks Pablo. -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 21 80 http://www.biozentrum.unibas.ch
Re: [easybuild] upgrade EB from 1.13 to 1.14
2014-09-09 9:37 GMT+02:00 Arnau Bria : > > > > That worked like a charm! > > And for coming releases you don't even need to download a new easyconfig :) This is how I did my last upgrade: $> eb EasyBuild-1.13.0.eb --try-software-version=1.14.0 regards, Pablo. -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 15 82 http://www.biozentrum.unibas.ch
Re: [easybuild] Getting started with EB - integrating into existing Lmod environment
2014-08-25 18:00 GMT+02:00 Trey Dockendorf : > > towhee - http://sourceforge.net/projects/towhee/ - not sure what it's > used for but requested by local Petroleum engineering users. > tesseract - OCR application used by our local Digital Humanities group > ior - I use this for benchmarking our parallel filesystem (FhGFS/BeeGFS) > > They are somewhat uncommon I would guess so once I get to them I'll try my > hand at making easyblocks for them. > There is a IOR pull request waiting in the queue...it's just that Kenneth is too lazy to merge it :) https://github.com/hpcugent/easybuild-easyconfigs/pull/949 -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 15 82 http://www.biozentrum.unibas.ch
Re: [easybuild] Support for other job schedulers
Hola Miguel :) as you already mentioned neither LSF or slurm is officially supported yet, anyway even if it were supported, I would suggest to start learning how easybuild works without the --job option because that is not a widely tested option. So I think it´s better to start learning how easybuild works without submitting to a scheduler and once you are used to how easybuild works then start testing with the --job option. The approach I use to run easybuild in different clusters is to have a different easybuild config files for each of my clusters (where I define different paths for install_dir or modules_dir) and then run the same easyconfig (.eb file) in the different login nodes using the specific easybuild config file for that cluster. This way, I write a single easyconfig which I execute in each of my clusters login nodes so the compilation is optimized for each machine. Automatizing this is quite simple. If you want more details about this specific setup just email to the list. un saludo Pablo. 2014-08-08 18:08 GMT+02:00 Kenneth Hoste : > HI Ricardo, > > On 08/08/14 17:48, Riccardo Murri wrote: > > Hi Miguel, all, > > > > On 8 August 2014 12:41, Miguel Bernabeu Diaz > wrote: > >> I'm not sure if all or at least the most common schedulers' CLI could be > >> abstracted in this manner as I've only worked with Slurm and LSF. Either > >> way, would the community be interested in this kind of abstraction? > Also, > >> has someone worked on something similar or a port to Slurm or LSF we > could > >> extend or reuse? > > We too would probably be interested in batch-system independence, > > although we're in no hurry. (This would fit in the framework of a project > > that will only start later on this year.) > I agree this would be a very nice feature indeed. --job is very useful > for us, but it probably really only works for us. > You basically need Torque + pbs_python (and maybe even align the > versions a bit, to make it worse). > > > Actually, if I am allowed a shameless self-plug, we already have a > > Python framework that can submit and manage jobs on different > > batch-queuing systems, see http://gc3pie.googlecode.com/ > > That sounds interesting! > > Let me pick up a crazy project idea we wrote up some time ago: > https://gist.github.com/boegel/9225891 . > > How does gc3pie relate to that? > > > I am not familiar with EasyBuild internals, but GC3Pie's job control > > reduces to a few lines that should be relatively quick to plug in: > > > > from gc3libs import Application > > from gc3libs.core import Engine > > > > task = Application(['some', '-unix', '+command', 'here'], ...) > > engine = Engine(...) > > engine.add(task) > > # run task and wait for it to finish > > engine.progress() > > > > If there is interest, I can look at the sources and try to estimate > > how much work it would be to integrate GC3Pie and EasyBuild. > The first step should be to abstract the current support for --job into > a generic class, and make what's there now derive from that (probably > naming it PbsPython). > > Then, SLURM & LSF could be just another version of that, and so can > gc3pie and DRMAA. > > Unless gc3pie solves all our problems, that would even be better. ;-) > > As the project idea gist shows, supporting different batch systems is > really a project on its own. > > > K. > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 15 82 http://www.biozentrum.unibas.ch
Re: [easybuild] upgrade EB from 1.13 to 1.14
Hi Arnau If you already have EasyBuild working I think the easier aproach is to use easybuild itself to install it. Kenneth always attach a easyconfig file with the release announce email so you would only need to run "eb newEBrelease.eb". That way you will have two different modules for your two different easybuild version and you can use whichever you prefer. If you find an problem using the easyconfig that Kenneth attached with his email apply dos2unix to it. For me it doesn't work until I apply dos2unix Pablo. 2014-07-22 15:39 GMT+02:00 Inigo Aldazabal Mensa : > Hi, > > On Tue, 22 Jul 2014 15:19:06 +0200 > Arnau Bria wrote: > > > Hello all, > > > > some weeks ago you announced a new version of EB. At the end of the > > announcement you recommended some ways for upgrading the software: > > > > > > * reinstalling EasyBuild from PyPi (don't forget --upgrade if > > you're using easy_install), > > * updating (the master branch of) your GitHub repository clones, > > or > > * installing EasyBuild with EasyBuild, using the easyconfig file > > available via [8] (this requires EasyBuild v1.8.2 or more recent) > > --- > > > > I did install EB following the bootstrapping method, may I use it > > again to overwrite my existing installation? > > > > wget --no-check-certificate > > > https://raw.github.com/hpcugent/easybuild-framework/develop/easybuild/scripts/bootstrap_eb.py > > export EASYBUILD_PREFIX=/software/as/el6.3/EasyBuild/ python > > bootstrap_eb.py $EASYBUILD_PREFIX > > I don't think it will overwrite your installation but it will install > it in /software/as/el6.3/EasyBuild/1.14.0 and you'll just load the new > module in order to use it. But this is a guess from what I see in here, > check it out first in a virtual machine! > > Iñigo > > > > > > > TIA, > > Arnau > -- Pablo Escobar López HPC systems engineer Biozentrum, University of Basel Swiss Institute of Bioinformatics SIB Email: pablo.escobarlo...@unibas.ch Phone: +41 61 267 15 82 http://www.biozentrum.unibas.ch
