Re: [easybuild] non-standard location of setup.py for easyconfig
Hi again,
I just learned about `start_dir`, and the following seems to work:
('ESMpy', '7.1.0r', {
'source_tmpl': 'esmf_7_1_0r_src.tar.gz',
'source_urls':
['http://www.earthsystemmodeling.org/esmf_releases/public/ESMF_7_1_0r/'],
'checksums':
['ae9a5edb8d40ae97a35cbd4bd00b77061f995c77c43d36334dbb95c18b00a889'],
'start_dir': 'src/addon/ESMPy/',
'modulename': 'ESMF',
}),
Sorry for the noise!
Cheers,
Andreas
[easybuild] non-standard location of setup.py for easyconfig
Hi EasyBuilders,
I'd like to add ESMpy to an easyconfig as a PythonPackage
extension.
The problem is that the `setup.py` is not in the root of the
source archive, but instead in a subfolder `src/addon/ESMPy/`.
Currently I have
('ESMpy', '7.1.0r', {
'source_tmpl': 'esmf_7_1_0r_src.tar.gz',
'source_urls':
['http://www.earthsystemmodeling.org/esmf_releases/public/ESMF_7_1_0r/'],
'checksums':
['ae9a5edb8d40ae97a35cbd4bd00b77061f995c77c43d36334dbb95c18b00a889'],
}),
in my `exts_list`.
Is there any way to add the information that EasyBuild should go
to the correct subdirectory in the sources before executing
`setup.py install`?
Thanks for your help!
Cheers,
Andreas
Re: [easybuild] preparing R-3.6.1-foss-2019b
Hi,
I think it would be good to include those extensions needed to run
the IRkernel (https://irkernel.github.io/) from this module. That
way, one could use the EasyBuild module in a JupyterHub
installation for interactive use.
We included the extensions
('repr', '1.0.1', {'checksums':
['ecde22c17fd800e1ff5c2b2962689119aa486fba40fbc6f2c50e8d4cc61bc44b']}),
('IRdisplay', '0.7.0', {'checksums':
['91eac9acdb92ed0fdc58e5da284aa4bb957ada5eef504fd89bec136747999089']}),
('pbdZMQ', '0.3-3', {'checksums':
['ae26c13400e2acfb6463ff9b67156847a22ec79f3b53baf65119efaba1636eca']}),
('IRkernel', '1.0.2', {'checksums':
['976f63a3c76f03ec793213255022d7b9aa91c4dcd5aa1c90d35dfd72cc8b1810']}),
in our intel-2019a build of R-3.6.1 and that seems to work without
problemms so far.
Cheers,
Andreas
--
Dr. Andreas Hilboll
Institute of Environmental Physics (IUP)
University of Bremen
NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Inconsistent versions within one toolchain
Dear Kenneth, thanks for the explanation! Kenneth Hoste writes: Dear Andres, We've been keeping a close (automated) eye on which dependencies are used in recent common toolchains (foss* & intel*, since 2018a) in that respect: we have a test that verifies that there's only a single "variant" (version + versionsuffix) of a software package used as a runtime dependency (with some exceptions, e.g. Python 2.x & 3.x). We're not doing this for build dependencies though, which is why there are two versions of Bison used in easyconfigs that use a 2018b toolchain. Ah, okay, I didn't think about that distinction. And I understand that for build dependencies, one might want the extra freedom of not enforcing this policy. For Bison, it's a bit silly, but we have seen situations where at some point a newer CMake was required than the one that was already being used... The main reason for the "one dep variant" policy is to avoid version conflicts as much as possible between easyconfigs using the same toolchain. We learned the hard way that you run into hard to fix problems if you don't maintain a policy like this... Any particular reason for your question? No, not really. I just noticed that for one install (I'm updating Octave to 4.4.1), eb wants to build both Bison versions, which I found silly, so I got curious. But your explanation makes perfect sense. Cheers, Andreas
[easybuild] Inconsistent versions within one toolchain
Hey,
I just noticed that library versions are not necessarily used
consistently within one toolchain. For example, in foss-2018b,
there's both Bison-3.0.4 and Bison-3.0.5:
$ grep Bison */*/*.eb | grep foss-2018b
b/bioawk/bioawk-1.0-foss-2018b.eb:builddependencies =
[('Bison', '3.0.5')]
g/GObject-Introspection/GObject-Introspection-1.54.1-foss-2018b-Python-2.7.15.eb:
('Bison', '3.0.5'),
i/ITK/ITK-4.13.1-foss-2018b-Python-2.7.15.eb:('Bison',
'3.0.5'),
i/ITK/ITK-4.13.1-foss-2018b-Python-3.6.6.eb:('Bison',
'3.0.5'),
m/Mesa/Mesa-18.1.1-foss-2018b.eb:('Bison', '3.0.5'),
m/motif/motif-2.3.8-foss-2018b.eb:('Bison', '3.0.5'),
o/OpenFOAM/OpenFOAM-6-foss-2018b.eb:('Bison', '3.0.4'),
o/OpenFOAM/OpenFOAM-v1806-foss-2018b.eb:('Bison',
'3.0.4'),
v/ViennaRNA/ViennaRNA-2.4.10-foss-2018b-Python-2.7.15.eb:
('Bison', '3.0.5'),
$ grep Bison */*/*.eb | grep GCCcore-7.3.0.eb
b/binutils/binutils-2.30-GCCcore-7.3.0.eb:('Bison',
'3.0.4'),
b/Bison/Bison-3.0.4-GCCcore-7.3.0.eb:name = 'Bison'
b/Bison/Bison-3.0.4-GCCcore-7.3.0.eb:description = """Bison is
a general-purpose parser generator that converts an annotated
context-free grammar
b/Bison/Bison-3.0.5-GCCcore-7.3.0.eb:name = 'Bison'
b/Bison/Bison-3.0.5-GCCcore-7.3.0.eb:description = """Bison is
a general-purpose parser generator that converts an annotated
context-free grammar
d/Doxygen/Doxygen-1.8.14-GCCcore-7.3.0.eb:('Bison',
'3.0.4'),
f/flex/flex-2.6.4-GCCcore-7.3.0.eb:('Bison', '3.0.4'),
l/libmatheval/libmatheval-1.1.11-GCCcore-7.3.0.eb:
('Bison', '3.0.5'),
x/X11/X11-20180604-GCCcore-7.3.0.eb:('Bison', '3.0.5'),
Is this done on purpose, or does it "just happen"? Should this be
fixed, or is it not important at all?
Cheers,
Andreas
Re: [easybuild] Cannot find dependency with --minimal-toolchains and --hidden-deps
Thanks, Kenneth --
One thing you also need to take into account is that the
tcsh-6.20.00-GCCcore-7.3.0.eb easyconfig must be available in
your
robot search path ("eb --search tcsh-6.20.00-GCCcore-7.3.0"
should
fine it).
That's a known problem which we haven't gotten around to fixing
yet,
see
https://github.com/easybuilders/easybuild-framework/issues/2375
.
that was indeed the problem. Building now.
Cheers,
Andreas
[easybuild] Cannot find dependency with --minimal-toolchains and --hidden-deps
Hi,
I'm trying to build WRF-4.0.1 for foss-2018b. My easyconfig has
(among other settings)
toolchain = {'name': 'foss', 'version': '2018b'}
builddependencies = [('tcsh', '6.20.00')]
dependencies = [
('JasPer', '2.0.14'),
('netCDF', '4.6.1'),
('netCDF-Fortran', '4.4.4'),
]
My EasyBuild (I'm using 3.7.1) config looks like this:
[override]
add-dummy-to-minimal-toolchains=True
hide-deps=M4,zlib,help2man,Bison,flex,binutils,Autoconf,Automake,libtool,ncurses,xorg-macros,gettext,numactl,libpciaccess,XZ,libxml2,hwloc,Autotools,pkg-config,ncurses,cURL,Szip,CMake,Doxygen,icc,ifort,iccifort,impi,iimpi,imkl,gompi,GCCcore,GCC,OpenBLAS,OpenMPI,FFTW,ScaLAPACK,tcsh
hide-toolchains=GCCcore,GCC,gompi,iimpi,iccifort
minimal-toolchains=True
use-existing-modules=True
I built tcsh-6.20.00-GCCcore-7.3.0.eb from by previous WRF4 PR,
using
eb tcsh-6.20.00-GCCcore-7.3.0.eb --hidden -f
The module is there:
$ module --show-hidden avail | grep tcsh
XZ/.5.2.4-GCCcore-7.3.0
tcsh/.6.20.00-GCCcore-7.3.0
However, when I try building the WRF easyconfig,
eb WRF-4.0.1-foss-2018b-dmpar.eb
--include-easyblocks=/home2/hilboll/src/easybuild/repos/easybuild-easyblocks/easybuild/easyblocks/w/wrf.py
-r --debug --ignore-checksums -D
I get
ERROR: Irresolvable dependencies encountered:
tcsh/6.20.00-foss-2018b
On my other system, where I don't hide dependencies, and don't
have minimal-toolchains turned on, the tcsh dependency is
automatically resolved from the GCCcore toolchain, but here on my
test system (where I'm trying to learn about --hide-deps and
--minimal-toolchains) apparently it doesn't.
I'd appreciate if you could point me to my mistake.
Sunny greetings,
Andreas
Re: [easybuild] Updated Lmod RPM package needed for EasyBuild v3.7.0
We just yesterday installed 7.8.1 from http://build.openhpc.community/OpenHPC:/1.3:/Update6:/Factory/CentOS_7/x86_64/lmod-ohpc-7.8.1-5.1.ohpc.1.3.6.x86_64.rpm, and everything's working smoothly. Cheers, Andreas André Gemünd writes: Dear Ole, maybe the OpenHPC for CentOS 7 package can be useful for that? It provides 7.7.14 currently: http://build.openhpc.community/OpenHPC:/1.3:/Update5/CentOS_7/src/lmod-ohpc-7.7.14-3.1.src.rpm Greetings André - Am 26. Sep 2018 um 13:25 schrieb Ole Holm Nielsen ole.h.niel...@fysik.dtu.dk: Dear Kenneth, Thanks a lot for the answer. Since CentOS 7 is a very popular HPC cluster OS, it would be great to obtain an authoritative RPM package for Lmod as required by EasyBuild. It's very unfortunate that EPEL hasn't updated the Lmod RPM in 2 years. Your .spec file for Lmod 6.6 seems to me not to be very much plug-and-play :-( If no-one can provide an Lmod 6.6.3 RPM for CentOS 7, then it would save us a lot of trouble if EB could still use Lmod 6.5.1 until a newer RPM becomes available. Thanks a lot, Ole On 26-09-2018 11:38, Kenneth Hoste wrote: Dear Ole, We build our own Lmod RPMs so we can stay on top of recent developments. You can find our .spec file in https://github.com/hpcugent/Lmod-UGent. If you go back in history a bit, you should be able to find a .spec file for Lmod 6.x. Of course, you'll need to customize this w.r.t. Lmod configuration & such. I should also mention that the Lmod 6.6.3 requirement may be a bit more than is strictly required... In theory a slightly older Lmod 6.x should be fine too, but I kept running into problems left & right when testing on top of older Lmod 6.x versions, so I figured going with the latest 6.x was a reasonable compromise (I didn't want to force people to switch to Lmod 7). If that's a big problem, I can try and reconsider that version requirement to loosen it up a bit (as long as all the tests pass, that it), and issue a quick EasyBuild 3.7.1. On the other hand, this is good motivation to update your Lmod installation, there have been *a lot* of improvements to Lmod since version 6.5.1 (which was released Aug 2016...). regards, Kenneth On 26/09/2018 11:25, Ole Holm Nielsen wrote: Regarding upgrading to EB 3.7.0: On 25-09-2018 15:09, Kenneth Hoste wrote: Note that the minimal version requirement for Lmod has been bumped to 6.6.3. We use CentOS 7 and the 2 years old Lmod RPM package provided by the EPEL repository: Lmod-6.5.1-2.el7.x86_64.rpm Can anyone suggest the best way to get or build an RPM package of a recent version of Lmod [1] that works well on CentOS 7? Thanks, Ole >> [1] https://github.com/TACC/Lmod
Re: [easybuild] Re: installing octave
Dear Joseph, the package you have to install is called "openssl-devel" (at least on CentOS 7.2). This package has to be present in your image. The error message you're getting tells you that easybuild cannot find that package in your image. When, in your image, you look at the output of the command yum info openssl-devel you should see Repo : installed If you don't see that, you need to change your image until the "yum info" shows the package as installed. Sunny greetings, Andreas
[easybuild] mpdboot in intel/2017a MPI
Hi, I just noticed that the intel/2017a MPI doesn't have the `mpdboot` binary. In the intel/2016a one it was present. Is this to be expected, or is there something wrong with my setup? If it is expected, what can I do when I have software relying on mpdboot in scripts? Cheers for your help, Andreas
Re: [easybuild] installing octave
Hi Joseph,
you should rather install openssl development package through your
distribution's package manager. As you can see in the error message,
> One or more OS dependencies were not found: [('openssl-devel',
> 'libssl-dev', 'libopenssl-devel')]
EasyBuild is looking for a package in your distribution called one of
'openssl-devel', 'libssl-dev', 'libopenssl-devel'). Install that (using
yum, or apt, or whatever your distribution uses), and you should be fine.
Cheers,
Andreas
Re: [easybuild] Including pip in EasyBuild module
Hi Kennth,
Kenneth Hoste writes:
> Hi Andreas,
>
> On 25/08/2017 11:23, Andreas Hilboll wrote:
>> PS: I should add that this is a Python3-based easyconfig, so there is no
>> `pip` executable in Python/3.6.1-intel2017a; it is called `pip3`. Maybe
>> the PythonPackage easyblock should somehow check if the correct pip is
>> called pip or pip3?
>
> I think is your actual problem...
> When you specify "use_pip = True", EasyBuild only tries to use 'pip', it
> doesn't take into account that it may need to use 'pip3' when Python 3
> is being used...
>
> The logic for this is in the generic PythonPackage easyblock [1].
>
> Are you up for taking a stab to fix that?
sure, shouldn't be too hard.
> Basically you need to do something like this when Python 3 is being used:
>
> self.install_cmd = self.install_cmd.replace('pip', 'pip3')
>
> This should probably be done in the 'prepare_python' method, since only
> there the Python module used as a dependency will be loaded.
I opened an issue
(https://github.com/easybuilders/easybuild-easyblocks/issues/1232) with
some questions regarding the implementation.
Cheers,
Andreas
Re: [easybuild] Including pip in EasyBuild module
PS: I should add that this is a Python3-based easyconfig, so there is no `pip` executable in Python/3.6.1-intel2017a; it is called `pip3`. Maybe the PythonPackage easyblock should somehow check if the correct pip is called pip or pip3?
[easybuild] Including pip in EasyBuild module
Hi, I want to include nbdime [1] in an easyconfig as an extension. nbdime wants to be installed with `pip install .`. So I add `'use_pip': True` to the extension dict. However, there is no pip installed on the machines where I run the eb jobs, and the OS (EL7) only provides pip 7.1, which is too old for nbdime, which wants pip>=8. Is there any way I can add a recent pip to the EasyBuild module itself, so that pip is always and everywhere available for eb? The existing pip easyconfigs all depend on a toolchain, and when I just try loading the pip module together with EasyBuild, eb complains that there are unallowed modules loaded. Cheers, Andreas [1] https://nbdime.readthedocs.io/ -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Intel toolchain / architecture selection
Thanks, Åke, I just remember that our login nodes are KVM virtualized, which should be the reason for this behavior. I'll have my colleague look into libvirt configuration to hopefully fix that. To answer your question, the full /proc/cpuinfo is below. Cheers, Andreas Compute node: ---8<-- processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz stepping: 1 microcode : 0xb1b cpu MHz : 3199.929 cache size : 35840 KB physical id : 0 siblings: 14 core id : 0 cpu cores : 14 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc bogomips: 5188.38 clflush size: 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: ---8<-- Login virtual machine ---8<-- processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 61 model name : Intel Core Processor (Broadwell) stepping: 2 microcode : 0x1 cpu MHz : 2399.996 cache size : 4096 KB physical id : 0 siblings: 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt bogomips: 4799.99 clflush size: 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: ---8<-- Master virtualization guest ---8<-- processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz stepping: 1 microcode : 0xb1d cpu MHz : 2599.968 cache size : 25600 KB physical id : 0 siblings: 20 core id : 0 cpu cores : 10 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc bogomips: 4799.91 clflush size: 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: ---8<--
Re: [easybuild] Intel toolchain / architecture selection
Hi Kenneth, thanks for your answer! > By default, EasyBuild uses -xHost when building with the Intel compilers > to optimize for the processor architecture of the system you are > building on. Yes, that's what I thought. However, I'm a bit confused, because both CPUs are from the Xeon E-5 26XXv4 family, so according to the Intel docs (or, at least, those parts which I understand) they should support the same instruction sets. So I'm not sure what to put into -axcode (at least that's how I understood the icc manpage, that using -axcode I can specify optional instruction sets which will be used if present). However, /proc/cpuinfo is different for the two CPUs flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc Any idea what's going on here? cheers, Andreas
[easybuild] Intel toolchain / architecture selection
Hi Easybuilders, when I build the Python-3.6.1-intel-2017a.eb easyconfig on the compute nodes of our cluster instead of the login node, I can only start the generated python executable on the compute nodes: [hilboll@login1 ~]$ module load Python [hilboll@login1 ~]$ python Please verify that both the operating system and the processor support Intel(R) F16C and LZCNT instructions. What do I have to do in so that the executable a. also runs on the login nodes and b. makes optimal use of the available architecture? Our both logins and compute nodes have Broadwell processors; login is 2620v4 I believe, and compute nodes are 2690v4. Thanks, Andreas
Re: [easybuild] Regarding New weather application
Hi Satish, I have updated the wrf easyblock so that it can also build WRF-Chem, and I also have easyconfigs for WRF and WPS 3.8.1. All that's not merged yet into the official repos, but I'll see if I can put the code online sometime next week. Cheers, Andreas Am 11. August 2017 07:20:40 MESZ schrieb Satish Sherikar <sherikar1...@gmail.com>: >Hi, > >I am working at Indian Institute of tropical meteorology (IITM) Pune, >India >as a HPC application Engineer. > >The users of this organization needs weather related application like >WRF >and WPS which already available in EasyBuild. > >I want to know do you have any planning for the below models? > >1. CFS (Climate Forecast System) model >2. GFS (Global Forecast System) model >3. HWRF( Hurricane WRF) model >4. WRF-Chem >5. MOM5 (Modular Ocean Model) >6. DNS (Direct Numerical Simulation) Model > >Please let us know if you have any planning for these above models. > >Thank You! > >Regards, >Satish Sherikar Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] patch just one R extension
Hi,
it's in the documentation, see
https://easybuild.readthedocs.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-parameters.
Cheers,
Andreas
Jure Pečar writes:
> Hi,
>
> I want to apply this patch to R data.table to get it to behave in civilized
> manner:
>
> diff -ur data.table/R/fread.R data.table.fix/R/fread.R
> --- data.table/R/fread.R2017-01-31 03:10:52.0 +
> +++ data.table.fix/R/fread.R2017-06-29 08:34:42.764465000 +
> @@ -85,8 +85,11 @@
> tt = tempfile()
> on.exit(unlink(tt), add = TRUE)
> if (.Platform$OS.type == "unix") {
> -if (file.exists('/dev/shm') && file.info('/dev/shm')$isdir) {
> -tt = tempfile(tmpdir = '/dev/shm')
> +Sys.getenv(tmp("TMPDIR"))
> +if (file.exists(tmp) && file.info(tmp)$isdir) {
> +tt = tempfile(tmpdir = tmp)
> +} else {
> +tt = tempfile(tmpdir = '/tmp')
> }
> system(paste('(', input, ') > ', tt, sep=""))
> } else {
>
>
> What would be the proper syntax in R easyconfig to apply it?
Re: [easybuild] how to specify choices for dependencies?
Thanks, Alan and Gizo,
I'll take a look into this.
Would it be welcome to include this functionality in an upcoming PR to
include ecCodes and EMOS? Or rather not, since this apparently depends
on lmod being the module system of choice?
Cheers,
Andreas
Alan O'Cais writes:
> This sounds like a reasonable approach to me. You should probably specify one
> python as one of your `builddependencies` and then create a modluafooter that
> ensures you have a Python loaded, something like
> if not isloaded("Python") then
> load("Python")
> end
> That should still allow you to keep the python bindings as part of your
> sanity check too.
>
> Alan
>
> On 2 May 2017 at 21:12,
> <nan...@luis.uni-hannover.de<mailto:nan...@luis.uni-hannover.de>> wrote:
>
> Hi,
>
> you could implemet this logic as some lua function defined in SitePackage.lua
> and then use the
> option modluafooter in easyconfig of emos. Not very nice, but should work.
>
> http://lmod.readthedocs.io/en/latest/050_lua_modulefiles.html
>
> Best,
> Gizo
>
> Quoting Andreas Hilboll <hilb...@uni-bremen.de<mailto:hilb...@uni-bremen.de>>:
>
> Hi all,
>
> I want to create easyconfigs for ecCodes[1] and EMOS[2]. ecCodes comes
> with Python bindings, so I'm setting a
>
> versionsuffix = '-Python-%(pyver)s'
>
> EMOS depends on ecCodes, but does not depend on Python at all.
>
> I have built two ecCodes modules, one for Python-2.7.11 and one for
> Python-3.5.1. I'm wondering how to specify the dependencies in the EMOS
> easyconfig, without having to provide a Python-dependent easyconfig for
> EMOS. Ideally, when loading EMOS, lmod should pick the ecCodes module
> which causes the least conflict:
>
> - if no other Python is loaded, chose any default
> - if either Python/2.7.11 or Python/3.5.1 are already loaded, chose the
> matching ecCodes module
>
> Is this somehow possible?
>
> Cheers,
> Andreas.
>
> [1] https://software.ecmwf.int/wiki/display/ECC
> [2] https://software.ecmwf.int/wiki/display/EMOS
>
> --
> Dr. Andreas Hilboll
>
> Center for Marine Environmental Sciences (MARUM)
> - AND -
> Institute of Environmental Physics (IUP)
>
> University of Bremen
>
> NW1 / S3132
> Otto-Hahn-Allee 1
> D-28359 Bremen
> Germany
>
> +49(0)421 218 62133<tel:%2B49%280%29421%20218%2062133> (phone)
> +49(0)421 218 98 62133<tel:%2B49%280%29421%20218%2098%2062133> (fax)
> http://www.iup.uni-bremen.de/~hilboll
>
>
> --
> Dr. Gizo Nanava
> Leibniz Universitaet IT Services
> Leibniz Universitaet Hannover
> Schlosswender Str. 5
> D-30159 Hannover
> Tel +49 511 762 7919085<tel:%2B49%20511%20762%207919085>
> http://www.luis.uni-hannover.de
--
Dr. Andreas Hilboll
Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)
University of Bremen
NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
[easybuild] how to specify choices for dependencies?
Hi all, I want to create easyconfigs for ecCodes[1] and EMOS[2]. ecCodes comes with Python bindings, so I'm setting a versionsuffix = '-Python-%(pyver)s' EMOS depends on ecCodes, but does not depend on Python at all. I have built two ecCodes modules, one for Python-2.7.11 and one for Python-3.5.1. I'm wondering how to specify the dependencies in the EMOS easyconfig, without having to provide a Python-dependent easyconfig for EMOS. Ideally, when loading EMOS, lmod should pick the ecCodes module which causes the least conflict: - if no other Python is loaded, chose any default - if either Python/2.7.11 or Python/3.5.1 are already loaded, chose the matching ecCodes module Is this somehow possible? Cheers, Andreas. [1] https://software.ecmwf.int/wiki/display/ECC [2] https://software.ecmwf.int/wiki/display/EMOS -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
[easybuild] Path to test scripts
Hi all, I'd like to use the `tests` option to include testing by a custom script. I'm unsure where to put such a script, and how to set its path in the EasyConfig. When I simply specify tests = ['run_my_test.sh'] I get the error message Test specifies invalid path: /home/eb/sources/PnetCDF/_eb_run_tests.sh However, I'm puzzled why eb looks in that path, as the sources are located in /home/eb/sources/p/PnetCDF. Is there a possibility to run test scripts which are inside the package's source tree? Cheers, Andreas.
[easybuild] SHA1 checksums
Hi all, I'm writing an EasyConfig for some software which provides SHA1 checksums for their source packages. Is there a way to tell eb that the checksums are SHA1 instead of MD5? Cheers, Andreas.
[easybuild] Strategy for hiding modules
Hi all, when I started using EasyBuild, I didn't know about the option to hide modules. Slowly, the list of installed modules becomes too long for `module avail` to be useful for the user. So I would like to hide modules. The question is, how should I do this? 1. Re-generate all modules (can I use EasyBuild to regenerate modules, hiding them if I want to, without recompiling the code?) 2. Manually hide modules (probably a bad idea) 3. ??? It would be nice to have a best-practices document somewhere, but I couldn't find any. [*] Also, from what I understand, using minimal toolchains is generally a good idea to keep the number of modules low -- correct? I'd appreciate any guidance on how I can clean up my installation. Cheers, Andreas. PS: I'm using lmod [*] I'd be happy to write a blog post from a non-expert's perspective (I'm no sysadmin, and our cluster is rather smallish, almost no technical staff to maintain the software).
[easybuild] FPATH variable overwritten by imkl module
Hi, I'm using ZSH as my shell. Apparently ZSH uses an environment variable FPATH for internal purposes. I have the problem that after loading the imkl module from intel-2016a, my ZSH autocomplete doesn't work any more. This is because the imkl module apparently overwrites the FPATH variable. I'd appreciate any suggestions how to deal with this problem. Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Support for ROOT and/or Julia?
>> has anyone here worked on installing ROOT (https://root.cern.ch/) and/or >> Julia (http://julialang.org/)? > > There are easyconfigs for ROOT included with EasyBuild. Wow, that's embarassing. I wonder why I didn't see that > You can find an easyconfig for Julia here: > > http://www.siliconslick.com/easybuild/easyconfigs/ada/ Great, thanks a lot, I'll give it a try. Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
[easybuild] Support for ROOT and/or Julia?
Hi, has anyone here worked on installing ROOT (https://root.cern.ch/) and/or Julia (http://julialang.org/)? Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] A question on Intel 2017 cluster edition
PS: I put the easyconfigs here: https://gist.github.com/andreas-h/9a0b3fc50fd1445fecb6ece8cd367a83 HTH, Andreas Andreas Hilboll writes: > Hi Exequiel, > > I did this some months ago; it is possible to get somewhere with > specifying the 'components'. (However, now that I think about it, I'm > not sure if I needed to tweak the easyblock for this.) > > The installations took up this much space: > > $ du -sh icc/2016.2.181-GCC-4.9.3-2.25 ifort/2016.2.181-GCC-4.9.3-2.25 > impi/5.1.3.181-iccifort-2016.2.181-GCC-4.9.3-2.25 > imkl/11.3.2.181-iimpi-2016.02-GCC-4.9.3-2.25 > 559M icc/2016.2.181-GCC-4.9.3-2.25 > 550M ifort/2016.2.181-GCC-4.9.3-2.25 > 764M impi/5.1.3.181-iccifort-2016.2.181-GCC-4.9.3-2.25 > 2.7G imkl/11.3.2.181-iimpi-2016.02-GCC-4.9.3-2.25 > > This is comparable to our new system, where I was able to install the > intel toolchain from the individual source files: > > $ du -sh icc/2016.1.150-GCC-4.9.3-2.25/ ifort/2016.1.150-GCC-4.9.3-2.25/ > impi/5.1.2.150-iccifort-2016.1.150-GCC-4.9.3-2.25/ > imkl/11.3.1.150-iimpi-8.1.5-GCC-4.9.3-2.25/ > 563M icc/2016.1.150-GCC-4.9.3-2.25/ > 493M ifort/2016.1.150-GCC-4.9.3-2.25/ > 817M impi/5.1.2.150-iccifort-2016.1.150-GCC-4.9.3-2.25/ > 2.8G imkl/11.3.1.150-iimpi-8.1.5-GCC-4.9.3-2.25/ > > Cheers, > Andreas. > > > Exequiel Sepúlveda writes: > >> Hi everyone! >> >> Starting this 2017 with installing Intel 2017 :-) >> >> I have two questions if there is someone willing to answer them. >> >> 1.- Why are GCC and binutils needed on the easyconfigure file >> intel-2017.00.eb >> 2.- Because I have only a big tar.gz file with Intel Cluster Edition, how >> could I separate the required parts to use easyconfigs file for icc, ifort, >> imkl, etc. >> >> Cheers and a great 2017! -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Python extensions from .whl
Thanks, Kenneth,
for the super-quick reply! As always, I really appreciate all your
help.
Unfortunately, use_pip = True doesn't seem to change anything (actually,
according to the debug log, see
https://gist.githubusercontent.com/andreas-h/6355a160082db1920fce6cb5d0f3655b/raw/efe52f557b289c7b322e633ed31e9568f4675004/easybuild-eKoEuJ.log,
easybuild 3.0.1 still tries to use setup.py ?)
Cheers,
Andreas.
Kenneth Hoste writes:
> Hi Andreas,
>
> On 27/12/2016 19:12, Andreas Hilboll wrote:
>> Hi all,
>>
>> I'm trying to create an easyconfig for jupyterhub 0.7. I copied the
>> existing 0.6.1 easyconfig and bumped the version number, so it now reads
>>
>> easyblock = 'Bundle'
>> name = 'jupyterhub'
>> version = '0.7.0'
>> versionsuffix = '-Python-%(pyver)s'
>> homepage = 'http://jupyter.org'
>> description = """JupyterHub is a multiuser version of the Jupyter
>> (IPython) notebook designed for
>> centralized deployments in companies, university classrooms and
>> research labs."""
>> toolchain = {'name': 'intel', 'version': '2016a'}
>> dependencies = [
>> ('Python', '3.5.1'),
>> ('IPython', '5.1.0', versionsuffix),
>> ('configurable-http-proxy', '1.3.0', '-nodejs-4.4.7'),
>> ]
>> # this is a bundle of Python packages
>> exts_defaultclass = 'PythonPackage'
>> exts_list = [
>> ('SQLAlchemy', '1.0.13', {
>> 'source_urls':
>> ['https://pypi.python.org/packages/source/S/SQLAlchemy/'],
>> }),
>> ('requests', '2.10.0', {
>> 'source_urls':
>> ['https://pypi.python.org/packages/source/r/requests/'],
>> }),
>> ('pamela', '0.2.1', {
>> 'source_urls':
>> ['https://pypi.python.org/packages/source/p/pamela/'],
>> }),
>> ('jupyterhub', version, {
>> 'source_urls':
>> ['https://pypi.python.org/packages/source/j/jupyterhub/'],
>> }),
>> ]
>> modextrapaths = {'PYTHONPATH':
>> ['lib/python%(pyshortver)s/site-packages']}
>> sanity_check_paths = {
>> 'files': ['bin/jupyterhub'],
>> 'dirs': ['lib/python%(pyshortver)s/site-packages/jupyterhub'],
>> }
>> sanity_check_commands = ['jupyterhub --help']
>> moduleclass = 'tools'
>>
>> However, I get the error
>>
>> == 2016-12-27 19:07:51,707 build_log.py:147 ERROR EasyBuild crashed with
>> an error (at ?:124 in __init__): cmd "
>> /home/eb/software/Python/3.5.1-intel-2016a/bin/python setup.py install
>> --prefix=/home/eb/software/jupyterhub/0.7.0-intel-2016a-Python-3.5.1 "
>> exited with exitcode 1 and output:
>> running install
>> running bdist_egg
>> Aborting implicit building of eggs. Use `pip install .` to install from
>> source.
>> (at easybuild/tools/run.py:438 in parse_cmd_output)
>> == 2016-12-27 19:07:51,707 easyblock.py:2499 WARNING build failed (first
>> 300 chars): cmd " /home/eb/software/Python/3.5.1-intel-2016a/bin/python
>> setup.py install
>> --prefix=/home/eb/software/jupyterhub/0.7.0-intel-2016a-Python-3.5.1 "
>> exited with exitcode 1 and output:
>> running install
>> running bdist_egg
>> Aborting implicit building of eggs. Use `pip install .` to install from
>> source.
>>
>> Any idea how I can build this extension with EasyBuild? The dirty way
>> might be to do a postinstallcmd using pip, but that would not be nice
>> ...
>
> Can you including "use_pip = True" in the easyconfig (outside of exts_list)?
>
> That will instruct the PythonPackage easyblock to install each of the
> extensions using "pip install" rather than "python setup.py install".
>
> If that works, please contribute back the easyconfig file, since not
> using pip worked fine before for jupyterhub (see
> https://github.com/hpcugent/easybuild-easyconfigs/pull/3380/files#diff-413ed864a21f21af061b5140b30f3643).
>
>
> regards,
>
> Kenneth
--
Dr. Andreas Hilboll
Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)
University of Bremen
NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
[easybuild] How to install .whl from pypi
Hi again,
I'm trying to install bash_kernel version 0.5 from pypi, but there's
only a .whl file, no tgz source file. I'm having trouble using this in
the exts_list; when I use
exts_list = [
('bash_kernel', '0.5', {
'source_urls':
['https://pypi.python.org/packages/b8/c4/dd0bb29461401a89dcf7f6ffed97c6afadb95a3734b1cf3b52c8a2ac0808/'],
'source_tmpl': 'bash_kernel-0.5-py2.py3-none-any.whl'
}),
]
I get the error
== 2016-12-27 19:16:10,454 build_log.py:147 ERROR EasyBuild crashed with an
error (at ?:124 in __init__): build failed (first 300 chars): Unknown file type
for file bash_kernel-0.5-py2.py3-none-any.whl (at easybuild/main.py:118 in
build_and_install_software)
How can I tell EasyBuild that I want to install this extension from the
Wheel?
Cheers,
Andreas.
--
Dr. Andreas Hilboll
Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)
University of Bremen
NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Python extension installation using pip (was: Python extensions from .whl)
Sorry, the subject line was wrong ...
Andreas Hilboll writes:
> Hi all,
>
> I'm trying to create an easyconfig for jupyterhub 0.7. I copied the
> existing 0.6.1 easyconfig and bumped the version number, so it now reads
>
>easyblock = 'Bundle'
>name = 'jupyterhub'
>version = '0.7.0'
>versionsuffix = '-Python-%(pyver)s'
>homepage = 'http://jupyter.org'
>description = """JupyterHub is a multiuser version of the Jupyter
> (IPython) notebook designed for
> centralized deployments in companies, university classrooms and research
> labs."""
>toolchain = {'name': 'intel', 'version': '2016a'}
>dependencies = [
>('Python', '3.5.1'),
>('IPython', '5.1.0', versionsuffix),
>('configurable-http-proxy', '1.3.0', '-nodejs-4.4.7'),
>]
># this is a bundle of Python packages
>exts_defaultclass = 'PythonPackage'
>exts_list = [
>('SQLAlchemy', '1.0.13', {
>'source_urls':
> ['https://pypi.python.org/packages/source/S/SQLAlchemy/'],
>}),
>('requests', '2.10.0', {
>'source_urls':
> ['https://pypi.python.org/packages/source/r/requests/'],
>}),
>('pamela', '0.2.1', {
>'source_urls':
> ['https://pypi.python.org/packages/source/p/pamela/'],
>}),
>('jupyterhub', version, {
>'source_urls':
> ['https://pypi.python.org/packages/source/j/jupyterhub/'],
>}),
>]
>modextrapaths = {'PYTHONPATH': ['lib/python%(pyshortver)s/site-packages']}
>sanity_check_paths = {
>'files': ['bin/jupyterhub'],
>'dirs': ['lib/python%(pyshortver)s/site-packages/jupyterhub'],
>}
>sanity_check_commands = ['jupyterhub --help']
>moduleclass = 'tools'
>
> However, I get the error
>
>== 2016-12-27 19:07:51,707 build_log.py:147 ERROR EasyBuild crashed with
> an error (at ?:124 in __init__): cmd "
> /home/eb/software/Python/3.5.1-intel-2016a/bin/python setup.py install
> --prefix=/home/eb/software/jupyterhub/0.7.0-intel-2016a-Python-3.5.1 " exited
> with exitcode 1 and output:
>running install
>running bdist_egg
>Aborting implicit building of eggs. Use `pip install .` to install from
> source.
> (at easybuild/tools/run.py:438 in parse_cmd_output)
>== 2016-12-27 19:07:51,707 easyblock.py:2499 WARNING build failed (first
> 300 chars): cmd " /home/eb/software/Python/3.5.1-intel-2016a/bin/python
> setup.py install
> --prefix=/home/eb/software/jupyterhub/0.7.0-intel-2016a-Python-3.5.1 " exited
> with exitcode 1 and output:
>running install
>running bdist_egg
>Aborting implicit building of eggs. Use `pip install .` to install from
> source.
>
> Any idea how I can build this extension with EasyBuild? The dirty way
> might be to do a postinstallcmd using pip, but that would not be nice
> ...
>
> Cheers,
> Andreas.
--
Dr. Andreas Hilboll
Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)
University of Bremen
NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
[easybuild] Python extensions from .whl
Hi all,
I'm trying to create an easyconfig for jupyterhub 0.7. I copied the
existing 0.6.1 easyconfig and bumped the version number, so it now reads
easyblock = 'Bundle'
name = 'jupyterhub'
version = '0.7.0'
versionsuffix = '-Python-%(pyver)s'
homepage = 'http://jupyter.org'
description = """JupyterHub is a multiuser version of the Jupyter (IPython)
notebook designed for
centralized deployments in companies, university classrooms and research
labs."""
toolchain = {'name': 'intel', 'version': '2016a'}
dependencies = [
('Python', '3.5.1'),
('IPython', '5.1.0', versionsuffix),
('configurable-http-proxy', '1.3.0', '-nodejs-4.4.7'),
]
# this is a bundle of Python packages
exts_defaultclass = 'PythonPackage'
exts_list = [
('SQLAlchemy', '1.0.13', {
'source_urls':
['https://pypi.python.org/packages/source/S/SQLAlchemy/'],
}),
('requests', '2.10.0', {
'source_urls':
['https://pypi.python.org/packages/source/r/requests/'],
}),
('pamela', '0.2.1', {
'source_urls': ['https://pypi.python.org/packages/source/p/pamela/'],
}),
('jupyterhub', version, {
'source_urls':
['https://pypi.python.org/packages/source/j/jupyterhub/'],
}),
]
modextrapaths = {'PYTHONPATH': ['lib/python%(pyshortver)s/site-packages']}
sanity_check_paths = {
'files': ['bin/jupyterhub'],
'dirs': ['lib/python%(pyshortver)s/site-packages/jupyterhub'],
}
sanity_check_commands = ['jupyterhub --help']
moduleclass = 'tools'
However, I get the error
== 2016-12-27 19:07:51,707 build_log.py:147 ERROR EasyBuild crashed with an
error (at ?:124 in __init__): cmd "
/home/eb/software/Python/3.5.1-intel-2016a/bin/python setup.py install
--prefix=/home/eb/software/jupyterhub/0.7.0-intel-2016a-Python-3.5.1 " exited
with exitcode 1 and output:
running install
running bdist_egg
Aborting implicit building of eggs. Use `pip install .` to install from
source.
(at easybuild/tools/run.py:438 in parse_cmd_output)
== 2016-12-27 19:07:51,707 easyblock.py:2499 WARNING build failed (first 300
chars): cmd " /home/eb/software/Python/3.5.1-intel-2016a/bin/python setup.py
install --prefix=/home/eb/software/jupyterhub/0.7.0-intel-2016a-Python-3.5.1 "
exited with exitcode 1 and output:
running install
running bdist_egg
Aborting implicit building of eggs. Use `pip install .` to install from
source.
Any idea how I can build this extension with EasyBuild? The dirty way
might be to do a postinstallcmd using pip, but that would not be nice
...
Cheers,
Andreas.
--
Dr. Andreas Hilboll
Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)
University of Bremen
NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] How to specify the source file name with URL parameters for Python extensions
Hi Kenneth,
Kenneth Hoste writes:
> Hi Andreas,
>
> On 22/12/16 23:14, Andreas Hilboll wrote:
>> Hi,
>>
>> I'm trying to add the ecmwf-api-client
>> (https://software.ecmwf.int/wiki/display/WEBAPI/Downloads) to my custom
>> Python easyconfig. I currently have the following in the `exts_list`:
>>
>> ('ecmwf-api-client', '1.4.1', {
>> 'modulename': 'ecmwfapi',
>> 'source_urls':
>> ['https://software.ecmwf.int/wiki/download/attachments/56664858/'],
>> 'sources':
>> ['ecmwf-api-client-python.tgz?version=3=1469183826340=v2'],
>> }),
>>
>> The problem is that EasyBuild still looks for a source file
>> ecmwf-api-client-1.4.1.tgz:
>>
>> Couldn't find file ecmwf-api-client-1.4.1.tar.gz anywhere, and
>> downloading it didn't work either... Paths attempted (in order): [...]
>> https://software.ecmwf.int/wiki/download/attachments/56664858/ecmwf-api-client-1.4.1.tar.gz
>>
>> So there's two things here:
>>
>> 1. EasyBuild inserts the version number into the file name even though
>> I'm specifying another name
>> 2. EasyBuild doesn't append the URL parameters to the URL
>>
>> Of course I can manually download the file, rename it and place it into
>> a location where EasyBuild finds it, but maybe there's a cleaner
>> solution?
>
> Things are a bit different for extensions, which is indeed confusing.
> Fixing that is on our TODO list (see
> https://github.com/hpcugent/easybuild-framework/issues/991), but it
> hasn't happened yet...
>
> EasyBuild uses the value for 'source_tmpl' for extensions to determine
> the name of the source tarball, and it uses
> '%(name)s-%(version)s.tar.gz' as default value.
>
> If downloading a tarball requires stuff after the tarball name, you
> should use a tuple (..., ...) in 'source_urls', rather than just a
> string (that's the same for non-extensions).
>
> So, try this:
>
> ('ecmwf-api-client', '1.4.1', {
> 'modulename': 'ecmwfapi',
> 'source_urls':
> [('https://software.ecmwf.int/wiki/download/attachments/56664858/',
> '?version=3=1469183826340=v2')],
> 'source_tmpl': ['ecmwf-api-client-python.tgz'],
> }),
Thanks so much for your super-quick answer! Unfortunately, I have two
issues with it:
1. the `source_tmpl` complains when it is a list of strings; after
removing the brackets, i.e. when passing only the string
'source_tmpl': 'ecmwf-api-client-python.tgz',
it works.
2. When downloading, EB inserts a `/` between the filename and the URL
extension, according to the logs:
https://software.ecmwf.int/wiki/download/attachments/56664858/ecmwf-api-client-python.tgz/?version=3=1469183826340=v2
and that leads to a 404 error.
If this is a bug in EB, should I open an issue?
Cheers,
Andreas.
PS: I'm now using
('ecmwf-api-client', '1.4.1', {
'modulename': 'ecmwfapi',
'source_urls':
[('https://software.ecmwf.int/wiki/download/attachments/56664858',
'?version=3=1469183826340=v2')],
'source_tmpl': 'ecmwf-api-client-python.tgz',
}),
--
Dr. Andreas Hilboll
Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)
University of Bremen
NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Setting environment variables
Thanks, Gizo, that helped =) nan...@luis.uni-hannover.de writes: > Hi, > > you can use any of the configopts, prebuildopts or similar EB variables to > set env variables you need > > Eg. > > buildopts = 'CXXFLAGS="$CXXFLAGS -DMPICH_IGNORE_CXX_SEEK"' > > There are a lot of ready for use configs that you could just grep under your > EB installation > ~/lib/python2.6/site-packages/easybuild_easyconfigs-2.8.0-py2.6.egg/easybuild/easyconfigs > > Cheers, > Gizo > > On Friday, June 24, 2016 15:34 CEST, Andreas Hilboll <hilb...@uni-bremen.de> > wrote: > >> Hi, >> >> I'm trying to create an easyconfig for UDUNITS1 (I know, old and >> deprecated, but still needed unfortunately), and am running into the >> problem that I need to pass some environment variables to the build >> environment (i.e., set CPPFLAGS to a custom value). I'm sure this is >> possible somehow, but I couldn't find any reference to this in >> http://easybuild.readthedocs.io/en/latest/eb_a.html. Any pointers would >> be greatly appreciated ... >> >> Cheers, >> Andreas. >> >> -- >> Dr. Andreas Hilboll >> >> Center for Marine Environmental Sciences (MARUM) >> - AND - >> Institute of Environmental Physics (IUP) >> >> University of Bremen >> >> NW1 / S3132 >> Otto-Hahn-Allee 1 >> D-28359 Bremen >> Germany >> >> +49(0)421 218 62133 (phone) >> +49(0)421 218 98 62133 (fax) >> http://www.iup.uni-bremen.de/~hilboll > > > > -- -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] How to name easyconfigs for alternative build options
Ward Poelmans writes: > On 01-07-16 10:53, Andreas Hilboll wrote: >> >> How should I go about naming the easyconfig, so that in the end I have >> two distinct module files, one with 'nonetcdf' somewhere in its name? > > The current approach is to use a versionsuffix for this. In the > `nonetcdf` you simple add: > versionsuffix = '-nonetcdf' > and rename the easyconfig with the suffix. Great, thanks a lot, Ward! Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
[easybuild] How to name easyconfigs for alternative build options
Hi, I'm using the HDF4 library and have created a HDF easyconfig for it, using the ConfigureMake easyblock. The problem is that with HDF4, it is common to use it with the configopt --disable-netcdf. I would like to have two easyconfigs, one with the default build, and one with netcdf disabled. How should I go about naming the easyconfig, so that in the end I have two distinct module files, one with 'nonetcdf' somewhere in its name? Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
[easybuild] Setting environment variables
Hi, I'm trying to create an easyconfig for UDUNITS1 (I know, old and deprecated, but still needed unfortunately), and am running into the problem that I need to pass some environment variables to the build environment (i.e., set CPPFLAGS to a custom value). I'm sure this is possible somehow, but I couldn't find any reference to this in http://easybuild.readthedocs.io/en/latest/eb_a.html. Any pointers would be greatly appreciated ... Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen NW1 / S3132 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Extending an easyconfig
>> You did ;) Your addition to the wrf easyblock did the trick for me. At >> least with intel/2016a. I still cannot build WRF with foss, but that's >> another issue, as the existing WRF-3.6/foss also doesn't build on my >> machine. I'll report the issue next week, I guess, but as I have a >> working WRF now, it's not high on my priority list. And since you don't >> seem to have general problems with WRF, I suppose it's somehow related >> to my machine. > > Definitely open an issue on this, other people may also hit the problem > you're seeing... Here you go: https://github.com/hpcugent/easybuild-easyconfigs/issues/2983 Cheers, Andreas -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Extending an easyconfig
Hi Kenneth, Kenneth Hoste writes: > How did you fix the problems you were running into? You did ;) Your addition to the wrf easyblock did the trick for me. At least with intel/2016a. I still cannot build WRF with foss, but that's another issue, as the existing WRF-3.6/foss also doesn't build on my machine. I'll report the issue next week, I guess, but as I have a working WRF now, it's not high on my priority list. And since you don't seem to have general problems with WRF, I suppose it's somehow related to my machine. >>Now I want to >> run WRF-Chem, and if I understand the documentation correctly, this is >> done by >> >> a) extracting an additional source tarball into the WRFV3 directory >> b) setting some environment variables telling WRF to include the >> chemistry code >> >> Does EasyBuild foresee a mechanism to extend easyconfigs, so that I can >> build on the existing WRF easyconfig? Or do I have to make a seperate >> WRF-Chem easyconfig by copy-pasting and amending the existing WRF? > > There's no "include" or "extend" mechanism for easyconfigs, no. I like > the idea though... > > We've discussed something like this in the past (cfr. > https://github.com/hpcugent/easybuild-framework/issues/544), but what > you're asking here is sort of different. > > So, you should either create a WRF-Chem easyconfig (and maybe easyblock > too, depends), or just enhance the existing WRF easyconfig/easyblock to > include WRF-Chem. > > I'm not sure if the latter would actually make any sense though... Actually, I think it would. I'll look into extending the existing wrf easyblock, and introduce a boolean config value `build_chemistry` or the like in the easyconfig, defaulting to `False`. Have a nice week-end, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
[easybuild] Extending an easyconfig
Hi easybuilders, so there's the WRF easyconfig, which works fine for me. Now I want to run WRF-Chem, and if I understand the documentation correctly, this is done by a) extracting an additional source tarball into the WRFV3 directory b) setting some environment variables telling WRF to include the chemistry code Does EasyBuild foresee a mechanism to extend easyconfigs, so that I can build on the existing WRF easyconfig? Or do I have to make a seperate WRF-Chem easyconfig by copy-pasting and amending the existing WRF? Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Supporting WRF 3.8
Hi Kenneth, Kenneth Hoste writes: > Hi Andreas, > > On 20/04/16 20:44, Andreas Hilboll wrote: >> Okay, it seems I have the regexp now. Now I need to fiddle with the >> patches ... but that should be okay. I'll submit a PR once I'm done. > > Any luck? I guess so. I still have trouble getting the tests to run properly, but that doesn't seem to be related to the easyblock. I'd appreciate if someone could test this: https://github.com/hpcugent/easybuild-easyblocks/pull/902 Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Packaging of Intel compilers
Thanks, guys, turns out the existing easyconfigs for impi and imkl seem to work just fine (i.e., build without problems) when I just hard-code the Parallel Studio tgz filename in the easyconfig files. I'll report back in case I run into problems, but this seems to work just fine for me. Cheers, Andreas. Kenneth Hoste writes: > Hi Andreas, > > You should be able to download the separate tarballs for impi and imkl > from the Intel website. > Just the RPMs is not enough for what's in place in EasyBuild, since you > need to install script too provided by Intel... > > There's currently no supported to install the whole Parallel Studio > suite in one go, although that could be looked into... > > > regards, > > Kenneth > > On 22/04/16 14:20, Andreas Hilboll wrote: >> Hi EasyBuilders, >> >> I'm trying to get the Intel toolchains working on my system. However, >> there seems to be something off with the packaging. What I have is .tgz >> files parallel_studio_xe_2016.tgz, parallel_studio_xe_2016_update1.tgz, >> parallel_studio_xe_2016_update2.tgz. >> >> I could build the icc and ifort and iccifort easyconfigs from these >> files, but do have problems with impi and imkl, as the .tgz files I have >> don't include them as .tgz file, as seems to be required by the impi and >> imkl easyconfigs. However, these .tgz files do contain RPM files, e.g., >> >> - intel-mpi-doc-5.1.3-181.x86_64.rpm >> - intel-mpirt-l-ps-181-16.0.2-181.i486.rpm >> - intel-mpi-sdk-mic-181-5.1.3-181.x86_64.rpm >> - intel-mpi-psxe-062-5.1.3-062.x86_64.rpm >> - intel-mpi-rt-mic-181-5.1.3-181.x86_64.rpm >> - intel-mpi-rt-core-181-5.1.3-181.x86_64.rpm >> - intel-mpi-sdk-core-181-5.1.3-181.x86_64.rpm >> >> - intel-mkl-181-11.3.2-181.x86_64.rpm >> - intel-mkl-common-181-11.3.2-181.noarch.rpm >> - intel-mkl-devel-181-11.3.2-181.i486.rpm >> - intel-mkl-devel-181-11.3.2-181.x86_64.rpm >> >> I'd appreciate some guidance in building the impi and imkl easyconfigs >> from these rpm files. Any ideas? >> >> Cheers, >>Andreas. >> >> >> -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Supporting WRF 3.8
>> How would the decision which easyblock version to use be done properly?
>> I'd somehow have to add support for WRF >= 3.8 without taking away
>> support for earlier versions. Would that go via subclassing the EB_WRF
>> class? If so, how would I ensure that EasyBuild automatically picks the
>> correct easyblock depending on the version?
>
> You would insert a check like:
>
> if LooseVersion(self.version) >= LooseVersion('3.8'):
> # new regexp
> else:
> # old regexp
>>
>> But maybe I should start working on the right regexp ... ;)
>
> Let me know if you nee help with this, because these can be daunting...
>
> The key is to include a 'named group' in the regex with the label 'nr',
> which will indicate the actual number that would be used, as is also the
> case now in the existing value for build_type_question with (?P[0-9]+) .
>
> You'll need to provide new definitions for build_type_option for the
> newer WRF versions, based on the compiler family (see existing code),
> and then construct build_type_question differently using both
> build_type_option and bt (the build type being selected, e.g. dmpar).
>
> Taking a look at the way in which WRF 3.6 and earlier prints the list of
> options would help a great deal probably.
>
> If you need help, maybe you should consider jumping into the #easybuild
> IRC channel on the FreeNode IRC network...
Okay, it seems I have the regexp now. Now I need to fiddle with the
patches ... but that should be okay. I'll submit a PR once I'm done.
Cheers,
Andreas.
--
Dr. Andreas Hilboll
Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)
University of Bremen
U3145
Otto-Hahn-Allee 1
D-28359 Bremen
Germany
+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] How to update easybuild?
Hi Kenneth, I basically copied your email into the Installation.rst file, see my branch at https://github.com/andreas-h/easybuild/tree/updatedoc. I wanted to build the documentation before submitting the PR, just to make sure everything is fine with my additions (my rst is a bit rusty ...), but couldn't find the Sphinxdoc Makefile in the repo. Is there an easy way to build the documentation locally on my machine? Otherwise, I can submit the PR as-is ... Cheers, Andreas. Kenneth Hoste writes: > Hi Andreas, > > On 19/04/16 17:17, Andreas Hilboll wrote: >> Hi easybuild experts, >> >> I'm wondering what is the canonical method to update the easybuild >> installation? I currently have 2.4.0 installed, and it's working well, >> but I'd like to update at least the easyconfigs to 2.7.0. >> >> I cannot find any dedicated information about this in the documentation, >> so I'm asking here ... > > From the release announcements: > > To upgrade to EasyBuild v2.7.0, there are several options: > > (i) (re)bootstrap EasyBuild to obtain an EasyBuild/2.7.0 module to > load [1] > > (ii) install EasyBuild v2.7.0 with a previous version of EasyBuild, > using the easyconfig file available in [2] > > (iii) install EasyBuild v2.7.0 from PyPI, using one of the standard > Python installation tools (easy_install, pip, ...) > > (iv) update your Git working copies of the different EasyBuild > repositories > > [1] > http://easybuild.readthedocs.org/en/latest/Installation.html#bootstrapping-easybuild > [2] https://github.com/hpcugent/easybuild-easyconfigs/pull/2740 > > Let us know if you have any questions on this. > > > regards, > > Kenneth -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] Problems installing: ImportError on pkg_resources
Hi Kenneth, > Is this a recent version of the bootstrap script? What's the MD5sum? > Can you try re-downloading it (although that won't change anything if it > was downloading in the last couple of days). Just to be sure, I re-downloaded using the curl command given in the documentation: curl -O https://raw.githubusercontent.com/hpcugent/easybuild-framework/develop/easybuild/scripts/bootstrap_eb.py > If the problem persists, please provide a full (debug) log of the whole > bootstrap procedure, which you can obtain with: > > EASYBUILD_BOOTSTRAP_DEBUG=1 python bootstrap_eb.py > $EASYBUILD_PREFIX 2>&1 | tee eb.log > > Please upload it somewhere, or mailed a zipped version to me off-list. The problem indeed persists, and I put the debug log here: https://gist.github.com/andreas-h/267a2ea4512c4691ab9ba6048d9c825a Would you prefer me opening an issue for this? If so, against which repository (easybuild or easybuild-framework)? Cheers, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
Re: [easybuild] How to update easybuild?
Thanks, Kenneth, I'm trying to go for your option 1), see my other mail about the problems I'm having there. It would be helpful if the documentation on readthedocs contained a small section "Updating an existing easybuild installation", just giving the options you mention. Would you consider a PR I could prepare to include this in the documentation? Cheers, Andreas. Kenneth Hoste writes: > Hi Andreas, > > On 19/04/16 17:17, Andreas Hilboll wrote: >> Hi easybuild experts, >> >> I'm wondering what is the canonical method to update the easybuild >> installation? I currently have 2.4.0 installed, and it's working well, >> but I'd like to update at least the easyconfigs to 2.7.0. >> >> I cannot find any dedicated information about this in the documentation, >> so I'm asking here ... > > From the release announcements: > > To upgrade to EasyBuild v2.7.0, there are several options: > > (i) (re)bootstrap EasyBuild to obtain an EasyBuild/2.7.0 module to > load [1] > > (ii) install EasyBuild v2.7.0 with a previous version of EasyBuild, > using the easyconfig file available in [2] > > (iii) install EasyBuild v2.7.0 from PyPI, using one of the standard > Python installation tools (easy_install, pip, ...) > > (iv) update your Git working copies of the different EasyBuild > repositories > > [1] > http://easybuild.readthedocs.org/en/latest/Installation.html#bootstrapping-easybuild > [2] https://github.com/hpcugent/easybuild-easyconfigs/pull/2740 > > Let us know if you have any questions on this. > > > regards, > > Kenneth -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
[easybuild] Problems installing: ImportError on pkg_resources
Hi again, I'm having a problem with the bootstrapping procedure for easybuild 2.7.0: hilboll@heron:~> python --version Python 2.7.9 hilboll@heron:~> python bootstrap_eb.py $EASYBUILD_PREFIX [...] [[ERROR]] Stage 1 failed, could not determine EasyBuild version (txt: Traceback (most recent call last): File "", line 1, in File "/tmpx/jtmp.hilboll/tmpWybQAn/eb_stage1/lib/python2.7/site-packages/easybuild_easyblocks-2.7.0-py2.7.egg/easybuild/__init__.py", line 31, in import pkg_resources ImportError: No module named pkg_resources ). I thought that the bootstrap process makes sure that setuptools are installed ... and, alas, it seems to be there: hilboll@heron:~> ls /tmpx/jtmp.hilboll/tmpbx8cTB/lib64/python2.7/site-packages/ hilboll@heron:~> ls /tmpx/jtmp.hilboll/tmpbx8cTB/lib/python2.7/site-packages/ distribute-0.6.49-py2.7.egg easy-install.pth setuptools-0.6c11-py2.7.egg-info setuptools.pth site.py site.pyc Or is it an issue that the packages are in the lib folder while my Python might expect them to be in lib64? Confused greetings, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
[easybuild] How to update easybuild?
Hi easybuild experts, I'm wondering what is the canonical method to update the easybuild installation? I currently have 2.4.0 installed, and it's working well, but I'd like to update at least the easyconfigs to 2.7.0. I cannot find any dedicated information about this in the documentation, so I'm asking here ... Thanks for your helping me use this great software, Andreas. -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 (phone) +49(0)421 218 98 62133 (fax) http://www.iup.uni-bremen.de/~hilboll
