If you rely on the easyconfigs that ship with easybuild, you can get reproducible results by sticking to the same easybuild version.
PRs in easyconfigs have been accepted in the past, if they fixed bugs or enhanced functionality, and it should be like that IMO. That said, I don’t use OpenBLAS, so I have no strong feeling in either direction for this particular case. Cheers, Damian From: <easybuild-requ...@lists.ugent.be> on behalf of "Dey, John F" <jf...@fredhutch.org> Reply-To: "easybuild@lists.ugent.be" <easybuild@lists.ugent.be> Date: Tuesday, 28. May 2019 at 17:26 To: "easybuild@lists.ugent.be" <easybuild@lists.ugent.be> Subject: Re: [easybuild] Openblas(foss) matrix issue If this fix was submitted as a PR for OpenBLAS 0.3.1 which is part of the foss-2018b toolchain would it be accepted? One of the main purposes of EasyBuild is to facilitate reproducible science. Changing a published easyconfig, is breaking the rules. There is a huge amount of flawed software that we need to deal with, but the OpenBLAS issue will affect analysis results. Rebuilding OpenBLAS in place should not create any issues; I assume that nothing is statically linked to OpenBLAS. Moving to 2019a is not an option for me at this time, -- John Dey HPC Operations Scientific Computing O 206.667.4308<tel:(206)%20667-4308> M 360.649.2731<tel:(360)%20649-2731> E jf...@fredhutch.org<mailto:jf...@fredhutch.org> [signature_639330222] Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N<https://maps.google.com/?q=1100+Fairview+Ave.+N&entry=gmail&source=g>., Mail Stop J3-516 Seattle, WA 98109 fredhutch.org<http://www.fredhutch.org/> From: <easybuild-requ...@lists.ugent.be> on behalf of Carlos Fenoy <fenoym...@gmail.com> Reply-To: "easybuild@lists.ugent.be" <easybuild@lists.ugent.be> Date: Tuesday, May 28, 2019 at 5:32 AM To: easybuild <easybuild@lists.ugent.be> Subject: Re: [easybuild] Openblas(foss) matrix issue Hi, After fighting a long time with this, we managed to get a solution that passes both the "Openblas_matrix_issue" and "BLAS_tester" test suites. To solve the issue we had to apply a patch and add a new build parameter (USE_SIMPLE_THREADED_LEVEL3=1) to OpenBLAS to make it work with multiple openmp threads. This is how the buildopts line looks like for us: buildopts = ' USE_SIMPLE_THREADED_LEVEL3=1 BINARY=64 USE_THREAD=1 USE_OPENMP=1 CC="$CC" FC="$F77" DYNAMIC_ARCH=1' And the patch, we got it from this commit on the OpenBLAS repo: https://github.com/xianyi/OpenBLAS/commit/b14f44d2adbe1ec8ede0cdf06fb8b09f3c4b6e43<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xianyi_OpenBLAS_commit_b14f44d2adbe1ec8ede0cdf06fb8b09f3c4b6e43&d=DwMFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=3TR-iteG1SyRqQ5yubQg-_2KIAToz9bj5dZrRdW36Hc&m=7KHaWVsIxKa5sDpJjJICIsYxR8q-MepJUDMgU7AA-3o&s=OablXo14euTScZg4SKMDgHQje2Ush-zVGAbHbOnmfYY&e=> (you can get the patch by adding .patch at the end of the URL) Regards, Carlos On Mon, May 27, 2019 at 6:15 PM Pablo Escobar Lopez <pablo.escobarlo...@unibas.ch<mailto:pablo.escobarlo...@unibas.ch>> wrote: Hi, did anyone found a working patch or workaround for the matrix issue when using OpenBLAS-0.3.1 ? After a lot of try&error I couldn't pass the tests in https://github.com/eylenth/Openblas_matrix_issue<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eylenth_Openblas-5Fmatrix-5Fissue&d=DwMFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=3TR-iteG1SyRqQ5yubQg-_2KIAToz9bj5dZrRdW36Hc&m=7KHaWVsIxKa5sDpJjJICIsYxR8q-MepJUDMgU7AA-3o&s=TExoymruAD12Fab-0XkLZdeeTtYMhJwNUWSMer8Pf4Y&e=> when using https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/o/OpenBLAS/OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_easybuilders_easybuild-2Deasyconfigs_blob_master_easybuild_easyconfigs_o_OpenBLAS_OpenBLAS-2D0.3.1-2DGCC-2D7.3.0-2D2.30.eb&d=DwMFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=3TR-iteG1SyRqQ5yubQg-_2KIAToz9bj5dZrRdW36Hc&m=7KHaWVsIxKa5sDpJjJICIsYxR8q-MepJUDMgU7AA-3o&s=faT0z2Z2fXzk3DW3aSoXMe1TUc86cGIhPds_d8XHUFg&e=> . No matter what patches, toolchainopts or buildopts I use (and I have tried few different combinations) . Is anyone able to pass the tests using openblas-0.3.1 ? I could pass the tests using openblas-0.3.5 but upgrading my foss/2018b toolchain would be quite messy because I use RPATH. The less intrusive solution for my users would be to be able to patch openblas-0.3.1 somehow but I couldn't find a working solution. Any suggestions? regards, Pablo. p.s. in a related topic, IMHO unless there is a proper workaround I would suggest to stop providing openblas-0.3.1 with easybuild. Right now we are distributing a broken library On Tue, May 7, 2019 at 6:34 PM Mikael Öhman <micket...@gmail.com<mailto:micket...@gmail.com>> wrote: Hi Thomas, I can also confirm these issues. I tried rebuilding OpenBLAS+R after the fix in #7180, but I still saw the same problems. Very large matrix-matrix multiplications randomly gave the wrong result. Very large errors. The larger the matrix, the more frequent the errors. In the end, I compiled an intel-version (but I had to remove a few extensions that didn't build) and removed my Foss version from our installations. Perhaps it's related to hardware; I saw this on happen skylake servers. I haven't had time to check if this https://github.com/easybuilders/easybuild-easyconfigs/issues/8197<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_easybuilders_easybuild-2Deasyconfigs_issues_8197&d=DwMFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=3TR-iteG1SyRqQ5yubQg-_2KIAToz9bj5dZrRdW36Hc&m=7KHaWVsIxKa5sDpJjJICIsYxR8q-MepJUDMgU7AA-3o&s=RMROVzKOqcVBnJr-PzJIYIB3wcJGjeHZfx07zOfk29Y&e=> also affects 0.3.1 Best regards, Mikael On Tue, May 7, 2019 at 6:12 PM Thomas Eylenbosch <thomas.eylenbosch....@agro.basf-se.com<mailto:thomas.eylenbosch....@agro.basf-se.com>> wrote: Hello Some of our end users reported a calculation issue with matrices when they are working with a foss/2018b module I reproduced this error with Python and R that are compiled with the foss/2018b toolchain, the output returns unexcepted results. Then I reproduced this error with Python and R that are compiled with the foss/2016b toolchain , then it gives me the expected behavior. You can reproduce this error with the following github repository: https://github.com/eylenth/Openblas_matrix_issue<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eylenth_Openblas-5Fmatrix-5Fissue&d=DwMFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=3TR-iteG1SyRqQ5yubQg-_2KIAToz9bj5dZrRdW36Hc&m=7KHaWVsIxKa5sDpJjJICIsYxR8q-MepJUDMgU7AA-3o&s=TExoymruAD12Fab-0XkLZdeeTtYMhJwNUWSMer8Pf4Y&e=> I have also tried to recompile the OpenBLAS-0.3.1-GCC-7.3.0-2.30.eb easyconfig file with “toolchainopts = {'vectorize': False}” ( cfr. https://github.com/easybuilders/easybuild-easyconfigs/issues/7180<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_easybuilders_easybuild-2Deasyconfigs_issues_7180&d=DwMFaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=3TR-iteG1SyRqQ5yubQg-_2KIAToz9bj5dZrRdW36Hc&m=7KHaWVsIxKa5sDpJjJICIsYxR8q-MepJUDMgU7AA-3o&s=lNOZuvwF_eL4Al8AIZrEB-s9R4-uGoEpDhpcI6syET4&e=>) But is still giving me unexpected behavior Can someone try to reproduce the error with the R/Python(foss/2018b) modules. Or can someone give me feedback on this? Thank you in advance. Met vriendelijke groet / Kind regards / Beste Grüße Thomas Eylenbosch Ext: Gluo N.V. BASF Agricultural Solutions Belgium NV Technologiepark 101 B-9052 Ghent (Zwijnaarde) BELGIUM E-mail: thomas.eylenbosch....@agro.basf-se.com<mailto:thomas.eylenbosch....@agro.basf-se.com> [cid:image001.jpg@01D42B1E.FC3C98F0] BASF Agricultural Solutions Belgium NV, Registered Office: 9052 Gent, Belgium Registration: RPR Gent: 0685.756.742 -- Pablo Escobar López Linux/HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------
