Re: [petsc-dev] PETSc optimization

[Satish Balay via petsc-dev]

Well in spack world - the idea is to get flags from spack and use it in petsc 
build. Its possible that there are issues in this implementation.

Something spack does is - it internally adds in flags (to the compiler - via 
its compiler wrapper) - that petsc configure doesn't see.


I see 'spack install cflags=-O3' is working - but not spack install cflags='-O3 
-g' [something to debug]

[there is also the alternative of adding flags to compiler spec file]

The other issue is mapping cflags from spack to CFLAGS/COPTFLAGS - in some 
cases petsc build requires some default CLFAGS to work - and overriding them 
from spack might cause issues. [again this part is not properly handed]..

Satish

On Tue, 8 Aug 2023, Liu Wei   AWE via petsc-dev wrote:

> Hi all
> 
> I am currently building a large software stack using Spack with PETSc 3.19 as 
> part of the dependency library.
> 
> Spotted the following message during the build process
> 
> Using default optimization C flags "-g -O". You might consider manually 
> setting optimal optimization flags for your system with
> COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples
> 
> Previously when we install PETSc manually, optimisation flags are enforced 
> via configure script e.g.
> COPTFLAGS="-g -O3 -march=native"
> CXXOPTFLAGS="-g -O3 -march=native"
> ...
> (or "-g -O3 -xhost" for intel compiler)
> 
> Whilst spack spec syntax allows compiler parameters via cflags/cxxflags/fflags
> https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/petsc/package.py
> the requirement is not obvious, especially in a long build process.
> 
> The question is should optimisation be amended at spack recipe stage or 
> should PETSc configure enforce some compiler optimisation (if 
> -with-debugging=0 is set) ?
> 
> Regards
> 
> Dr Wei Liu
> High Performance Computing
> To chat with me on Teams click 
> here
> T: +44 01189 856209
> M: wei@awe.co.uk
> AWE Aldermaston,
> Reading, Berkshire, RG7 4PR
> 
> The information in this email and in any attachment(s) is commercial in 
> confidence. If you are not the named addressee(s) or if you receive this 
> email in error then any distribution, copying or use of this communication or 
> the information in it is strictly prohibited. Please notify us immediately by 
> email at admin.internet(at)awe.co.uk, and then delete this message from your 
> computer. While attachments are virus checked, AWE plc does not accept any 
> liability in respect of any virus which is not detected. AWE Plc Registered 
> in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 
> 4PR
> 

Re: [petsc-dev] PETSc optimization

[Jacob Faibussowitsch]

>  The question is should optimisation be amended at spack recipe stage or 
> should PETSc configure enforce some compiler optimisation (if 
> –with-debugging=0 is set) ?

This sounds reasonable, however there are pitfalls to automatically enabling 
optimization.

The naive approach would be to check if `--with-debugging=0` is set, and if so, 
append `-O3 -march=native -mtune=native` (assuming those work with the 
compiler). This might work well enough for GCC/Clang (and probably MSVC, but 
who knows), but not all compilers play nice. For example:

- Intel compilers (before the clang transition) automatically (and silently!) 
enable the equivalent of `-ffast-math` on any optimization level greater than 
base.
- PGI compilers have had some long running bugs when optimization has been 
enabled. For example, there has been a bug with 
__attribute__((visibility(“hidden”))) (conversely, “default” if using 
`-fvisibility=hidden`) variables, where their order in the source file (i.e. is 
variable A literally defined before or after variable B) causes their 
definition to be lost at link-time leading to “undefined reference to Foo” 
errors.


All this is to say, configure now has to keep track of, and add special rules 
for all these compilers (and versions). Since there is no clear answer we have 
found it’s easier to punt responsibility to the user.

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)

> On Aug 8, 2023, at 12:35, Liu Wei AWE via petsc-dev  
> wrote:
> 
> Hi all
>  I am currently building a large software stack using Spack with PETSc 3.19 
> as part of the dependency library.
>  Spotted the following message during the build process
>  Using default optimization C flags “-g -O”. You might consider manually 
> setting optimal optimization flags for your system with
> COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples
>  Previously when we install PETSc manually, optimisation flags are enforced 
> via configure script e.g.
> COPTFLAGS=”-g -O3 -march=native”
> CXXOPTFLAGS=”-g -O3 -march=native”
> …
> (or “-g -O3 -xhost” for intel compiler)
>  Whilst spack spec syntax allows compiler parameters via 
> cflags/cxxflags/fflags  
> https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/petsc/package.py
> the requirement is not obvious, especially in a long build process.
>  The question is should optimisation be amended at spack recipe stage or 
> should PETSc configure enforce some compiler optimisation (if 
> –with-debugging=0 is set) ?
>  Regards
>  Dr Wei Liu
> High Performance Computing
> To chat with me on Teams click here
> T: +44 01189 856209
> M: wei@awe.co.uk
> AWE Aldermaston,
> Reading, Berkshire, RG7 4PR
>  The information in this email and in any attachment(s) is commercial in 
> confidence. If you are not the named addressee(s) or if you receive this 
> email in error then any distribution, copying or use of this communication or 
> the information in it is strictly prohibited. Please notify us immediately by 
> email at admin.internet(at)awe.co.uk, and then delete this message from your 
> computer. While attachments are virus checked, AWE plc does not accept any 
> liability in respect of any virus which is not detected. AWE Plc Registered 
> in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 
> 4PR

Re: [petsc-dev] PETSc optimization

[Barry Smith]

  The space of "best" choices for optimization flags for PETSc is huge; each 
OS, each compiler, and each particular piece of hardware may have different 
"best" choices, in addition, the previous choice may not even allow compiling 
with any small change in any of these things. Finally particular choices may 
not be compatible with choices may be other packages being linked with PETSc. 
Hence PETSc's configure punts on trying to select a good choice (and just uses 
-O).  Surely this is an issue with most portable open source packages not being 
able to select "good" optimization flags in a portable way?

  Does the Spack system contain any infrastructure to help solve this common 
problem? That ensures all the packages it builds have good compatible 
optimization flags for the particular system?

  Barry

Perhaps the configure message you mention printed by PETSc is inappropriate for 
Spack builds of PETSc and we should print something different to help guide the 
user to the correct Spack solution to the issue?

   

  



> On Aug 8, 2023, at 12:35 PM, Liu Wei AWE via petsc-dev 
>  wrote:
> 
> Hi all
>  
> I am currently building a large software stack using Spack with PETSc 3.19 as 
> part of the dependency library.
>  
> Spotted the following message during the build process
>  
> Using default optimization C flags “-g -O”. You might consider manually 
> setting optimal optimization flags for your system with
> COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples
>  
> Previously when we install PETSc manually, optimisation flags are enforced 
> via configure script e.g.
> COPTFLAGS=”-g -O3 -march=native”
> CXXOPTFLAGS=”-g -O3 -march=native”
> …
> (or “-g -O3 -xhost” for intel compiler)
>  
> Whilst spack spec syntax allows compiler parameters via 
> cflags/cxxflags/fflags  
> https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/petsc/package.py
>  
> 
> the requirement is not obvious, especially in a long build process.
>  
> The question is should optimisation be amended at spack recipe stage or 
> should PETSc configure enforce some compiler optimisation (if 
> –with-debugging=0 is set) ?
>  
> Regards
>  
> Dr Wei Liu
> High Performance Computing
> To chat with me on Teams click here 
> 
> T: +44 01189 856209
> M: wei@awe.co.uk 
> AWE Aldermaston,
> Reading, Berkshire, RG7 4PR
>  
> The information in this email and in any attachment(s) is commercial in 
> confidence. If you are not the named addressee(s) or if you receive this 
> email in error then any distribution, copying or use of this communication or 
> the information in it is strictly prohibited. Please notify us immediately by 
> email at admin.internet(at)awe.co.uk , and then delete 
> this message from your computer. While attachments are virus checked, AWE plc 
> does not accept any liability in respect of any virus which is not detected. 
> AWE Plc Registered in England and Wales Registration No 02763902 AWE, 
> Aldermaston, Reading, RG7 4PR

Re: [petsc-dev] PETSC optimization

[Zhang, Hong]

> 
> On Aug 16, 2017, at 12:01 PM, Jiyuan Li  wrote:
> 
> Dear developers, 
> 
> I am wondering if PETSC is optimized for modern intel architectures (Xeon E5, 
> Xeon phi, ...),

We have been optimizing PETSc for KNL (2nd generation Xeon Phi). If you happen 
to be using KNL, please refer to /config/examples/arch-cray-xc40-knl-opt.py for 
a configuration example. If you do not need to deal with the batch system, the 
only important options would be

--COPTFLAGS=-g -xMIC-AVX512 -O3 -mP2OPT_hpo_vec_remainder=F
--CXXOPTFLAGS=-g -xMIC-AVX512 -O3 -mP2OPT_hpo_vec_remainder=F
--FOPTFLAGS=-g -xMIC-AVX512 -O3 -mP2OPT_hpo_vec_remainder=F
--with-memalign=64

> i.e., if I compile PETSC using Intel compilers (icc, icpc, intelmpi, etc.) 
> instead of GNU compilers, can I get better performance?
> 

Yes, you should. Based on my experience, using Intel compilers on intel 
architectures can usually result in >2X better performance than using GNU 
compilers. Of course, the speedup may vary depending on applications.

> Best,
> Jiyuan

Hong (Mr.)