Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

2019-03-01 Thread Zhang, Hong via petsc-users
On Mar 1, 2019, at 11:00 AM, Sajid Ali mailto:sajidsyed2...@u.northwestern.edu>> wrote: Hi Hong, So, the speedup was coming from increased DRAM bandwidth and not the usage of MCDRAM. Certainly the speedup was coming from the usage of MCDRAM (which has much higher bandwidth than DRAM). Wha

Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

2019-03-01 Thread Sajid Ali via petsc-users
Hi Hong, So, the speedup was coming from increased DRAM bandwidth and not the usage of MCDRAM. There is moderate MPI imbalance, large amount of Back-End stalls and good vectorization. I'm attaching my submit script, PETSc log file and Intel APS summary (all as non-HTML text). I can give more det

Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

2019-02-28 Thread Zhang, Hong via petsc-users
On Feb 28, 2019, at 6:10 AM, Sajid Ali mailto:sajidsyed2...@u.northwestern.edu>> wrote: Hi Hong, Thanks for the advice. I see that the example takes ~180 seconds to run but I can't see the DRAM vs MCDRAM info from Intel APS. I'll try to fix the profiling and get back with further questions.

Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

2019-02-28 Thread Sajid Ali via petsc-users
Hi Hong, Thanks for the advice. I see that the example takes ~180 seconds to run but I can't see the DRAM vs MCDRAM info from Intel APS. I'll try to fix the profiling and get back with further questions. Also, the intel-mpi manpages say that the use of tmi is now deprecated : https://software.int

Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

2019-02-27 Thread Zhang, Hong via petsc-users
For LCRC, it is better to follow the example on their website (http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/) #!/bin/bash #SBATCH -J mympijobname #SBATCH -A myaccount #SBATCH -p knlall #SBATCH -C knl,quad,cache #SBATCH -N 2 #SBATCH --ntasks-per-node=64 #SBATCH

Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

2019-02-27 Thread Zhang, Junchao via petsc-users
On Wed, Feb 27, 2019 at 7:03 PM Sajid Ali mailto:sajidsyed2...@u.northwestern.edu>> wrote: Hi Junchao, I’m confused with the syntax. If I submit the following as my job script, I get an error : #!/bin/bash #SBATCH --job-name=petsc_test #SBATCH -N 1 #SBATCH -C knl,quad,flat #SBATCH -p apsxr

Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

2019-02-27 Thread Sajid Ali via petsc-users
Hi Junchao, I’m confused with the syntax. If I submit the following as my job script, I get an error : #!/bin/bash #SBATCH --job-name=petsc_test #SBATCH -N 1 #SBATCH -C knl,quad,flat #SBATCH -p apsxrmd #SBATCH --time=1:00:00 module load intel/18.0.3-d6gtsxs module load intel-parallel-studio/clus

Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

2019-02-27 Thread Zhang, Junchao via petsc-users
Use srun numactl -m 1 ./app OR srun numactl -p 1 ./app See bottom of https://www.nersc.gov/users/computational-systems/cori/configuration/knl-processor-modes/ --Junchao Zhang On Wed, Feb 27, 2019 at 4:16 PM Sajid Ali via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: Hi, I ran a TS in