Re: [petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

[Zhang, Junchao via petsc-users]

On Wed, Feb 27, 2019 at 7:03 PM Sajid Ali 
<sajidsyed2...@u.northwestern.edu<mailto:sajidsyed2...@u.northwestern.edu>> 
wrote:

Hi Junchao,

I’m confused with the syntax. If I submit the following as my job script, I get 
an error :

#!/bin/bash
#SBATCH --job-name=petsc_test
#SBATCH -N 1
#SBATCH -C knl,quad,flat
#SBATCH -p apsxrmd
#SBATCH --time=1:00:00

module load intel/18.0.3-d6gtsxs
module load intel-parallel-studio/cluster.2018.3-xvnfrfz
module load numactl-2.0.12-intel-18.0.3-wh44iog
srun -n 64 -c 64 --cpu_bind=cores numactl -m 1 aps ./ex_modify -ts_type cn 
-prop_steps 25 -pc_type gamg -ts_monitor -log_view


The error is :
srun: cluster configuration lacks support for cpu binding

This cluster does not support cpu binding.  You need to remove 
--cpu_bind=cores. In addition, I don't know what is the 'aps' argument


srun: error: Unable to create step for job 916208: More processors requested 
than permitted

I remember the product of -n -c has to be 256.  You can try srun -n 64 -c 4 
numactl -m 1 ./ex_modify ...


I’m following the advice as given at slide 33 of 
https://www.nersc.gov/assets/Uploads/02-using-cori-knl-nodes-20170609.pdf

For further info, I’m using LCRC at ANL.

Thank You,
Sajid Ali
Applied Physics
Northwestern University