The tests are run in parallel - but most other tests are sequential. When I tried to reproduce I ran sequentially [as in 'make -f gmakefile' without -j] - and some of them hung..
Satish On Fri, 6 Apr 2018, Karl Rupp wrote: > Hi again, > > the reason for the higher number of timeouts is likely to be due to the higher > number of GPU tests. GPU builds that formerly only used CUSP now run against > the CUDA backend, which has a higher number of tests. Also, the CUDA backend > uses CUBLAS and CUSPARSE, whereas CUSP used its own kernels. As far as I know, > CUBLAS and CUSPARSE initialization is fairly slow on the M2090. > > Best regards, > Karli > > > On 04/06/2018 09:13 PM, Karl Rupp wrote: > > Hi, > > > >> The CUDA tests are hanging/timing-out more often now. For eg: > >> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/06/examples_next_arch-cuda-double_es.log > >> > >> > >> > >> And I did see some build where they didn't get killed due to timeout. For > >> eg: > >> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/05/examples_next_arch-cuda-double_es.log > >> > >> > >> > >> This is on M2090. I can see them getting stuck on es.mcs [when I run > >> manually - and check with nvidia-smi] > >> > >> When i run these tests manually on GTX1050 (frog.mcs) - they zip through.. > >> Any idea why they get stuck on M2090? [more frequently than random hangs..] > > > > no, I don't know why this is the case. All my local tests finish quickly, > > too. I noticed last summer that there is higher startup overhead on the > > M2090 than on more recent GPUs, but that was in the seconds regime, not in > > minutes. > > > > Are the tests run in parallel? If so, then maybe the parallel initialization > > of GPUs is slowing things down. > > > > Best regards, > > Karli >