Re: [petsc-dev] [petsc-users] Convergence of AMG

On Mon, Oct 29, 2018 at 2:35 PM Smith, Barry F. wrote: > >Why not just stop it once it is equal to or less than the minimum > values set by the person. That is what it does now. It stops when it is below the value given. > Thus you need not "backtrack" by removing levels but the user stil

[petsc-dev] Plex - Metis warnings

I am building a fresh PETSc with GNU on Titan and I get these warnings about incompatible pointers in calls in PlexPartition to MarMetis. Mark /lustre/atlas1/geo127/proj-shared/petsc/src/dm/impls/plex/plexpartition.c: In function 'PetscPartitionerPartition_ParMetis': /lustre/atlas1/geo127/proj-sh

[petsc-dev] Error running on Titan with GPUs & GNU

I get this error running the tests using GPUs. An error in an LAPACK routine. 16:50 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda PETSC_ARCH="" test Running test examples to verify correct installation Using PETSC_DIR=

Re: [petsc-dev] Plex - Metis warnings

On Mon, Oct 29, 2018 at 5:01 PM Matthew Knepley wrote: > On Mon, Oct 29, 2018 at 4:56 PM Mark Adams via petsc-dev < > petsc-dev@mcs.anl.gov> wrote: > >> I am building a fresh PETSc with GNU on Titan and I get these warnings >> about incompatible pointers in calls i

Re: [petsc-dev] Error running on Titan with GPUs & GNU

On Mon, Oct 29, 2018 at 5:07 PM Matthew Knepley wrote: > On Mon, Oct 29, 2018 at 5:01 PM Mark Adams via petsc-dev < > petsc-dev@mcs.anl.gov> wrote: > >> I get this error running the tests using GPUs. An error in an LAPACK >> routine. >> > > From the comma

Re: [petsc-dev] Plex - Metis warnings

am just getting this machine/compiler working. > > Satish > > On Mon, 29 Oct 2018, Matthew Knepley via petsc-dev wrote: > > > On Mon, Oct 29, 2018 at 4:56 PM Mark Adams via petsc-dev < > > petsc-dev@mcs.anl.gov> wrote: > > > > > I am building a fresh

Re: [petsc-dev] Error running on Titan with GPUs & GNU

Still getting this error with downloaded lapack. I sent the logs on the other thread. 18:02 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda PETSC_ARCH="" test Running test examples to verify correct installation Using P

Re: [petsc-dev] Plex - Metis warnings

> > > Pushing language C > Popping language C > Executing: cc -o /tmp/petsc-yiGfSd/config.packages.MPI/conftest -O > /tmp/petsc-yiGfSd/config.packages.MPI/conftest.o -ldl > Testing executable /tmp/petsc-yiGfSd/config.packages.MPI/conftes

Re: [petsc-dev] Plex - Metis warnings

I can I can hand if off to my users with some assurance that it _can_ work! Thanks, Mark On Mon, Oct 29, 2018 at 7:23 PM Balay, Satish wrote: > On Mon, 29 Oct 2018, Mark Adams via petsc-dev wrote: > > > > > > > > > >

Re: [petsc-dev] Error running on Titan with GPUs & GNU

be an optimizing > compiler error. > Configuring debug now. Note, I was able to run ex56 (ksp) which does not use GMRES. This error was from a GMRES method so maybe this is an isolated problem. > >Barry > > > > On Oct 29, 2018, at 3:56 PM, Mark Adams via petsc-dev <

Re: [petsc-dev] Error running on Titan with GPUs & GNU

And a debug build seems to work: 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda PETSC_ARCH="" test Running test examples to verify correct installation Using PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_tita

Re: [petsc-dev] Error running on Titan with GPUs & GNU

> > > > Are there newer versions of the Gnu compiler for this system? Yes: -- /opt/modulefiles -- gc

Re: [petsc-dev] Error running on Titan with GPUs & GNU

Karli > > > > On 10/31/18 8:15 AM, Mark Adams via petsc-dev wrote: > > After loading a cuda module ... > > > > On Wed, Oct 31, 2018 at 2:58 AM Mark Adams > <mailto:mfad...@lbl.gov>> wrote: > > > > I get an error with --with-cuda=1 > >

Re: [petsc-dev] Error running on Titan with GPUs & GNU

gt; gcc/4.8.2 gcc/5.3.0 gcc/6.2.0 gcc/7.1.0 > gcc/7.3.0 > > > >> >> Best regards, >> Karli >> >> >> >> On 10/31/18 8:15 AM, Mark Adams via petsc-dev wrote: >> > After loading a cuda module ... >> &g

Re: [petsc-dev] Error running on Titan with GPUs & GNU

On Wed, Oct 31, 2018 at 12:30 PM Mark Adams wrote: > > > On Wed, Oct 31, 2018 at 6:59 AM Karl Rupp wrote: > >> Hi Mark, >> >> ah, I was confused by the Python information at the beginning of >> configure.log. So it is picking up the correct compiler. >> >> Have you tried uncommenting the check f

Re: [petsc-dev] Error running on Titan with GPUs & GNU

FYI, I seem to have the new GPU machine at ORNL (summitdev) working with GPUs. That is good enough for now. Thanks, 14:00 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres -snes_monitor_short -snes_rtol 1.e-5 -k

Re: [petsc-dev] Error running on Titan with GPUs & GNU

And I just tested it with GAMG and it seems fine. And hypre ran, but it is not clear that it used GPUs 14:13 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type hypre -ksp_type fgmres -snes_monitor_short -snes_rtol 1.e-5 -ksp_view

Re: [petsc-dev] Error running on Titan with GPUs & GNU

I did not configure hypre manually, so I guess it is not using GPUs. On Fri, Nov 2, 2018 at 2:40 PM Smith, Barry F. wrote: > > > > On Nov 2, 2018, at 1:25 PM, Mark Adams wrote: > > > > And I just tested it with GAMG and it seems fine. And hypre ran, but it > is not clear that it used GPUs

Re: [petsc-dev] FW: Re[2]: Implementing of a variable block size BILU preconditioner

If you zero a row out then put something on the diagonal. And your matrix data file (it does not look like it has any sparsity meta-data) has about 18 orders of scales. When you diagonally scale, which most solvers implicitly do, it looks like some of these numbers will just go away and you will n

[petsc-dev] GPU web page out of date

The GPU web page looks like it is 8 years old ... this link is dead (but ex47cu seems to be in the repo): - Example that uses CUDA directly in the user function evaluation Thanks, Mark

Re: [petsc-dev] HYPRE_LinSysCore.h

On Tue, Jan 29, 2019 at 5:20 PM Victor Eijkhout via petsc-dev < petsc-dev@mcs.anl.gov> wrote: > > > On Jan 29, 2019, at 3:58 PM, Balay, Satish wrote: > > -args.append('--without-fei') > > > The late-1990s Finite Element Interface? > I would guess FEI is still actively used and interfaces to

[petsc-dev] is DMSetDS not in master?

10:37 master= ~/Codes/petsc$ git grep DMSetDS src/dm/interface/dm.c:.seealso: DMGetDS(), DMSetDS() 10:37 master= ~/Codes/petsc$

Re: [petsc-dev] is DMSetDS not in master?

OK, it's not in Changes and there is one comment to it. On Fri, Feb 1, 2019 at 10:50 AM Matthew Knepley wrote: > I removed it, since no one should use it anymore. You use > DMSetField()+DMCreateDS() instead. > > THanks, > > Matt > > On Fri, Feb 1, 2019 at 10:3

Re: [petsc-dev] [petsc-users] Bad memory scaling with PETSc 3.10

I'm probably screwing up some sort of history by jumping into dev, but this is a dev comment ... (1) -matptap_via hypre: This call the hypre package to do the PtAP trough > an all-at-once triple product. In our experiences, it is the most memory > efficient, but could be slow. > FYI, I visited L

Re: [petsc-dev] [petsc-users] Bad memory scaling with PETSc 3.10

> > > Could you explain this more by adding some small examples? > > Since you are considering implementing all-at-once (four nested loops, right?) I'll give you my old code. This code is hardwired for two AMG and for a geometric-AMG, where the blocks of the R (and hence P) matrices are scaled ide

Re: [petsc-dev] MatNest and FieldSplit

I think he is saying that this line seems to have no effect (and the comment is hence wrong): KSPSetOperators(subksp[nsplits - 1], S, S); // J2 = [[4, 0] ; [0, 0.1]] J2 is a 2x2 but this block has been changed into two single equation fields. Does this KSPSetOperators supposed to copy this

Re: [petsc-dev] [petsc-users] Bad memory scaling with PETSc 3.10

> > > The way to reduce the memory is to have the all-at-once algorithm (Mark is > an expert on this). But I am not sure how efficient it could be > implemented. > I have some data from a 3D elasticity problem with 1.4B equations on: /home1/04906/bonnheim/olympus-keaveny/Olympus/olympus.petsc-3.

Re: [petsc-dev] [petsc-users] Bad memory scaling with PETSc 3.10

On Wed, Mar 27, 2019 at 12:06 AM Victor Eijkhout wrote: > > > On Mar 26, 2019, at 6:25 PM, Mark Adams via petsc-dev < > petsc-dev@mcs.anl.gov> wrote: > > /home1/04906/bonnheim/olympus-keaveny/Olympus/olympus.petsc-3.9.3.skx-cxx-O > on a skx-cxx-O named c478-062.sta

Re: [petsc-dev] [petsc-users] Bad memory scaling with PETSc 3.10

So is this the instructions that I should give him? This grad student is a quick study but he has not computing background. So we don't care what we use, we just want to work (easily). Thanks Do not use "--download-fblaslapack=1". Set it to 0. Same for "--download-mpich=1". Now do: > module loa

Re: [petsc-dev] SNESSolve and changing dimensions

I agree that you want to adapt around a converged solution. I have code that runs time step(s), adapts, Transfers solutions and state, creates a new TS & SNES, if you want to clone that. It works with PForest, but Toby and Matt are working on these abstractions so it might not be the most u

Re: [petsc-dev] New implementation of PtAP based on all-at-once algorithm

Interesting, nice work. It would be interesting to get the flop counters working. This looks like GMG, I assume 3D. The degree of parallelism is not very realistic. You should probably run a 10x smaller problem, at least, or use 10x more processes. I guess it does not matter. This basically like

Re: [petsc-dev] New implementation of PtAP based on all-at-once algorithm

On Thu, Apr 11, 2019 at 11:42 PM Smith, Barry F. wrote: > > > > On Apr 11, 2019, at 9:07 PM, Mark Adams via petsc-dev < > petsc-dev@mcs.anl.gov> wrote: > > > > Interesting, nice work. > > > > It would be interesting to get the flop counters work

Re: [petsc-dev] New implementation of PtAP based on all-at-once algorithm

> > > I guess you are interested in the performance of the new algorithms on > small problems. I will try to test a petsc example such as > mat/examples/tests/ex96.c. > It's not a big deal. And the fact that they are similar on one node tells us the kernels are similar. > > >> >> And are you su

Re: [petsc-dev] New implementation of PtAP based on all-at-once algorithm

Mark Adams wrote: >>> >>>> >>>> >>>> On Thu, Apr 11, 2019 at 11:42 PM Smith, Barry F. >>>> wrote: >>>> >>>>> >>>>> >>>>> > On Apr 11, 2019, at 9:07 PM, Mark Adams vi

Re: [petsc-dev] New implementation of PtAP based on all-at-once algorithm

> > So you could reorder your equations and see a block diagonal matrix with >> 576 blocks. right? >> > > I not sure I understand the question correctly. For each mesh vertex, we > have a 576x576 diagonal matrix. The unknowns are ordered in this way: > v0, v2.., v575 for vertex 1, and another 576

Re: [petsc-dev] New implementation of PtAP based on all-at-once algorithm

> >> I wonder if the their symbolic setup is getting called every time. You do >> 50 solves it looks like and that should be enough to amortize a one time >> setup cost. >> > > Hypre does not have concept called symbolic. They do everything from > scratch, and won't reuse any data. > Really, Hypre

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

I am stumped with this GPU bug(s). Maybe someone has an idea. I did find a bug in the cuda transpose mat-vec that cuda-memcheck detected, but I still have differences between the GPU and CPU transpose mat-vec. I've got it down to a very simple test: bicg/none on a tiny mesh with two processors. It

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

for MatMultTranspose()? > > Barry > > BTW: Unrelated comment, the code > > ierr = VecSet(yy,0);CHKERRQ(ierr); > ierr = VecCUDAGetArrayWrite(yy,&yarray);CHKERRQ(ierr); > > has an unneeded ierr = VecSet(yy,0);CHKERRQ(ierr); here. > VecCUDAGetArrayWrite() requires

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

or->size() against A->cmap->n. > > Stefano > > Il giorno mer 10 lug 2019 alle ore 15:54 Mark Adams via petsc-dev < > petsc-dev@mcs.anl.gov> ha scritto: > >> >> >> On Wed, Jul 10, 2019 at 1:13 AM Smith, Barry F. >> wrote: >> &

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

ark > > > > On Wed, Jul 10, 2019 at 9:22 AM Stefano Zampini < > stefano.zamp...@gmail.com> wrote: > > Mark, > > > > if the difference is on lvec, I suspect the bug has to do with > compressed row storage. I have fixed a similar bug in MatMult. > > you want

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

> > > 3) Is comparison between pointers appropriate? For example if (dptr != > zarray) { is scary if some arrays are zero length how do we know what the > pointer value will be? > > Yes, you need to consider these cases, which is kind of error prone. Also, I think merging transpose,and not,is a go

[petsc-dev] CUDA GAMG coarse grid solver

I am running ex56 with -ex56_dm_vec_type cuda -ex56_dm_mat_type aijcusparse and I see no GPU communication in MatSolve (the serial LU coarse grid solver). I am thinking the dispatch of the CUDA version of this got dropped somehow. I see that this is getting called: PETSC_EXTERN PetscErrorCode Mat

Re: [petsc-dev] CUDA GAMG coarse grid solver

DDT, but I can try stepping through from MatGetFactor to see what its doing. Thanks, Mark On Sun, Jul 21, 2019 at 11:14 AM Smith, Barry F. wrote: > > > > On Jul 21, 2019, at 8:55 AM, Mark Adams via petsc-dev < > petsc-dev@mcs.anl.gov> wrote: > > > > I

Re: [petsc-dev] CUDA GAMG coarse grid solver

y F. > wrote: > >> >> >> > On Jul 21, 2019, at 8:55 AM, Mark Adams via petsc-dev < >> petsc-dev@mcs.anl.gov> wrote: >> > >> > I am running ex56 with -ex56_dm_vec_type cuda -ex56_dm_mat_type >> aijcusparse and I see no GPU communication

[petsc-dev] MatPinToCPU

I've tried to add pining the matrix and prolongator to the CPU on coarse grids in GAMG with this: /* pin reduced coase grid - could do something smarter */ ierr = MatPinToCPU(*a_Amat_crs,PETSC_TRUE);CHKERRQ(ierr); ierr = MatPinToCPU(*a_P_inout,PETSC_TRUE);CHKERRQ(ierr); It does not se

Re: [petsc-dev] MatPinToCPU

> > > What are the symptoms of it not working? Does it appear to be still > copying the matrices to the GPU? then running the functions on the GPU? > > The object is dispatching the CUDA mat-vec etc. I suspect the pinning is incompletely done for CUDA (and MPIOpenCL) > matrices. > > Yes, git g

Re: [petsc-dev] MatPinToCPU

I'm not sure what to do here. The problem is that pinned-to-cpu vectors are calling *VecCUDACopyFromGPU* here. Should I set *x->valid_GPU_array *to something else, like PETSC_OFFLOAD_CPU, in PinToCPU so this block of code i s not executed? PetscErrorCode VecGetArray(Vec x,PetscScalar **a) { Pet

Re: [petsc-dev] MatPinToCPU

Yea, I just figured out the problem. VecDuplicate_MPICUDA did not call PinToCPU or even copy pinnedtocpu. It just copied ops, so I added and am testing: ierr = VecCreate_MPICUDA_Private(*v,PETSC_TRUE,w->nghost,0);CHKERRQ(ierr); vw = (Vec_MPI*)(*v)->data; ierr = PetscMemcpy((*v)->ops,win->o

Re: [petsc-dev] MatPinToCPU

On Sat, Jul 27, 2019 at 11:39 AM Smith, Barry F. wrote: > > Good catch. Thanks. Maybe the SeqCUDA has the same problem? > THis is done (I may have done it). Now it seems to me that when you call VecPinToCPU you are setting up and don't have data, so this copy does not seem necessary. Maybe r

Re: [petsc-dev] MatPinToCPU

Barry, I fixed CUDA to pin to CPUs correctly for GAMG at least. There are some hacks here that we can work on. I will start testing it tomorrow, but I am pretty sure that I have not regressed. I am hoping that this will fix the numerical problems, which seem to be associated with empty processors.

Re: [petsc-dev] PCREDUNDANT

On Sun, Jul 28, 2019 at 2:54 AM Pierre Jolivet via petsc-dev < petsc-dev@mcs.anl.gov> wrote: > Hello, > I’m facing multiple issues with PCREDUNDANT and MATMPISBAIJ: > 1) > https://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/sbaij/mpi/mpisbaij.c.html#line3354 > shouldn’t > this be sum != N?

Re: [petsc-dev] MatPinToCPU

This is looking good. I'm not seeing the numerical problems, but I've just hid them by avoiding the GPU on coarse grids. Should I submit a pull request now or test more or wait for Karl? On Sat, Jul 27, 2019 at 7:37 PM Mark Adams wrote: > Barry, I fixed CUDA to pin to CPUs correctly for GAMG at

Re: [petsc-dev] MatPinToCPU

On Mon, Jul 29, 2019 at 11:27 PM Smith, Barry F. wrote: > > Thanks. Could you please send the 24 processors with the GPU? > That is in out_cuda_24 >Note the final column of the table gives you the percentage of flops > (not rates, actual operations) on the GPU. For you biggest r

[petsc-dev] Is master broken?

I am seeing this when I pull master into my branch: "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/ densecuda.cu" , line 243: error: function call is not allowed in a constant expression #if PETSC_PKG_CUDA_VERSION_GE(10,1,0) and I see that this macro does

Re: [petsc-dev] Is master broken?

arch-linux2-c-debug/include/petscpkg_version.h >> contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send >> configure.lo >> >> check what is in arch-linux2-c-debug/include/petscpkg_version.h it >> nothing or broken send configure.lo >> >> >> B

Re: [petsc-dev] Is master broken?

at top of the "bad" source file crashes so in theory everything > is in order check that arch-linux2-c-debug/include/petscpkg_version.h > contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send > configure.lo > > > > check what is in arch-linux2-c-debug/i

Re: [petsc-dev] Is master broken?

Adams wrote: > >>>> Tried again after deleting the arch dirs and still have it. > >>>> This is my branch that just merged master. I will try with just > master. > >>>> Thanks, > >>>> > >>>> On Thu, Aug 1, 2019 at 1

Re: [petsc-dev] Is master broken?

I have been cherry-picking, etc, branch mark/gamg-fix-viennacl-rebased and it is very messed up. Can someone please update this branch when all the fixes are settled down? eg, I am seeing dozens of modified files that I don't know anything about and I certainly don't want to put in a PR for them.

Re: [petsc-dev] Is master broken?

. [presumably a single commit for all > > the changes is not the correct thing here. But the current set of 21 > > commits are all over the place] > > > > If you are able to migrate to this branch - its best to delete the old > > one [i.e origin/mark/gamg-fix-viennacl-rebase

Re: [petsc-dev] Is master broken?

Satish, Your new branch mark/gamg-fix-viennacl-rebased-v2 does not seem to have Barry's fixes (the source of this thread): ... , line 243: error: function call is not allowed in a constant expression #if PETSC_PKG_CUDA_VERSION_GE(10,1,0) Here is the reflog of the cherry pi

Re: [petsc-dev] Is master broken?

Satish, I think I can do this now. Mark On Mon, Aug 12, 2019 at 6:26 AM Mark Adams wrote: > Satish, > > Your new branch mark/gamg-fix-viennacl-rebased-v2 does not seem to have > Barry's fixes (the source of this thread): > > ... > , line 243: error: function call is not allowed in a c

Re: [petsc-dev] Is master broken?

t; > > the changes is not the correct thing here. But the current set of > 21 > > > commits are all over the place] > > > > > > If you are able to migrate to this branch - its best to delete > > the old > > > one [

Re: [petsc-dev] Is master broken?

> > >> several issues: >> >> >> https://bitbucket.org/petsc/petsc/pull-requests/1954/cuda-fixes-to-pinning-onto-cpu/diff >> >> > These links are dead. > I found one issue with not protecting the pinnedtocpu member variable in Mat and Vec. Will fix asap.

[petsc-dev] running test

I want to test a test as it will be executed. Can someone please tell me how do I run a "cuda" test SNES ex56 exactly? This is the test in ex56.c test: suffix: cuda nsize: 2 requires: cuda args: -cel Thanks, Mark

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

On Wed, Aug 14, 2019 at 2:35 PM Smith, Barry F. wrote: > > Mark, > >Would you be able to make one run using single precision? Just single > everywhere since that is all we support currently? > > Experience in engineering at least is single does not work for FE elasticity. I have tried it ma

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

On Wed, Aug 14, 2019 at 3:37 PM Jed Brown wrote: > Mark Adams via petsc-dev writes: > > > On Wed, Aug 14, 2019 at 2:35 PM Smith, Barry F. > wrote: > > > >> > >> Mark, > >> > >>Would you be able to make one run using single pr

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

On Wed, Aug 14, 2019 at 2:19 PM Smith, Barry F. wrote: > > Mark, > > This is great, we can study these for months. > > 1) At the top of the plots you say SNES but that can't be right, there is > no way it is getting such speed ups for the entire SNES solve since the > Jacobians are CPUs an

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

> > > > Do you have any applications that specifically want Q2 (versus Q1) > elasticity or have some test problems that would benefit? > > No, I'm just trying to push things.

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

Here is the times for KSPSolve on one node with 2,280,285 equations. These nodes seem to have 42 cores. There are 6 "devices" (GPUs) and 7 core attached to the device. The anomalous 28 core result could be from only using 4 "devices". I figure I will use 36 cores for now. I should really do this w

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

have linear variations in stress. > > On 8/14/19 2:51 PM, Mark Adams via petsc-dev wrote: > > > > > > Do you have any applications that specifically want Q2 (versus Q1) > > elasticity or have some test problems that would benefit? > > > > > > No, I'm just trying to push things. >

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

OK, I'll run single. It a bit perverse to run with 4 byte floats and 8 byte integers ... I could use 32 bit ints and just not scale out. On Wed, Aug 14, 2019 at 6:48 PM Smith, Barry F. wrote: > > Mark, > >Oh, I don't even care if it converges, just put in a fixed number of > iterations. The

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

I can run single, I just can't scale up. But I can use like 1500 processors. On Wed, Aug 14, 2019 at 9:31 PM Smith, Barry F. wrote: > > Oh, are all your integers 8 bytes? Even on one node? > > Once Karl's new middleware is in place we should see about reducing to 4 > bytes on the GPU. > >

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

I am getting this error with single: 22:21 /gpfs/alpine/geo127/scratch/adams$ jsrun -n 1 -a 1 -c 1 -g 1 ./ex56_single -cells 2,2,2 -ex56_dm_vec_type cuda -ex56_dm_mat_type aijcusparse -fp_trap [0] 81 global equations, 27 vertices [0]PETSC ERROR: *** unknown floating point error occurred *** [0]PE

[petsc-dev] hypre and CUDA

I have configured with Hypre on SUMMIT, with cuda, and it ran. I'm now trying to verify that it used GPUs (I doubt it). Any ideas on how to verify this? Should I use the cuda vecs and mats, or does Hypre not care. Can I tell hypre not to use GPUs other than configuring an non-cude PETSc? I'm not su

Re: [petsc-dev] hypre and CUDA

ed. > I have just been timing it. It is very slow. 3x slower than GAMG/CPU and 20x slower than GAMG/GPU. But Hypers parameters tend to be optimized for 2D and I have not optimized parameters. But clearly its not using GPUs. > >Barry > > > > On Aug 15, 2019, at 10:47 AM

Re: [petsc-dev] hypre and CUDA

./ex56 ... > > > https://devblogs.nvidia.com/cuda-pro-tip-nvprof-your-handy-universal-gpu-profiler/ > > If it uses the GPU, then you will get some information on the GPU > kernels called. If it doesn't use the GPU, the list will be (almost) empty. > > Best regards, > Karli > >

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

On Sat, Aug 31, 2019 at 4:28 PM Smith, Barry F. wrote: > > Any explanation for why the scaling is much better for CPUs and than > GPUs? Is it the "extra" time needed for communication from the GPUs? > The GPU work is well load balanced so it weak scales perfectly. When you put that work in the

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

Junchao and Barry, I am using mark/fix-cuda-with-gamg-pintocpu, which is built on barry's robustify branch. Is this in master yet? If so, I'd like to get my branch merged to master, then merge Junchao's branch. Then us it. I think we were waiting for some refactoring from Karl to proceed. Anyway

Re: [petsc-dev] Should we add something about GPU support to the user manual?

> > >> And are there any thoughts on where this belongs in the manual? >> > > I think just make another chapter. > > Agreed. That way we can make it very clear that this is WIP, interfaces will change, etc. > Thanks, > > Matt > > >> --Richard >> > > > -- > What most experimenters take for g

Re: [petsc-dev] MatMult on Summit

On Sat, Sep 21, 2019 at 12:48 AM Smith, Barry F. via petsc-dev < petsc-dev@mcs.anl.gov> wrote: > > Junchao, > >Very interesting. For completeness please run also 24 and 42 CPUs > without the GPUs. Note that the default layout for CPU cores is not good. > You will want 3 cores on each socket

Re: [petsc-dev] MatMult on Summit

I came up with 36 cores/node for CPU GAMG runs. The memory bus is pretty saturated at that point. On Sat, Sep 21, 2019 at 1:44 AM Zhang, Junchao via petsc-dev < petsc-dev@mcs.anl.gov> wrote: > Here are CPU version results on one node with 24 cores, 42 cores. Click > the links for core layout. > >

Re: [petsc-dev] MatMult on Summit

Note, the numerical problems that we have look a lot like a race condition of some sort. Happens with empty processors and goes away under cuda-memcheck (valgrind like thing). I did try adding WaitForGPU() , but maybe I did do it right or there are other synchronization mechanisms. On Mon, Sep 2

Re: [petsc-dev] MatMult on Summit

Yes, please, thank you. On Tue, Sep 24, 2019 at 1:46 AM Mills, Richard Tran via petsc-dev < petsc-dev@mcs.anl.gov> wrote: > Karl, that would be fantastic. Much obliged! > > --Richard > > On 9/23/19 8:09 PM, Karl Rupp wrote: > > Hi, > > `git grep cudaStreamCreate` reports that vectors, matrices an

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

t; > On 9/25/19 11:12 AM, Mark Adams via petsc-dev wrote: > > I am using karlrupp/fix-cuda-streams, merged with master, and I get this > > error: > > > > Could not execute "['jsrun -g\\ 1 -c\\ 1 -a\\ 1 --oversubscribe -n 1 > > printenv']": > >

[petsc-dev] getting eigen estimates from GAMG to CHEBY

It's been a few years since we lost the ability to cache the eigen estimates, that smoothed aggregation computes, to chebyshev smoothers. I'd like to see if we bring this back. This is slightly (IMO) complicated by the fact that the smoother PC may not be Jacobi, but I think it is close enough (an

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

s for configure, speeds up the configure 10:18 (0542e31...)|BISECTING ~/petsc-karl$ > Best regards, > Karli > > > > > > > On Wed, Sep 25, 2019 at 5:26 AM Karl Rupp via petsc-dev > > mailto:petsc-dev@mcs.anl.gov>> wrote: > > > > > > > >

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

> > > Mark, > > > > Can you send configure.log from mark/fix-cuda-with-gamg-pintocpu branch? > > > > Satish > > > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote: > > > > > I double checked that a clean build of your (master) bran

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

r this purpose on summit? > > And then there is this code to escape spaces in path - for > windows. [but we have to make sure this is not in code-path for user > specified --with-mpiexec="jsrun -g 1" > > Satish > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote: >

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

Oh, and I tested the branch and it didn't work. file was attached. On Wed, Sep 25, 2019 at 2:38 PM Mark Adams wrote: > > > On Wed, Sep 25, 2019 at 2:23 PM Balay, Satish wrote: > >> On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote: >> >> > On Wed,

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

I did test this and sent the log (error). On Wed, Sep 25, 2019 at 2:58 PM Balay, Satish wrote: > I made changes and asked to retest with the latest changes. > > Satish > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote: > > > Oh, and I tested the branch and

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

> Yes, it's supported, but it's a little different than what "-n" usually > does in mpiexec, where it means the number of processes. For 'jsrun', it > means the number of resource sets, which is multiplied by the "tasks per > resource set" specified by "-a" to get the MPI process count. I think if

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

; balay/fix-mpiexec-shell-escape, origin/balay/fix-mpiexec-shell-escape) > > Try: > > git fetch > git checkout origin/balay/fix-mpiexec-shell-escape > > Satish > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote: > > > On Wed, Sep 25, 2019 at 4:57 PM Balay,

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

So perhaps its easier to > deal with in detached mode - which makes this obvious] > I got this <> and "fixed" it by deleting the branch and repulling it. I guess I needed to fetch also. Mark > > Satish > > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wro

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

e/geo127/scratch/adams$ jsrun -g 1 -n 1 printenv GIT_PS1_SHOWDIRTYSTATE=1 XDG_SESSION_ID=494 SHELL=/bin/bash HISTSIZE=100 PETSC_ARCH=arch-summit-opt64-pgi-cuda SSH_CLIENT=160.91.202.152 48626 22 LC_ALL= USER=adams ... > > Satish > > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

> > If jsrun is not functional from configure, alternatives are > --with-mpiexec=/bin/true or --with-batch=1 > > --with-mpiexec=/bin/true seems to be working. Thanks, Mark > Satish >

Re: [petsc-dev] getting eigen estimates from GAMG to CHEBY

> > Okay, it seems like they should be stored in GAMG. > Before we stored them in the matrix. When you get to the test in Cheby you don't have caller anymore (GAMG). > Why would the PC type change anything? > Oh, the eigenvalues are the preconditioned ones, the PC (Jacobi) matters but it is not

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

regards, > Karli > > > > > > > On Wed, Sep 25, 2019 at 5:26 AM Karl Rupp via petsc-dev > > mailto:petsc-dev@mcs.anl.gov>> wrote: > > > > > > > > On 9/25/19 11:12 AM, Mark Adams via petsc-dev wrote: > >

Re: [petsc-dev] getting eigen estimates from GAMG to CHEBY

As I recall we attached the eigenstates to the matrix. Is that old attach mechanism still the used/recommended? Or is there a better way to do this now? Thanks, Mark On Thu, Sep 26, 2019 at 7:45 AM Mark Adams wrote: > > >> Okay, it seems like they should be stored in GAMG. >> > > Before we store

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

The logic is basically correct because I simple zero out yy vector (the output vector) and it runs great now. The numerics look fine without CPU pinning. AND, it worked with 1,2, and 3 GPUs (one node, one socket), but failed with 4 GPU's which uses the second socket. Strange. On Sat, Sep 28, 2019

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

On Sat, Sep 28, 2019 at 12:55 AM Karl Rupp wrote: > Hi Mark, > > > OK, so now the problem has shifted somewhat in that it now manifests > > itself on small cases. It is somewhat random and anecdotal but it does happen on the smaller test problem now. When I try to narrow down when the problem m

  1   2   >