Re: [petsc-users] [EXTERNAL] Re: Call to DMSetMatrixPreallocateSkip not changing allocation behavior

LLSetPreallocation_C", )); > + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatSeqSELLSetPreallocation_C", )); > +} > +if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, > "MatISSetPreallocation_C", )); &

[petsc-users] Call to DMSetMatrixPreallocateSkip not changing allocation behavior

I'm using the following sequence of functions related to the Jacobian matrix: DMDACreate1d(..., ); DMSetFromOptions(da); DMSetUp(da); DMSetMatType(da, MATAIJKOKKOS); DMSetMatrixPreallocateSkip(da, PETSC_TRUE); Mat J; DMCreateMatrix(da, ); MatSetPreallocationCOO(J, ...); I recently added the call

Re: [petsc-users] [Xolotl-psi-development] [EXTERNAL] Re: Unexpected performance losses switching to COO interface

ZWQ=owjSIigcMR9I7l2CwJBbVn5A6D7SU6RaEKwSNLafLr2msXHmjhMhZWSPCqdwWRAI=dxTWlHKB4itRnwMh05b4rPf4V4axP7XpXAJIUNJYWoQ=> ] "Fackler, Philip via petsc-users" writes: > That makes sense. Here are the arguments that I think are relevant: > > -fieldsplit_1_pc_type redundant -fieldsplit_0_pc_typ

Re: [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface

te: Hi, Philip, I will look into the tarballs and get back to you. Thanks. --Junchao Zhang On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: We finally have xolotl ported to use the new COO interface and the aijkokkos implementation f

Re: [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface

with COO enabled? [Screenshot 2023-10-05 at 10.55.29 AM.png] --Junchao Zhang On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang mailto:junchao.zh...@gmail.com>> wrote: Hi, Philip, I will look into the tarballs and get back to you. Thanks. --Junchao Zhang On Mon, Oct 2, 2023 at

Re: [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface

o Zhang On Mon, Oct 2, 2023 at 9:52 AM Junchao Zhang mailto:junchao.zh...@gmail.com>> wrote: Hi, Philip, I will look into the tarballs and get back to you. Thanks. --Junchao Zhang On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>>

[petsc-users] Unexpected performance losses switching to COO interface

We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version (using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for

Re: [petsc-users] [EXTERNAL] Re: Initializing kokkos before petsc causes a problem

ok at the issue. --Junchao Zhang On Wed, Jun 7, 2023 at 9:30 AM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: I'm encountering a problem in xolotl. We initialize kokkos before initializing petsc. Therefore... The pointer referenced here: https://gitlab.

Re: [petsc-users] [EXTERNAL] Re: Initializing kokkos before petsc causes a problem

Thanks for reporting. I will have a look at the issue. --Junchao Zhang On Wed, Jun 7, 2023 at 9:30 AM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: I'm encountering a problem in xolotl. We initialize kokkos before initializing petsc. Therefore... The pointer referen

Re: [petsc-users] [EXTERNAL] Re: Initializing kokkos before petsc causes a problem

] Initializing kokkos before petsc causes a problem Hi, Philip, Thanks for reporting. I will have a look at the issue. --Junchao Zhang On Wed, Jun 7, 2023 at 9:30 AM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: I'm encountering a problem in xolotl. We init

[petsc-users] Initializing kokkos before petsc causes a problem

I'm encountering a problem in xolotl. We initialize kokkos before initializing petsc. Therefore... The pointer referenced here: https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/impls/basic/kokkos/sfkok.kokkos.cxx#L363

Re: [petsc-users] [EXTERNAL] Re: Kokkos backend for Mat and Vec diverging when running on CUDA device.

>; Zhang, Junchao mailto:jczh...@mcs.anl.gov>>; Roth, Philip mailto:rot...@ornl.gov>> Subject: [EXTERNAL] Re: [petsc-users] Kokkos backend for Mat and Vec diverging when running on CUDA device. Hi, Philip, Sorry to hear that. It seems you could run the same code on CPUs but not no GP

Re: [petsc-users] [EXTERNAL] Re: Performance problem using COO interface

Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: The following is the log_view output for the ported case using 4 MPI

Re: [petsc-users] [EXTERNAL] Re: Kokkos backend for Mat and Vec diverging when running on CUDA device.

ers@mcs.anl.gov>>; Blondel, Sophie mailto:sblon...@utk.edu>>; Zhang, Junchao mailto:jczh...@mcs.anl.gov>>; Roth, Philip mailto:rot...@ornl.gov>> Subject: [EXTERNAL] Re: [petsc-users] Kokkos backend for Mat and Vec diverging when running on CUDA device. Hi, Philip, Sorr

Re: [petsc-users] Performance problem using COO interface

The following is the log_view output for the ported case using 4 MPI tasks. ***WIDEN YOUR WINDOW TO 160

[petsc-users] Performance problem using COO interface

In Xolotl's feature-petsc-kokkos branch I have ported the code to use petsc's COO interface for creating the Jacobian matrix (and the Kokkos interface for interacting with Vec entries). As the attached plots show for one case, while the code for computing the RHSFunction and RHSJacobian perform

Re: [petsc-users] [EXTERNAL] Re: Kokkos backend for Mat and Vec diverging when running on CUDA device.

ailto:petsc-users@mcs.anl.gov>>; Blondel, Sophie mailto:sblon...@utk.edu>>; Zhang, Junchao mailto:jczh...@mcs.anl.gov>>; Roth, Philip mailto:rot...@ornl.gov>> Subject: [EXTERNAL] Re: [petsc-users] Kokkos backend for Mat and Vec diverging when running on CUDA device. Hi, Ph

Re: [petsc-users] [EXTERNAL] Re: Kokkos backend for Mat and Vec diverging when running on CUDA device.

lto:jczh...@mcs.anl.gov>>; Roth, Philip mailto:rot...@ornl.gov>> Subject: [EXTERNAL] Re: [petsc-users] Kokkos backend for Mat and Vec diverging when running on CUDA device. Hi, Philip, Sorry to hear that. It seems you could run the same code on CPUs but not no GPUs (wi

Re: [petsc-users] [EXTERNAL] Re: Kokkos backend for Mat and Vec diverging when running on CUDA device.

t not no GPUs (with either petsc/Kokkos backend or petsc/cuda backend, is it right? --Junchao Zhang On Mon, Nov 14, 2022 at 12:13 PM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: This is an issue I've brought up before (and discussed in-person with Richard).

Re: [petsc-users] [EXTERNAL] Re: Using multiple MPI ranks with COO interface crashes in some cases

EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrot

Re: [petsc-users] [EXTERNAL] Re: Using multiple MPI ranks with COO interface crashes in some cases

; Blondel, Sophie mailto:sblon...@utk.edu>> Subject: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fac

Re: [petsc-users] [EXTERNAL] Re: Using multiple MPI ranks with COO interface crashes in some cases

t; Subject: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl

Re: [petsc-users] [EXTERNAL] Re: Kokkos backend for Mat and Vec diverging when running on CUDA device.

kend or petsc/cuda backend, is it right? --Junchao Zhang On Mon, Nov 14, 2022 at 12:13 PM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: This is an issue I've brought up before (and discussed in-person with Richard). I wanted to bring it up again because I'm hit

Re: [petsc-users] [EXTERNAL] Re: Using multiple MPI ranks with COO interface crashes in some cases

i, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use

Re: [petsc-users] [EXTERNAL] Re: Kokkos backend for Mat and Vec diverging when running on CUDA device.

at 12:13 PM Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: This is an issue I've brought up before (and discussed in-person with Richard). I wanted to bring it up again because I'm hitting the limits of what I know to do, and I need help figuring this out. The probl

[petsc-users] Using multiple MPI ranks with COO interface crashes in some cases

In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a

[petsc-users] Kokkos backend for Mat and Vec diverging when running on CUDA device.

This is an issue I've brought up before (and discussed in-person with Richard). I wanted to bring it up again because I'm hitting the limits of what I know to do, and I need help figuring this out. The problem can be reproduced using Xolotl's "develop" branch built against a petsc build with

Re: [petsc-users] [EXTERNAL] Re: Kokkos Interface for PETSc

vice and it'll use GPU-aware MPI. There are a few examples of residual evaluation and matrix assembly on the device using Kokkos. You can also see libCEED examples for assembly on the device into Kokkos matrices and vectors without touching host memory. "Fackler, Philip via petsc-users" <ma

[petsc-users] Kokkos Interface for PETSc

We're intending to transitioning the Xolotl interfaces with PETSc. I am hoping someone (can) point us to some documentation (and examples) for using PETSc's Kokkos-based interface. If this does not yet exist, then perhaps some slides (like the ones Richard Mills showed at the NE-SciDAC

Re: [petsc-users] [EXTERNAL] Re: Redirecting petsc output

Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory From: petsc-users on behalf of Fackler, Philip via petsc-users Sent: Wednesday, September 8, 2021 11:24 To: Barry Smith Cc: petsc-users@mcs.anl.gov ; xolotl-psi-developm

Re: [petsc-users] [EXTERNAL] Re: Redirecting petsc output

by setting PETSC_STDOUT = fopen(...) Barry On Sep 8, 2021, at 10:59 AM, Fackler, Philip via petsc-users mailto:petsc-users@mcs.anl.gov>> wrote: Is there a way to customize how petsc writes information? Instead of writing to stdout (for example: 0 TS dt 0.1 time 0.), what if we want

[petsc-users] Redirecting petsc output

Is there a way to customize how petsc writes information? Instead of writing to stdout (for example: 0 TS dt 0.1 time 0.), what if we want to log that message to a file other output from Xolotl? I'm assuming there are multiple ways of getting this result. What's common practice with petsc