Re: [petsc-dev] GPU counters

No. For each vector/matrix operation, PETSc can get its flop count based on number of nonzeros, for example. --Junchao Zhang On Wed, Nov 6, 2019 at 8:44 AM Mark Adams via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: I am puzzled. I am running AMGx now, and I am getting flop counts/rates.

Re: [petsc-dev] PetscLayoutFindOwner and PetscLayoutFindOwnerIndex

The value of "owner" should fit in PetscMPIInt. But if you change prototype of the two functions, you have to change all their uses. In petsc, values representing MPI ranks are not always of type PetscMPIInt. Only those closely tied to MPI routines are in PetscMPIInt. --Junchao Zhang On Wed,

Re: [petsc-dev] Should v->valid_GPU_array be a bitmask?

Oct 1, 2019, at 10:24 PM, Zhang, Junchao via petsc-dev > mailto:petsc-dev@mcs.anl.gov>> wrote: > > Stafano recently modified the following code, > > > PetscErrorCode VecCreate_SeqCUDA(Vec V) > > { > > PetscErrorCode ierr; > > > > PetscFunct

Re: [petsc-dev] Feed back on report on performance of vector operations on Summit requested

*Better to have an abstract for readers to know your intention/conclusion *p.5 "We also launch all jobs using the --launch_distribution cyclic option so that MPI ranks are assigned to resource sets in a circular fashion, which we deem appropriate for most high performance computing (HPC)

Re: [petsc-dev] Should v->valid_GPU_array be a bitmask?

touch valid_GPU_array, then we should also use a better name or refactor completely. Code like (V->valid_GPU_array & PETSC_OFFLOAD_GPU) simply isn't intuitive (nor does it make sense) when read aloud. Best regards, Karli On 10/2/19 5:24 AM, Zhang, Junchao via petsc-dev wrote: >

[petsc-dev] Should v->valid_GPU_array be a bitmask?

Stafano recently modified the following code, PetscErrorCode VecCreate_SeqCUDA(Vec V) { PetscErrorCode ierr; PetscFunctionBegin; ierr = PetscLayoutSetUp(V->map);CHKERRQ(ierr); ierr = VecCUDAAllocateCheck(V);CHKERRQ(ierr); ierr =

Re: [petsc-dev] MatMult on Summit

(but no WaitForGPU() inside the timed region). I believe this is >> another area in which we get a meaningless timing. It looks like >> we need a WaitForGPU() there, and then maybe inside the timed >> region handling the scatter. (I don't know if this stuff happens >>

Re: [petsc-dev] MatMult on Summit

region handling the scatter. (I don't know if this stuff happens asynchronously or not.) But do we potentially want two WaitForGPU() calls in one function, just to help with getting timings? I don't have a good idea of how much overhead this adds. --Richard On 9/21/19 12:03 PM, Zhang, Junc

Re: [petsc-dev] MatMult on Summit

attached 'lstopo' PDF output from a Summit compute node to see an illustration of the node layout. --Richard On 9/23/19 9:01 AM, Zhang, Junchao via petsc-dev wrote: I also did OpenMP stream test and then I found mismatch between OpenMPI and MPI. That reminded me a subtle issue on summit: pair o

Re: [petsc-dev] MatMult on Summit

tem, and 270 GB/s STREAM Triad according to this post > > > > > > https://openpowerblog.wordpress.com/2018/07/19/epyc-skylake-vs-power9-stream-memory-bandwidth-comparison-via-zaius-barreleye-g2/ > > > > Is this 60% of Triad the best we can get for SpMV? > > > > "Zhan

Re: [petsc-dev] MatMult on Summit

ttps://devblogs.nvidia.com/introduction-cuda-aware-mpi/ has detailed explanation of CUDA-aware MPI. In short, it avoids CPU involvement and redundant memory copies. Best regards, Karli On 9/21/19 6:22 AM, Zhang, Junchao via petsc-dev wrote: > I downloaded a sparse matrix (HV15R > <ht

Re: [petsc-dev] MatMult on Summit

ltiply for this problem size. > On Sep 21, 2019, at 6:40 PM, Mark Adams > mailto:mfad...@lbl.gov>> wrote: > > I came up with 36 cores/node for CPU GAMG runs. The memory bus is pretty > saturated at that point. > > On Sat, Sep 21, 2019 at 1:44 AM Zhang, Junchao via petsc-de

Re: [petsc-dev] MatMult on Summit

I made the following changes: 1) In MatMultAdd_SeqAIJCUSPARSE, use this code sequence at the end ierr = WaitForGPU();CHKERRCUDA(ierr); ierr = PetscLogGpuTimeEnd();CHKERRQ(ierr); ierr = PetscLogGpuFlops(2.0*a->nz);CHKERRQ(ierr); PetscFunctionReturn(0); 2) In MatMult_MPIAIJCUSPARSE, use the

Re: [petsc-dev] MatMult on Summit

times more to do the multiplies in the MPS case but where is that time coming from in the other numbers? Communication time doesn't drop that much? I can't present these numbers with this huge inconsistency Thanks, Barry > On Sep 20, 2019, at 11:22 PM, Zhang, Junchao via

Re: [petsc-dev] MatMult on Summit

l want 3 cores on each socket then 12 on each. Thanks Barry Since Tim is one of our reviewers next week this is a very good test matrix :-) > On Sep 20, 2019, at 11:39 PM, Zhang, Junchao via petsc-dev > mailto:petsc-dev@mcs.anl.gov>> wrote: > > Click the links to v

Re: [petsc-dev] MatMult on Summit

20, 2019 at 11:34 PM Mills, Richard Tran via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Junchao, Can you share your 'jsrun' command so that we can see how you are mapping things to resource sets? --Richard On 9/20/19 11:22 PM, Zhang, Junchao via petsc-dev wrote: I downloaded a sparse

[petsc-dev] MatMult on Summit

I downloaded a sparse matrix (HV15R) from Florida Sparse Matrix Collection. Its size is about 2M x 2M. Then I ran the same MatMult 100 times on one node of Summit with -mat_type aijcusparse -vec_type cuda. I found MatMult was almost dominated by VecScatter

Re: [petsc-dev] Configure hangs on Summit

ouple days ago), I didn't have this problem. I'm working from the example template that's in the PETSc repo at config/examples/arch-olcf-summit-opt.py. Can you point me to your configure script on Summit so I can try to reproduce your problem? --Richard On 9/20/19 4:25 PM, Zhang, Junchao via

Re: [petsc-dev] Configure hangs on Summit

Satish's trick --with-mpiexec=/bin/true solved the problem. Thanks. --Junchao Zhang On Fri, Sep 20, 2019 at 3:50 PM Junchao Zhang mailto:jczh...@mcs.anl.gov>> wrote: My configure hangs on Summit at TESTING: configureMPIEXEC from

[petsc-dev] Configure hangs on Summit

My configure hangs on Summit at TESTING: configureMPIEXEC from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:170) On the machine one has to use script to submit jobs. So why do we need configureMPIEXEC? Do I need to use --with-batch? I remember we removed that. --Junchao

Re: [petsc-dev] PetscCUDAInitialize

with -cuda_initialize 0 on frog but I could not > reproduce it. I'm doing investigation. > > Barry > > > > > --Junchao Zhang > > > > > > > > On Wed, Sep 18, 2019 at 11:42 PM Smith, Barry F. > > mailto:bsm...@mcs.anl.gov>> wrote: > &

Re: [petsc-dev] Master broken after changes to PetscSection headers

Could this problem be fixed right now? It shows up in stage-1 tests and block further tests. --Junchao Zhang On Thu, Sep 19, 2019 at 4:15 AM Lisandro Dalcin via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: The warnings below are from a C build. A C++ build fails right away. We need to

Re: [petsc-dev] PetscCUDAInitialize

on frog but I could not reproduce it. I'm doing investigation. Barry > > --Junchao Zhang > > > > On Wed, Sep 18, 2019 at 11:42 PM Smith, Barry F. > mailto:bsm...@mcs.anl.gov>> wrote: > > Fixed the docs. Thanks for pointing out the lack of clarity > &g

Re: [petsc-dev] PetscCUDAInitialize

On Sep 18, 2019, at 11:25 PM, Zhang, Junchao via petsc-dev > mailto:petsc-dev@mcs.anl.gov>> wrote: > > Barry, > > I saw you added these in init.c > > > + -cuda_initialize - do the initialization in PetscInitialize() > > > > > > > > > Notes:

[petsc-dev] PetscCUDAInitialize

Barry, I saw you added these in init.c + -cuda_initialize - do the initialization in PetscInitialize() Notes: Initializing cuBLAS takes about 1/2 second there it is done by default in PetscInitialize() before logging begins But I did not get otherwise with -cuda_initialize 0, when will

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT

On Sat, Aug 31, 2019 at 8:04 PM Mark Adams mailto:mfad...@lbl.gov>> wrote: On Sat, Aug 31, 2019 at 4:28 PM Smith, Barry F. mailto:bsm...@mcs.anl.gov>> wrote: Any explanation for why the scaling is much better for CPUs and than GPUs? Is it the "extra" time needed for communication from

Re: [petsc-dev] DMDAGlobalToNatural errors with Ubuntu:latest; gcc 7 & Open MPI 2.1.1

Some updates for this OpenMPI bug: 1) It appears to OpenMPI 2.1.x when configured with --enable-heterogeneous, which is not a default option and is not commonly used. But Ubuntu somehow used that. 2) OpenMPI fixed it in 3.x 3) It was reported to Ubuntu two years ago but is still unassigned.

Re: [petsc-dev] DMDAGlobalToNatural errors with Ubuntu:latest; gcc 7 & Open MPI 2.1.1

Hi, Fabian, I found it is an OpenMPI bug w.r.t self-to-self MPI_Send/Recv using MPI_ANY_SOURCE for message matching. OpenMPI does not put correct value in recv buffer. I have a workaround

Re: [petsc-dev] DMDAGlobalToNatural errors with Ubuntu:latest; gcc 7 & Open MPI 2.1.1

Fabian, I happen have a Ubuntu virtual machine and I could reproduce the error with your mini-test, even with two processes. It is horrible to see wrong results in such a simple test. We'd better figure out whether it is a PETSc bug or an OpenMPI bug. If it is latter, which MPI call is at

Re: [petsc-dev] Issues with Fortran Interfaces for PetscSort routines

Fixed in jczhang/fix-sort-fortran-binding and will be in master later. Thanks. --Junchao Zhang On Mon, Jul 29, 2019 at 10:14 AM Fabian.Jakub via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Dear Petsc, Commit 051fd8986cf23c0556f4229193defe128fafa1f7 changed the C signature of the sorting

Re: [petsc-dev] (no subject)

We should be able to overlap PetscSFReduce. I will have a look. Thanks. --Junchao Zhang On Mon, Jul 22, 2019 at 5:23 AM Stefano Zampini mailto:stefano.zamp...@gmail.com>> wrote: Junchao, I found an issue with PetscSFReduceBegin/End. It seems that we can no longer do (always worked before)

Re: [petsc-dev] Fwd: PETSc blame digest (next-tmp) 2019-07-17

I changed the include file from petscmat.h to petsc.h, one more similar warning showed up in petsctao.h. So we can conclude Sun C++ works for headers except petscmat.h and petsctao.h My previous fix does not work on Linux because PETSC_EXTERN conflicts with PETSC_STATIC_INLINE. I have to give

Re: [petsc-dev] Fwd: PETSc blame digest (next-tmp) 2019-07-17

Fande did a fantastic job using C++11 in the test code. I modified it so it does not use C++11 anymore. But I think Satish's code is still useful and we should have it in a new PR. Another serious issue using Sun C++ with this C++ test code: "/export/home/jczhang/petsc/include/petscmat.h", line

Re: [petsc-dev] [Radev, Martin] Re: Adding a new encoding for FP data

A side question: Do lossy compressors have value for PETSc? --Junchao Zhang On Thu, Jul 11, 2019 at 9:06 AM Jed Brown via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Zstd is a remarkably good compressor. I've experimented with it for compressing column indices for sparse matrices on

Re: [petsc-dev] Slowness of PetscSortIntWithArrayPair in MatAssembly

Fande, I ran your code with two processes and found the poor performance of PetscSortIntWithArrayPair() was due to duplicates. In particular, rank 0 has array length = 0 and rank 1 has array length = 4,180,070. On rank 1, each unique array value has ~95 duplicates; The duplicates are

Re: [petsc-dev] Slowness of PetscSortIntWithArrayPair in MatAssembly

Is the code public for me to test? --Junchao Zhang On Mon, Jul 8, 2019 at 3:06 PM Fande Kong mailto:fdkong...@gmail.com>> wrote: Thanks Junchao, Tried your code. I did not hit seg fault this time, but the assembly was still slow time mpirun -n 2 ./matrix_sparsity-opt -matstash_legacy

Re: [petsc-dev] Slowness of PetscSortIntWithArrayPair in MatAssembly

Could you debug it or paste the stack trace? Since it is a segfault, it should be easy. --Junchao Zhang On Wed, Jul 3, 2019 at 5:16 PM Fande Kong mailto:fdkong...@gmail.com>> wrote: Thanks Junchao, But there is still segment fault. I guess you could write some continuous integers to test

Re: [petsc-dev] Slowness of PetscSortIntWithArrayPair in MatAssembly

Fande and John, Could you try jczhang/feature-better-quicksort-pivot? It passed Jenkins tests and I could not imagine why it failed on yours. Hash table has its own cost. We'd better get quicksort right and see how it performs before rewriting code. --Junchao Zhang On Tue, Jul 2, 2019 at

Re: [petsc-dev] Slowness of PetscSortIntWithArrayPair in MatAssembly

Try this to see if it helps: diff --git a/src/sys/utils/sorti.c b/src/sys/utils/sorti.c index 1b07205a..90779891 100644 --- a/src/sys/utils/sorti.c +++ b/src/sys/utils/sorti.c @@ -294,7 +294,8 @@ static PetscErrorCode PetscSortIntWithArrayPair_Private(PetscInt *L,PetscInt *J, }

Re: [petsc-dev] Slowness of PetscSortIntWithArrayPair in MatAssembly

Is it because the array is already sorted? --Junchao Zhang On Tue, Jul 2, 2019 at 12:13 PM Fande Kong via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Hi Developers, John just noticed that the matrix assembly was slow when having sufficient amount of off-diagonal entries. It was not a MPI

Re: [petsc-dev] PETSc blame digest (next-tmp) 2019-06-27

Fixed. --Junchao Zhang On Thu, Jun 27, 2019 at 3:28 PM PETSc checkBuilds mailto:petsc-checkbui...@mcs.anl.gov>> wrote: Dear PETSc developer, This email contains listings of contributions attributed to you by `git blame` that caused compiler errors or warnings in PETSc automated testing.

Re: [petsc-dev] Is DMGetDMKSPWrite() really not collective?

mailto:jczh...@mcs.anl.gov>> wrote: On Fri, Jun 14, 2019 at 1:01 PM Lawrence Mitchell mailto:we...@gmx.li>> wrote: > On 14 Jun 2019, at 18:44, Zhang, Junchao via petsc-dev > mailto:petsc-dev@mcs.anl.gov>> wrote: > > Hello, >I am investigating petsc is

Re: [petsc-dev] Is DMGetDMKSPWrite() really not collective?

On Fri, Jun 14, 2019 at 1:01 PM Lawrence Mitchell mailto:we...@gmx.li>> wrote: > On 14 Jun 2019, at 18:44, Zhang, Junchao via petsc-dev > mailto:petsc-dev@mcs.anl.gov>> wrote: > > Hello, >I am investigating petsc issue 306. One can produce the problem with &g

Re: [petsc-dev] Is DMGetDMKSPWrite() really not collective?

In PCSetUp_MG, processors can diverge in the red line, DMGetDMKSPWrite for (i=n-2; i>-1; i--) { DMKSP kdm; PetscBool dmhasrestrict, dmhasinject; ierr = KSPSetDM(mglevels[i]->smoothd,dms[i]);CHKERRQ(ierr); if (!needRestricts) {ierr =

[petsc-dev] Is DMGetDMKSPWrite() really not collective?

Hello, I am investigating petsc issue 306. One can produce the problem with src/snes/examples/tutorials/ex9.c and mpirun -n 3 ./ex9 -snes_grid_sequence 3 -snes_converged_reason -pc_type mg The program can run

Re: [petsc-dev] Bad use of defined(MPI_XXX)

PetscSF has many PETSC_HAVE_MPI_REDUCE_LOCAL. It is disturbing. But consider the time gap between MPI-2.0 (1998) and MPI-2.2 (2009), it is better to keep it. On Fri, May 24, 2019 at 3:53 PM Jed Brown mailto:j...@jedbrown.org>> wrote: "Zhang, Junchao" mailto:jczh...@mcs.anl.gov>> writes: > How

Re: [petsc-dev] Bad use of defined(MPI_XXX)

How about stuff in MPI-2.2 (approved in 2009), the last of MPI-2.x, e.g., PETSC_HAVE_MPI_REDUCE_LOCAL? On Fri, May 24, 2019 at 2:51 PM Jed Brown via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Lisandro Dalcin via petsc-dev mailto:petsc-dev@mcs.anl.gov>> writes: > These two are definitely

Re: [petsc-dev] Deprecation strategy for Enums

>> wrote: > > "Zhang, Junchao via petsc-dev" > mailto:petsc-dev@mcs.anl.gov>> writes: > >> We should have a mechanism to auto-detect API-breaking commits and then we >> can fix them before release. > > We should have our CI system flag PRs that

Re: [petsc-dev] Deprecation strategy for Enums

We should have a mechanism to auto-detect API-breaking commits and then we can fix them before release. --Junchao Zhang On Tue, Apr 9, 2019 at 12:39 PM Matthew Knepley via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: This change:

Re: [petsc-dev] MPI_UB is deprecated in MPI-2.0

I pushed an update to this branch, which adopts MPI_Type_create_resized. --Junchao Zhang On Tue, Mar 19, 2019 at 11:56 AM Balay, Satish via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: For now - I'm merging this branch to next. If better fix comes up later we can merge it then. thanks,

Re: [petsc-dev] https://www.dursi.ca/post/hpc-is-dying-and-mpi-is-killing-it.html

Let's see how the author thought about PETSc. The author likes Chapel -- a PGAS language. In https://www.dursi.ca/post/julia-vs-chapel.html he said his concerns about Chapel "the beginnings of a Chapel-native set of solvers from Scalapack or PETSc (both of which are notoriously hard to get

Re: [petsc-dev] errors with cuda + mumps

search='snes_tutorials-ex69_q2p1fetidp_deluxe snes_tutorials-ex62_fetidp_2d_quad snes_tutorials-ex69_q2p1fetidp_deluxe_adaptive ksp_ksp_tutorials-ex52f_mumps' Already made a PR. --Junchao Zhang On Wed, Mar 13, 2019 at 4:55 PM Balay, Satish mailto:ba...@mcs.anl.gov>> wrote: On Wed, 13 Mar

Re: [petsc-dev] errors with cuda + mumps

On Wed, Mar 13, 2019 at 3:49 PM Balay, Satish mailto:ba...@mcs.anl.gov>> wrote: mailers that only format html and not the text form are annoying :( On Wed, 13 Mar 2019, Zhang, Junchao wrote: > Satish, I found something strange. I configured with --with-cuda > --with-precision=single,

Re: [petsc-dev] errors with cuda + mumps

Satish, I found something strange. I configured with --with-cuda --with-precision=single, then with -log_view, I saw Compiled with single precision PetscScalar and PetscReal Compiled with full precision matrices (default) I found that confused mumps. When I added --with-precision=double, I

[petsc-dev] Fwd: MPI_UB is deprecated in MPI-2.0

eplacing use of MPI_UB . Bill On Mar 13, 2019 1:34 PM, "Zhang, Junchao via petsc-dev" mailto:petsc-dev@mcs.anl.gov>> wrote: On Wed, Mar 13, 2019 at 12:48 PM Isaac, Tobin G via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Pushed a fix that just uses MPI_Type_contiguo

Re: [petsc-dev] MPI_UB is deprecated in MPI-2.0

On Wed, Mar 13, 2019 at 12:48 PM Isaac, Tobin G via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Pushed a fix that just uses MPI_Type_contiguous(MPI_BYTE, sizeof(), ...), which is not great but I'm only creating the type to work with PetscSF, so it does the job. Satish, do you want this

Re: [petsc-dev] errors with cuda + mumps

On Wed, Mar 13, 2019 at 11:28 AM Matthew Knepley mailto:knep...@gmail.com>> wrote: On Wed, Mar 13, 2019 at 12:16 PM Zhang, Junchao via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: I met some errors with cuda + mumps. It does not look like CUDA is being used in these runs. Is it

Re: [petsc-dev] errors with cuda + mumps

I do not know the reason. --Junchao Zhang On Wed, Mar 13, 2019 at 11:15 AM Junchao Zhang mailto:jczh...@mcs.anl.gov>> wrote: I met some errors with cuda + mumps. It was tested with make -f gmakefile test search='snes_tutorials-ex69_q2p1fetidp_deluxe snes_tutorials-ex62_fetidp_2d_quad

[petsc-dev] errors with cuda + mumps

I met some errors with cuda + mumps. It was tested with make -f gmakefile test search='snes_tutorials-ex69_q2p1fetidp_deluxe snes_tutorials-ex62_fetidp_2d_quad snes_tutorials-ex69_q2p1fetidp_deluxe_adaptive ksp_ksp_tutorials-ex52f_mumps' I can reproduce it with petsc master. The first line of

Re: [petsc-dev] Unwanted Fortran stub

l page but no stub then use /*@C Barry > On Mar 12, 2019, at 1:46 PM, Zhang, Junchao via petsc-dev > mailto:petsc-dev@mcs.anl.gov>> wrote: > > I declared PETSC_INTERN PetscErrorCode VecLockWriteSet_Private(Vec,PetscBool) > in vecimp.h and defined it in src/vec/vec/i

[petsc-dev] Unwanted Fortran stub

I declared PETSC_INTERN PetscErrorCode VecLockWriteSet_Private(Vec,PetscBool) in vecimp.h and defined it in src/vec/vec/interface/rvector.c. I used PETSC_INTERN and _Private since currently the function is only used in the Vec packaged and is not public. I met this compilation warning

[petsc-dev] Fwd: Arm DDT feature questions

In an Arm training class, I requested a DDT feature from John Linford from Arm Inc, which now owns Allinea DDT. Basically, I want DDT to correctly display petsc variable-lengthed arrays and void* pointers. From John's feedback, it looks DDT could support it. Does anyone already have custom

[petsc-dev] No VecRestoreArrayRead_Nest()?

I did not see VecRestoreArrayRead_Nest. On VecNest, VecRestoreArrayRead defaults to VecRestoreArray(x,a) , which copies a[] to x and is expensive. Is it an oversight? --Junchao Zhang

Re: [petsc-dev] How to add root values to leaves in PetscSF?

On Tue, Jan 22, 2019 at 5:07 PM Jed Brown mailto:j...@jedbrown.org>> wrote: "Zhang, Junchao" mailto:jczh...@mcs.anl.gov>> writes: > On Tue, Jan 22, 2019 at 4:08 PM Jed Brown > mailto:j...@jedbrown.org>>> > wrote: >> It is not supported at

Re: [petsc-dev] How to add root values to leaves in PetscSF?

ATTER_FORWARD. Now I wan to do SCATTER_REVERSE with ADD_VALUES. To solve this problem without creating another SF and without breaking current SF API, I propose to add PetscSFBcastAndOp(sf, unit, rootdata, leafdata, op) "Zhang, Junchao via petsc-dev" mailto:petsc-dev@mcs.anl.gov>&g

Re: [petsc-dev] How to add root values to leaves in PetscSF?

On Tue, Jan 22, 2019 at 1:35 PM Matthew Knepley mailto:knep...@gmail.com>> wrote: On Tue, Jan 22, 2019 at 2:23 PM Zhang, Junchao via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: I want to add root values to leaves, and keep root unchanged. PetscSFBcast came to my mind, but u

[petsc-dev] How to add root values to leaves in PetscSF?

I want to add root values to leaves, and keep root unchanged. PetscSFBcast came to my mind, but unfortunately it only broadcasts roots and does not have an MPI_Op argument like PetscSFReduce for me to choose from INSERT_VALUES, ADD_VALUES, etc. Any tips? Thanks. PS: I met this problem when I