There was a bug in my fix for parallel which I have fixed. You will need to
git fetch git checkout main git branch -D barry/2024-05-27/fix-bug-baij-setvaluesblocked/release git checkout barry/2024-05-27/fix-bug-baij-setvaluesblocked/release I get the same results with your example with 1,2, and 3 ranks > On May 28, 2024, at 7:45 PM, Adrian Croucher <a.crouc...@auckland.ac.nz> > wrote: > > hi Barry, > > Thanks, that change has fixed the error on 2 ranks for me. > > When I run on 3 ranks, there is no error, but it doesn't actually add the > extra values in to the matrix. Do you see that behaviour too? > > - Adrian > > On 29/05/24 4:33 am, Barry Smith wrote: >> >> Adrian, >> >> I could reproduce with 3 MPI ranks. >> >> Another error I had to fix. I also added a test example >> >> SInce I rebased the branch you will need to do something like >> >> git fetch >> git checkout main >> git branch -D barry/2024-05-27/fix-bug-baij-setvaluesblocked/release >> git checkout barry/2024-05-27/fix-bug-baij-setvaluesblocked/release >> >> Thanks for your patience >> >> Barry >> >> >>> On May 27, 2024, at 10:42 PM, Adrian Croucher <a.crouc...@auckland.ac.nz> >>> <mailto:a.crouc...@auckland.ac.nz> wrote: >>> >>> Hmm, that's a bit weird. I haven't modified the test code - I checked the >>> file date to make sure. I also tried deleting my PETSc build dir and >>> rebuilding it, then rebuilding the test code. I still get the error if I >>> run on 2 ranks with -dm_mat_type (or -mat_type) baij or mpibaij, but it's >>> fine on aij or mpiaij. >>> >>> My actual (non-test) code is however working ok now, in serial or parallel. >>> So I don't think this should hold up merging your bugfix. >>> >>> - Adrian >>> >>> On 28/05/24 2:13 pm, Barry Smith wrote: >>>> >>>> When I run the exact code you sent with two ranks and—mat_type mpibaij, >>>> it runs as expected. If you modified the code in any way to demonstrate >>>> the bug, please send the modified code. >>>> >>>> >>>> >>>>> On May 27, 2024, at 9:37 PM, Adrian Croucher <a.crouc...@auckland.ac.nz> >>>>> <mailto:a.crouc...@auckland.ac.nz> wrote: >>>>> >>>>> hi Barry, >>>>> >>>>> On 28/05/24 7:46 am, Barry Smith wrote: >>>>>> >>>>>> Thanks for reporting this. It is a bug. I have a fixed branch >>>>>> barry/2024-05-27/fix-bug-baij-setvaluesblocked/release and associated >>>>>> merge request >>>>>> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7578__;!!G_uCfscf7eWS!dngbcXmJfofRckcEMElrf4BvXnHLnRh4FePsIwherUCHmsPBqTWSGvW36-62EUO3uDDIffqW8q9rei8toRdYLUY$ >>>>>> >>>>>> >>>>> Thanks very much, that does appear to fix the bug when I run my test >>>>> program in serial. >>>>> >>>>> If I run it on 2 processes, it is OK with AIJ matrix type, but with BAIJ >>>>> I get an error (see below). Is there another problem, or I am doing >>>>> something else wrong? >>>>> >>>>> - Adrian >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>> [0]PETSC ERROR: Incorrect colmap >>>>> [0]PETSC ERROR: See >>>>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dngbcXmJfofRckcEMElrf4BvXnHLnRh4FePsIwherUCHmsPBqTWSGvW36-62EUO3uDDIffqW8q9rei8tCH9MbY4$ >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.21.1, unknown >>>>> [0]PETSC ERROR: ./matmodify on a main-debug named EN438880 by acro018 Tue >>>>> May 28 13:33:46 2024 >>>>> [0]PETSC ERROR: Configure options --with-x --download-hdf5 >>>>> --download-zlib --download-netcdf --download-pnetcdf --download-exodusii >>>>> --download-triangle --download-ptscotch --download-chaco --download-hypre >>>>> [0]PETSC ERROR: #1 MatSetValuesBlocked_MPIBAIJ() at >>>>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:448 >>>>> [0]PETSC ERROR: #2 MatSetValuesBlocked() at >>>>> /home/acro018/software/PETSc/code/src/mat/interface/matrix.c:2030 >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: It appears a new error in the code was triggered after >>>>> a previous error, possibly because: >>>>> [0]PETSC ERROR: - The first error was not properly handled via (for >>>>> example) the use of >>>>> [0]PETSC ERROR: PetscCall(TheFunctionThatErrors()); or >>>>> [0]PETSC ERROR: - The second error was triggered while handling the >>>>> first error. >>>>> [0]PETSC ERROR: Above is the traceback for the previous unhandled >>>>> error, below the traceback for the next error >>>>> [0]PETSC ERROR: ALL ERRORS in the PETSc libraries are fatal, you should >>>>> add the appropriate error checking to the code >>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>> [0]PETSC ERROR: Incorrect colmap >>>>> [0]PETSC ERROR: See >>>>> https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dngbcXmJfofRckcEMElrf4BvXnHLnRh4FePsIwherUCHmsPBqTWSGvW36-62EUO3uDDIffqW8q9rei8tCH9MbY4$ >>>>> for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.21.1, unknown >>>>> [0]PETSC ERROR: ./matmodify on a main-debug named EN438880 by acro018 Tue >>>>> May 28 13:33:46 2024 >>>>> [0]PETSC ERROR: Configure options --with-x --download-hdf5 >>>>> --download-zlib --download-netcdf --download-pnetcdf --download-exodusii >>>>> --download-triangle --download-ptscotch --download-chaco --download-hypre >>>>> [0]PETSC ERROR: #1 MatSetValuesBlocked_MPIBAIJ() at >>>>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:448 >>>>> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIBAIJ() at >>>>> /home/acro018/software/PETSc/code/src/mat/impls/baij/mpi/mpibaij.c:943 >>>>> [0]PETSC ERROR: #3 MatAssemblyEnd() at >>>>> /home/acro018/software/PETSc/code/src/mat/interface/matrix.c:5820 >>>>> [0]PETSC ERROR: #4 matmodify.F90:40 >>>>> >>>>> >>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>>> On May 26, 2024, at 10:45 PM, Adrian Croucher >>>>>>> <a.crouc...@auckland.ac.nz> <mailto:a.crouc...@auckland.ac.nz> wrote: >>>>>>> >>>>>>> hi, >>>>>>> >>>>>>> I've been trying creating a matrix with DMCreateMatrix() and then >>>>>>> adding extra blocks of nonzeros into it using MatSetValuesBlocked(), >>>>>>> but getting some unexpected results if I set the matrix type to BAIJ. >>>>>>> It seems to behave as expected if I use matrix type AIJ. >>>>>>> >>>>>>> I've attached a minimal example program. It reads in the DMPlex from >>>>>>> file, sets up a section on it, creates a matrix (blocksize 2) and then >>>>>>> inserts a single 2x2 block at global block indices (0,7). It views the >>>>>>> matrix before and after the insertion. >>>>>>> >>>>>>> If I run with "-dm_mat_type aij" it gives the expected results, but >>>>>>> with "-dm_mat_type baij" it doesn't - e.g. if run in serial, it adds >>>>>>> the new nonzeros in the right place but also adds a whole lot of other >>>>>>> duplicated entries in block row 0. >>>>>>> >>>>>>> Is there something I'm not understanding about BAIJ, or about >>>>>>> MatSetValuesBlocked()? or possibly some other mistake? >>>>>>> >>>>>>> - Adrian >>>>>>> >>>>>>> On 20/05/24 12:24 pm, Barry Smith wrote: >>>>>>>> >>>>>>>> You can call MatSetOption(mat,MAT_NEW_NONZERO_LOCATION_ERR) then >>>>>>>> insert the new values. If it is just a handful of new insertions the >>>>>>>> extra time should be small. >>>>>>>> >>>>>>>> Making a copy of the matrix won't give you a new matrix that is >>>>>>>> any faster to insert into so best to just use the same matrix. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>>> On May 19, 2024, at 7:44 PM, Adrian Croucher >>>>>>>>> <a.crouc...@auckland.ac.nz> <mailto:a.crouc...@auckland.ac.nz> wrote: >>>>>>>>> >>>>>>>>> This Message Is From an External Sender >>>>>>>>> This message came from outside your organization. >>>>>>>>> hi, >>>>>>>>> >>>>>>>>> I have a Jacobian matrix created using DMCreateMatrix(). What would >>>>>>>>> be >>>>>>>>> the best way to add extra nonzero entries into it? >>>>>>>>> >>>>>>>>> I'm guessing that DMCreateMatrix() allocates the storage so the >>>>>>>>> nonzero >>>>>>>>> structure can't really be easily modified. Would it be a case of >>>>>>>>> creating a new matrix, copying the nonzero entries from the original >>>>>>>>> one >>>>>>>>> and then adding the extra ones, before calling MatSetUp() or similar? >>>>>>>>> If >>>>>>>>> so, how exactly would you copy the nonzero structure from the >>>>>>>>> original >>>>>>>>> matrix? >>>>>>>>> >>>>>>>>> Background: the flow problem I'm solving (on a DMPlex with finite >>>>>>>>> volume >>>>>>>>> method) has complex source terms that depend on the solution (e.g. >>>>>>>>> pressure), and can also depend on other source terms. A simple >>>>>>>>> example >>>>>>>>> is when fluid is extracted from one location, with a >>>>>>>>> pressure-dependent >>>>>>>>> flow rate, and some of it is then reinjected in another location. >>>>>>>>> This >>>>>>>>> can result in poor nonlinear solver convergence. I think the reason >>>>>>>>> is >>>>>>>>> that there are effectively missing Jacobian entries in the row for >>>>>>>>> the >>>>>>>>> reinjection cell, which should have an additional dependence on the >>>>>>>>> solution in the cell where fluid is extracted. >>>>>>>>> >>>>>>>>> - Adrian >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dr Adrian Croucher >>>>>>>>> Senior Research Fellow >>>>>>>>> Department of Engineering Science >>>>>>>>> Waipapa Taumata Rau / University of Auckland, New Zealand >>>>>>>>> email: a.crouc...@auckland.ac.nz <mailto:a.crouc...@auckland.ac.nz> >>>>>>>>> tel: +64 (0)9 923 4611 >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Dr Adrian Croucher >>>>>>> Senior Research Fellow >>>>>>> Department of Engineering Science >>>>>>> Waipapa Taumata Rau / University of Auckland, New Zealand >>>>>>> email: a.crouc...@auckland.ac.nz <mailto:a.crouc...@auckland.ac.nz> >>>>>>> tel: +64 (0)9 923 4611 >>>>>>> <3x3grid.exo><matmodify.F90> >>>>>> >>>>> -- >>>>> Dr Adrian Croucher >>>>> Senior Research Fellow >>>>> Department of Engineering Science >>>>> Waipapa Taumata Rau / University of Auckland, New Zealand >>>>> email: a.crouc...@auckland.ac.nz <mailto:a.crouc...@auckland.ac.nz> >>>>> tel: +64 (0)9 923 4611 >>>> >>> -- >>> Dr Adrian Croucher >>> Senior Research Fellow >>> Department of Engineering Science >>> Waipapa Taumata Rau / University of Auckland, New Zealand >>> email: a.crouc...@auckland.ac.nz <mailto:a.crouc...@auckland.ac.nz> >>> tel: +64 (0)9 923 4611 >> > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > Waipapa Taumata Rau / University of Auckland, New Zealand > email: a.crouc...@auckland.ac.nz <mailto:a.crouc...@auckland.ac.nz> > tel: +64 (0)9 923 4611