Probably matrix assembly on GPU is more important. Do you have an example for me to play to see what GPU interface we should have? --Junchao Zhang
On Mon, Apr 13, 2020 at 5:44 PM Mark Adams <mfad...@lbl.gov> wrote: > I was looking into assembling matrices with threads. I have a coloring to > avoid conflicts. > > Turning off all the logging seems way overkill and for methods that can > get called in a thread then we could use PETSC_HAVE_THREADSAFTEY (thingy) > to protect logging functions. So one can still get timings for the whole > assembly process, just not for MatSetValues. Few people are going to do > this. I don't think it will be a time sink, and if it is we just revert > back to saying 'turn logging off'. I don't see a good argument for > insisting on turning off logging, it is pretty important, if we just say > that we are going to protect methods as needed. > > It is not a big deal, I am just exploring this idea. It is such a basic > concept in shared memory sparse linear algebra that it seems like a good > thing to be able to support and have in an example to say we can assemble > matrices in threads (not that it is a great idea). We have all the tools > (eg, coloring methods) that it is just a matter of protecting code a few > methods. I use DMPlex MatClosure instead of MatSetValues and this is where > I die now with non-thread safe code. We have an idea, from Jed, on how to > fix it. > > Anyway, thanks for your help, but I think we should hold off on doing > anything until we have some consensus that this would be a good idea to put > some effort into getting a thread safe PETSc that can support OMP matrix > assembly with a nice compact example. > > Thanks again, > Mark > > On Mon, Apr 13, 2020 at 5:44 PM Junchao Zhang <junchao.zh...@gmail.com> > wrote: > >> Mark, >> I saw you had "--with-threadsaftey --with-log=0". Do you really want to >> call petsc from multiple threads (in contrast to letting petsc call >> other libraries, e.g., BLAS, doing multithreading)? If not, you can >> drop --with-threadsaftey. >> I have https://gitlab.com/petsc/petsc/-/merge_requests/2714 that should >> fix your original compilation errors. >> >> --Junchao Zhang >> >> On Mon, Apr 13, 2020 at 2:07 PM Mark Adams <mfad...@lbl.gov> wrote: >> >>> https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html >>> >>> and I see this on my Mac: >>> >>> 14:23 1 mark/feature-xgc-interface-rebase *= ~/Codes/petsc$ >>> ../arch-macosx-gnu-O-omp.py >>> >>> >>> >>> =============================================================================== >>> Configuring PETSc to compile on your system >>> >>> >>> =============================================================================== >>> =============================================================================== >>> >>> >>> Warning: PETSC_ARCH from environment does not match >>> command-line or name of script. >>> >>> Warning: Using from command-line or >>> name of script: arch-macosx-gnu-O-omp, ignoring environment: >>> arch-macosx-gnu-g >>> >>> =============================================================================== >>> >>> >>> TESTING: configureLibraryOptions from >>> PETSc.options.libraryOptions(config/PETSc/options/libraryOptions.py:37) >>> >>> >>> >>> ******************************************************************************* >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >>> for details): >>> >>> ------------------------------------------------------------------------------- >>> Must use --with-log=0 with --with-threadsafety >>> >>> ******************************************************************************* >>> >>> >>> On Mon, Apr 13, 2020 at 2:54 PM Junchao Zhang <junchao.zh...@gmail.com> >>> wrote: >>> >>>> >>>> >>>> >>>> On Mon, Apr 13, 2020 at 12:06 PM Mark Adams <mfad...@lbl.gov> wrote: >>>> >>>>> BTW, I can build on SUMMIT with logging and OMP, apparently. I also >>>>> seem to be able to build with debugging. Both of which are not allowed >>>>> according the the docs. I am puzzled. >>>>> >>>> What are "the docs"? >>>> >>>>> >>>>> On Mon, Apr 13, 2020 at 12:05 PM Mark Adams <mfad...@lbl.gov> wrote: >>>>> >>>>>> I think the problem is that you have to turn off logging with openmp >>>>>> and the (newish) GPU timers did not protect their timers. >>>>>> >>>>>> I don't see a good reason to require logging be turned off with OMP. >>>>>> We could use PETSC_HAVE_THREADSAFETY to protect logs that we care about >>>>>> (eg, in MatSetValues) and as users discover more things that they want to >>>>>> call in an OMP thread block, then tell them to turn logging off and we >>>>>> will >>>>>> fix it when we can. >>>>>> >>>>>> Any thoughts on the idea of letting users keep logging with openmp? >>>>>> >>>>>> On Mon, Apr 13, 2020 at 11:40 AM Junchao Zhang < >>>>>> junchao.zh...@gmail.com> wrote: >>>>>> >>>>>>> Yes. Looks we need to include petsclog.h. Don't know why OMP >>>>>>> triggered the error. >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Mon, Apr 13, 2020 at 9:59 AM Mark Adams <mfad...@lbl.gov> wrote: >>>>>>> >>>>>>>> Should I do an MR to fix this? >>>>>>>> >>>>>>>