[petsc-dev] PETSc Social on line in 2 hours. 5pm Central time

We are having a PETSc social event at 5 pm central time today (in 2 hours). Everyone is welcome. Hope to see many of you, Barry Begin forwarded message: From: Todd Munson via BlueJeans Network mailto:inv...@bluejeans.com>> Subject: PETSc Social Date: March 31, 2020 at 5:16:32 PM CDT T

[petsc-dev] Fwd: [Xlab] El Capitan CPU announcement

Begin forwarded message: From: "Thakur, Rajeev" mailto:tha...@anl.gov>> Subject: [Xlab] El Capitan CPU announcement Date: March 4, 2020 at 1:33:13 PM CST To: "x...@cels.anl.gov" mailto:x...@cels.anl.gov>> AMD https://www.anandtech.com/show/15581/el-capitan-supercomput

Re: [petsc-dev] [petsc-users] Matrix-free method in PETSc

PETSc developers and users, I apologize for my rude and unfair message to PETSc developers. I have removed myself from the PETSc mailing lists and removed my access to the PETSc repository and will no longer be involved in PETSc. Barry > On Feb 18, 2020, at 9:10 PM, Smith, B

[petsc-dev] Fwd: [petsc-users] Matrix-free method in PETSc

In the past you needed a brain to get a Stanford email account Begin forwarded message: From: Yuyun Yang mailto:yyan...@stanford.edu>> Subject: Re: [petsc-users] Matrix-free method in PETSc Date: February 18, 2020 at 8:26:11 AM CST To: Matthew Knepley mailto:knep...@gmail.com>> Cc: "Smith, Ba

Re: [petsc-dev] First call to cudaMalloc or cudaFree is very slow on summit

> On Feb 13, 2020, at 5:39 PM, Zhang, Hong wrote: > > > >> On Feb 13, 2020, at 7:39 AM, Smith, Barry F. wrote: >> >> >> How are the two being compiled and linked? The same way, one with the PETSc >> library in the path and the other without? Or does the PETSc one have lots >> of flags an

Re: [petsc-dev] First call to cudaMalloc or cudaFree is very slow on summit

How are the two being compiled and linked? The same way, one with the PETSc library in the path and the other without? Or does the PETSc one have lots of flags and stuff while the non-PETSc one is just simple by hand? Barry > On Feb 12, 2020, at 7:29 PM, Zhang, Hong wrote: > > > >> O

Re: [petsc-dev] First call to cudaMalloc or cudaFree is very slow on summit

ldd -o on the petsc program (static) and the non petsc program (static), what are the differences? nm -o both executables | grep cudaFree() > On Feb 12, 2020, at 1:51 PM, Munson, Todd via petsc-dev > wrote: > > > There are some side effects when loading shared libraries, such as >

Re: [petsc-dev] First call to cudaMalloc or cudaFree is very slow on summit

gprof or some similar tool? > On Feb 10, 2020, at 11:18 AM, Zhang, Hong via petsc-dev > wrote: > > -cuda_initialize 0 does not make any difference. Actually this issue has > nothing to do with PetscInitialize(). I tried to call cudaFree(0) before > PetscInitialize(), and it still took 7.

Re: [petsc-dev] First call to cudaMalloc or cudaFree is very slow on summit

Perhaps the intent is that you build or install (--prefix) your libraries in a different place than /autofs/nccs-svm1_home1 > On Feb 7, 2020, at 3:09 PM, Zhang, Hong wrote: > > Note that the overhead was triggered by the first call to a CUDA function. So > it seems that the first CUDA

Re: [petsc-dev] First call to cudaMalloc or cudaFree is very slow on summit

ldd -o on the executable of both linkings of your code. My guess is that without PETSc it is linking the static version of the needed libraries and with PETSc the shared. And, in typical fashion, the shared libraries are off on some super slow file system so take a long time to be loaded

[petsc-dev] Fwd: [SIAM-CSE] Introducing hIPPYlib, a python-based inverse problems solver library

Lois sent out this announcement on hIPPYlib 3.0 Begin forwarded message: From: "McInnes, Lois Curfman" mailto:curf...@anl.gov>> Subject: FW: [SIAM-CSE] Introducing hIPPYlib, a python-based inverse problems solver library Date: February 4, 2020 at 8:52:46 AM CST To: "Smith, Barry F." mailto:bs

Re: [petsc-dev] is make alldoc tested?

We should still be doing the testing now > On Feb 4, 2020, at 2:46 PM, Jed Brown wrote: > > Moving toward Sphinx (as Patrick has been working on) will help reduce > the number of stages and improve identification and reporting of > diagnostics/errors. > > Satish Balay via petsc-dev writes

Re: [petsc-dev] is make alldoc tested?

es in doc build targets - and errors in each > stage are ignored. So will have to fix them to stop on error [in the subset > of targets we build in CI] > > And all doc-only changes would have to be run through this CI test - before > merge.. > > Satish > > On Tue,

Re: [petsc-dev] is make alldoc tested?

- but not > in gitlab-ci. > > I have the fixes at https://gitlab.com/petsc/petsc/-/merge_requests/2503 > > Satish > > On Sat, 1 Feb 2020, Smith, Barry F. via petsc-dev wrote: > >> >> Generating manual example links >> Unexpected argument -print

[petsc-dev] is make alldoc tested?

Generating manual example links Unexpected argument -printmatch-link! Unexpected argument -printmatch-link! manualpages in: /Users/barrysmith/Src/petsc/src/snes/linesearch/impls/bt Error: Error reading html.def in path .:/Users/barrysmith/Src/petsc/arch-master/share or TEXTFILTER_PATH environm

Re: [petsc-dev] complex fix and -Wfloat-equal

> On Jan 27, 2020, at 3:42 PM, Lisandro Dalcin wrote: > > I usually compile my code with almost most warning flags on, including > -Wfloat-equal. > My implementation of the C++ complex fix that is not included by default is > an obvious offender (see warning messages at the end). > > A simp

Re: [petsc-dev] how do see artifacts?

Do they link with iCloud? ☠️ > On Jan 20, 2020, at 5:07 AM, Matthew Knepley wrote: > > Try Firefox or Chrome? > >Matt > > On Sun, Jan 19, 2020 at 11:41 PM Smith, Barry F. via petsc-dev > wrote: > > > > On Jan 19, 2020, at 10:39 PM, Jed Brown

Re: [petsc-dev] how do see artifacts?

> On Jan 19, 2020, at 10:39 PM, Jed Brown wrote: > > "Smith, Barry F. via petsc-dev" writes: > >> I have no left hand side. > > Right hand side? I certainly see them. Are you using a funny browser? Apple's :-) > >> If I grab the lef

Re: [petsc-dev] how do see artifacts?

ng a different > one > > https://gitlab.com/petsc/petsc/-/jobs/407945185 > > Attaching what I see [with the 'artifact' section in the right side column] > > Satish > > On Mon, 20 Jan 2020, Smith, Barry F. via petsc-dev wrote: > >> >> >>

Re: [petsc-dev] how do see artifacts?

g > > Satish > > On Sun, 19 Jan 2020, Smith, Barry F. via petsc-dev wrote: > >> >> With the new Gitlab interfaces how do you see the artifacts for failed >> pipelines/tests? I can't find them anywhere. It just takes me straight to >> the output with no sidebar for artifacts? >> >>Thanks >> >> Barry >> >

[petsc-dev] how do see artifacts?

With the new Gitlab interfaces how do you see the artifacts for failed pipelines/tests? I can't find them anywhere. It just takes me straight to the output with no sidebar for artifacts? Thanks Barry

Re: [petsc-dev] Fortran equivalent + separate output with output_file

Better to send more details. What function are you calling, what is the full error stack. Much quicker for us to help you then repeated email (you can never send too much information about a crash). PETSC_NULL_FUNCTION is the thing to use, but it is completely manual to for us to put them

Re: [petsc-dev] Fortran equivalent + separate output with output_file

> On Jan 13, 2020, at 9:32 AM, Pierre Jolivet > wrote: > > Hello, > This is actually two separate questions, sorry. > 1) I’m looking for the Fortran equivalent of the following, but I couldn’t > get any help looking at the sources. > ierr = PetscOptionsBegin(PETSC_COMM_WORLD,"","","");CHKERR

Re: [petsc-dev] Weird errors with enum in 64-bit ints

I looked at the code and see nothing glaring wrong. I guess it requires valgrind and the debugger on the machine where the trouble occurs. Your enum should not need to be changed. Barry > On Jan 13, 2020, at 3:52 AM, Matthew Knepley wrote: > > I have some unexplainable errors with 64-b

[petsc-dev] https://software.intel.com/en-us/devcloud/oneapi

Re: [petsc-dev] KSP in DMPlex

> On Nov 14, 2019, at 12:30 PM, Faibussowitsch, Jacob via petsc-dev > wrote: > > Hello, > > So I am trying to make a simple 5 pt stencil finite difference laplacian > operator (https://akmitra.public.iastate.edu/aero361/design_web/Laplace.pdf) > work on a dmplex mesh, but all of the example

Re: [petsc-dev] Gitlab notifications and labels

Scott, Thanks for researching this and send it out. Yes, I think all MR on DMPLEX should use a label DMPLEX; I don't think having an additional DM label is worth the effort since all the meat at the moment is DMPLEX. We should probably also have a TS label. Maybe others wil

Re: [petsc-dev] Parmetis bug

Nice. Anyway to add this exact reproducibility into a PETSc example that runs daily? The truism: all codes are buggy, even those that haven't been touched in 15 years, is definitely represented here. Barry > On Nov 10, 2019, at 7:31 PM, Fande Kong via petsc-dev > wrote: > > Valgr

[petsc-dev] Place to capture all our work on GPUs (and ECP ...)

Please do not respond to this email: use https://gitlab.com/petsc/petsc/issues/490 Mark Adams has been generating some great information on Summit with GAMG and now AMGx and other people such as Hannah and Junchao generating information important to our education about GPUs and, of cours

Re: [petsc-dev] Right-preconditioned GMRES

https://gitlab.com/petsc/petsc/merge_requests/2290 > On Nov 7, 2019, at 4:24 AM, Pierre Jolivet wrote: > > > >> On 7 Nov 2019, at 5:32 AM, Smith, Barry F. wrote: >> >> >> Some idiot logged what they did, but not why they did it. >> >> commit bf108f309acab50613e150419c680842cf4b8a05 (HEA

Re: [petsc-dev] Right-preconditioned GMRES

Some idiot logged what they did, but not why they did it. commit bf108f309acab50613e150419c680842cf4b8a05 (HEAD) Author: Barry Smith Date: Thu Mar 18 20:40:53 2004 -0600 bk-changeset-1.2063.1.1 barrysmith@barry-smiths-computer.local|ChangeSet|20040319024053|12244 ChangeSet

Re: [petsc-dev] Feed back on report on performance of vector operations on Summit requested

Jed, Thanks, this is very useful. Barry > On Oct 31, 2019, at 11:47 AM, Jed Brown wrote: > > "Smith, Barry F." writes: > >>> On Oct 23, 2019, at 7:15 PM, Jed Brown wrote: >>> >>> IMO, Figures 2 and 7+ are more interesting when the x axis (vector size) >>> is replaced by ex

Re: [petsc-dev] Feed back on report on performance of vector operations on Summit requested

ou give me access to the repository with data and current plotting > scripts, I can take a crack at slicing it in the way that I think would > be useful. Hannah will give you the data. Barry > > "Smith, Barry F. via petsc-dev" writes: > >> We've prepare

Re: [petsc-dev] Feed back on report on performance of vector operations on Summit requested

will report a much higher latency than the latter, because > synchronizations are expensive (i.e. your latency consists of kernel launch > latency plus device synchronization latency). Approach B is slightly > over-optimistic, but I've found it to better match what one observes for a

Re: [petsc-dev] AVX kernels, old gcc, still broken

> On Oct 26, 2019, at 9:09 AM, Jed Brown wrote: > > "Smith, Barry F." writes: > >> The proposed fix is #if defined(PETSC_USE_AVX512_KERNELS) && && && && && >> in https://gitlab.com/petsc/petsc/merge_requests/2213/diffs > > Looks fine; approved. > >> but note that PETSC_USE_AVX512_KERNE

Re: [petsc-dev] AVX kernels, old gcc, still broken

think it should be used minimally, you hate configure and think it should be used minimally. Barry > On Oct 25, 2019, at 1:54 PM, Jed Brown wrote: > > "Smith, Barry F. via petsc-dev" writes: > >> This needs to be fixed properly with a configure test(s)

Re: [petsc-dev] AVX kernels, old gcc, still broken

https://gitlab.com/petsc/petsc/issues/434 > On Oct 25, 2019, at 9:16 AM, Smith, Barry F. wrote: > > > This needs to be fixed properly with a configure test(s) and not with huge > and inconsistent checks like this > > #if defined(PETSC_HAVE_IMMINTRIN_H) && defined(__AVX512F__) && > def

Re: [petsc-dev] AVX kernels, old gcc, still broken

This needs to be fixed properly with a configure test(s) and not with huge and inconsistent checks like this #if defined(PETSC_HAVE_IMMINTRIN_H) && defined(__AVX512F__) && defined(PETSC_USE_REAL_DOUBLE) && !defined(PETSC_USE_COMPLEX) && !defined(PETSC_USE_64BIT_INDICES) or this #elif d

Re: [petsc-dev] Right-preconditioned GMRES

Send the code and exact instructions to run a "good" and a "bad" ASM Barry > On Oct 14, 2019, at 10:44 AM, Pierre Jolivet > wrote: > > Here are the three logs. > FGMRES also gives a wrong first iterate. > I think Mark was right in the sense that the problem is _most likely_ in my > RH

Re: [petsc-dev] PetscLayoutFindOwner and PetscLayoutFindOwnerIndex

These routines should be fixed. > On Oct 16, 2019, at 5:19 AM, Pierre Jolivet via petsc-dev > wrote: > > Hello, > These two functions use a parameter “owner” of type PetscInt*. > Shouldn’t this be PetscMPIInt*? > This implies changes left and right, so I want to check I’m not pushing an >

Re: [petsc-dev] I think the test system is broken in master

The test system really has to dump all the output including stderr to GitLab; the stack frames etc. Drop just little tidbits without all the error output is worse than users who cut and paste one line from configure.log and expect it to help. I don't know how you decide to redirect stuff an

Re: [petsc-dev] [Suggestion] Configure QOL Improvements

https://www.youtube.com/watch?v=NVopDink4uQ https://gitlab.com/petsc/petsc/merge_requests/2207 > On Oct 23, 2019, at 8:34 PM, Matthew Knepley via petsc-dev > wrote: > > On Wed, Oct 23, 2019 at 5:08 PM Faibussowitsch, Jacob > wrote: > I think Jed is referring to the fact that configure app

Re: [petsc-dev] Suggestion regarding CI issues

Great idea and good text: https://gitlab.com/petsc/petsc/issues/360 https://gitlab.com/petsc/petsc/wikis/home > On Oct 23, 2019, at 4:13 AM, Hapla Vaclav via petsc-dev > wrote: > > Issues related to CI testing often affect multiple independent MRs. There > should be a single point where

Re: [petsc-dev] Wrong "failed tests" command

May need more work on the tester infrastructure? > On Oct 21, 2019, at 12:30 PM, Pierre Jolivet via petsc-dev > wrote: > > Hello, > In this pipeline build log, https://gitlab.com/petsc/petsc/-/jobs/326525063, > it shows that I can rerun failed tests using the following command: > /usr/bin/m

Re: [petsc-dev] "participants" on gitlab

gt; > On Mon, Oct 21, 2019 at 9:47 AM Smith, Barry F. via petsc-dev > wrote: > > > > On Oct 21, 2019, at 10:27 AM, Zhang, Hong via petsc-dev > > wrote: > > > > How is the list of participants determined when a MR is created on gitlab? > > It seems

Re: [petsc-dev] ksp_error_if_not_converged in multilevel solvers

> On Oct 21, 2019, at 12:55 AM, Pierre Jolivet > wrote: > > > > On Oct 20, 2019, at 6:07 PM, "Smith, Barry F." wrote: > >> >> The reason the code works this way is that normally >> -ksp_error_if_not_converged is propagated into the inner (and innerer) >> solves and normally it is desir

Re: [petsc-dev] "participants" on gitlab

> On Oct 21, 2019, at 10:27 AM, Zhang, Hong via petsc-dev > wrote: > > How is the list of participants determined when a MR is created on gitlab? It > seems to include everybody by default. Is there any way to shorten the list? > Ideally only the participants involved in the particular MR sh

Re: [petsc-dev] BlockGetIndices and GetBlockIndices

Fix one of presumably many bugs related to the rush to add PetscLayout too IS without understanding the ramifications. https://gitlab.com/petsc/petsc/merge_requests/2191 > On Oct 21, 2019, at 8:34 AM, Pierre Jolivet via petsc-dev > wrote: > > >> On 21 Oct 2019, at 3:01 PM, Jed Brown wrote

Re: [petsc-dev] BlockGetIndices and GetBlockIndices

> On Oct 21, 2019, at 12:23 AM, Pierre Jolivet > wrote: > > > >> On 21 Oct 2019, at 7:11 AM, Smith, Barry F. wrote: >> >> >> >>> On Oct 20, 2019, at 11:52 PM, Pierre Jolivet >>> wrote: >>> >>> >>> On 21 Oct 2019, at 6:42 AM, Smith, Barry F. wrote: Could you provide

Re: [petsc-dev] BlockGetIndices and GetBlockIndices

> On Oct 20, 2019, at 11:52 PM, Pierre Jolivet > wrote: > > > >> On 21 Oct 2019, at 6:42 AM, Smith, Barry F. wrote: >> >> Could you provide a use case where you want to access/have a block size of >> a IS that is not an ISBlock? > > In the end, all I really want is get access to the un

Re: [petsc-dev] BlockGetIndices and GetBlockIndices

Could you provide a use case where you want to access/have a block size of a IS that is not an ISBlock? > On Oct 16, 2019, at 2:50 AM, Pierre Jolivet via petsc-dev > wrote: > > Hello, > I’m trying to understand what is the rationale for naming a function > ISBlockGetIndices and another IS

Re: [petsc-dev] BlockGetIndices and GetBlockIndices

> On Oct 16, 2019, at 9:41 AM, Stefano Zampini via petsc-dev > wrote: > > I just took a look at the ISGENERAL code. ISSetBlockSize_General just sets > the block size of the layout (??) > ISGetIndices always return the data->idx memory. > So, a more profound question is: what is the model beh

Re: [petsc-dev] BlockGetIndices and GetBlockIndices

> On Oct 16, 2019, at 2:50 AM, Pierre Jolivet via petsc-dev > wrote: > > Hello, > I’m trying to understand what is the rationale for naming a function > ISBlockGetIndices and another ISLocalToGlobalMappingGetBlockIndices (BlockGet > vs. GetBlock). ISBlockGetIndices returns the indices fo

Re: [petsc-dev] ksp_error_if_not_converged in multilevel solvers

The reason the code works this way is that normally -ksp_error_if_not_converged is propagated into the inner (and innerer) solves and normally it is desirable that these inner solves do not error simply because they reach the maximum number of iterations since for nested iterative methods g

Re: [petsc-dev] SuperLU + GPUs

https://gitlab.com/petsc/petsc/merge_requests/2048 All discussion etc should go into the Discussion for that MR. > On Oct 18, 2019, at 12:21 PM, Mark Adams via petsc-dev > wrote: > > What is the status of supporting SuperLU_DIST with GPUs? > Thanks, > Mark

Re: [petsc-dev] Should v->valid_GPU_array be a bitmask?

I'd like to see both of them allocated on demand. > On Oct 13, 2019, at 6:56 PM, Zhang, Junchao wrote: > > I had an MR (already merged to master) that changed the name to > v->offloadmask. > But the behavior is not changed. VecCreate_SeqCUDA still allocates on both > CPU and GPU. I belie

Re: [petsc-dev] Right-preconditioned GMRES

Is this one process with one subdomain? (And hence no meaningful overlap since there is nothing to overlap?) And you expect to get the "exact" answer on one iteration? Please run the right preconditioned GMRES with -pc_asm_type [restrict and basic and none] -ksp_monitor_true_solution an

Re: [petsc-dev] Should v->valid_GPU_array be a bitmask?

Yikes, forget about bit flags and names. Does this behavior make sense? EVERY CUDA vector allocates memory on both GPU and CPU ? Or do I misunderstand the code? This seems fundamentally wrong and is different than before. What about the dozens of work vectors on the GPU (for example f

Re: [petsc-dev] Better error message for missing components

PetscErrorCode MatGetFactor(Mat mat, MatSolverType type,MatFactorType ftype,Mat *f) { PetscErrorCode ierr,(*conv)(Mat,MatFactorType,Mat*); PetscBool foundpackage,foundmtype; PetscFunctionBegin; PetscValidHeaderSpecific(mat,MAT_CLASSID,1); PetscValidType(mat,1); if (mat->factor

Re: [petsc-dev] SNES_DIVERGED_TR_DELTA

I clarified the text in changes/312.html I don't think it can be depreciated in the usual way since it no longer exists and its previous usage was nonsense. Barry > On Oct 6, 2019, at 1:10 AM, Pierre Jolivet via petsc-dev > wrote: > > Hello, > Shouldn’t there be a deprecation warning f

Re: [petsc-dev] MR 2036 : Better implementation of multipreconditioning

> On Oct 11, 2019, at 9:19 AM, Pierre Gosselet via petsc-dev > wrote: > > Dear all, > Barry kindly identified many flaws in our implementation, which gives > us many opportunities to improve our code (which is currently under MR > https://gitlab.com/petsc/petsc/merge_requests/2036). Among hi

[petsc-dev] PETSc 3.12 release

We are pleased to announce the release of PETSc version 3.12 at http://www.mcs.anl.gov/petsc The major changes and updates can be found at http://www.mcs.anl.gov/petsc/documentation/changes/312.html We recommend upgrading to PETSc 3.12 soon. As always, please report problems to petsc-ma...@m

Re: [petsc-dev] MatMult on Summit

Junchao, Great, thanks Barry Eventually I think this should all got into a MR that includes these tests and the PetscSF ping-pongs so later someone can reproduce these numbers on Summit and on the new machines that come out. > On Sep 23, 2019, at 11:01 AM, Zhang, Junchao wrote:

Re: [petsc-dev] It would be really nice if you could run a single job on the pipeline with a branch

> On Sep 23, 2019, at 10:43 AM, Jed Brown wrote: > > "Smith, Barry F. via petsc-dev" writes: > >>> On Sep 22, 2019, at 11:26 PM, Balay, Satish wrote: >>> >>> Even-though a fix addresses a breakage in a single build - that change >>&g

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

Hong, As a hack for this release could you have the Numeric portion of the multiply routines check if the symbolic data is there and if not just call the symbolic an attach the needed data? You might need to have a utility function that does all the symbolic part except the allocation

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

We would like avoid allocating a huge array for the matrix and then having the user place on top of it. In the new paradigm there could be options called on the resulting C of MatMatGetProduct() that would take effect before the C is fully formed to prevent the allocating and freeing f

Re: [petsc-dev] It would be really nice if you could run a single job on the pipeline with a branch

ns [within this pipeline] YES! > > The status in the MR will reflect that the pipeline failed [due to all the > other canceled jobs] > > Satish > > On Mon, 23 Sep 2019, Smith, Barry F. via petsc-dev wrote: > >> >> When you fix something on a branch that broke

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

Hong, We need to support this "hack" for the current release since so many people rely on it but I agree with you we should eliminate it in the future. Barry > On Sep 22, 2019, at 9:16 PM, h...@aspiritech.org wrote: > > Now I understand why the latest changes in MatMatMult_MPIAIJ_M

[petsc-dev] PETSc release testing and schedule

Petsc-developers, We are planning on a PETSc release for Sunday Sept 29 (about 5 pm CST) For this - we'll have a feature freeze (on merges) on Tuesday Sept 24th (about 5 pm CST] After this time we will be accepting only bug fix/doc fix MR. Any unfinished MR in the gitlab site will remain

[petsc-dev] It would be really nice if you could run a single job on the pipeline with a branch

When you fix something on a branch that broke a particular job in gitlab-ci it would be nice to be able to run that single job on the updated branch instead of having to submit an entirely new pipeline Does this exist? Should this be requested in gitlab-ci issues? Could we make a work

Re: [petsc-dev] MatMult on Summit

Junchao, For completeness could you please run with a single core? But leave the ratio as you have with over 2 ranks since that is the correct model. Thanks Barry > On Sep 22, 2019, at 11:14 AM, Zhang, Junchao wrote: > > I did stream test on Summit. I used the MPI version f

Re: [petsc-dev] MatMult on Summit

Here is how the bandwidth improves with more cores. Terrible in going from 1 to 2 cores per socket > On Sep 21, 2019, at 2:03 PM, Zhang, Junchao wrote: > > I made the following changes: > 1) In MatMultAdd_SeqAIJCUSPARSE, use this code sequence at the end > ierr = WaitForGPU();CHKERRCUDA(

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

Since this a common used feature we will need to support it in the release or it will break a variety of codes. I am not sure how to "deprecate it" in a useful way. How would the code actively tell the user that the approach is deprecated and they should update their code before the next r

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

gt; > It is the same as MatMatMultSymbolic_MPIAIJ_MPIDense() except does not > create C > */ > > > Il giorno dom 22 set 2019 alle ore 20:11 Smith, Barry F. via petsc-dev > ha scritto: > >Jose, > > Thanks for the pointer. > > Will this ch

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

Jose, Thanks for the pointer. Will this change dramatically affect the organization of SLEPc? As noted in my previous email eventually we need to switch to a new API where the REUSE with a different matrix is even more problematic. If you folks have use cases that fundamen

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

> On Sep 22, 2019, at 11:49 AM, Pierre Jolivet > wrote: > > >> On 22 Sep 2019, at 6:33 PM, Smith, Barry F. wrote: >> >> >> Ok. So we definitely need better error checking and to clean up the code, >> comments and docs >> >> As the approaches for these computations of products get more

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

Ok. So we definitely need better error checking and to clean up the code, comments and docs As the approaches for these computations of products get more complicated it becomes a bit harder to support the use of a raw product matrix so I don't think we want to add all the code needed to

Re: [petsc-dev] Broken MatMatMult_MPIAIJ_MPIDense

> On Sep 22, 2019, at 10:14 AM, Pierre Jolivet via petsc-dev > wrote: > > FWIW, I’ve fixed MatMatMult and MatTransposeMatMult here > https://gitlab.com/petsc/petsc/commit/93d7d1d6d29b0d66b5629a261178b832a925de80 > (with MAT_INITIAL_MATRIX). > I believe there is something not right in your MR

Re: [petsc-dev] MatMult on Summit

> On Sep 22, 2019, at 9:56 AM, Jed Brown wrote: > > Run two resource sets on one side versus separate nodes. I don't know what this is suppose to mean. Is it a toy situation where you show the problem is measurable or a real application run properly at scale where you show the problem has

Re: [petsc-dev] MatMult on Summit

I'm guessing it would be very difficult to connect this particular performance bug with a decrease in performance for an actual full application since models don't catch this level of detail well (and since you cannot run the application without the bug to see the better performance)? IBM

Re: [petsc-dev] MatMult on Summit

Ok, thanks. Then one has to be careful in HPC when using the term so each time it is used everyone in the conversation knows which one it is referring to. > On Sep 22, 2019, at 8:33 AM, Jed Brown wrote: > > "Smith, Barry F." writes: > >>> On Sep 21, 2019, at 11:43 PM, Jed Brown wrote:

Re: [petsc-dev] MatMult on Summit

> On Sep 21, 2019, at 11:43 PM, Jed Brown wrote: > > "Smith, Barry F." writes: > >> Jed, >> >> What does latency as a function of message size mean? It is in the plots > > It's just the wall-clock time to ping-pong a message of that size. All > the small sizes take the same amount of

Re: [petsc-dev] MatMult on Summit

Junchao could try the PETSc (and non-PETSc) streams tests on the machine. There are a few differences, compiler, the reported results are with OpenMP, different number of cores but yes the performance is a bit low. For DOE that is great, makes GPUs look better :-) > On Sep 21, 2019, at

Re: [petsc-dev] MatMult on Summit

Jed, What does latency as a function of message size mean? It is in the plots > On Sep 21, 2019, at 11:15 PM, Jed Brown via petsc-dev > wrote: > > Karl Rupp via petsc-dev writes: > >> Hi Junchao, >> >> thanks, these numbers are interesting. >> >> Do you have an easy way to evalua

Re: [petsc-dev] MatMult on Summit

Thanks > On Sep 21, 2019, at 10:17 PM, Zhang, Junchao wrote: > > 42 cores have better performance. > > 36 MPI ranks > MatMult 100 1.0 2.2435e+00 1.0 1.75e+09 1.3 2.9e+04 4.5e+04 > 0.0e+00 6 99 97 28 0 100100100100 0 25145 0 0 0.00e+000 > 0.00e+00 0 > VecS

Re: [petsc-dev] MatMult on Summit

Junchao, Mark has a good point; could you also try for completeness the CPU with 36 cores and see if it is any better than the 42 core case? Barry So extrapolating about 20 nodes of the CPUs is equivalent to 1 node of the GPUs for the multiply for this problem size. > On Sep 21, 2

Re: [petsc-dev] MatMult on Summit

Thanks! This is great stuff, very useful. Barry > On Sep 21, 2019, at 2:03 PM, Zhang, Junchao wrote: > > I made the following changes: > 1) In MatMultAdd_SeqAIJCUSPARSE, use this code sequence at the end > ierr = WaitForGPU();CHKERRCUDA(ierr); > ierr = PetscLogGpuTimeEnd();CHKERRQ

Re: [petsc-dev] MatMult on Summit

Sorry, forgot Could you please put the GPU wait call before each of the log ends in that routine and see what kind of new numbers you get? Thanks Barry > On Sep 21, 2019, at 11:00 AM, Zhang, Junchao wrote: > > We log gpu time before/after cusparse calls. > https://gitlab

Re: [petsc-dev] MatMult on Summit

> On Sep 21, 2019, at 11:00 AM, Zhang, Junchao wrote: > > We log gpu time before/after cusparse calls. > https://gitlab.com/petsc/petsc/blob/master/src%2Fmat%2Fimpls%2Faij%2Fseq%2Fseqcusparse%2Faijcusparse.cu#L1441 > But according to > https://docs.nvidia.com/cuda/cusparse/index.html#asynchr

Re: [petsc-dev] MatMult on Summit

Hannah, Junchao and Richard, The on-GPU flop rates for 24 MPI ranks and 24 MPS GPUs looks totally funky. 951558 and 973391 they are so much lower than unvirtualized 3084009 and 3133521 and yet the total time to solution is similar for the runs. Is it possible these are being count

Re: [petsc-dev] MatMult on Summit

Dang, makes the GPUs less impressive :-). > On Sep 21, 2019, at 12:44 AM, Zhang, Junchao wrote: > > Here are CPU version results on one node with 24 cores, 42 cores. Click the > links for core layout. > > 24 MPI ranks, https://jsrunvisualizer.olcf.ornl.gov/?s4f1o01n6c4g1r14d1b21l0= > MatM

[petsc-dev] Tip while using valgrind

When using valgrind it is important to understand that it does not immediately make a report when it finds an uninitialized memory, it only makes a report when an uninitialized memory would cause a change in the program flow (like in an if statement). This is why sometimes it seems to report

Re: [petsc-dev] Configure hangs on Summit

Then the hang is curious. > On Sep 20, 2019, at 11:28 PM, Mills, Richard Tran wrote: > > Everything that Barry says about '--with-batch' is valid, but let me point > out one thing about Summit: You don't need "--with-batch" at all, because the > Summit login/compile nodes run the same har

Re: [petsc-dev] MatMult on Summit

Junchao, Very interesting. For completeness please run also 24 and 42 CPUs without the GPUs. Note that the default layout for CPU cores is not good. You will want 3 cores on each socket then 12 on each. Thanks Barry Since Tim is one of our reviewers next week this is a very good

Re: [petsc-dev] test harness: output of actually executed command for V=1 gone?

> On Sep 20, 2019, at 4:18 PM, Scott Kruger via petsc-dev > wrote: > > > > > > On 9/20/19 2:49 PM, Jed Brown wrote: >> Hapla Vaclav via petsc-dev writes: >>> On 20 Sep 2019, at 19:59, Scott Kruger >>> mailto:kru...@txcorp.com>> wrote: >>> >>> >>> On 9/20/19 10:44 AM, Hapla Vaclav via

Re: [petsc-dev] Configure hangs on Summit

--with-batch is still there and should be used in such circumstances. The difference is that --with-branch does not generate a program that you need to submit to the batch system before continuing the configure. Instead --with-batch guesses at and skips some of the tests (with clear warnin

Re: [petsc-dev] PetscCUDAInitialize

> On Sep 19, 2019, at 9:30 PM, Balay, Satish wrote: > > On Fri, 20 Sep 2019, Smith, Barry F. via petsc-dev wrote: > >> >> >>> On Sep 19, 2019, at 9:11 PM, Balay, Satish wrote: >>> >>> On Fri, 20 Sep 2019, Smith, Barry F. via petsc-dev

[petsc-dev] Do not use PetscSequentialPhaseBegin/End to attempt to control stdout

It does not work! Use PetscViewerASCIISyncronizedPrintf() and PetscViewerGetSubviewer() as needed These codes break the automated testing because once in a great while the output is in a different order. ierr = PetscSequentialPhaseBegin(PETSC_COMM_WORLD,1);CHKERRQ(ierr); ierr = Pet

Re: [petsc-dev] PetscCUDAInitialize

> On Sep 19, 2019, at 9:11 PM, Balay, Satish wrote: > > On Fri, 20 Sep 2019, Smith, Barry F. via petsc-dev wrote: > >> >> This should be reported on gitlab, not in email. >> >> Anyways, my interpretation is that the machine runs low on swap space s

Re: [petsc-dev] PetscCUDAInitialize

This should be reported on gitlab, not in email. Anyways, my interpretation is that the machine runs low on swap space so the OS is killing things. Once Satish and I sat down and checked the system logs on one machine that had little swap and we saw system messages about low swap at exa

  1   2   3   4   >