Thank you all! I will check with a smaller example to see how my allocation is doing.
-- José Abell *PhD Candidate* Computational Geomechanics Group Dept. of Civil and Environmental Engineering UC Davis www.joseabell.com On Thu, Dec 17, 2015 at 10:59 AM, Barry Smith <bsm...@mcs.anl.gov> wrote: > > > On Dec 17, 2015, at 11:12 AM, Dave May <dave.mayhe...@gmail.com> wrote: > > > > > > > > On Thursday, 17 December 2015, Jose A. Abell M. <jaab...@ucdavis.edu> > wrote: > > Thank you Dave! > > > > Do you have a rough idea of how long a matrix like that should take to > assemble? > > Not hours. Right? > > > > If the preallocation is correct, and most of the entries to be inserted > live locally (and don't need to be scattered to another rank), it should > definitely not take hours. > > If the preallocation is correct then the time to do the first assembly > may take maybe twice as long as subsequent assemblies but it should not be > much more than that. > > Barry > > > > > > > > > Regards, > > Jose > > > > -- > > > > José Abell > > PhD Candidate > > Computational Geomechanics Group > > Dept. of Civil and Environmental Engineering > > UC Davis > > www.joseabell.com > > > > > > On Thu, Dec 17, 2015 at 12:58 AM, Dave May <dave.mayhe...@gmail.com> > wrote: > > > > > > On 17 December 2015 at 08:06, Jose A. Abell M. <jaab...@ucdavis.edu> > wrote: > > Hello dear PETSc users, > > > > This is a problem that pops up often, from what I see, in the mailing > list. My program takes a long time assembling the matrix. > > > > What I know: > > > > • Matrix Size is (MatMPIAIJ) 2670402 > > • Number of processes running PETSc: 95 > > • Not going to virtual memory (no swapping, used mem well withing > each node's capacity) > > • System is partitioned with ParMETIS for load balancing > > • I see memory moving around in each node (total used memory > changes a bit, grows and then frees) > > • Matrix is filled in blocks of size 81x81 (FEM code, so this ends > up being a sparse matrix) > > • I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the > MatSetValues are done. > > Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to store > the buffers? If so, how often? Every 100 blocks? > > > > What else could it be? > > > > Its taking several hours to asseble this matrix. I re-use the sparsity > pattern, so subsequent assemblies are fast. Does this mean that my > preallocation is wrong? > > > > The preallocation could be wrong. That is the usual cause of very slow > matrix assembly. To confirm this hypothesis, run your code with the command > line option -info. You will get an enormous amount of information in > stdout. You might consider using -info with a smallish problem size / core > count. > > > > Inspect the output generated by -info and look for lines like this: > > > > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 > > > > If the number of mallocs during MatSetValues() is not zero, then your > preallocation is not exactly correct. A small number of mallocs, say less > than 10, might be accepted (performance wise). However if the number of > mallocs is > 100, then assembly time will be terribly slow. > > > > Thanks, > > Dave > > > > > > > > > > Regards, > > > > > > > > > > -- > > > > José Abell > > PhD Candidate > > Computational Geomechanics Group > > Dept. of Civil and Environmental Engineering > > UC Davis > > www.joseabell.com > > > > > > > >