On Friday March 4 2011 08:48:14 Garth N. Wells wrote: > On 04/03/11 16:38, Johan Hake wrote: > > On Friday March 4 2011 03:29:32 Garth N. Wells wrote: > >> On 03/03/11 19:48, Johan Hake wrote: > >>> On Thursday March 3 2011 11:20:03 Marie E. Rognes wrote: > >>>> On 03/03/2011 08:03 PM, Johan Hake wrote: > >>>>> Hello! > >>>>> > >>>>> I am using Mixed spaces with Reals quite alot. It turnes out that > >>>>> assemble forms with functions from MixedFunctionSpaces containing > >>>>> Real spaces are dead slow. The time spent also increase with the > >>>>> number of included Real spaces, even if none of them are included in > >>>>> form which is assembled. > >>>>> > >>>>> The attached test script illustrates this. > >>>> > >>>> By replacing "CG", 1 by "R", 0 or? > >>> > >>> OMG!! Yes, *flush* > >>> > >>> That explains the memory usage :P > >>> > >>>>> The test script also reviels that an unproportial time is spent in > >>>>> FFC generating the code. This time also increase with the number of > >>>>> Real spaces included. Turning of FErari helped a bit with this > >>>>> point. > >>>> > >>>> I can take a look on the FFC side, but not today. > >>> > >>> Nice! > >>> > >>> With the update correction from Marie the numbers now looks like: > >>> > >>> With PETSc backend > >>> > >>> Tensor without Mixed space | 0.11211 0.11211 1 > >>> With 1 global dofs | 1.9482 1.9482 1 > >>> With 2 global dofs | 2.8725 2.8725 1 > >>> With 4 global dofs | 5.1959 5.1959 1 > >>> With 8 global dofs | 10.524 10.524 1 > >>> With 16 global dofs | 25.574 25.574 1 > >>> > >>> With Epetra backend > >>> > >>> Tensor without Mixed space | 0.87544 0.87544 1 > >>> With 1 global dofs | 1.7089 1.7089 1 > >>> With 2 global dofs | 2.6868 2.6868 1 > >>> With 4 global dofs | 4.28 4.28 1 > >>> With 8 global dofs | 8.123 8.123 1 > >>> With 16 global dofs | 17.394 17.394 1 > >>> > >>> Still a pretty big increase in time for just adding 16 scalar dofs to a > >>> system of 274625 dofs in teh first place. > >> > >> I have seen this big slow down for large problems. The first issue, > >> which was the computation of the sparsity pattern, has been 'resolved' > >> by using boost::unordered_set. This comes at the expense of the a small > >> slow down for regular problems. > >> > >> I also noticed that Epetra performs much better for these problems than > >> PETSc does. We need to check the matrix initialisation, but it could > >> ultimately be a limitation of the backends. Each matrix row > >> corresponding to a global dof is full, and it may be that backends > >> designed for large sparse matrices do not handle this well. > > > > How could inserting into the matrix be the bottle neck. In the test > > script I attached I do not assemble any global dofs. > > I think that you've find that it is. It will be assembling zeroes in the > global dof positions.
I guess you are right. Is the sparsity pattern and also the tabulated tensor based only on the MixedSpace formulation, and not on the actual integral? Is this a bug or feature? > >> The best approach is probably to add the entire row at once for global > >> dofs. This would require a modified assembler. > >> > >> There is a UFC Blueprint to identify global dofs: > >> https://blueprints.launchpad.net/ufc/+spec/global-dofs > >> > >> If we can identify global dofs, we have a better chance of dealing with > >> the problem properly. This includes running in parallel with global > >> dofs. > > > > Do you envision to have integrals over global dofs to get separated into > > its own tabulate tensor function? Then in DOLFIN we can assemble the > > whole row/column in one loop and insert it into the Matrix in one go. > > No - I had in mind possibly adding only cell-based dofs to the matrix, > and adding the global rows into a global vector, then is that inserted > at the end (as one row) into a matrix. I'm not advocating at this stage > a change to the tabulate_foo interface. Ok, but you need its own tabulate_tensor function for the global dofs? > > Do you think we also need to recognise global dofs in UFL to properly > > flesh out these integrals? > > Yes. For one to handle them properly in parallel since global dofs to > not reside at mesh entity, and the domains are parallelised mesh-wise. Ok Johan > Garth > > > Johan > > > >> Garth > >> > >>> Johan > >>> > >>>> -- > >>>> Marie > >>>> > >>>>> I have not profiled any of this, but I just throw it out there. I do > >>>>> not recognize any difference between for example Epetra or PETSc > >>>>> backend as suggested in the fixed bug for building of sparsity > >>>>> pattern with global dofs. > >>>>> > >>>>> My test has been done on a DOLFIN 0.9.9+. I haven't profiled it yet. > >>>>> > >>>>> Output from summary: > >>>>> Tensor without Mixed space | 0.11401 0.11401 1 > >>>>> With 1 global dofs | 0.40725 0.40725 1 > >>>>> With 2 global dofs | 0.94694 0.94694 1 > >>>>> With 4 global dofs | 2.763 2.763 1 > >>>>> With 8 global dofs | 9.6149 9.6149 1 > >>>>> > >>>>> Also the amount of memory used to build the sparsity patter seams to > >>>>> double for each step. The memory print for a 32x32x32 unit cube with > >>>>> 16 global dofs was 1.6 GB memory(!?). > >>>>> > >>>>> Johan > >>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> Mailing list: https://launchpad.net/~dolfin > >>>>> Post to : [email protected] > >>>>> Unsubscribe : https://launchpad.net/~dolfin > >>>>> More help : https://help.launchpad.net/ListHelp > >>>> > >>>> _______________________________________________ > >>>> Mailing list: https://launchpad.net/~dolfin > >>>> Post to : [email protected] > >>>> Unsubscribe : https://launchpad.net/~dolfin > >>>> More help : https://help.launchpad.net/ListHelp > >>>> > >>>> > >>>> _______________________________________________ > >>>> Mailing list: https://launchpad.net/~dolfin > >>>> Post to : [email protected] > >>>> Unsubscribe : https://launchpad.net/~dolfin > >>>> More help : https://help.launchpad.net/ListHelp > >> > >> _______________________________________________ > >> Mailing list: https://launchpad.net/~dolfin > >> Post to : [email protected] > >> Unsubscribe : https://launchpad.net/~dolfin > >> More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~dolfin Post to : [email protected] Unsubscribe : https://launchpad.net/~dolfin More help : https://help.launchpad.net/ListHelp
