On 04/03/11 17:11, Johan Hake wrote: > On Friday March 4 2011 08:48:14 Garth N. Wells wrote: >> On 04/03/11 16:38, Johan Hake wrote: >>> On Friday March 4 2011 03:29:32 Garth N. Wells wrote: >>>> On 03/03/11 19:48, Johan Hake wrote: >>>>> On Thursday March 3 2011 11:20:03 Marie E. Rognes wrote: >>>>>> On 03/03/2011 08:03 PM, Johan Hake wrote: >>>>>>> Hello! >>>>>>> >>>>>>> I am using Mixed spaces with Reals quite alot. It turnes out that >>>>>>> assemble forms with functions from MixedFunctionSpaces containing >>>>>>> Real spaces are dead slow. The time spent also increase with the >>>>>>> number of included Real spaces, even if none of them are included in >>>>>>> form which is assembled. >>>>>>> >>>>>>> The attached test script illustrates this. >>>>>> >>>>>> By replacing "CG", 1 by "R", 0 or? >>>>> >>>>> OMG!! Yes, *flush* >>>>> >>>>> That explains the memory usage :P >>>>> >>>>>>> The test script also reviels that an unproportial time is spent in >>>>>>> FFC generating the code. This time also increase with the number of >>>>>>> Real spaces included. Turning of FErari helped a bit with this >>>>>>> point. >>>>>> >>>>>> I can take a look on the FFC side, but not today. >>>>> >>>>> Nice! >>>>> >>>>> With the update correction from Marie the numbers now looks like: >>>>> >>>>> With PETSc backend >>>>> >>>>> Tensor without Mixed space | 0.11211 0.11211 1 >>>>> With 1 global dofs | 1.9482 1.9482 1 >>>>> With 2 global dofs | 2.8725 2.8725 1 >>>>> With 4 global dofs | 5.1959 5.1959 1 >>>>> With 8 global dofs | 10.524 10.524 1 >>>>> With 16 global dofs | 25.574 25.574 1 >>>>> >>>>> With Epetra backend >>>>> >>>>> Tensor without Mixed space | 0.87544 0.87544 1 >>>>> With 1 global dofs | 1.7089 1.7089 1 >>>>> With 2 global dofs | 2.6868 2.6868 1 >>>>> With 4 global dofs | 4.28 4.28 1 >>>>> With 8 global dofs | 8.123 8.123 1 >>>>> With 16 global dofs | 17.394 17.394 1 >>>>> >>>>> Still a pretty big increase in time for just adding 16 scalar dofs to a >>>>> system of 274625 dofs in teh first place. >>>> >>>> I have seen this big slow down for large problems. The first issue, >>>> which was the computation of the sparsity pattern, has been 'resolved' >>>> by using boost::unordered_set. This comes at the expense of the a small >>>> slow down for regular problems. >>>> >>>> I also noticed that Epetra performs much better for these problems than >>>> PETSc does. We need to check the matrix initialisation, but it could >>>> ultimately be a limitation of the backends. Each matrix row >>>> corresponding to a global dof is full, and it may be that backends >>>> designed for large sparse matrices do not handle this well. >>> >>> How could inserting into the matrix be the bottle neck. In the test >>> script I attached I do not assemble any global dofs. >> >> I think that you've find that it is. It will be assembling zeroes in the >> global dof positions. > > I guess you are right. Is the sparsity pattern and also the tabulated tensor > based only on the MixedSpace formulation, and not on the actual integral? >
The sparsity pattern is based on the dof map, which depends on the function spaces. > Is this a bug or feature? > I would say just the natural approach. There is/was a Blueprint to avoid computing and assembling the zeroes in problems like Stokes, but I'm not sure that this would be worthwhile, since it would involve assembling non-matrix values, and most backends want to assemble dense local matrices into sparse global matrices. Garth >>>> The best approach is probably to add the entire row at once for global >>>> dofs. This would require a modified assembler. >>>> >>>> There is a UFC Blueprint to identify global dofs: >>>> https://blueprints.launchpad.net/ufc/+spec/global-dofs >>>> >>>> If we can identify global dofs, we have a better chance of dealing with >>>> the problem properly. This includes running in parallel with global >>>> dofs. >>> >>> Do you envision to have integrals over global dofs to get separated into >>> its own tabulate tensor function? Then in DOLFIN we can assemble the >>> whole row/column in one loop and insert it into the Matrix in one go. >> >> No - I had in mind possibly adding only cell-based dofs to the matrix, >> and adding the global rows into a global vector, then is that inserted >> at the end (as one row) into a matrix. I'm not advocating at this stage >> a change to the tabulate_foo interface. > > Ok, but you need its own tabulate_tensor function for the global dofs? > >>> Do you think we also need to recognise global dofs in UFL to properly >>> flesh out these integrals? >> >> Yes. For one to handle them properly in parallel since global dofs to >> not reside at mesh entity, and the domains are parallelised mesh-wise. > > Ok > > Johan > >> Garth >> >>> Johan >>> >>>> Garth >>>> >>>>> Johan >>>>> >>>>>> -- >>>>>> Marie >>>>>> >>>>>>> I have not profiled any of this, but I just throw it out there. I do >>>>>>> not recognize any difference between for example Epetra or PETSc >>>>>>> backend as suggested in the fixed bug for building of sparsity >>>>>>> pattern with global dofs. >>>>>>> >>>>>>> My test has been done on a DOLFIN 0.9.9+. I haven't profiled it yet. >>>>>>> >>>>>>> Output from summary: >>>>>>> Tensor without Mixed space | 0.11401 0.11401 1 >>>>>>> With 1 global dofs | 0.40725 0.40725 1 >>>>>>> With 2 global dofs | 0.94694 0.94694 1 >>>>>>> With 4 global dofs | 2.763 2.763 1 >>>>>>> With 8 global dofs | 9.6149 9.6149 1 >>>>>>> >>>>>>> Also the amount of memory used to build the sparsity patter seams to >>>>>>> double for each step. The memory print for a 32x32x32 unit cube with >>>>>>> 16 global dofs was 1.6 GB memory(!?). >>>>>>> >>>>>>> Johan >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Mailing list: https://launchpad.net/~dolfin >>>>>>> Post to : [email protected] >>>>>>> Unsubscribe : https://launchpad.net/~dolfin >>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>> >>>>>> _______________________________________________ >>>>>> Mailing list: https://launchpad.net/~dolfin >>>>>> Post to : [email protected] >>>>>> Unsubscribe : https://launchpad.net/~dolfin >>>>>> More help : https://help.launchpad.net/ListHelp >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Mailing list: https://launchpad.net/~dolfin >>>>>> Post to : [email protected] >>>>>> Unsubscribe : https://launchpad.net/~dolfin >>>>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~dolfin >>>> Post to : [email protected] >>>> Unsubscribe : https://launchpad.net/~dolfin >>>> More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~dolfin Post to : [email protected] Unsubscribe : https://launchpad.net/~dolfin More help : https://help.launchpad.net/ListHelp
