>> Besides, Ben's code (with an expensive assembly combined with >> experimenting with a matrix-free solve) isn't typical. Jed was right >> that, for most users, threading the assembly isn't going to help much >> when you've got an un-threaded solve going on in the background. > > Even just insertion won't scale great, locking would be tricky since > PETSc doesn't require you to preallocate correctly. We could make > MatSetValues thread-safe when allocation is correct, but this has not > been done, and taking the locks would have nontrivial cost for the > people who typically insert very few values per row.
>From a PETSc point of view that is certainly true, but for libMesh applications we allocate the sparsity exactly, so there is no allocation inside the MatSetValues. Eventhough MatSetValues is not thread-safe, when the allocation is correct putting a mutex around the element matrix insertion has suffered no noticeable performance degradation out to 16 threads... I should definitely point out that the threading in libMesh is designed to augment MPI based parallelism, certainly not to replace it. Depending on the architecture, we've seen 4-socket 4-core nodes perform *better* (lower wall-clock time) using 4 MPI tasks on the node that 16. The current libMesh threading approach is ideal in this case - use 4 MPI tasks with 4 threads each, and the matrix assembly per task will realize a ~4x speedup. Of course, as previously discussed, this only has a real impact on runtime if you spend a nontrivial amount of time in the matrix assembly. This is true in some of my applications, where the matrix assembly time is comparable to (or sometimes greater than) the linear solve time. -Ben ------------------------------------------------------------------------------ Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev _______________________________________________ Libmesh-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/libmesh-users
