Hi all, I'm wondering if there's a simple way to get statistics from OpenMPI as to how much memory the *MPI* layer in an application is taking. For example, I'm running a model and I can get the RSS size at various points in the code, and that reflects the user data for the application, *plus*, surely, buffers for MPI messages that are either allocated at runtime or, maybe, a pool from start-up. The memory use -which I assume is tied to internal buffers? differs considerably with *how* I run MPI - eg, TCP vs UCX, and with UCX, a UD vs RC mode.
Here's an example of this: 60km (163842 columns), 2304 ranks [OpenMPI] UCX Transport Changes (environment variable) (No recompilation; all runs done on same nodes) Showing memory after ATM-TO-MED Step [RSS Memory in MB] Standard Decomposition UCX_TLS value ud default rc Run 1 347.03 392.08 750.32 Run 2 346.96 391.86 748.39 Run 3 346.89 392.18 750.23 I'd love a way to trace how much *MPI alone* is using, since here I'm still measuring the *process's* RSS. My feeling is that if, for example, I'm running on N nodes and have a 1GB dataset + (for the sake of discussion) 100MB of MPI info, then at 2N, with good scaling of domain memory, that's 500MB + 100MB, at 4N it's 250MB/100MB, and eventually, at 16N, the MPI memory dominates. As a result, when we scale out, even with perfect scaling of *domain* memory, at some point memory associated with MPI will cause this curve to taper off, and potentially invert. But I'm admittedly *way* out of date on how modern MPI implementations allocate buffers. In short, any tips on ways to better characterize MPI memory use would be *greatly* appreciated! If this is purely on the UCX (or other transport) level, that's good to know too. Thanks, - Brian
