Here are my thoughts about limiting the memory footprint for IPOIB CM (NOSRQ) patch:
By default, cap the NOSRQ memory usage to 1GB. The default recvq_size is set to 128. Therefore for 64KB packets this would imply a maximum of 128 endpoints. -Make the maximum number of endpoints a module parameter with a default value of 128. -The NOSRQ limit of 1GB is also made a module parameter. However, 1GB is the default limit and could be changed as needed (by the administrator) depending on the system configuration, application needs and so on. The server would return a "REJ" message upon receiving a "REQ", whenever one of these limits (i.e. max number of endpoints or the max NOSRQ memory usage) is reached. Currently, we only check for the maximum number of endpoints -hard coded to 1024. -The IPOIB CM (NOSRQ) patch is practically transparent to HCAs that support SRQ like the Topspin HCA and, such HCAs should not be impacted at all. -Currently we allocate a default of 64KB for the ring buffer elements, and this buffer size is not linked to the mtu. In the future, we could allocate buffers based on the mtu and link that into the computation of the memory cap. This way customers who might want to use a smaller mtu could use a larger number of endpoints, or a larger recvq_size without exceeding the memory cap. Would this approach address the issues of scalability and enable IPOIB CM to be turned as the default? Pradeep _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
