Thomas, you are correct, in the verbsPorts configuration variable description
verbsPorts = <device>/<port>/<fabric>/<service level> <service level> refers to the service level which is set for all RC queue pairs on the specified adapter port. This information is used by the subnet manager to control QoS. Unfortunately I'm not familiar with configuring QoS so I cannot provide additional information on this. If found the following white paper from Mellanox/Nvidia providing some more information: https://network.nvidia.com/pdf/whitepapers/deploying_qos_wp_10_19_2005.pdf . It is at least a starting point. Mit freundlichen Grüßen / Kind regards Ralph Würthner IBM Storage Scale Development IBM Systems & Technology Group, Systems Software Development ________________________________ Mobile: +49 (0) 171 3089472 IBM Deutschland Research & Development GmbH [cid:4c6d823aedac83b4cc99633a0db02b7738e8aa77.camel@de.ibm.com-1] Email: ralph.wuerth...@de.ibm.com<mailto:ralph.wuerth...@de.ibm.com> Wilhelm-Fay-Str. 32 65936 Frankfurt Germany ________________________________ IBM Data Privacy Statement<https://www.ibm.com/privacy/us/en/> IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Gregor Pillen Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 -----Original Message----- From: Thomas Bernecker <thomas.bernec...@emea.nec.com<mailto:thomas%20bernecker%20%3cthomas.bernec...@emea.nec.com%3e>> Reply-To: gpfsug main discussion list <gpfsug-discuss@gpfsug.org<mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-disc...@gpfsug.org%3e>> To: gpfsug main discussion list <gpfsug-discuss@gpfsug.org<mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-disc...@gpfsug.org%3e>> Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS Date: 09/07/2023 11:02:02 AM Hi Luis, Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of said presentation says to configure the RDMA device as follows to use QoS verbsPorts ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7pa6Y4IryZDRZDY_pFV_gcuWmVhOYY_MnKo0CryfRuaBuRmNkPYpLYCHVyCAO62RdiqmuBcGrQ1bbfxDJ3CzqkCGbKUj4BJ_KOcmBuYQq_Z4sy-kxOV4aF_bfA4U$> Report Suspicious ZjQcmQRYFpfptBannerEnd Hi Luis, Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of said presentation says to configure the RDMA device as follows to use QoS verbsPorts = <device>/<port>/<fabric>/<service level> − List of RDMA ports to be used − <device> : RDMA device, required, e.g. mlx5_0, mlx5_1 − <port>: RDMA port on device, default 1, valid values are 1 or 2 − <fabric>: virtual fabric number, default 0, valid values are >= 0 Only verbsPorts using a common <fabric> are connected − <service level>: QoS level, default is 0, valid values defined in SM configuration The references to SM and verbsPorts being part of RDMA configuration indicates (to me) that this is fabric QoS … -- Best regards / Mit freundlichem Gruß Thomas Bernecker ------------------------------------------------------------------------------------------------------------------ Manager System Integration and Support, HPCE Division Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091 NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941 ------------------------------------------------------------------------------------------------------------------- From: gpfsug-discuss <gpfsug-discuss-boun...@gpfsug.org>On Behalf Of Luis Bolinches Sent: Thursday, September 7, 2023 10:51 AM To: gpfsug main discussion list <gpfsug-discuss@gpfsug.org> Subject: Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS Hi Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW. Limits IOPS per class https: //www. ibm. com/docs/en/storage-scale/5. 1. 8?topic=reference-mmchqos-command ZjQcmQRYFpfptBannerStart Be Careful With This Message The sender's identity could not be verified and someone may be impersonating the sender. Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/BQNorrFsuw!1ltCYpPUsLPrqvNt6ouhTp6NGkpGHjneRgeZQ4EL8X29-niUBGCBdKKkDls1qqoVuPo1KL8kYyJpf4yhW6ztYNlnYstlXTWSM5mLl3UqEvexW0whUlE-4P3Yj_DDkWnTV-cejP_cc5NNq29-2g$> ZjQcmQRYFpfptBannerEnd Hi Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW. Limits IOPS per class https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command [ibm.com]<https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command> I recommend (depending on number of nodes and seconds) fine-stats so then you can visualize which PID on which client node is doing what. I believe I did something of that on SSUG London 18 or 19 -- Ystävällisin terveisin/Regards/Saludos/Salutations/Salutacions Luis Bolinches Executive IT Specialist IBM Storage ScaleServer (formerly ESS) developer Phone: +358503112585 Ab IBM Finland Oy Toinen linja 7 00530 Helsinki Uusimaa - Finland Visitors entrance: Siltasaarenkatu 22 "If you always give you will always have" -- Anonymous https://www.credly.com/users/luis-bolinches/badges [credly.com]<https://www.credly.com/users/luis-bolinches/badges> From: gpfsug-discuss <gpfsug-discuss-boun...@gpfsug.org<mailto:gpfsug-discuss-boun...@gpfsug.org>>On Behalf Of Thomas Bernecker Sent: Thursday, 7 September 2023 11.36 To: gpfsug-discuss@gpfsug.org<mailto:gpfsug-discuss@gpfsug.org> Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS Dear all, we recently found this presentation from 2021: https: //www. spectrumscaleug. org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA. pdf On page 10 it explains by setting the RDMA device in a specific ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7ra7I0IqiHHRBBi-CjVQYUsssj6dhwnB7Lo2ZohM0He5_MhQUpUfgGfOHPXlTwIhxLUDAE1nbLv0ArY1Ygr3eS1cGzhvDzAy1DtrxLl5hHS5FHw6KsNKzpBBnc2d$> ZjQcmQRYFpfptBannerEnd Dear all, we recently found this presentation from 2021:https://www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf [spectrumscaleug.org]<https://www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf> On page 10 it explains by setting the RDMA device in a specific way one could employ QoS with InfiniBand, which is something we would like to achieve as well. Has anyone a working environment using QoS with InfiniBand outside of the ESS domain? If so, would you share your experience? Sorry for asking a rather broad question, but it seems that this is not well-known stuff … -- Best regards / Mit freundlichem Gruß Thomas Bernecker ------------------------------------------------------------------------------------------------------------------ Manager System Integration and Support, HPCE Division Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091 NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941 ------------------------------------------------------------------------------------------------------------------- Unless otherwise stated above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
