Re: [ewg] Questions about CC in OFED
I am not familiar with how psm may set CC parameters. Perhaps someone from QLogic/Intel can speak up here? Intel's Fabric Manager User guide provides a brief section about enabling and using CCA. It can be downloaded from www.intel.com/infiniband. If you cannot download it, please contact me or our customer support offline and we can get you the manual and additional information about using CCA with PSM. Basically there are configuration parameters in the Fabric Manager (FM) config file and there are PSM environment variable settings which can be modified. Once CCA is enabled in the FM config file, the fabric will use CCA since PSM has it enabled by default. The default values of PSM environment variables for CCA should be sufficient for most applications since we have taken care to find the best combination of parameters. If it becomes necessary to tune performance, additional PSM debug logging can be enabled to facilitate analysis of PSM CCA events. - Edward - - - -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg- boun...@lists.openfabrics.org] On Behalf Of Ira Weiny Sent: Wednesday, June 27, 2012 9:26 AM To: Michael Zhang Cc: ewg@lists.openfabrics.org Subject: Re: [ewg] Questions about CC in OFED On Wed, 27 Jun 2012 11:41:03 +0800 Michael Zhang ziwen1...@gmail.com wrote: Hi everybody, I focus on the source code of IB congestion control in OFED, such as CCT related data structure and FECN/BECN process. But the only code related to the congestion control mechanism is found in the module of infinipath-psm. As I know, psm works for MPI applications and actually I haven't installed psm in our system. I have done experiment through IPoIB and the congestion control still works that makes me confused. I am not familiar with how psm may set CC parameters. Perhaps someone from QLogic/Intel can speak up here? Could someone tell me where is the CC-related source code located besides infinipath-psm? Or some functions are realized in hardware such as the movement of CCT Index when receives BECN message? Thank you for your help! The latest version of infiniband-diags (1.6.0)[*] has 2 tools (ibccquery and ibccconfig) which will allow one to query and set the CC parameters. As you will note in the man page as well as the help output these tools are not to be used lightly. You can cause instability in your fabric if you set things wrong. However, when used properly they work well. In addition, Al Chu from LLNL has posted a patch to OpenSM which allows for the setting of CC parameters in OpenSM. Feedback from this patch has been minimal but Alex Netes (maintainer of OpenSM) says he will review it soon. (http://www.spinics.net/lists/linux-rdma/msg11615.html) Finally, if you are interested in the ongoing development of this and other features of Open Fabrics code you may want to sign up for the linux-rdma mailing list (linux-r...@vger.kernel.org or http://vger.kernel.org/vger- lists.html). Ira [*] git://beany.openfabrics.org/~iraweiny/infiniband-diags.git Best, Ziwen -- Ziwen(Michael) Zhang Ph.D candidate College of Computer National University of Defense Technology (NUDT) , P.R.China Email: zi...@nudt.edu.cn ziwen1...@gmail.com -- Ira Weiny Member of Technical Staff Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Questions about CC in OFED
On 6/28/12, Ira Weiny wei...@llnl.gov wrote: On Wed, 27 Jun 2012 11:41:03 +0800 Michael Zhang ziwen1...@gmail.com wrote: Hi everybody, I focus on the source code of IB congestion control in OFED, such as CCT related data structure and FECN/BECN process. But the only code related to the congestion control mechanism is found in the module of infinipath-psm. As I know, psm works for MPI applications and actually I haven't installed psm in our system. I have done experiment through IPoIB and the congestion control still works that makes me confused. I am not familiar with how psm may set CC parameters. Perhaps someone from QLogic/Intel can speak up here? Could someone tell me where is the CC-related source code located besides infinipath-psm? Or some functions are realized in hardware such as the movement of CCT Index when receives BECN message? Thank you for your help! The latest version of infiniband-diags (1.6.0)[*] has 2 tools (ibccquery and ibccconfig) which will allow one to query and set the CC parameters. As you will note in the man page as well as the help output these tools are not to be used lightly. You can cause instability in your fabric if you set things wrong. However, when used properly they work well. In addition, Al Chu from LLNL has posted a patch to OpenSM which allows for the setting of CC parameters in OpenSM. Feedback from this patch has been minimal but Alex Netes (maintainer of OpenSM) says he will review it soon. (http://www.spinics.net/lists/linux-rdma/msg11615.html) Finally, if you are interested in the ongoing development of this and other features of Open Fabrics code you may want to sign up for the linux-rdma mailing list (linux-r...@vger.kernel.org or http://vger.kernel.org/vger-lists.html). As I tested, the tool of ibccquery and ibccconfig in infiniband-diags (1.6.0) works well to get and set CC parameters. It is helpful for us. I really appreciate it. The tools resolve the configuration of cc related parameters. On the other side, we also care about the implementation of congestion control mechanism itself, such as CCTI is increased by CCTI_Increase when receiving BECN and feedback BECN if receiving data packet with FECN tag. I find no data structure of CC (like CCTI) in QP Context since CC is based on QP and no realization of ECN mechanism as well in the source code of OFED. Can you give me some cluses of this in OFED source code? Thank you for ur help! Best, Ziwen Ira [*] git://beany.openfabrics.org/~iraweiny/infiniband-diags.git Best, Ziwen -- Ziwen(Michael) Zhang Ph.D candidate College of Computer National University of Defense Technology (NUDT) , P.R.China Email: zi...@nudt.edu.cn ziwen1...@gmail.com -- Ira Weiny Member of Technical Staff Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- Ziwen(Michael) Zhang Ph.D candidate College of Computer National University of Defense Technology (NUDT) , P.R.China Email: zi...@nudt.edu.cn ziwen1...@gmail.com ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Questions about CC in OFED
Michael, PSM is a specific feature of the QLogic (now Intel) line of Infinipath HCAs, to does not work with Mellanox HCAs. The congestion and routing abilities in PSM also require using the QLogic/Intel subnet manager. From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Michael Zhang Sent: Tuesday, June 26, 2012 11:41 PM To: ewg@lists.openfabrics.org Subject: [ewg] Questions about CC in OFED Hi everybody, I focus on the source code of IB congestion control in OFED, such as CCT related data structure and FECN/BECN process. But the only code related to the congestion control mechanism is found in the module of infinipath-psm. As I know, psm works for MPI applications and actually I haven't installed psm in our system. I have done experiment through IPoIB and the congestion control still works that makes me confused. Could someone tell me where is the CC-related source code located besides infinipath-psm? Or some functions are realized in hardware such as the movement of CCT Index when receives BECN message? Thank you for your help! Best, Ziwen -- Ziwen(Michael) Zhang Ph.D candidate College of Computer National University of Defense Technology (NUDT) , P.R.China Email: zi...@nudt.edu.cnmailto:zi...@nudt.edu.cn ziwen1...@gmail.commailto:ziwen1...@gmail.com ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Questions about CC in OFED
On Wed, 27 Jun 2012 11:41:03 +0800 Michael Zhang ziwen1...@gmail.com wrote: Hi everybody, I focus on the source code of IB congestion control in OFED, such as CCT related data structure and FECN/BECN process. But the only code related to the congestion control mechanism is found in the module of infinipath-psm. As I know, psm works for MPI applications and actually I haven't installed psm in our system. I have done experiment through IPoIB and the congestion control still works that makes me confused. I am not familiar with how psm may set CC parameters. Perhaps someone from QLogic/Intel can speak up here? Could someone tell me where is the CC-related source code located besides infinipath-psm? Or some functions are realized in hardware such as the movement of CCT Index when receives BECN message? Thank you for your help! The latest version of infiniband-diags (1.6.0)[*] has 2 tools (ibccquery and ibccconfig) which will allow one to query and set the CC parameters. As you will note in the man page as well as the help output these tools are not to be used lightly. You can cause instability in your fabric if you set things wrong. However, when used properly they work well. In addition, Al Chu from LLNL has posted a patch to OpenSM which allows for the setting of CC parameters in OpenSM. Feedback from this patch has been minimal but Alex Netes (maintainer of OpenSM) says he will review it soon. (http://www.spinics.net/lists/linux-rdma/msg11615.html) Finally, if you are interested in the ongoing development of this and other features of Open Fabrics code you may want to sign up for the linux-rdma mailing list (linux-r...@vger.kernel.org or http://vger.kernel.org/vger-lists.html). Ira [*] git://beany.openfabrics.org/~iraweiny/infiniband-diags.git Best, Ziwen -- Ziwen(Michael) Zhang Ph.D candidate College of Computer National University of Defense Technology (NUDT) , P.R.China Email: zi...@nudt.edu.cn ziwen1...@gmail.com -- Ira Weiny Member of Technical Staff Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Questions about CC in OFED
Hi everybody, I focus on the source code of IB congestion control in OFED, such as CCT related data structure and FECN/BECN process. But the only code related to the congestion control mechanism is found in the module of infinipath-psm. As I know, psm works for MPI applications and actually I haven't installed psm in our system. I have done experiment through IPoIB and the congestion control still works that makes me confused. Could someone tell me where is the CC-related source code located besides infinipath-psm? Or some functions are realized in hardware such as the movement of CCT Index when receives BECN message? Thank you for your help! Best, Ziwen -- Ziwen(Michael) Zhang Ph.D candidate College of Computer National University of Defense Technology (NUDT) , P.R.China Email: zi...@nudt.edu.cn ziwen1...@gmail.com ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
