Hi Giovanni! Thank you very much for your offer , we really would be very grateful to be allowed to come if we run into troubles!
Well, the implementation will not happen before June or later, but may I ask only one question meanwhile? Did you ever run into problems with IBM support or did you get a special “OK” from them? Or do you accept to sove any rdma specific problems without support ? (it´s only because of the FAQ “not supported” ) Have a great day and keep healthy! Best regards walter -----Original Message----- From: Giovanni Bracco <giovanni.bra...@enea.it> Sent: Montag, 1. Februar 2021 20:42 To: Walter Sklenka <walter.skle...@edv-design.at> Cc: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org> Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled On 30/01/21 21:01, Walter Sklenka wrote: > Hi Giovanni! > Thats great! Many thanks for your fast and detailed answer!!!! > So this is the way we will go too! > > Have a nice weekend and keep healthy! > Best regards > Walter > I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works! If yu have any issues I am ready to discuss! Regards Giovanni > -----Original Message----- > From: Giovanni Bracco > <giovanni.bra...@enea.it<mailto:giovanni.bra...@enea.it>> > Sent: Samstag, 30. Jänner 2021 18:08 > To: gpfsug main discussion list > <gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>>; > Walter Sklenka > <walter.skle...@edv-design.at<mailto:walter.skle...@edv-design.at>> > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD > Server with only ib rdma enabled > > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of > them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge > cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 > nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it > works nicely using RDMA since 2018. GPFS 4.2.3-19. > See > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a > multifabric GPFS Spectrum Scale layout," 2019 International Conference > on High Performance Computing & Simulation (HPCS), Dublin, Ireland, > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 > > When setting up the system the main trick has been: > just use CentOS drivers and do not install OFED We do not use IPoIB. > > Giovanni > > On 30/01/21 06:45, Walter Sklenka wrote: >> Hi! >> >> Is it possible to mix OPAcards and Infininiband HCAs on the same server? >> >> In the faq >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. >> html#rdma >> >> >> They talk about RDMA : >> >> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel >> Omni-Path HFIs are ENABLED for RDMA." >> >> So do I understand right: When we do NOT enable the opa interface we >> can still enable IB ? >> >> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers >> (wih access to storage) with opa interfaces which provide access to >> remote cluster also via OPA. >> >> A new cluster with HDR interfaces will be implemented soon >> >> They shell have access to the same filesystems >> >> When we add HDR interfaces to NSD servers and enable rdma on this >> network while disabling rdma on opa we would accept the worse >> performance via opa . We hope that this provides still better perf >> and less technical overhead than using routers >> >> Or am I totally wrong? >> >> Thank you very much and keep healthy! >> >> Best regards >> >> Walter >> >> Mit freundlichen Grüßen >> */Walter Sklenka/* >> */Technical Consultant/* >> >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 >> Wien >> Tel: +43 1 29 22 165-31 >> Fax: +43 1 29 22 165-90 >> E-Mail: skle...@edv-design.at<mailto:skle...@edv-design.at> >> <mailto:skle...@edv-design.at> >> Internet: www.edv-design.at<http://www.edv-design.at> >> <http://www.edv-design.at/> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > -- > Giovanni Bracco > phone +39 351 8804788 > E-mail giovanni.bra...@enea.it<mailto:giovanni.bra...@enea.it> > WWW http://www.afs.enea.it/bracco > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bra...@enea.it<mailto:giovanni.bra...@enea.it> WWW http://www.afs.enea.it/bracco
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss