Hello Douglas! Many thanks for your advice ! Well we are in a horrible situation regarding firmware and MOFED of old equipment Mellanox advised us to use a special version of subnetmanager 5.0-2.1.8.0 from MOFED I hope this helps
Let´s see how we can proceed Best regards Walter From: gpfsug-discuss-boun...@spectrumscale.org <gpfsug-discuss-boun...@spectrumscale.org> On Behalf Of Douglas O'flaherty Sent: Donnerstag, 9. Dezember 2021 17:04 To: gpfsug-discuss@spectrumscale.org Subject: Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration Walter: Though not directly about your design, our work with NVIDIA on GPUdirect Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both MOFED and Firmware version compatibility can be. I would suggest anyone debugging RDMA issues should look at those closely. Doug by carrier pigeon ________________________________ On Dec 9, 2021, 5:04:36 AM, gpfsug-discuss-requ...@spectrumscale.org<mailto:gpfsug-discuss-requ...@spectrumscale.org> wrote: From: gpfsug-discuss-requ...@spectrumscale.org<mailto:gpfsug-discuss-requ...@spectrumscale.org> To: gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org> Cc: Date: Dec 9, 2021, 5:04:36 AM Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 119, Issue 5 ________________________________ Send gpfsug-discuss mailing list submissions to gpfsug-discuss@spectrumscale.orgTo<mailto:gpfsug-discuss@spectrumscale.orgTo> subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request@spectrumscale.orgYou<mailto:gpfsug-discuss-request@spectrumscale.orgYou> can reach the person managing the list at gpfsug-discuss-owner@spectrumscale.orgWhen<mailto:gpfsug-discuss-owner@spectrumscale.orgWhen> replying, please edit your Subject line so it is more specificthan "Re: Contents of gpfsug-discuss digest..." Send gpfsug-discuss mailing list submissions to gpfsug-discuss@spectrumscale.orgTo<mailto:gpfsug-discuss@spectrumscale.orgTo> subscribe or unsubscribe via the World Wide Web, visit http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' to gpfsug-discuss-request@spectrumscale.orgYou<mailto:gpfsug-discuss-request@spectrumscale.orgYou> can reach the person managing the list at gpfsug-discuss-owner@spectrumscale.orgWhen<mailto:gpfsug-discuss-owner@spectrumscale.orgWhen> replying, please edit your Subject line so it is more specificthan "Re: Contents of gpfsug-discuss digest..."Today's Topics: 1. alternate path between ESS Servers for Datamigration (Walter Sklenka) Dear spectrum scale users! May I ask you a design question? We have an IB environment which is very mixed at the moment ( connecX3 … connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 and HDR switches. We still have some big troubles in this fabric when using RDMA , a case at Mellanox and IBM is open . The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where we want to migrate the data to ess5000 , ( mmdelvdisk +qos) Due to the current problems with RDMA we though eventually we could try a workaround : If you are interested there is Maybe you can find the attachment ? We build 2 separate fabrics , the ess-IO servers attached to both blue and green and all other cluster members and all remote clusters only to fabric blue The daemon interfaces (IPoIP) are on fabric blue It is the aim to setup rdma only on the ess-ioServers in the fabric green , in the blue we must use IPoIB (tcp) Do you think datamigration would work between ess01,ess02,… to ess07,ess08 via RDMA ? Or is it principally not possible to make a rdma network only for a subset of a cluster (though this subset would be reachable via other fabric) ? Thank you very much for any input ! Best regards walter Mit freundlichen Grüßen Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31<tel:+43%201%2029%2022%20165-31> Fax: +43 1 29 22 165-90<tel:+43%201%2029%2022%20165-90> E-Mail: skle...@edv-design.at<mailto:skle...@edv-design.at> Internet: www.edv-design.at<http://www.edv-design.at/>
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
