Re: does i need compat-rdma package
Hello Vasilly Tolstov This URL might be help. http://www.rdmamojo.com There are RDMA configuration for many OSs. (RHEL, Ubuntu, SLES..) -- Hiroyuki Sato 2014-12-24 17:19 GMT+09:00 Or Gerlitz : > On 12/23/2014 4:47 PM, Vasiliy Tolstov wrote: >> >> Hello. I want to develop some software under linux 3.14 and using >> infiniband libraries. > > > You can use the inbox libraries and install them through your distro package > installer, with RHEL, Fedora and such > it would be just > > $ yum groupinstall "infiniband support" > > to get you the set of required RPMS [1] SLES and Ubuntu should have similar > means, no need to use OFED, compat, etc. > > Or. > > [1] you mentioned ConnectX-3, so your basic needs are libmlx4, libibverbs > and librdmacm > >> I'm download latest ofed (3.12 ?) and see that >> some packages i don't need (i'm use only mellanox connect x3) >> But i can't understand in case of linux 3.14 does i need compat-rdma ? >> >> > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Hiroyuki Sato -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Q] How to tranfer a file which is over 2GB(2^31) size in RDMA network?
Hello Sean Thank you for your information. I'll try it. P.S. I'm reading rockets(librdmacm) commit log for study Infiniband. :-) 2012/7/4 Hefty, Sean : >> Hello Parav.Pandit >> >> Thank you for your advice. >> >> I'll try it. > > You can also look at rsockets in the latest librdmacm library. You'd need to > download and build the library yourself, since rsockets is not yet available > in any release. But there's a sample program (rcopy) that will copy a file > between two systems over that API. > > I haven't tried using it to copy a very large files, but in theory it should > work. (Hey, maybe you'll get lucky!) If you do try it and hit into any > problems, please let me know. > > - Sean -- Hiroyuki Sato -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Q] How to tranfer a file which is over 2GB(2^31) size in RDMA network?
Hello Parav.Pandit Thank you for your advice. I'll try it. -- Hiroyuki Sato 2012/7/3 : > > >> -Original Message- >> From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma- >> ow...@vger.kernel.org] On Behalf Of Hiroyuki Sato >> Sent: Monday, July 02, 2012 7:20 PM >> To: linux-rdma >> Subject: [Q] How to tranfer a file which is over 2GB(2^31) size in RDMA >> network? >> >> Dear developers. >> >> I'm writing simple file transfer program. >> >> I would like to know about the following. >> Q: How to tranfer a file which is over 2GB(2^31) size in RDMA network? >> >> Please imagine to transfer whole DVD(4.7GB) file via RDMA networok. >> >> The maximum message size of RC is 2^31. >> so I can't transfer it with one RDMA message. >> >> Maybe It must split multiple parts. >> When I sent first part with the following sequence, how to transfer second >> part? >> >> Do I have to call qp ibv_destroy_qp and recreate new qp? >> Or can I reuse current qp? >> > You can use the same QP to send multiple messages of same or different size. > Receive side needs to have sufficient memory to receive 2GB of data, which > you should have posted using post_recv(). > Since you are going to divide a file/consumer application data to multiple > RDMA messages, make sure sender and receiver agrees to a message size of > interest. > This is the simplest way to do via RDMA_SEND and RECV buffers. > > You can further do it via sharing the stag before invoking post_send/recv and > use RDMA Write or READ operations and don't need to synchronize the message > size. > Sender can send/split data into one or more WRITE messages followed by ending > it with write_immidiate or send_immidiate to notify the peer of file transfer > done. > > In a third way, you can void writing the application and use SDP protocol to > send your file via FTP/SFTP, in which FTP server and client application > sockets use the SDP sockets instead of TCP sockets. > > >> I'm looking for similar example but I can't find it. >> Any information is welcome. >> >> Thank you for your advice. >> >> Sincerely >> >> -- >> Hiroyuki Sato. >> >> >> File transfer sequence (1st part) >> >> ibv_open_device >> ibv_alloc_pd >> ibv_reg_mr >> ibv_create_cq >> ibv_create_qp >> ibv_modify_qp(RESET->INIT) >> ibv_post_recv >> exchange lid, sid,qpn >> ibv_connect >> ibv_modify_qp(INIT->RTR) >> ibv_modify_qp(RTR->RTS) >> ibv_post_send >> ... >> >> >> >> Environment >> OS Scientific Linux 6.2 >> OFED: 1.5.4.1 >> >> -- >> Hiroyuki Sato >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the >> body of a message to majord...@vger.kernel.org More majordomo info at >> http://vger.kernel.org/majordomo-info.html -- Hiroyuki Sato -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Q] How to tranfer a file which is over 2GB(2^31) size in RDMA network?
Dear developers. I'm writing simple file transfer program. I would like to know about the following. Q: How to tranfer a file which is over 2GB(2^31) size in RDMA network? Please imagine to transfer whole DVD(4.7GB) file via RDMA networok. The maximum message size of RC is 2^31. so I can't transfer it with one RDMA message. Maybe It must split multiple parts. When I sent first part with the following sequence, how to transfer second part? Do I have to call qp ibv_destroy_qp and recreate new qp? Or can I reuse current qp? I'm looking for similar example but I can't find it. Any information is welcome. Thank you for your advice. Sincerely -- Hiroyuki Sato. File transfer sequence (1st part) ibv_open_device ibv_alloc_pd ibv_reg_mr ibv_create_cq ibv_create_qp ibv_modify_qp(RESET->INIT) ibv_post_recv exchange lid, sid,qpn ibv_connect ibv_modify_qp(INIT->RTR) ibv_modify_qp(RTR->RTS) ibv_post_send ... Environment OS Scientific Linux 6.2 OFED: 1.5.4.1 -- Hiroyuki Sato -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Q] Mellanox ConnectX QDR on SandyBridge motherboard works only PCIex GEN1 speed.
Dears developers. We are now investigating Mellanox ConnectX QDR on SandyBridge motherboard(P67/H67/C206), and have a serious performance problem on this configuration. Any information is appreciated. * Questions (1) kerenel parameters. What parameter should I change to detect ConnectX GEN2 speed?? with vanilla kernel?? It is mean that, enable CONFIG_ parameters, or change module parameters, apply patch .. and so on (2) What kernel function should I check to work ConnectX with PCIex GEN2 speed?? on C206 chipset. check kernel output, execute some commands, add printk in functions * Problem (1) ConnectX works only PCI express GEN1 speed (2.5GT/s) on CentOS5.5 and Ubuntu 10.10 on P67/H67 chipset. This is lspci output. (complete output attached below) LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- But, Scientific Linux6, and Ubuntu 10.10 works GEN2(5.0GT/s) speed on same motherboard. And I tried to build vanilla kernel on Ubuntu 10.04 with default configuration. The kerenel newer than Ubuntu 10.10, but It still works only GEN1 speeed. (2) ConnectX works only PCI express GEN1 speed on Scientific Linux6, CentOS 5.5 Ubuntu 10.10 and 10.04 * Environment (one of example) - OFED 1.5.3.1 - Ubuntu 10.4 x64 server, and CentOS 5.5 - ASUS P8P67-WS-Revolution - 1-port ConnectX QDR : Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) * lspci output 04:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) Subsystem: Mellanox Technologies Device 0022 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [9c] MSI-X: Enable+ Mask- TabSize=256 Vector table: BAR=0 offset=0007c000 PBA: BAR=0 offset=0007d000 Capabilities: [60] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited ClockPM- Suprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [100] #0e Kernel driver in use: mlx4_core Kernel modules: mlx4_en, mlx4_core 00: b3 15 3c 67 06 04 10 00 b0 00 06 0c 10 00 00 00 10: 04 00 10 fe 00 00 00 00 0c 00 00 d0 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 b3 15 22 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 05 01 00 00 40: 01 48 03 00 00 00 00 00 03 9c ff 7f 11 11 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 10 00 02 00 01 8e 00 00 00 20 00 00 82 f4 03 08 70: 00 00 81 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 1f 00 00 00 00 00 00 00 00 00 00 00 90: 02 00 00 00 00 00 00 00 00 00 00 00 11 60 ff 80 a0: 00 c0 07 00 00 d0 07 00 05 00 8a 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Sincerely -- Hiroyuki Sato. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: IPoIB to Ethernet routing performance
Basic Question. MTU size matched?? on 10GbE and Infiniband BTW, could you tell me more detail about 10GbE configuration. what kind of teaming are you using?? (ex IEEE802.3ad) (It is my interest.) -- Hiroyuki Sato. 2010/12/7 sebastien dugue : > On Tue, 7 Dec 2010 12:33:32 +0200 > Or Gerlitz wrote: > >> On 12/7/2010 12:27 PM, sebastien dugue wrote: >> > >> > Huh? How do you do that? >> see Documentation/infiniband/ipoib.txt on your clone of Linus tree or if >> you're not a developer, see >> http://lxr.linux.no/#linux+v2.6.36/Documentation/infiniband/ipoib.txt >> the HCA you're using does support offloads, so you're probably running >> connected mode, use datagram > > Right, I'm running in connected mode. Last time I checked, datagram mode > was far from connected mode performance wise. > > Sébastien. > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Q] I tried to mount nfs_server:/mnt volume, but It mounted another volume.
Thank you useful information. I'll study about NFSv4 -- Hiroyuki Sato 2010/11/22 J. Bruce Fields : > On Sun, Nov 21, 2010 at 03:41:18PM +0200, Boaz Harrosh wrote: >> On 11/21/2010 02:23 PM, Hiroyuki Sato wrote: >> > Hello lists >> > >> > OS: CentOS 5.5 >> > kernel: 2.6.36 rebuild myself. >> > >> > >> > I'm trying to test NFS/RDMA. >> > I tried to mount nfs_server:/mnt volume on on nfs_client >> > but It mounted nfs_server:/nfstest volume >> > >> > note: /nfstest is tmpfs >> > >> > this is mount output >> > tmpfs on /nfstest type tmpfs (rw,size=4g) >> > /dev/sdb1 on /mnt type ext3 (rw) >> > >> > Is this bug?? >> > >> > NFS server config >> > >> > # ls -1 /mnt >> > This_is_mnt_volume >> > >> > # ls -1 /nfstest >> > This_is_nfstest_volume >> > >> > # cat /etc/exports >> > /nfstest >> > 192.168.100.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) >> > /mnt >> > 192.168.100.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) >> > >> >> You must not have two exports with fsid=0. First one is picked. >> >> nfs4 will only export a single name space point, other exports are >> subdirs of that root export. (use bind mounts to present a single >> directory tree) >> >> <http://www.citi.umich.edu/projects/nfsv4/linux/using-nfsv4.html> > > We should update and/or replace that documentation; with new nfs-utils > and kernel, if you omit the "fsid=0", you'll end up with an nfsv4 > namespace that's the same as v2/v3's always was. > > --b. > >> >> NFSv4 exports on linux >> ~~ >> >> NFSv4 no longer has a separate "mount" protocol. Instead of exporting a >> number of distinct exports, an NFSv4 client sees the NFSv4 server's exports >> as existing inside a single filesystem, called the nfsv4 "pseudofilesystem". >> >> On the current linux implementation, the pseudofilesystem is a single real >> filesystem, identified at export with the fsid=0 option. >> >> In the example above, we exported only a single filesystem, which the client >> mounted as "/". You can provide clients with multiple filesystems to mount, >> producing NFSv3-like-behavior, by creative use of mount --bind. For example, >> you could export /usr/local/bin to clients as /bin and /usr/local/etc as >> /etc as follows: >> >> mkdir /export >> mkdir /export/bin >> mkdir /export/etc >> mount --bind /usr/local/bin /export/bin >> mount --bind /usr/local/etc /export/etc >> exportfs -ofsid=0,insecure,no_subtree_check *:/export >> exportfs -orw,nohide,insecure,no_subtree_check *:/export/bin >> exportfs -orw,nohide,insecure,no_subtree_check *:/export/etc >> >> Note that the paths returned by the "showmount" program are meaningful only >> to clients using nfs versions 2 and 3; in the above example, "showmount" >> will list the paths /export, /export/bin/, and /export/etc, but nfsv4 >> clients should mount yourserver:/, yourserver:/bin, or yourserver:/etc. >> >> http://www.citi.umich.edu/projects/nfsv4/linux/using-nfsv4.html> >> >> Boaz >> > >> > # modprobe svcrdma >> > >> > # /sbin/service nfs start >> > >> > # echo rdma 20049 > /proc/fs/nfsd/portlist >> > >> > >> > Client Setting >> > >> > /sbin/modprobe xprtrdma >> > /sbin/mount.rnfs 192.168.100.231:/mnt /mnt -i -o rdma,port=20049 >> > >> > # ls -1 /mnt >> > This_is_nfstest_volume >> > >> > NFS Server log >> > sysctl -w sunrpc.nfsd_debug=1023 >> > >> > Nov 21 20:47:37 dell1435 mountd[3575]: authenticated mount request >> > from 192.168.100.232:766 for /mnt (/mnt) >> > Nov 21 20:47:37 dell1435 mountd[3575]: /nfstest and /mnt have same >> > filehandle for 192.168.100.0/255.255.255.0, using first >> > Nov 21 20:48:55 dell1435 mountd[3575]: authenticated unmount request >> > from 192.168.100.232:912 for /mnt (/mnt) >> > Nov 21 20:48:55 dell1435 mountd[3575]: authenticated unmount request >> > from 192.168.100.232:913 for /mnt (/mnt) >> > Nov 21 20:49:00 dell1435 mountd[3575]: authenticated unmount request >> > from 192.168.100.232:917 for /mnt (/mnt) >> > Nov 21 20:49:16 dell1435 mountd[3
[Q] I tried to mount nfs_server:/mnt volume, but It mounted another volume.
Hello lists OS: CentOS 5.5 kernel: 2.6.36 rebuild myself. I'm trying to test NFS/RDMA. I tried to mount nfs_server:/mnt volume on on nfs_client but It mounted nfs_server:/nfstest volume note: /nfstest is tmpfs this is mount output tmpfs on /nfstest type tmpfs (rw,size=4g) /dev/sdb1 on /mnt type ext3 (rw) Is this bug?? NFS server config # ls -1 /mnt This_is_mnt_volume # ls -1 /nfstest This_is_nfstest_volume # cat /etc/exports /nfstest 192.168.100.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) /mnt 192.168.100.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) # modprobe svcrdma # /sbin/service nfs start # echo rdma 20049 > /proc/fs/nfsd/portlist Client Setting /sbin/modprobe xprtrdma /sbin/mount.rnfs 192.168.100.231:/mnt /mnt -i -o rdma,port=20049 # ls -1 /mnt This_is_nfstest_volume NFS Server log sysctl -w sunrpc.nfsd_debug=1023 Nov 21 20:47:37 dell1435 mountd[3575]: authenticated mount request from 192.168.100.232:766 for /mnt (/mnt) Nov 21 20:47:37 dell1435 mountd[3575]: /nfstest and /mnt have same filehandle for 192.168.100.0/255.255.255.0, using first Nov 21 20:48:55 dell1435 mountd[3575]: authenticated unmount request from 192.168.100.232:912 for /mnt (/mnt) Nov 21 20:48:55 dell1435 mountd[3575]: authenticated unmount request from 192.168.100.232:913 for /mnt (/mnt) Nov 21 20:49:00 dell1435 mountd[3575]: authenticated unmount request from 192.168.100.232:917 for /mnt (/mnt) Nov 21 20:49:16 dell1435 mountd[3575]: authenticated mount request from 192.168.100.232:865 for /mnt (/mnt) Nov 21 21:02:22 dell1435 mountd[3575]: authenticated unmount request from 192.168.100.232:955 for /mnt (/mnt) Nov 21 21:02:26 dell1435 mountd[3575]: authenticated mount request from 192.168.100.232:884 for /mnt (/mnt) Nov 21 21:02:26 dell1435 kernel: nfsd: exp_rootfh(/mnt [88011f586740] 192.168.100.0/255.255.255.0:sdb1/2) -- Hiroyuki Sato -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Q] I would like to test NFS/RDMA with multiple ports on same fabric
Dear members. I'm newbie about Infiniband. I would like to test NFS/RDMA on the following environment. I succeed to setup NFS/RDMA between NFS Server and NFS Client1 Probelm: I want to add second NFS client (NFS client2) but I can't setup IP over Infiniband between NFS server and NFS client2 (I don't have Infiniband switch so I want to use multiple ports on same fabric) NFS Serer% ping nfs_client2 no response Question: (1) Is this environment work fine if I setup correctly?? ex) I have to change subnet address I have to use Infiniband Switch (2) What is the problem?? Could you tell me how to check what is the problem. or Please tell me good example if avairable . Thank you for your information. Sincrely -- Hiroyuki Sato. opensmd NFS Server 192.168.100.231(port1) 192.168.200.231(port2)NFS client1(192.168.100.232) +--+ +--+ | | 192.168.100.0/24 | | | port1port1| | port2--+ port2| | | | | | +--+| +--+ | | | NFS client2(192.168.100.233) | +--+ | | | +--port1 | 192.168.200.0/24 port2 | | | +--+ Environment OS: CentOS 5.5 Infiniband Card: MHEA28-XTC Kernel: 2.6.36 This is ibstat output on NFS Server # ibstat CA 'mthca0' CA type: MT25208 Number of ports: 2 Firmware version: 5.3.0 Hardware version: a0 Node GUID: 0x0002c90200220eb8 System image GUID: 0x0005ad000100d050 Port 1: State: Active Physical state: LinkUp Rate: 10 Base lid: 1 LMC: 0 SM lid: 1 Capability mask: 0x02590a6a Port GUID: 0x0002c90200220eb9 Link layer: IB Port 2: State: Active Physical state: LinkUp Rate: 10 Base lid: 1 LMC: 0 SM lid: 3 Capability mask: 0x02590a68 Port GUID: 0x0002c90200220eba Link layer: IB -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
