Re: [lustre-discuss] Limit to number of OSS?
On Oct 10, 2019, at 11:20, Michael Di Domenico mailto:mdidomeni...@gmail.com>> wrote: On Mon, Oct 7, 2019 at 6:33 PM Andreas Dilger mailto:adil...@whamcloud.com>> wrote: With socklnd there are 3 TCP connections per client-server pair. For IB there is no such connection limit that I'm aware of. just out of morbid curiosity, can very briefly explain the connectivity differences between TCP/IB. Does IB use the same 3 connections as TCP? If not, is that why the connectivity limit doesn't exist with IB or is there some other overriding design principal in IB that allows lustre to push past TCP? Not that any of this has any relevance to anything i do, i'm just curious. i'd love to have 2000 OSS's and 20k clients, but sadly i do not... :( This is a fundamental difference between TCP and IB. TCP needs a persistent connection between peers (socket) to manage state, and the (very ancient) IP protocol on which TCP is built has a limit of 65536 connections on a single node. When computers had 1-2MB of RAM that was more than enough... IB does not have this limitation, though it does consume some memory for each peer that that it is communicating with. o2iblnd can establish multiple connections to a single peer to get better bandwidth, and this is important for OPA performance, but is not critical for IB networks. Cheers, Andreas -- Andreas Dilger Principal Lustre Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Not able to start changelog in MDS
Hi Arnab, Are you running "lfs changelog" on the MDS or on the client? It needs to be run on the client. -Olaf From: lustre-discuss on behalf of Arnab Kumar Paul Sent: Thursday, October 10, 2019 10:42 AM To: lustre-discuss@lists.lustre.org Subject: [lustre-discuss] Not able to start changelog in MDS Hello, I have a Lustre system with version 2.10.3 with 1 MDS and 8 OSS On the client system: lfs df -h gives this output: UUID bytesUsed Available Use% Mounted on scratch0-MDT_UUID 27.8G 61.0M 25.2G 0% /mnt/lustre[MDT:0] scratch0-OST0001_UUID9.2G 37.2M8.7G 0% /mnt/lustre[OST:1] scratch0-OST0002_UUID9.2G 37.2M8.7G 0% /mnt/lustre[OST:2] . . . On MDS: $ lctl set_param mdt.scratch0-MDT.hsm_control=enabled mdt.scratch0-MDT.hsm_control=enabled $ lctl --device scratch0-MDT changelog_register scratch0-MDT: Registered changelog userid 'cl1' $ lctl get_param mdd.scratch0-MDT.changelog_users mdd.scratch0-MDT.changelog_size mdd.scratch0-MDT.changelog_users= current index: 1 IDindex (idle seconds) cl1 0 (56886) mdd.scratch0-MDT.changelog_size=33112 $ lfs changelog scratch0-MDT Can't start changelog: No such file or directory Why does this happen? Any suggestions how to start changelog? - Arnab Paul ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Not able to start changelog in MDS
Hello, I have a Lustre system with version 2.10.3 with 1 MDS and 8 OSS On the client system: lfs df -h gives this output: UUID bytesUsed Available Use% Mounted on scratch0-MDT_UUID 27.8G 61.0M 25.2G 0% /mnt/lustre[MDT:0] scratch0-OST0001_UUID9.2G 37.2M8.7G 0% /mnt/lustre[OST:1] scratch0-OST0002_UUID9.2G 37.2M8.7G 0% /mnt/lustre[OST:2] . . . On MDS: $* lctl set_param mdt.scratch0-MDT.hsm_control=enabled* mdt.scratch0-MDT.hsm_control=enabled $ *lctl --device scratch0-MDT changelog_register* scratch0-MDT: Registered changelog userid 'cl1' $ *lctl get_param mdd.scratch0-MDT.changelog_users mdd.scratch0-MDT.changelog_size* mdd.scratch0-MDT.changelog_users= current index: 1 IDindex (idle seconds) cl1 0 (56886) mdd.scratch0-MDT.changelog_size=33112 $ *lfs changelog scratch0-MDT* *Can't start changelog: No such file or directory* Why does this happen? Any suggestions how to start changelog? - Arnab Paul ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Limit to number of OSS?
On Mon, Oct 7, 2019 at 6:33 PM Andreas Dilger wrote: > > With socklnd there are 3 TCP connections per client-server pair. > For IB there is no such connection limit that I'm aware of. just out of morbid curiosity, can very briefly explain the connectivity differences between TCP/IB. Does IB use the same 3 connections as TCP? If not, is that why the connectivity limit doesn't exist with IB or is there some other overriding design principal in IB that allows lustre to push past TCP? Not that any of this has any relevance to anything i do, i'm just curious. i'd love to have 2000 OSS's and 20k clients, but sadly i do not... :( ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
