Re: [lustre-discuss] lnet routing issue - 2.12.5 client with 2.10.3 server
Hi Mark, [Tue Dec 1 11:07:55 2020] LNetError: 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 10.110.0.21@tcp from I would suggest checking lnetctl routing show and remove the route to 10.110.0.21@tcp and try to mount. https://wiki.lustre.org/LNet_Router_Config_Guide On Tue, Dec 1, 2020 at 2:41 PM Mark Lundie wrote: > Hi all, > > I've just run in to an issue mounting on a newly upgraded client running > 2.12.5 with 2.10.3 servers. Just to give some background, we're about to > replace our existing Lustre storage, but will run it concurrently with the > replacement for a couple of months. We'll be running 2.12.5 server on the > new MDS and OSSs and I plan to update all clients to the same version. I > would like to avoid updating the existing servers though. > > The problem is this. The servers have two tcp LNET networks, tcp and tcp1, > on separate subnets and VLANs. The clients only see tcp1 (a small number > are also on tcp3, routed via 2 lnet routers), which has been fine until > now. With the 2.12.5 client, however, it is trying to mount from tcp. > 2.10.3 to 2.12.5 is obviously a bit of a jump, but does anyone have any > ideas on what has changed and what I could do here please? > > meta# lnetctl net show > net: > - net type: lo > local NI(s): > - nid: 0@lo > status: up > - net type: tcp > local NI(s): > - nid: 10.110.0.21@tcp > status: up > interfaces: > 0: bond0.22 > - net type: tcp1 > local NI(s): > - nid: 10.10.0.91@tcp1 > status: up > interfaces: > 0: bond0 > > meta# lnetctl route show > route: > - net: tcp2 > gateway: 10.10.0.254@tcp1 > - net: tcp3 > gateway: 10.10.0.254@tcp1 > > > client# lnetctl net show > net: > - net type: lo > local NI(s): > - nid: 0@lo > status: up > - net type: o2ib > local NI(s): > - nid: 10.12.170.47@o2ib > status: up > interfaces: > 0: ib0 > - net type: tcp1 > local NI(s): > - nid: 10.10.170.47@tcp1 > status: up > interfaces: > 0: em1 > > [Tue Dec 1 11:07:55 2020] LNetError: > 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to > 10.110.0.21@tcp from > [Tue Dec 1 11:08:01 2020] LustreError: > 1792:0:(mgc_request.c:249:do_config_log_add()) MGC10.10.0.91@tcp1: failed > processing log, type 1: rc = -5 > [Tue Dec 1 11:08:08 2020] LustreError: > 2169:0:(mgc_request.c:599:do_requeue()) failed processing log: -5 > [Tue Dec 1 11:08:19 2020] LNetError: > 2127:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to > 10.110.0.22@tcp from > [Tue Dec 1 11:08:30 2020] LustreError: 15c-8: MGC10.10.0.91@tcp1: The > configuration from log 'lustre-client' failed (-5). This may be the result > of communication errors between this node and the MGS, a bad configuration, > or other errors. See the syslog for more information. > > client# lctl ping 10.10.0.91@tcp1 > 12345-0@lo > 12345-10.110.0.21@tcp > 12345-10.10.0.91@tcp1 > > Any suggestions will be greatly appreciated! > > Many thanks, > > Mark > > *Dr Mark Lundie* | Research IT Systems Administrator | Research IT | > Directorate of IT Services | B39, Sackville Street Building | The > University of Manchester | Manchester | M1 3WE | 0161 275 8403 | > ri.itservices.manchester.ac.uk > > Working Hours: Tues - Thurs 0730-1730; Fri 0730-1630 > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] new mounted client shows lower disk space
Hi Thomas And Raj, Thank you for the feedback Thomas, I have checked the recovery_status on oss's I assume this recovery durations are second and i have to check it while mount operation takes long time. These values are updated every minute. Good to learn that. oss1 status: COMPLETE recovery_start: 1539937227 *recovery_duration: 97* completed_clients: 3/3 replayed_requests: 0 last_transno: 73019446366 VBR: DISABLED IR: ENABLED oss2 status: COMPLETE recovery_start: 1540380323 *recovery_duration: 436* completed_clients: 196/197 replayed_requests: 0 last_transno: 77309411331 VBR: ENABLED IR: ENABLED oss3 status: COMPLETE recovery_start: 1539937210 *recovery_duration: 150* completed_clients: 0/3 replayed_requests: 0 last_transno: 73019440310 VBR: ENABLED IR: ENABLED oss4 status: COMPLETE recovery_start: 1539937234 *recovery_duration: 151* completed_clients: 0/3 replayed_requests: 0 last_transno: 55839576629 VBR: ENABLED IR: ENABLED oss5 status: COMPLETE recovery_start: 1539937257 *recovery_duration: 96* completed_clients: 3/3 replayed_requests: 0 last_transno: 51544609437 VBR: DISABLED IR: ENABLED oss6 recovery_start: 1539937194 *recovery_duration: 96* completed_clients: 3/3 replayed_requests: 0 last_transno: 47249690300 VBR: DISABLED IR: ENABLED Filesystem mounts on system boot process and its solid that after each reboot lctl ping from client to server finishes with no error and vice versa. It seems like when there is a high I/O on the filesystem, mount operation takes longer. Best Regards. On Wed, Nov 14, 2018 at 6:13 PM Raj wrote: > I would check if LNET address gets setup properly before mounting lustre > FS from client. You can try manually loading lustre module and try pinging > (lctl ping oss-nid) all the OSS nodes and observe any abnormalities and > dmesg before mounting FS. > It could be as simple as duplicate IP address in your ib interface or > unstable IB fabric. > > On Wed, Nov 14, 2018 at 8:08 AM Thomas Roth wrote: > >> Hi, >> >> your error messages are all well known - the one on the OSS will show up >> as soon as the Lustre modules >> are loaded, provided you have some clients asking for the OSTs (and your >> MDT, which should be up by >> then, is also looking for the OSTs). >> The kiblnd_check_conns message I have also seen quite often, never with >> any OST problems. >> >> Rather seems your OST take a lot of time to mount or to recover - did you >> check >> /proc/fs/lustre/obdfilter/lustre-OST00*/recovery_status >> ? >> >> Regards >> Thomas >> >> On 11/12/18 9:46 AM, fırat yılmaz wrote: >> > Hi All >> > OS=Redhat 7.4 >> > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 >> > >> > I have 72 osts over 6 oss with HA and 1 mdt serving to 195 clients over >> > infiniband EDR. >> > >> > After a reboot on client, lustre filesystem mounts on startup. It >> should be >> > 2.1 TB area but lt starts with 350TB. >> > >> > lfs osts command shows 90 percent of even numbered osts are ACTIVE and >> no >> > information about other OSTs, as time passes like 1 hour or so, all OSTs >> > become active and lustre area can be seen as 2.1 PB >> > >> > >> > dmesg on lustre oss server: >> > LustreError: 137-5: lustre-OST0009_UUID: not available for connect from >> > 10.0.0.130@o2ib (no target). If you are running an HA pair check that >> the >> > target is mounted on the other server. >> > >> > dmesg on client: >> > LNet: 5419:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for >> > 10.0.0.5@o2ib: 15 seconds >> > Lustre: 5546:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request >> sent >> > has failed due to network error: [sent 1542009416/real 1542009426] >> > req@885f4761 x1616909446641136/t0(0) >> > o8->lustre-OST0030-osc-885f75219800@10.0.0.8@o2ib:28/4 lens >> 520/544 e 0 >> > to 1 dl 1542009696 ref 1 fl Rpc:eXN/0/ rc 0/-1 >> > >> > I tested infiniband with ib_send_lat, ib_read_lat and no error occured >> > I tested lnet ping with lctl ping 10.0.0.8@o2ib , no error occured >> > 12345-0@lo >> > 12345-10.51.22.8@o2ib >> > >> > Why some OST's can be accesible while some are not in the same server? >> > Best Regards. >> > >> > >> > ___ >> > lustre-discuss mailing list >> > lustre-discuss@lists.lustre.org >> > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> > >> >> >> ___ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] new mounted client shows lower disk space
Hi All OS=Redhat 7.4 Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 I have 72 osts over 6 oss with HA and 1 mdt serving to 195 clients over infiniband EDR. After a reboot on client, lustre filesystem mounts on startup. It should be 2.1 TB area but lt starts with 350TB. lfs osts command shows 90 percent of even numbered osts are ACTIVE and no information about other OSTs, as time passes like 1 hour or so, all OSTs become active and lustre area can be seen as 2.1 PB dmesg on lustre oss server: LustreError: 137-5: lustre-OST0009_UUID: not available for connect from 10.0.0.130@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. dmesg on client: LNet: 5419:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.0.5@o2ib: 15 seconds Lustre: 5546:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1542009416/real 1542009426] req@885f4761 x1616909446641136/t0(0) o8->lustre-OST0030-osc-885f75219800@10.0.0.8@o2ib:28/4 lens 520/544 e 0 to 1 dl 1542009696 ref 1 fl Rpc:eXN/0/ rc 0/-1 I tested infiniband with ib_send_lat, ib_read_lat and no error occured I tested lnet ping with lctl ping 10.0.0.8@o2ib , no error occured 12345-0@lo 12345-10.51.22.8@o2ib Why some OST's can be accesible while some are not in the same server? Best Regards. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Second read or write performance
Hi Patrick, Thank you for clarifying flock capabilities. So many think can cause the difference between 2 test results i saw in dashboard, now i learn that flock has no effect on it. Best Regars. 22 Eyl 2018 Cmt 04:14 tarihinde Patrick Farrell şunu yazdı: > Firat, > > I strongly suspect that careful remeasurement of flock on/off will show > that removing the flock option had no effect at all. It simply doesn’t DO > anything like that - it controls a single flag that says, if you use flock > operations, they work one way, or if it is not set, they work another way. > It does nothing else, and has no impact on any part of file system > operation except when flocks are used, and dd does not use flocks. It is > simply impossible for the setting of the flock option to affect dd or > performance level or variation, unless something using flocks is running at > the same time. (And even then, it would be affecting it indirectly) > > I’m pushing back strongly because I’ve repeatedly seen people on the > mailing speculate about turning flock off as a way to increase performance, > and it simply isn’t. > > - Patrick > > > -- > *From:* fırat yılmaz > *Sent:* Friday, September 21, 2018 7:50:51 PM > *To:* Patrick Farrell > *Cc:* adil...@whamcloud.com; lustre-discuss@lists.lustre.org > *Subject:* Re: [lustre-discuss] Second read or write performance > > The problem solved by adding lustre fine tuning parameter to oss servers > > lctl set_param obdfilter.lı-lustrefolder-OST*.brw_size=16 > > The flock is required by the application running in the filesystem so > flock option is enabled > > removing flock decrased the divergence of the flactuations and about %5 > performance gain from iml dashboard > > Best Regards. > > On Sat, Sep 22, 2018 at 12:56 AM Patrick Farrell wrote: > > Just 300 GiB, actually. But that's still rather large and could skew > things depending on OST size. > > - Patrick > > On 9/21/18, 4:43 PM, "lustre-discuss on behalf of Andreas Dilger" < > lustre-discuss-boun...@lists.lustre.org on behalf of adil...@whamcloud.com> > wrote: > > On Sep 21, 2018, at 00:43, fırat yılmaz > wrote: > > > > Hi Andreas, > > Tests are made with dd, The test folder is created by the related > application company, i will check that when i have connection. OST's has > %85-86 free space and filesystem mounted with flock option, i will ask for > it to remove and test again. > > The "flock" option shouldn't make any difference, unless the > application is actually doing userspace file locking in the code. > Definitely "dd" will not be using it. > > What does "lfs getstripe" on the first and second file as well as the > parent directory show, and "lfs df" for the filesystem? > > > Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null > bs=1M count=30 > > > > Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` > bs=1M count=300000 > > This is creating a single file of 300TB in size, so that is definitely > going to skew the space allocation. > > Cheers, Andreas > > > > > On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger < > adil...@whamcloud.com> wrote: > > On Sep 20, 2018, at 03:07, fırat yılmaz > wrote: > > > > > > Hi all, > > > > > > OS=Redhat 7.4 > > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > > İnterconnect: Mellanox OFED, ConnectX-5 > > > 72 OST over 6 OSS with HA > > > 1mdt and 1 mgt on 2 MDS with HA > > > > > > Lustre servers fine tuning parameters: > > > lctl set_param timeout=600 > > > lctl set_param ldlm_timeout=200 > > > lctl set_param at_min=250 > > > lctl set_param at_max=600 > > > lctl set_param obdfilter.*.read_cache_enable=1 > > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > > > Lustre clients fine tuning parameters: > > > lctl set_param osc.*.checksums=0 > > > lctl set_param timeout=600 > > > lctl set_param at_min=250 > > > lctl set_param at_max=600 > > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > > lctl set_param osc.*.max_pages_per_rpc=1024 > > > lctl set_param llite.*.max_read_ahead_mb=1024 &
Re: [lustre-discuss] Second read or write performance
The problem solved by adding lustre fine tuning parameter to oss servers lctl set_param obdfilter.lı-lustrefolder-OST*.brw_size=16 The flock is required by the application running in the filesystem so flock option is enabled removing flock decrased the divergence of the flactuations and about %5 performance gain from iml dashboard Best Regards. On Sat, Sep 22, 2018 at 12:56 AM Patrick Farrell wrote: > Just 300 GiB, actually. But that's still rather large and could skew > things depending on OST size. > > - Patrick > > On 9/21/18, 4:43 PM, "lustre-discuss on behalf of Andreas Dilger" < > lustre-discuss-boun...@lists.lustre.org on behalf of adil...@whamcloud.com> > wrote: > > On Sep 21, 2018, at 00:43, fırat yılmaz > wrote: > > > > Hi Andreas, > > Tests are made with dd, The test folder is created by the related > application company, i will check that when i have connection. OST's has > %85-86 free space and filesystem mounted with flock option, i will ask for > it to remove and test again. > > The "flock" option shouldn't make any difference, unless the > application is actually doing userspace file locking in the code. > Definitely "dd" will not be using it. > > What does "lfs getstripe" on the first and second file as well as the > parent directory show, and "lfs df" for the filesystem? > > > Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null > bs=1M count=30 > > > > Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` > bs=1M count=30 > > This is creating a single file of 300TB in size, so that is definitely > going to skew the space allocation. > > Cheers, Andreas > > > > > On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger < > adil...@whamcloud.com> wrote: > > On Sep 20, 2018, at 03:07, fırat yılmaz > wrote: > > > > > > Hi all, > > > > > > OS=Redhat 7.4 > > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > > İnterconnect: Mellanox OFED, ConnectX-5 > > > 72 OST over 6 OSS with HA > > > 1mdt and 1 mgt on 2 MDS with HA > > > > > > Lustre servers fine tuning parameters: > > > lctl set_param timeout=600 > > > lctl set_param ldlm_timeout=200 > > > lctl set_param at_min=250 > > > lctl set_param at_max=600 > > > lctl set_param obdfilter.*.read_cache_enable=1 > > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > > > Lustre clients fine tuning parameters: > > > lctl set_param osc.*.checksums=0 > > > lctl set_param timeout=600 > > > lctl set_param at_min=250 > > > lctl set_param at_max=600 > > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > > lctl set_param osc.*.max_pages_per_rpc=1024 > > > lctl set_param llite.*.max_read_ahead_mb=1024 > > > lctl set_param llite.*.max_read_ahead_per_file_mb=1024 > > > > > > Mountpoint stripe count:72 stripesize:1M > > > > > > I have a 2Pb lustre filesystem, In the benchmark tests i get the > optimum values for read and write, but when i start a concurrent I/O > operation, second job throughput stays around 100-200Mb/s. I have tried > lovering the stripe count to 36 but since the concurrent operations will > not occur in a way that keeps OST volume inbalance, i think that its not a > good way to move on, secondly i saw some discussion about turning off flock > which ended up unpromising. > > > > > > As i check the stripe behaviour, > > > first operation starts to use first 36 OST > > > when a second job starts during a first job, it uses second 36 OST > > > > > > But when second job starts after 1st job it uses first 36 OST's > which causes OST unbalance. > > > > > > Is there a round robin setup that each 36 OST pair used in a round > robin way? > > > > > > And any kind of suggestions are appreciated. > > > > Can you please describe what command you are using for testing. > Lustre is already using round-robin OST allocation by default, so the > second job should use the next set of 36 OSTs, unless the file layout has > been specified e.g. to start on OST or the space usage of the OSTs is > very imbalanced (more than 17% of the remaining free space). > > > > Cheers, Andreas > > --- > > Andreas Dilger > > Principal Lustre Architect > > Whamcloud > > > > > > > > > > > > > > > > Cheers, Andreas > --- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > > > > > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Second read or write performance
Hi Andreas, Tests are made with dd, The test folder is created by the related application company, i will check that when i have connection. OST's has %85-86 free space and filesystem mounted with flock option, i will ask for it to remove and test again. Thanks Andereas, Best Regards. Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null bs=1M count=30 Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` bs=1M count=30 On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger wrote: > On Sep 20, 2018, at 03:07, fırat yılmaz wrote: > > > > Hi all, > > > > OS=Redhat 7.4 > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > İnterconnect: Mellanox OFED, ConnectX-5 > > 72 OST over 6 OSS with HA > > 1mdt and 1 mgt on 2 MDS with HA > > > > Lustre servers fine tuning parameters: > > lctl set_param timeout=600 > > lctl set_param ldlm_timeout=200 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param obdfilter.*.read_cache_enable=1 > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > Lustre clients fine tuning parameters: > > lctl set_param osc.*.checksums=0 > > lctl set_param timeout=600 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > lctl set_param osc.*.max_pages_per_rpc=1024 > > lctl set_param llite.*.max_read_ahead_mb=1024 > > lctl set_param llite.*.max_read_ahead_per_file_mb=1024 > > > > Mountpoint stripe count:72 stripesize:1M > > > > I have a 2Pb lustre filesystem, In the benchmark tests i get the optimum > values for read and write, but when i start a concurrent I/O operation, > second job throughput stays around 100-200Mb/s. I have tried lovering the > stripe count to 36 but since the concurrent operations will not occur in a > way that keeps OST volume inbalance, i think that its not a good way to > move on, secondly i saw some discussion about turning off flock which ended > up unpromising. > > > > As i check the stripe behaviour, > > first operation starts to use first 36 OST > > when a second job starts during a first job, it uses second 36 OST > > > > But when second job starts after 1st job it uses first 36 OST's which > causes OST unbalance. > > > > Is there a round robin setup that each 36 OST pair used in a round robin > way? > > > > And any kind of suggestions are appreciated. > > Can you please describe what command you are using for testing. Lustre is > already using round-robin OST allocation by default, so the second job > should use the next set of 36 OSTs, unless the file layout has been > specified e.g. to start on OST or the space usage of the OSTs is very > imbalanced (more than 17% of the remaining free space). > > Cheers, Andreas > --- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > > > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Second read or write performance
Hi all, OS=Redhat 7.4 Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 İnterconnect: Mellanox OFED, ConnectX-5 72 OST over 6 OSS with HA 1mdt and 1 mgt on 2 MDS with HA Lustre servers fine tuning parameters: lctl set_param timeout=600 lctl set_param ldlm_timeout=200 lctl set_param at_min=250 lctl set_param at_max=600 lctl set_param obdfilter.*.read_cache_enable=1 lctl set_param obdfilter.*.writethrough_cache_enable=1 lctl set_param obdfilter.lfs3test-OST*.brw_size=16 Lustre clients fine tuning parameters: lctl set_param osc.*.checksums=0 lctl set_param timeout=600 lctl set_param at_min=250 lctl set_param at_max=600 lctl set_param ldlm.namespaces.*.lru_size=2000 lctl set_param osc.*OST*.max_rpcs_in_flight=256 lctl set_param osc.*OST*.max_dirty_mb=1024 lctl set_param osc.*.max_pages_per_rpc=1024 lctl set_param llite.*.max_read_ahead_mb=1024 lctl set_param llite.*.max_read_ahead_per_file_mb=1024 Mountpoint stripe count:72 stripesize:1M I have a 2Pb lustre filesystem, In the benchmark tests i get the optimum values for read and write, but when i start a concurrent I/O operation, second job throughput stays around 100-200Mb/s. I have tried lovering the stripe count to 36 but since the concurrent operations will not occur in a way that keeps OST volume inbalance, i think that its not a good way to move on, secondly i saw some discussion about turning off flock which ended up unpromising. As i check the stripe behaviour, first operation starts to use first 36 OST when a second job starts during a first job, it uses second 36 OST But when second job starts after 1st job it uses first 36 OST's which causes OST unbalance. Is there a round robin setup that each 36 OST pair used in a round robin way? And any kind of suggestions are appreciated. Best regards. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Lustre Filesystem mounted but having "Input/output error" on df command
Hi There, OS=Centos 7.4 Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 İnterconnect: Mellanox OFED, ConnectX-5 In one of my lustre client i have Input/output error in df command, i am unable to see the lustre mount point in df but mtab file shows that lustre is mounted df -h output: df: ‘/home’: Input/output error df: ‘/vol1’: Input/output error df: ‘/cm/shared’: Input/output error FilesystemSize Used Avail Use% Mounted on cat /etc/mtab |grep lustre 10.51.22.11@o2ib:10.51.22.10@o2ib:/lustre/home /home lustre rw,flock,lazystatfs 0 0 10.51.22.11@o2ib:10.51.22.10@o2ib:/lustre /vol1 lustre rw,flock,lazystatfs 0 0 10.51.22.11@o2ib:10.51.22.10@o2ib:/lustre/cmshared /cm/shared lustre rw,flock,lazystatfs 0 0 df -h output: df: ‘/home’: Input/output error df: ‘/vol1’: Input/output error df: ‘/cm/shared’: Input/output error FilesystemSize Used Avail Use% Mounted on When i cd to the mounted point i can reach the lustre filesystem, i can create and delete files and folders. But when i cd to a large fileand run ls -lah command, response from the lustre client freezes. dmesg output: [84276.460557] Lustre: 5617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1536408434/real 1536408489] req@882f31697800 x1610952588839712/t0(0) o8->lustre-OST0016-osc-885f5fa1f000@10.52.23.5@o2ib:28/4 lens 520/544 e 0 to 1 dl 1536408714 ref 1 fl Rpc:eXN/0/ rc 0/-1 [84276.460565] Lustre: 5617:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 910 previous similar messages [84386.986467] LustreError: 122750:0:(llite_lib.c:1772:ll_statfs_internal()) obd_statfs fails: rc = -5 [84386.986471] LustreError: 122750:0:(llite_lib.c:1772:ll_statfs_internal()) Skipped 29 previous similar messages [84704.429967] LNet: 5429:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.52.23.5@o2ib: 4379575 seconds [84704.429970] LNet: 5429:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 863 previous similar messages [84881.004949] Lustre: 5617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1536409034/real 1536409095] req@882f2a6e5700 x1610952588854608/t0(0) o8->lustre-OST002e-osc-885f5fa1f000@10.52.23.5@o2ib:28/4 lens 520/544 e 0 to 1 dl 1536409314 ref 1 fl Rpc:eXN/0/ rc 0/-1 [84881.004957] Lustre: 5617:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 863 previous similar messages [85065.953686] LustreError: 123635:0:(llite_lib.c:1772:ll_statfs_internal()) obd_statfs fails: rc = -5 [85065.953689] LustreError: 123635:0:(llite_lib.c:1772:ll_statfs_internal()) Skipped 26 previous similar messages fstab mount options: lustre flock,_netdev,x-systemd.requires=lnet.service 0 0 ib_* benchmark tests are as usual. Where should i check? Best Regards. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] lustre iml on mds server
Hi Hopko, Thanks for the information, i also set iml up on a virtual server. Have A Good Day. On Fri, Aug 3, 2018 at 4:13 PM H. Meijering wrote: > Hi, > > I don't advise to run IML on a separate machine, IML can consume a lot of > the resources from your system and on the mds'es and oss'es you need that > resources for Lustre itself. > We are running IML in our virtual server farm and have the compliments, > from hour administrators of the farm, that is one of the few systems which > is really using it resources. > > Op ma 16 jul. 2018 om 21:38 schreef fırat yılmaz : > >> Hi All, >> >> Can i install iml server to one of my mds server? >> >> >> ___ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> > > > -- > > > *Hopko Meijering* > > University of Groningen > Center for Information Technology (CIT) > > www.rug.nl/cit > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] lustre iml on mds server
Hi All, Can i install iml server to one of my mds server? ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
