Re: [lustre-discuss] Second read or write performance
Hi Patrick, Thank you for clarifying flock capabilities. So many think can cause the difference between 2 test results i saw in dashboard, now i learn that flock has no effect on it. Best Regars. 22 Eyl 2018 Cmt 04:14 tarihinde Patrick Farrell şunu yazdı: > Firat, > > I strongly suspect that careful remeasurement of flock on/off will show > that removing the flock option had no effect at all. It simply doesn’t DO > anything like that - it controls a single flag that says, if you use flock > operations, they work one way, or if it is not set, they work another way. > It does nothing else, and has no impact on any part of file system > operation except when flocks are used, and dd does not use flocks. It is > simply impossible for the setting of the flock option to affect dd or > performance level or variation, unless something using flocks is running at > the same time. (And even then, it would be affecting it indirectly) > > I’m pushing back strongly because I’ve repeatedly seen people on the > mailing speculate about turning flock off as a way to increase performance, > and it simply isn’t. > > - Patrick > > > -- > *From:* fırat yılmaz > *Sent:* Friday, September 21, 2018 7:50:51 PM > *To:* Patrick Farrell > *Cc:* adil...@whamcloud.com; lustre-discuss@lists.lustre.org > *Subject:* Re: [lustre-discuss] Second read or write performance > > The problem solved by adding lustre fine tuning parameter to oss servers > > lctl set_param obdfilter.lı-lustrefolder-OST*.brw_size=16 > > The flock is required by the application running in the filesystem so > flock option is enabled > > removing flock decrased the divergence of the flactuations and about %5 > performance gain from iml dashboard > > Best Regards. > > On Sat, Sep 22, 2018 at 12:56 AM Patrick Farrell wrote: > > Just 300 GiB, actually. But that's still rather large and could skew > things depending on OST size. > > - Patrick > > On 9/21/18, 4:43 PM, "lustre-discuss on behalf of Andreas Dilger" < > lustre-discuss-boun...@lists.lustre.org on behalf of adil...@whamcloud.com> > wrote: > > On Sep 21, 2018, at 00:43, fırat yılmaz > wrote: > > > > Hi Andreas, > > Tests are made with dd, The test folder is created by the related > application company, i will check that when i have connection. OST's has > %85-86 free space and filesystem mounted with flock option, i will ask for > it to remove and test again. > > The "flock" option shouldn't make any difference, unless the > application is actually doing userspace file locking in the code. > Definitely "dd" will not be using it. > > What does "lfs getstripe" on the first and second file as well as the > parent directory show, and "lfs df" for the filesystem? > > > Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null > bs=1M count=30 > > > > Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` > bs=1M count=30 > > This is creating a single file of 300TB in size, so that is definitely > going to skew the space allocation. > > Cheers, Andreas > > > > > On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger < > adil...@whamcloud.com> wrote: > > On Sep 20, 2018, at 03:07, fırat yılmaz > wrote: > > > > > > Hi all, > > > > > > OS=Redhat 7.4 > > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > > İnterconnect: Mellanox OFED, ConnectX-5 > > > 72 OST over 6 OSS with HA > > > 1mdt and 1 mgt on 2 MDS with HA > > > > > > Lustre servers fine tuning parameters: > > > lctl set_param timeout=600 > > > lctl set_param ldlm_timeout=200 > > > lctl set_param at_min=250 > > > lctl set_param at_max=600 > > > lctl set_param obdfilter.*.read_cache_enable=1 > > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > > > Lustre clients fine tuning parameters: > > > lctl set_param osc.*.checksums=0 > > > lctl set_param timeout=600 > > > lctl set_param at_min=250 > > > lctl set_param at_max=600 > > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > > lctl set_param osc.*.max_pages_per_rpc=1024 > > > lctl set_param llite.*.max_read_
Re: [lustre-discuss] Second read or write performance
Firat, I strongly suspect that careful remeasurement of flock on/off will show that removing the flock option had no effect at all. It simply doesn’t DO anything like that - it controls a single flag that says, if you use flock operations, they work one way, or if it is not set, they work another way. It does nothing else, and has no impact on any part of file system operation except when flocks are used, and dd does not use flocks. It is simply impossible for the setting of the flock option to affect dd or performance level or variation, unless something using flocks is running at the same time. (And even then, it would be affecting it indirectly) I’m pushing back strongly because I’ve repeatedly seen people on the mailing speculate about turning flock off as a way to increase performance, and it simply isn’t. - Patrick From: fırat yılmaz Sent: Friday, September 21, 2018 7:50:51 PM To: Patrick Farrell Cc: adil...@whamcloud.com; lustre-discuss@lists.lustre.org Subject: Re: [lustre-discuss] Second read or write performance The problem solved by adding lustre fine tuning parameter to oss servers lctl set_param obdfilter.lı-lustrefolder-OST*.brw_size=16 The flock is required by the application running in the filesystem so flock option is enabled removing flock decrased the divergence of the flactuations and about %5 performance gain from iml dashboard Best Regards. On Sat, Sep 22, 2018 at 12:56 AM Patrick Farrell mailto:p...@cray.com>> wrote: Just 300 GiB, actually. But that's still rather large and could skew things depending on OST size. - Patrick On 9/21/18, 4:43 PM, "lustre-discuss on behalf of Andreas Dilger" mailto:lustre-discuss-boun...@lists.lustre.org> on behalf of adil...@whamcloud.com<mailto:adil...@whamcloud.com>> wrote: On Sep 21, 2018, at 00:43, fırat yılmaz mailto:firatyilm...@gmail.com>> wrote: > > Hi Andreas, > Tests are made with dd, The test folder is created by the related application company, i will check that when i have connection. OST's has %85-86 free space and filesystem mounted with flock option, i will ask for it to remove and test again. The "flock" option shouldn't make any difference, unless the application is actually doing userspace file locking in the code. Definitely "dd" will not be using it. What does "lfs getstripe" on the first and second file as well as the parent directory show, and "lfs df" for the filesystem? > Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null bs=1M count=30 > > Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` bs=1M count=30 This is creating a single file of 300TB in size, so that is definitely going to skew the space allocation. Cheers, Andreas > > On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger mailto:adil...@whamcloud.com>> wrote: > On Sep 20, 2018, at 03:07, fırat yılmaz mailto:firatyilm...@gmail.com>> wrote: > > > > Hi all, > > > > OS=Redhat 7.4 > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > İnterconnect: Mellanox OFED, ConnectX-5 > > 72 OST over 6 OSS with HA > > 1mdt and 1 mgt on 2 MDS with HA > > > > Lustre servers fine tuning parameters: > > lctl set_param timeout=600 > > lctl set_param ldlm_timeout=200 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param obdfilter.*.read_cache_enable=1 > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > Lustre clients fine tuning parameters: > > lctl set_param osc.*.checksums=0 > > lctl set_param timeout=600 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > lctl set_param osc.*.max_pages_per_rpc=1024 > > lctl set_param llite.*.max_read_ahead_mb=1024 > > lctl set_param llite.*.max_read_ahead_per_file_mb=1024 > > > > Mountpoint stripe count:72 stripesize:1M > > > > I have a 2Pb lustre filesystem, In the benchmark tests i get the optimum values for read and write, but when i start a concurrent I/O operation, second job throughput stays around 100-200Mb/s. I have tried lovering the stripe count to 36 but since the concurrent operations will not occur in a way that keeps OST volume inbalance, i think that its not a good way to move on, secondly i saw some discussion about turning off flock which
Re: [lustre-discuss] Second read or write performance
The problem solved by adding lustre fine tuning parameter to oss servers lctl set_param obdfilter.lı-lustrefolder-OST*.brw_size=16 The flock is required by the application running in the filesystem so flock option is enabled removing flock decrased the divergence of the flactuations and about %5 performance gain from iml dashboard Best Regards. On Sat, Sep 22, 2018 at 12:56 AM Patrick Farrell wrote: > Just 300 GiB, actually. But that's still rather large and could skew > things depending on OST size. > > - Patrick > > On 9/21/18, 4:43 PM, "lustre-discuss on behalf of Andreas Dilger" < > lustre-discuss-boun...@lists.lustre.org on behalf of adil...@whamcloud.com> > wrote: > > On Sep 21, 2018, at 00:43, fırat yılmaz > wrote: > > > > Hi Andreas, > > Tests are made with dd, The test folder is created by the related > application company, i will check that when i have connection. OST's has > %85-86 free space and filesystem mounted with flock option, i will ask for > it to remove and test again. > > The "flock" option shouldn't make any difference, unless the > application is actually doing userspace file locking in the code. > Definitely "dd" will not be using it. > > What does "lfs getstripe" on the first and second file as well as the > parent directory show, and "lfs df" for the filesystem? > > > Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null > bs=1M count=30 > > > > Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` > bs=1M count=30 > > This is creating a single file of 300TB in size, so that is definitely > going to skew the space allocation. > > Cheers, Andreas > > > > > On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger < > adil...@whamcloud.com> wrote: > > On Sep 20, 2018, at 03:07, fırat yılmaz > wrote: > > > > > > Hi all, > > > > > > OS=Redhat 7.4 > > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > > İnterconnect: Mellanox OFED, ConnectX-5 > > > 72 OST over 6 OSS with HA > > > 1mdt and 1 mgt on 2 MDS with HA > > > > > > Lustre servers fine tuning parameters: > > > lctl set_param timeout=600 > > > lctl set_param ldlm_timeout=200 > > > lctl set_param at_min=250 > > > lctl set_param at_max=600 > > > lctl set_param obdfilter.*.read_cache_enable=1 > > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > > > Lustre clients fine tuning parameters: > > > lctl set_param osc.*.checksums=0 > > > lctl set_param timeout=600 > > > lctl set_param at_min=250 > > > lctl set_param at_max=600 > > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > > lctl set_param osc.*.max_pages_per_rpc=1024 > > > lctl set_param llite.*.max_read_ahead_mb=1024 > > > lctl set_param llite.*.max_read_ahead_per_file_mb=1024 > > > > > > Mountpoint stripe count:72 stripesize:1M > > > > > > I have a 2Pb lustre filesystem, In the benchmark tests i get the > optimum values for read and write, but when i start a concurrent I/O > operation, second job throughput stays around 100-200Mb/s. I have tried > lovering the stripe count to 36 but since the concurrent operations will > not occur in a way that keeps OST volume inbalance, i think that its not a > good way to move on, secondly i saw some discussion about turning off flock > which ended up unpromising. > > > > > > As i check the stripe behaviour, > > > first operation starts to use first 36 OST > > > when a second job starts during a first job, it uses second 36 OST > > > > > > But when second job starts after 1st job it uses first 36 OST's > which causes OST unbalance. > > > > > > Is there a round robin setup that each 36 OST pair used in a round > robin way? > > > > > > And any kind of suggestions are appreciated. > > > > Can you please describe what command you are using for testing. > Lustre is already using round-robin OST allocation by default, so the > second job should use the next set of 36 OSTs, unless the file layout has > been specified e.g. to start on OST or the space usage of the OSTs is > very imbalanced (more than 17% of the remaining free space). > > > > Cheers, Andreas > > --- > > Andreas Dilger > > Principal Lustre Architect > > Whamcloud > > > > > > > > > > > > > > > > Cheers, Andreas > --- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > > > > > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Second read or write performance
Just 300 GiB, actually. But that's still rather large and could skew things depending on OST size. - Patrick On 9/21/18, 4:43 PM, "lustre-discuss on behalf of Andreas Dilger" wrote: On Sep 21, 2018, at 00:43, fırat yılmaz wrote: > > Hi Andreas, > Tests are made with dd, The test folder is created by the related application company, i will check that when i have connection. OST's has %85-86 free space and filesystem mounted with flock option, i will ask for it to remove and test again. The "flock" option shouldn't make any difference, unless the application is actually doing userspace file locking in the code. Definitely "dd" will not be using it. What does "lfs getstripe" on the first and second file as well as the parent directory show, and "lfs df" for the filesystem? > Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null bs=1M count=30 > > Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` bs=1M count=30 This is creating a single file of 300TB in size, so that is definitely going to skew the space allocation. Cheers, Andreas > > On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger wrote: > On Sep 20, 2018, at 03:07, fırat yılmaz wrote: > > > > Hi all, > > > > OS=Redhat 7.4 > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > İnterconnect: Mellanox OFED, ConnectX-5 > > 72 OST over 6 OSS with HA > > 1mdt and 1 mgt on 2 MDS with HA > > > > Lustre servers fine tuning parameters: > > lctl set_param timeout=600 > > lctl set_param ldlm_timeout=200 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param obdfilter.*.read_cache_enable=1 > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > Lustre clients fine tuning parameters: > > lctl set_param osc.*.checksums=0 > > lctl set_param timeout=600 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > lctl set_param osc.*.max_pages_per_rpc=1024 > > lctl set_param llite.*.max_read_ahead_mb=1024 > > lctl set_param llite.*.max_read_ahead_per_file_mb=1024 > > > > Mountpoint stripe count:72 stripesize:1M > > > > I have a 2Pb lustre filesystem, In the benchmark tests i get the optimum values for read and write, but when i start a concurrent I/O operation, second job throughput stays around 100-200Mb/s. I have tried lovering the stripe count to 36 but since the concurrent operations will not occur in a way that keeps OST volume inbalance, i think that its not a good way to move on, secondly i saw some discussion about turning off flock which ended up unpromising. > > > > As i check the stripe behaviour, > > first operation starts to use first 36 OST > > when a second job starts during a first job, it uses second 36 OST > > > > But when second job starts after 1st job it uses first 36 OST's which causes OST unbalance. > > > > Is there a round robin setup that each 36 OST pair used in a round robin way? > > > > And any kind of suggestions are appreciated. > > Can you please describe what command you are using for testing. Lustre is already using round-robin OST allocation by default, so the second job should use the next set of 36 OSTs, unless the file layout has been specified e.g. to start on OST or the space usage of the OSTs is very imbalanced (more than 17% of the remaining free space). > > Cheers, Andreas > --- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > > > > > > Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Second read or write performance
On Sep 21, 2018, at 00:43, fırat yılmaz wrote: > > Hi Andreas, > Tests are made with dd, The test folder is created by the related application > company, i will check that when i have connection. OST's has %85-86 free > space and filesystem mounted with flock option, i will ask for it to remove > and test again. The "flock" option shouldn't make any difference, unless the application is actually doing userspace file locking in the code. Definitely "dd" will not be using it. What does "lfs getstripe" on the first and second file as well as the parent directory show, and "lfs df" for the filesystem? > Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null bs=1M > count=30 > > Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` bs=1M > count=30 This is creating a single file of 300TB in size, so that is definitely going to skew the space allocation. Cheers, Andreas > > On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger wrote: > On Sep 20, 2018, at 03:07, fırat yılmaz wrote: > > > > Hi all, > > > > OS=Redhat 7.4 > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > İnterconnect: Mellanox OFED, ConnectX-5 > > 72 OST over 6 OSS with HA > > 1mdt and 1 mgt on 2 MDS with HA > > > > Lustre servers fine tuning parameters: > > lctl set_param timeout=600 > > lctl set_param ldlm_timeout=200 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param obdfilter.*.read_cache_enable=1 > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > Lustre clients fine tuning parameters: > > lctl set_param osc.*.checksums=0 > > lctl set_param timeout=600 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > lctl set_param osc.*.max_pages_per_rpc=1024 > > lctl set_param llite.*.max_read_ahead_mb=1024 > > lctl set_param llite.*.max_read_ahead_per_file_mb=1024 > > > > Mountpoint stripe count:72 stripesize:1M > > > > I have a 2Pb lustre filesystem, In the benchmark tests i get the optimum > > values for read and write, but when i start a concurrent I/O operation, > > second job throughput stays around 100-200Mb/s. I have tried lovering the > > stripe count to 36 but since the concurrent operations will not occur in a > > way that keeps OST volume inbalance, i think that its not a good way to > > move on, secondly i saw some discussion about turning off flock which ended > > up unpromising. > > > > As i check the stripe behaviour, > > first operation starts to use first 36 OST > > when a second job starts during a first job, it uses second 36 OST > > > > But when second job starts after 1st job it uses first 36 OST's which > > causes OST unbalance. > > > > Is there a round robin setup that each 36 OST pair used in a round robin > > way? > > > > And any kind of suggestions are appreciated. > > Can you please describe what command you are using for testing. Lustre is > already using round-robin OST allocation by default, so the second job should > use the next set of 36 OSTs, unless the file layout has been specified e.g. > to start on OST or the space usage of the OSTs is very imbalanced (more > than 17% of the remaining free space). > > Cheers, Andreas > --- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > > > > > > Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud signature.asc Description: Message signed with OpenPGP ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Second read or write performance
Hi Andreas, Tests are made with dd, The test folder is created by the related application company, i will check that when i have connection. OST's has %85-86 free space and filesystem mounted with flock option, i will ask for it to remove and test again. Thanks Andereas, Best Regards. Read test dd if=/vol1/test_read/dd.test.`hostname` of=/dev/null bs=1M count=30 Write test dd if=/dev/zero of=/vol1/test_read/dd.test.2.`hostname` bs=1M count=30 On Thu, Sep 20, 2018 at 10:57 PM Andreas Dilger wrote: > On Sep 20, 2018, at 03:07, fırat yılmaz wrote: > > > > Hi all, > > > > OS=Redhat 7.4 > > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > > İnterconnect: Mellanox OFED, ConnectX-5 > > 72 OST over 6 OSS with HA > > 1mdt and 1 mgt on 2 MDS with HA > > > > Lustre servers fine tuning parameters: > > lctl set_param timeout=600 > > lctl set_param ldlm_timeout=200 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param obdfilter.*.read_cache_enable=1 > > lctl set_param obdfilter.*.writethrough_cache_enable=1 > > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > > > Lustre clients fine tuning parameters: > > lctl set_param osc.*.checksums=0 > > lctl set_param timeout=600 > > lctl set_param at_min=250 > > lctl set_param at_max=600 > > lctl set_param ldlm.namespaces.*.lru_size=2000 > > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > > lctl set_param osc.*OST*.max_dirty_mb=1024 > > lctl set_param osc.*.max_pages_per_rpc=1024 > > lctl set_param llite.*.max_read_ahead_mb=1024 > > lctl set_param llite.*.max_read_ahead_per_file_mb=1024 > > > > Mountpoint stripe count:72 stripesize:1M > > > > I have a 2Pb lustre filesystem, In the benchmark tests i get the optimum > values for read and write, but when i start a concurrent I/O operation, > second job throughput stays around 100-200Mb/s. I have tried lovering the > stripe count to 36 but since the concurrent operations will not occur in a > way that keeps OST volume inbalance, i think that its not a good way to > move on, secondly i saw some discussion about turning off flock which ended > up unpromising. > > > > As i check the stripe behaviour, > > first operation starts to use first 36 OST > > when a second job starts during a first job, it uses second 36 OST > > > > But when second job starts after 1st job it uses first 36 OST's which > causes OST unbalance. > > > > Is there a round robin setup that each 36 OST pair used in a round robin > way? > > > > And any kind of suggestions are appreciated. > > Can you please describe what command you are using for testing. Lustre is > already using round-robin OST allocation by default, so the second job > should use the next set of 36 OSTs, unless the file layout has been > specified e.g. to start on OST or the space usage of the OSTs is very > imbalanced (more than 17% of the remaining free space). > > Cheers, Andreas > --- > Andreas Dilger > Principal Lustre Architect > Whamcloud > > > > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Second read or write performance
On Sep 20, 2018, at 03:07, fırat yılmaz wrote: > > Hi all, > > OS=Redhat 7.4 > Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 > İnterconnect: Mellanox OFED, ConnectX-5 > 72 OST over 6 OSS with HA > 1mdt and 1 mgt on 2 MDS with HA > > Lustre servers fine tuning parameters: > lctl set_param timeout=600 > lctl set_param ldlm_timeout=200 > lctl set_param at_min=250 > lctl set_param at_max=600 > lctl set_param obdfilter.*.read_cache_enable=1 > lctl set_param obdfilter.*.writethrough_cache_enable=1 > lctl set_param obdfilter.lfs3test-OST*.brw_size=16 > > Lustre clients fine tuning parameters: > lctl set_param osc.*.checksums=0 > lctl set_param timeout=600 > lctl set_param at_min=250 > lctl set_param at_max=600 > lctl set_param ldlm.namespaces.*.lru_size=2000 > lctl set_param osc.*OST*.max_rpcs_in_flight=256 > lctl set_param osc.*OST*.max_dirty_mb=1024 > lctl set_param osc.*.max_pages_per_rpc=1024 > lctl set_param llite.*.max_read_ahead_mb=1024 > lctl set_param llite.*.max_read_ahead_per_file_mb=1024 > > Mountpoint stripe count:72 stripesize:1M > > I have a 2Pb lustre filesystem, In the benchmark tests i get the optimum > values for read and write, but when i start a concurrent I/O operation, > second job throughput stays around 100-200Mb/s. I have tried lovering the > stripe count to 36 but since the concurrent operations will not occur in a > way that keeps OST volume inbalance, i think that its not a good way to move > on, secondly i saw some discussion about turning off flock which ended up > unpromising. > > As i check the stripe behaviour, > first operation starts to use first 36 OST > when a second job starts during a first job, it uses second 36 OST > > But when second job starts after 1st job it uses first 36 OST's which causes > OST unbalance. > > Is there a round robin setup that each 36 OST pair used in a round robin way? > > And any kind of suggestions are appreciated. Can you please describe what command you are using for testing. Lustre is already using round-robin OST allocation by default, so the second job should use the next set of 36 OSTs, unless the file layout has been specified e.g. to start on OST or the space usage of the OSTs is very imbalanced (more than 17% of the remaining free space). Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud signature.asc Description: Message signed with OpenPGP ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Second read or write performance
Hi all, OS=Redhat 7.4 Lustre Version: Intel® Manager for Lustre* software 4.0.3.0 İnterconnect: Mellanox OFED, ConnectX-5 72 OST over 6 OSS with HA 1mdt and 1 mgt on 2 MDS with HA Lustre servers fine tuning parameters: lctl set_param timeout=600 lctl set_param ldlm_timeout=200 lctl set_param at_min=250 lctl set_param at_max=600 lctl set_param obdfilter.*.read_cache_enable=1 lctl set_param obdfilter.*.writethrough_cache_enable=1 lctl set_param obdfilter.lfs3test-OST*.brw_size=16 Lustre clients fine tuning parameters: lctl set_param osc.*.checksums=0 lctl set_param timeout=600 lctl set_param at_min=250 lctl set_param at_max=600 lctl set_param ldlm.namespaces.*.lru_size=2000 lctl set_param osc.*OST*.max_rpcs_in_flight=256 lctl set_param osc.*OST*.max_dirty_mb=1024 lctl set_param osc.*.max_pages_per_rpc=1024 lctl set_param llite.*.max_read_ahead_mb=1024 lctl set_param llite.*.max_read_ahead_per_file_mb=1024 Mountpoint stripe count:72 stripesize:1M I have a 2Pb lustre filesystem, In the benchmark tests i get the optimum values for read and write, but when i start a concurrent I/O operation, second job throughput stays around 100-200Mb/s. I have tried lovering the stripe count to 36 but since the concurrent operations will not occur in a way that keeps OST volume inbalance, i think that its not a good way to move on, secondly i saw some discussion about turning off flock which ended up unpromising. As i check the stripe behaviour, first operation starts to use first 36 OST when a second job starts during a first job, it uses second 36 OST But when second job starts after 1st job it uses first 36 OST's which causes OST unbalance. Is there a round robin setup that each 36 OST pair used in a round robin way? And any kind of suggestions are appreciated. Best regards. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
