Re: [lustre-discuss] [External] Re: obdfilter/mdt stats meaning ?

2019-07-16 Thread Degremont, Aurelien 
>   read  | write
>disk I/Os in flightios   % cum % |  ios % cum %
>1:   211177215  61  61   | 29305564  97  97
>2:41332944  11  72   | 498260   1  99
>[..]
>Does these lines means :
>Since last snapshot there was 211177215x1 and read 41332944x2 I/O in flight ?

It means (since the last time the statistics were cleared)

  *   11% of the time, 2 READ I/O requests were "in-flights" to disk, meaning 2 
I/O were sent to disks and not yet commit/acknowledged
  *   61 % of the time, only 1 READ I/O request.

Same principle for write.

What this means here is that your workload is not feeding the disks with lots 
of write (97% with 1 I/O in flight), but a bit more reads.
Disks and especially disk arrays are reordering I/O and distributing them 
across the various drives they are composed of to optimized bandwith. To really 
take benefits of all the possible bandwith/throughput your hardward can offer, 
you often need to be able to have lots of big I/O and possible multiple I/O in 
flights.
Few I/O in flight could means:

  *   your workload is not really big
  *   your hardward is fast compared to the throughput coming to this server 
(ratio disk BW vs network BW by example)
This could also help you identify bad performance numbers and find from where 
the bottleneck comes from.


De : lustre-discuss  au nom de Louis 
Bailleul 
Date : mardi 16 juillet 2019 à 17:49
À : lustre-discuss 
Objet : Re: [lustre-discuss] [External] Re: obdfilter/mdt stats meaning ?

Hi Aurélien,

Thanks for the prompt reply.
For the ost stats, any idea what the preprw and commitrw mean ?
And why there are two entries with different values for statfs ?

For brw_stats even with the doc I still struggle to read this.
For example how do you make sense of disk I/O in flight ?
   read  | write
disk I/Os in flightios   % cum % |  ios % cum %
1:   211177215  61  61   | 29305564  97  97
2:41332944  11  72   | 498260   1  99
[..]
Does these lines means :
Since last snapshot there was 211177215x1 and read 41332944x2 I/O in flight ?

Best regards,
Louis
On 16/07/2019 15:50, Degremont, Aurelien wrote:
Hi Louis,

About brw_stats, there are a bit of explanation in the Lustre Doc (not that 
detailed, but still)
http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.50438271_55057

> Last thing, is there any way to get the name of the filesystem an OST is part 
> of by using lctl ?

I don't know what you want exactly, but the OST names are self explanatory, 
there always are like: fsname-OST
Where fsname is the lustre filesystem they are part of.

For obdfilter stats, these are mostly action to OST objects or client 
connection management RPCs.

setattr: changing an OST object attributes (owner, group, ...)
punch: mostly used for truncate (theorically can do holes in files, like 
truncate with a start and length)
sync: straighforward, sync OST to disk
destroy: delete an OST object (mostly when a file is deleted)
create: create an OST object
statfs: like 'df' for this specific OST (used by 'lfs df' by example)
(re)connect: when a client connect/reconnect to this OST
ping: when a client ping this OST.


Aurélien

De : lustre-discuss 

 au nom de Louis Bailleul 

Date : mardi 16 juillet 2019 à 16:38
À : lustre-discuss 

Objet : [lustre-discuss] obdfilter/mdt stats meaning ?

Hi all,

I am trying to make sense of some of the OST/MDT stats for 2.12.
Can anybody point me to the doc that explain what the metrics are ?
The wiki only mention read/write/get_info : 
http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide
But the list I get is quite different :
obdfilter.OST001.stats=
snapshot_time 1563285450.647120173 secs.nsecs
read_bytes340177708 samples [bytes] 4096 4194304 
396712660910080
write_bytes   30008856 samples [bytes] 24 4194304 78618271501667
setattr   1755 samples [reqs]
punch 73463 samples [reqs]
sync  50606 samples [reqs]
destroy   31990 samples [reqs]
create956 samples [reqs]
statfs75378743 samples [reqs]
connect   5798 samples [reqs]
reconnect 3242 samples [reqs]
disconnect5820 samples [reqs]
statfs3737980 samples [reqs]
preprw370186566 samples [reqs]

Re: [lustre-discuss] [External] Re: obdfilter/mdt stats meaning ?

2019-07-16 Thread Louis Bailleul 
Hi Aurélien,

Thanks for the prompt reply.
For the ost stats, any idea what the preprw and commitrw mean ?
And why there are two entries with different values for statfs ?

For brw_stats even with the doc I still struggle to read this.
For example how do you make sense of disk I/O in flight ?
   read  | write
disk I/Os in flightios   % cum % |  ios % cum %
1:   211177215  61  61   | 29305564  97  97
2:41332944  11  72   | 498260   1  99
[..]
Does these lines means :
Since last snapshot there was 211177215x1 and read 41332944x2 I/O in flight ?

Best regards,
Louis

On 16/07/2019 15:50, Degremont, Aurelien wrote:
Hi Louis,

About brw_stats, there are a bit of explanation in the Lustre Doc (not that 
detailed, but still)
http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.50438271_55057

> Last thing, is there any way to get the name of the filesystem an OST is part 
> of by using lctl ?

I don't know what you want exactly, but the OST names are self explanatory, 
there always are like: fsname-OST
Where fsname is the lustre filesystem they are part of.

For obdfilter stats, these are mostly action to OST objects or client 
connection management RPCs.

setattr: changing an OST object attributes (owner, group, ...)
punch: mostly used for truncate (theorically can do holes in files, like 
truncate with a start and length)
sync: straighforward, sync OST to disk
destroy: delete an OST object (mostly when a file is deleted)
create: create an OST object
statfs: like 'df' for this specific OST (used by 'lfs df' by example)
(re)connect: when a client connect/reconnect to this OST
ping: when a client ping this OST.


Aurélien

De : lustre-discuss 

 au nom de Louis Bailleul 

Date : mardi 16 juillet 2019 à 16:38
À : lustre-discuss 

Objet : [lustre-discuss] obdfilter/mdt stats meaning ?

Hi all,

I am trying to make sense of some of the OST/MDT stats for 2.12.
Can anybody point me to the doc that explain what the metrics are ?
The wiki only mention read/write/get_info : 
http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide
But the list I get is quite different :
obdfilter.OST001.stats=
snapshot_time 1563285450.647120173 secs.nsecs
read_bytes340177708 samples [bytes] 4096 4194304 
396712660910080
write_bytes   30008856 samples [bytes] 24 4194304 78618271501667
setattr   1755 samples [reqs]
punch 73463 samples [reqs]
sync  50606 samples [reqs]
destroy   31990 samples [reqs]
create956 samples [reqs]
statfs75378743 samples [reqs]
connect   5798 samples [reqs]
reconnect 3242 samples [reqs]
disconnect5820 samples [reqs]
statfs3737980 samples [reqs]
preprw370186566 samples [reqs]
commitrw  370186557 samples [reqs]
ping  882096292 samples [reqs]
For the MDT, most are pretty much self explanatory, but I'll still be happy to 
be pointed to some doc.
mdt.MDT.md_stats=
snapshot_time 1563287416.006001068 secs.nsecs
open  3174644054 samples [reqs]
close 3174494603 samples [reqs]
mknod 107564 samples [reqs]
unlink99625 samples [reqs]
mkdir 199643 samples [reqs]
rmdir 45021 samples [reqs]
rename12728 samples [reqs]
getattr   50227431 samples [reqs]
setattr   103435 samples [reqs]
getxattr  9051470 samples [reqs]
setxattr  14 samples [reqs]
statfs7525513 samples [reqs]
sync  20597 samples [reqs]
samedir_rename207 samples [reqs]
crossdir_rename   12521 samples [reqs]
And anyone knows how to read the OST brw_stats ?
obdfilter.OST0014.brw_stats=
snapshot_time: 1563287631.511085465 (secs.nsecs)

   read  | write
pages per bulk r/w rpcs  % cum % |  rpcs% cum %
1:   231699298  66  66   | 180944   0   0
2:  855611   0  67   | 322359   1   1
4:  541749   0  67   | 5539716  18  20
8: 1281219   0  67   | 67837   0  20
16: 637808   0  67   | 114546   0  20
32:1342813

Re: [lustre-discuss] ZFS tunings

2019-07-16 Thread Degremont, Aurelien 
Nobody on this topic?
I'm pretty sure they are lots of people running Lustre on ZFS with various 
tuning applied. Don't be shy 


De : "Degremont, Aurelien" 
Date : mercredi 10 juillet 2019 à 10:35
À : "lustre-discuss@lists.lustre.org" 
Objet : ZFS tunings

Hi all,

I know good default tunings for ZFS when used with Lustre is a big topic. There 
are several pages on wiki or LUG/LAD slides which are few years old and it is 
difficult to know which ones are still relevant when used with recent Lustre 
and ZFS versions.

Does anybody have insight about which tunings are important. Especially for:

  *   zfs atime (does DMU when used by Lustre also updates atime ?)
  *   redundant_metadata=most
  *   metaslab_debug_unload=1
  *   zfs_prefetch_disable=1
  *   zfs_vdev_scheduler=deadline

Lustre already sets ‘xattr=sa’, ‘dnodesize=auto’ and ‘recordsize=1M’ 
automatically since Lustre 2.11.

Thanks

Aurélien


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] obdfilter/mdt stats meaning ?

2019-07-16 Thread Degremont, Aurelien 
Hi Louis,

About brw_stats, there are a bit of explanation in the Lustre Doc (not that 
detailed, but still)
http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.50438271_55057

> Last thing, is there any way to get the name of the filesystem an OST is part 
> of by using lctl ?

I don't know what you want exactly, but the OST names are self explanatory, 
there always are like: fsname-OST
Where fsname is the lustre filesystem they are part of.

For obdfilter stats, these are mostly action to OST objects or client 
connection management RPCs.

setattr: changing an OST object attributes (owner, group, ...)
punch: mostly used for truncate (theorically can do holes in files, like 
truncate with a start and length)
sync: straighforward, sync OST to disk
destroy: delete an OST object (mostly when a file is deleted)
create: create an OST object
statfs: like 'df' for this specific OST (used by 'lfs df' by example)
(re)connect: when a client connect/reconnect to this OST
ping: when a client ping this OST.


Aurélien

De : lustre-discuss  au nom de Louis 
Bailleul 
Date : mardi 16 juillet 2019 à 16:38
À : lustre-discuss 
Objet : [lustre-discuss] obdfilter/mdt stats meaning ?

Hi all,

I am trying to make sense of some of the OST/MDT stats for 2.12.
Can anybody point me to the doc that explain what the metrics are ?
The wiki only mention read/write/get_info : 
http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide
But the list I get is quite different :
obdfilter.OST001.stats=
snapshot_time 1563285450.647120173 secs.nsecs
read_bytes340177708 samples [bytes] 4096 4194304 
396712660910080
write_bytes   30008856 samples [bytes] 24 4194304 78618271501667
setattr   1755 samples [reqs]
punch 73463 samples [reqs]
sync  50606 samples [reqs]
destroy   31990 samples [reqs]
create956 samples [reqs]
statfs75378743 samples [reqs]
connect   5798 samples [reqs]
reconnect 3242 samples [reqs]
disconnect5820 samples [reqs]
statfs3737980 samples [reqs]
preprw370186566 samples [reqs]
commitrw  370186557 samples [reqs]
ping  882096292 samples [reqs]
For the MDT, most are pretty much self explanatory, but I'll still be happy to 
be pointed to some doc.
mdt.MDT.md_stats=
snapshot_time 1563287416.006001068 secs.nsecs
open  3174644054 samples [reqs]
close 3174494603 samples [reqs]
mknod 107564 samples [reqs]
unlink99625 samples [reqs]
mkdir 199643 samples [reqs]
rmdir 45021 samples [reqs]
rename12728 samples [reqs]
getattr   50227431 samples [reqs]
setattr   103435 samples [reqs]
getxattr  9051470 samples [reqs]
setxattr  14 samples [reqs]
statfs7525513 samples [reqs]
sync  20597 samples [reqs]
samedir_rename207 samples [reqs]
crossdir_rename   12521 samples [reqs]
And anyone knows how to read the OST brw_stats ?
obdfilter.OST0014.brw_stats=
snapshot_time: 1563287631.511085465 (secs.nsecs)

   read  | write
pages per bulk r/w rpcs  % cum % |  rpcs% cum %
1:   231699298  66  66   | 180944   0   0
2:  855611   0  67   | 322359   1   1
4:  541749   0  67   | 5539716  18  20
8: 1281219   0  67   | 67837   0  20
16: 637808   0  67   | 114546   0  20
32:1342813   0  68   | 3099780  10  31
64:1559834   0  68   | 173166   0  31
128:   1583127   0  69   | 211512   0  32
256:  10627583   3  72   | 499978   1  34
512:   3909601   1  73   | 1029686   3  37
1K:   92141161  26 100   | 18788597  62 100

   read  | write
discontiguous pagesrpcs  % cum % |  rpcs% cum %
0:   346179839 100 100   | 180946   0   0
1:   0   0 100   | 322363   1   1
2:   0   0 100   | 5521062  18  20
3:   0   0 100   | 18650   0  20
4:   0   0 100   | 18159   0  20
5:   0   0 100   | 26664   0  20
6:   0   0 100   | 10830   0  20
7:   0   0 100   | 12189   0  20
8:   0   0 100   | 11365   0  20
9:   0   0 100   | 10253   0  20
10:  0   0 100   | 8810   0  20
11:  0   0 100   | 9825   0  20
12:  0   0 100   | 16740   0  20
13:

[lustre-discuss] obdfilter/mdt stats meaning ?

2019-07-16 Thread Louis Bailleul 
Hi all,

I am trying to make sense of some of the OST/MDT stats for 2.12.
Can anybody point me to the doc that explain what the metrics are ?
The wiki only mention read/write/get_info : 
http://wiki.lustre.org/Lustre_Monitoring_and_Statistics_Guide
But the list I get is quite different :
obdfilter.OST001.stats=
snapshot_time 1563285450.647120173 secs.nsecs
read_bytes340177708 samples [bytes] 4096 4194304 
396712660910080
write_bytes   30008856 samples [bytes] 24 4194304 78618271501667
setattr   1755 samples [reqs]
punch 73463 samples [reqs]
sync  50606 samples [reqs]
destroy   31990 samples [reqs]
create956 samples [reqs]
statfs75378743 samples [reqs]
connect   5798 samples [reqs]
reconnect 3242 samples [reqs]
disconnect5820 samples [reqs]
statfs3737980 samples [reqs]
preprw370186566 samples [reqs]
commitrw  370186557 samples [reqs]
ping  882096292 samples [reqs]

For the MDT, most are pretty much self explanatory, but I'll still be happy to 
be pointed to some doc.
mdt.MDT.md_stats=
snapshot_time 1563287416.006001068 secs.nsecs
open  3174644054 samples [reqs]
close 3174494603 samples [reqs]
mknod 107564 samples [reqs]
unlink99625 samples [reqs]
mkdir 199643 samples [reqs]
rmdir 45021 samples [reqs]
rename12728 samples [reqs]
getattr   50227431 samples [reqs]
setattr   103435 samples [reqs]
getxattr  9051470 samples [reqs]
setxattr  14 samples [reqs]
statfs7525513 samples [reqs]
sync  20597 samples [reqs]
samedir_rename207 samples [reqs]
crossdir_rename   12521 samples [reqs]

And anyone knows how to read the OST brw_stats ?

obdfilter.OST0014.brw_stats=
snapshot_time: 1563287631.511085465 (secs.nsecs)

   read  | write
pages per bulk r/w rpcs  % cum % |  rpcs% cum %
1:   231699298  66  66   | 180944   0   0
2:  855611   0  67   | 322359   1   1
4:  541749   0  67   | 5539716  18  20
8: 1281219   0  67   | 67837   0  20
16: 637808   0  67   | 114546   0  20
32:1342813   0  68   | 3099780  10  31
64:1559834   0  68   | 173166   0  31
128:   1583127   0  69   | 211512   0  32
256:  10627583   3  72   | 499978   1  34
512:   3909601   1  73   | 1029686   3  37
1K:   92141161  26 100   | 18788597  62 100

   read  | write
discontiguous pagesrpcs  % cum % |  rpcs% cum %
0:   346179839 100 100   | 180946   0   0
1:   0   0 100   | 322363   1   1
2:   0   0 100   | 5521062  18  20
3:   0   0 100   | 18650   0  20
4:   0   0 100   | 18159   0  20
5:   0   0 100   | 26664   0  20
6:   0   0 100   | 10830   0  20
7:   0   0 100   | 12189   0  20
8:   0   0 100   | 11365   0  20
9:   0   0 100   | 10253   0  20
10:  0   0 100   | 8810   0  20
11:  0   0 100   | 9825   0  20
12:  0   0 100   | 16740   0  20
13:  0   0 100   | 14421   0  20
14:  0   0 100   | 10513   0  20
15:  0   0 100   | 32655   0  20
16:  0   0 100   | 1418677   4  25
17:  0   0 100   | 1477077   4  30
18:  0   0 100   | 6227   0  30
19:  0   0 100   | 7071   0  30
20:  0   0 100   | 7297   0  30
21:  0   0 100   | 8478   0  30
22:  0   0 100   | 34591   0  30
23:  0   0 100   | 35591   0  30
24:  0   0 100   | 8378   0  30
25:  0   0 100   | 8724   0  30
26:  0   0 100   | 52300   0  30
27:  0   0 100   | 14038   0  30
28:  0   0 100   | 4734   0  30
29:  0   0 100   | 4878   0  31
30:  0   0 100   | 6232   0  31
31:  0   0 100   | 20708383  68 100
   read  | write
disk I/Os in flightios   % cum % |  ios % cum %
1:   211177215  61  61   | 29305564  97  97
2:41332944  11  72   | 498260   1  99
3: