Re: [ceph-users] CBT results parsing/plotting

2016-07-06 Thread Pavan Rallabhandi 
Thanks Mark,  we did look at tools from Ben England here 
https://github.com/bengland2/cbt/blob/fio-thru-analysis/tools/parse-fio-result-log.sh
 but not with much luck, that’s partly because we didn’t bother to look into 
the gory details if things didn’t work.

Thanks for the scripts you have attached, will give a shot.

Thanks,
-Pavan.

On 7/6/16, 11:05 PM, "ceph-users on behalf of Mark Nelson" 
 wrote:

Hi Pavan,

A couple of us have some pretty ugly home-grown scripts for doing this. 
Basically just bash/awk that loop through the directories and grab the 
fio bw/latency lines.  Eventually the whole way that cbt stores data 
should be revamped since the way data gets laid out in a nested 
directory structure doesn't really scale.

Lately we've been more focused on parsing the fio log output.  We've had 
issues with strange clock skew and multiple concurrent clients not all 
running at exactly the same time, leading to bias in the results.  To 
get around this, we wrote a tool to parse the output of multiple fio 
bw/iops/latency log files.

It was upstreamed into fio itself recently here:

https://github.com/axboe/fio/blob/master/tools/fiologparser.py

a potentially better (but still experimental and maybe buggy) version of 
this is here:

https://github.com/markhpc/fio/blob/wip-interval/tools/fiologparser.py

The idea is to fit samples into intervals based on how much they 
overlap, but if you have multiple files (say from multiple simulatenous 
clients) only calculate averages based on the time the clients ran at 
the same time (in case some clients started late or ended early).  Ben 
England is working on speeding this up when fio records every IO rather 
than interval averages and we have an intern now that is looking at this 
and other methods for getting better logging data out of fio.

Anyway, I've included two scripts that will run through a cbt output 
directory and either read the fio output file or run this script to 
parse the output data.  Like I said, these are basically terrible (even 
embarrassing!) and will almost certainly need some slight hacking to 
work based on the iodepth/readahead/etc.  I think Ben England has 
something a bit nicer but he's on vacation at the moment so you'll have 
to suffer with my horrible hacked up bash for now. ;)

Mark

On 07/06/2016 03:22 AM, Pavan Rallabhandi wrote:
> Wanted to check if there are any readily available tools that the community 
> is aware of/using for parsing/plotting CBT run results. Am particularly 
> interested in tools for the CBT librbdfio runs, where in the aggregated 
> BW/IOPS/Latency reports are generated either in CSV/graphical.
>
> Thanks!
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CBT results parsing/plotting

2016-07-06 Thread Mark Nelson 

Hi Pavan,

A couple of us have some pretty ugly home-grown scripts for doing this. 
Basically just bash/awk that loop through the directories and grab the 
fio bw/latency lines.  Eventually the whole way that cbt stores data 
should be revamped since the way data gets laid out in a nested 
directory structure doesn't really scale.


Lately we've been more focused on parsing the fio log output.  We've had 
issues with strange clock skew and multiple concurrent clients not all 
running at exactly the same time, leading to bias in the results.  To 
get around this, we wrote a tool to parse the output of multiple fio 
bw/iops/latency log files.


It was upstreamed into fio itself recently here:

https://github.com/axboe/fio/blob/master/tools/fiologparser.py

a potentially better (but still experimental and maybe buggy) version of 
this is here:


https://github.com/markhpc/fio/blob/wip-interval/tools/fiologparser.py

The idea is to fit samples into intervals based on how much they 
overlap, but if you have multiple files (say from multiple simulatenous 
clients) only calculate averages based on the time the clients ran at 
the same time (in case some clients started late or ended early).  Ben 
England is working on speeding this up when fio records every IO rather 
than interval averages and we have an intern now that is looking at this 
and other methods for getting better logging data out of fio.


Anyway, I've included two scripts that will run through a cbt output 
directory and either read the fio output file or run this script to 
parse the output data.  Like I said, these are basically terrible (even 
embarrassing!) and will almost certainly need some slight hacking to 
work based on the iodepth/readahead/etc.  I think Ben England has 
something a bit nicer but he's on vacation at the moment so you'll have 
to suffer with my horrible hacked up bash for now. ;)


Mark

On 07/06/2016 03:22 AM, Pavan Rallabhandi wrote:

Wanted to check if there are any readily available tools that the community is 
aware of/using for parsing/plotting CBT run results. Am particularly interested 
in tools for the CBT librbdfio runs, where in the aggregated BW/IOPS/Latency 
reports are generated either in CSV/graphical.

Thanks!

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



getlibrbdgnuplot.sh
Description: application/shellscript


getlibrbdgnuplot2.sh
Description: application/shellscript
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CBT results parsing/plotting

2016-07-06 Thread Pavan Rallabhandi 
Wanted to check if there are any readily available tools that the community is 
aware of/using for parsing/plotting CBT run results. Am particularly interested 
in tools for the CBT librbdfio runs, where in the aggregated BW/IOPS/Latency 
reports are generated either in CSV/graphical.

Thanks!

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com