Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern
Hello, Any updates on when SPC-1 is going to be available as part of fio ? Thanks, Kiran. On Wed, Nov 26, 2014 at 7:23 PM, Michael O'Sullivan michael.osulli...@auckland.ac.nz wrote: Hi Luis, We worked with Jens Axobe for a little bit to try and merge things but then just got busy testing distributed file systems as opposed to raw storage. We had an email in 2012 from I encountered a couple of segfaults when modifying the sample configuration file. I've thought to revamp it and make it more fio like, possibly turning SPC into a profile so that someone can just run fio --profile=spc But the person that emailed did not follow up. I think having an fio --profile=spc-1 would be great and I'd be happy to help get this working, but fio-type testing is not my core research area/area of expertise. We used fio+spc-1 to test disks in order to get inputs for optimal infrastructure design research (which is one mof my core research areas). That said I did a lot of the original development, so I can probably help people understand what the code is trying to do. I hope this helps. Please let me know if you'd like to revamp fio+spc-1 and if you need my help. Thanks, Mike -Original Message- From: Luis Pabón [mailto:lpa...@redhat.com] Sent: Friday, 21 November 2014 3:24 a.m. To: Michael O'Sullivan; Justin Clift Cc: gluster-devel@gluster.org Subject: Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern Hi Michael, I noticed the code on the fio branch (that is where I grabbed the spc1.[hc] files :-) ). Do you know why that branch has not being merged to master? - Luis On 11/18/2014 11:56 PM, Michael O'Sullivan wrote: Hi Justin Luis, We did a branch of fio that implemented this SPC-1 trace a few years ago. I can dig up the code and paper we wrote if it is useful? Cheers, Mike On 19/11/2014, at 4:21 pm, Justin Clift jus...@gluster.org wrote: Nifty. :) (Yeah, catching up on old unread email, as the wifi in this hotel is so bad I can barely do anything else. 8-10 second ping times to www.gluster.org. :/) As a thought, would there be useful analysis/visualisation capabilities if you stored the data into a time series database (eg InfluxDB) then used Grafana (http://grafana.org) on it? + Justin On Fri, 07 Nov 2014 12:01:56 +0100 Luis Pabón lpa...@redhat.com wrote: Hi guys, I created a simple test program to visualize the I/O pattern of NetApp's open source spc-1 workload generator. SPC-1 is an enterprise OLTP type workload created by the Storage Performance Council (http://www.storageperformance.org/results). Some of the results are published and available here: http://www.storageperformance.org/results/benchmark_results_spc1_active . NetApp created an open source version of this workload and described it in their publication A portable, open-source implementation of the SPC-1 workload ( http://www3.lrgl.uqam.ca/csdl/proceedings/iiswc/2005/9461/00/0152601 4.pdf ) The code is available onGithub: https://github.com/lpabon/spc1 . All it does at the moment is capture the pattern, no real IO is generated. I will be working on a command line program to enable usage on real block storage systems. I may either extend fio or create a tool specifically tailored to the requirements needed to run this workload. On github, I have an example IO pattern for a simulation running 50 mil IOs using HRRW_V2. The simulation ran with an ASU1 (Data Store) size of 45GB, ASU2 (User Store) size of 45GB, and ASU3 (Log) size of 10GB. - Luis ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern
Hi Luis, We worked with Jens Axobe for a little bit to try and merge things but then just got busy testing distributed file systems as opposed to raw storage. We had an email in 2012 from I encountered a couple of segfaults when modifying the sample configuration file. I've thought to revamp it and make it more fio like, possibly turning SPC into a profile so that someone can just run fio --profile=spc But the person that emailed did not follow up. I think having an fio --profile=spc-1 would be great and I'd be happy to help get this working, but fio-type testing is not my core research area/area of expertise. We used fio+spc-1 to test disks in order to get inputs for optimal infrastructure design research (which is one mof my core research areas). That said I did a lot of the original development, so I can probably help people understand what the code is trying to do. I hope this helps. Please let me know if you'd like to revamp fio+spc-1 and if you need my help. Thanks, Mike -Original Message- From: Luis Pabón [mailto:lpa...@redhat.com] Sent: Friday, 21 November 2014 3:24 a.m. To: Michael O'Sullivan; Justin Clift Cc: gluster-devel@gluster.org Subject: Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern Hi Michael, I noticed the code on the fio branch (that is where I grabbed the spc1.[hc] files :-) ). Do you know why that branch has not being merged to master? - Luis On 11/18/2014 11:56 PM, Michael O'Sullivan wrote: Hi Justin Luis, We did a branch of fio that implemented this SPC-1 trace a few years ago. I can dig up the code and paper we wrote if it is useful? Cheers, Mike On 19/11/2014, at 4:21 pm, Justin Clift jus...@gluster.org wrote: Nifty. :) (Yeah, catching up on old unread email, as the wifi in this hotel is so bad I can barely do anything else. 8-10 second ping times to www.gluster.org. :/) As a thought, would there be useful analysis/visualisation capabilities if you stored the data into a time series database (eg InfluxDB) then used Grafana (http://grafana.org) on it? + Justin On Fri, 07 Nov 2014 12:01:56 +0100 Luis Pabón lpa...@redhat.com wrote: Hi guys, I created a simple test program to visualize the I/O pattern of NetApp's open source spc-1 workload generator. SPC-1 is an enterprise OLTP type workload created by the Storage Performance Council (http://www.storageperformance.org/results). Some of the results are published and available here: http://www.storageperformance.org/results/benchmark_results_spc1_active . NetApp created an open source version of this workload and described it in their publication A portable, open-source implementation of the SPC-1 workload ( http://www3.lrgl.uqam.ca/csdl/proceedings/iiswc/2005/9461/00/0152601 4.pdf ) The code is available onGithub: https://github.com/lpabon/spc1 . All it does at the moment is capture the pattern, no real IO is generated. I will be working on a command line program to enable usage on real block storage systems. I may either extend fio or create a tool specifically tailored to the requirements needed to run this workload. On github, I have an example IO pattern for a simulation running 50 mil IOs using HRRW_V2. The simulation ran with an ASU1 (Data Store) size of 45GB, ASU2 (User Store) size of 45GB, and ASU3 (Log) size of 10GB. - Luis ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern
The git repo for the fio/SPC-1 tool is here: http://git.kernel.dk/?p=fio.git;a=shortlog;h=refs/heads/spc1-test We talked about using this tool here: Lee, D., O'Sullivan, M. J., Walker, C. G. (2010). Practical Measurement of Typical Disk Performance and Power Consumption using Open Source SPC-1. In 2010 Annual Conference on GreenIT (pp. 8 pages). Singapore. doi:10.5176/978-981-08-7240-3_G-15 As you will see from the git repo, this hasn't been worked on since 2010, but it may be useful and it would be cool to see this work extended. It was quite a while ago, but I think we may have had to modify the original NetApp open source SPC-1 code a little to fix some bugs... Thanks, Mike -Original Message- From: gluster-devel-boun...@gluster.org [mailto:gluster-devel-boun...@gluster.org] On Behalf Of Michael O'Sullivan Sent: Wednesday, 19 November 2014 5:57 p.m. To: Justin Clift Cc: gluster-devel@gluster.org Subject: Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern Hi Justin Luis, We did a branch of fio that implemented this SPC-1 trace a few years ago. I can dig up the code and paper we wrote if it is useful? Cheers, Mike On 19/11/2014, at 4:21 pm, Justin Clift jus...@gluster.org wrote: Nifty. :) (Yeah, catching up on old unread email, as the wifi in this hotel is so bad I can barely do anything else. 8-10 second ping times to www.gluster.org. :/) As a thought, would there be useful analysis/visualisation capabilities if you stored the data into a time series database (eg InfluxDB) then used Grafana (http://grafana.org) on it? + Justin On Fri, 07 Nov 2014 12:01:56 +0100 Luis Pabón lpa...@redhat.com wrote: Hi guys, I created a simple test program to visualize the I/O pattern of NetApp's open source spc-1 workload generator. SPC-1 is an enterprise OLTP type workload created by the Storage Performance Council (http://www.storageperformance.org/results). Some of the results are published and available here: http://www.storageperformance.org/results/benchmark_results_spc1_active . NetApp created an open source version of this workload and described it in their publication A portable, open-source implementation of the SPC-1 workload ( http://www3.lrgl.uqam.ca/csdl/proceedings/iiswc/2005/9461/00/01526014 .pdf ) The code is available onGithub: https://github.com/lpabon/spc1 . All it does at the moment is capture the pattern, no real IO is generated. I will be working on a command line program to enable usage on real block storage systems. I may either extend fio or create a tool specifically tailored to the requirements needed to run this workload. On github, I have an example IO pattern for a simulation running 50 mil IOs using HRRW_V2. The simulation ran with an ASU1 (Data Store) size of 45GB, ASU2 (User Store) size of 45GB, and ASU3 (Log) size of 10GB. - Luis ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern
Interesting, I have never known about those tools. They look great! I will check them out. Thanks Justin. - Luis On 11/18/2014 10:20 PM, Justin Clift wrote: Nifty. :) (Yeah, catching up on old unread email, as the wifi in this hotel is so bad I can barely do anything else. 8-10 second ping times to www.gluster.org. :/) As a thought, would there be useful analysis/visualisation capabilities if you stored the data into a time series database (eg InfluxDB) then used Grafana (http://grafana.org) on it? + Justin On Fri, 07 Nov 2014 12:01:56 +0100 Luis Pabón lpa...@redhat.com wrote: Hi guys, I created a simple test program to visualize the I/O pattern of NetApp’s open source spc-1 workload generator. SPC-1 is an enterprise OLTP type workload created by the Storage Performance Council (http://www.storageperformance.org/results). Some of the results are published and available here: http://www.storageperformance.org/results/benchmark_results_spc1_active . NetApp created an open source version of this workload and described it in their publication A portable, open-source implementation of the SPC-1 workload ( http://www3.lrgl.uqam.ca/csdl/proceedings/iiswc/2005/9461/00/01526014.pdf ) The code is available onGithub: https://github.com/lpabon/spc1 . All it does at the moment is capture the pattern, no real IO is generated. I will be working on a command line program to enable usage on real block storage systems. I may either extend fio or create a tool specifically tailored to the requirements needed to run this workload. On github, I have an example IO pattern for a simulation running 50 mil IOs using HRRW_V2. The simulation ran with an ASU1 (Data Store) size of 45GB, ASU2 (User Store) size of 45GB, and ASU3 (Log) size of 10GB. - Luis ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern
Nifty. :) (Yeah, catching up on old unread email, as the wifi in this hotel is so bad I can barely do anything else. 8-10 second ping times to www.gluster.org. :/) As a thought, would there be useful analysis/visualisation capabilities if you stored the data into a time series database (eg InfluxDB) then used Grafana (http://grafana.org) on it? + Justin On Fri, 07 Nov 2014 12:01:56 +0100 Luis Pabón lpa...@redhat.com wrote: Hi guys, I created a simple test program to visualize the I/O pattern of NetApp’s open source spc-1 workload generator. SPC-1 is an enterprise OLTP type workload created by the Storage Performance Council (http://www.storageperformance.org/results). Some of the results are published and available here: http://www.storageperformance.org/results/benchmark_results_spc1_active . NetApp created an open source version of this workload and described it in their publication A portable, open-source implementation of the SPC-1 workload ( http://www3.lrgl.uqam.ca/csdl/proceedings/iiswc/2005/9461/00/01526014.pdf ) The code is available onGithub: https://github.com/lpabon/spc1 . All it does at the moment is capture the pattern, no real IO is generated. I will be working on a command line program to enable usage on real block storage systems. I may either extend fio or create a tool specifically tailored to the requirements needed to run this workload. On github, I have an example IO pattern for a simulation running 50 mil IOs using HRRW_V2. The simulation ran with an ASU1 (Data Store) size of 45GB, ASU2 (User Store) size of 45GB, and ASU3 (Log) size of 10GB. - Luis ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern
Hi Justin Luis, We did a branch of fio that implemented this SPC-1 trace a few years ago. I can dig up the code and paper we wrote if it is useful? Cheers, Mike On 19/11/2014, at 4:21 pm, Justin Clift jus...@gluster.org wrote: Nifty. :) (Yeah, catching up on old unread email, as the wifi in this hotel is so bad I can barely do anything else. 8-10 second ping times to www.gluster.org. :/) As a thought, would there be useful analysis/visualisation capabilities if you stored the data into a time series database (eg InfluxDB) then used Grafana (http://grafana.org) on it? + Justin On Fri, 07 Nov 2014 12:01:56 +0100 Luis Pabón lpa...@redhat.com wrote: Hi guys, I created a simple test program to visualize the I/O pattern of NetApp’s open source spc-1 workload generator. SPC-1 is an enterprise OLTP type workload created by the Storage Performance Council (http://www.storageperformance.org/results). Some of the results are published and available here: http://www.storageperformance.org/results/benchmark_results_spc1_active . NetApp created an open source version of this workload and described it in their publication A portable, open-source implementation of the SPC-1 workload ( http://www3.lrgl.uqam.ca/csdl/proceedings/iiswc/2005/9461/00/01526014.pdf ) The code is available onGithub: https://github.com/lpabon/spc1 . All it does at the moment is capture the pattern, no real IO is generated. I will be working on a command line program to enable usage on real block storage systems. I may either extend fio or create a tool specifically tailored to the requirements needed to run this workload. On github, I have an example IO pattern for a simulation running 50 mil IOs using HRRW_V2. The simulation ran with an ASU1 (Data Store) size of 45GB, ASU2 (User Store) size of 45GB, and ASU3 (Log) size of 10GB. - Luis ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Open source SPC-1 Workload IO Pattern
On Nov 7, 2014, at 1:53 PM, Mark Nelson mark.nel...@inktank.com wrote: On 11/07/2014 05:01 AM, Luis Pabón wrote: Hi guys, I created a simple test program to visualize the I/O pattern of NetApp’s open source spc-1 workload generator. SPC-1 is an enterprise OLTP type workload created by the Storage Performance Council (http://www.storageperformance.org/results). Some of the results are published and available here: http://www.storageperformance.org/results/benchmark_results_spc1_active . NetApp created an open source version of this workload and described it in their publication A portable, open-source implementation of the SPC-1 workload ( http://www3.lrgl.uqam.ca/csdl/proceedings/iiswc/2005/9461/00/01526014.pdf ) The code is available onGithub: https://github.com/lpabon/spc1 . All it does at the moment is capture the pattern, no real IO is generated. I will be working on a command line program to enable usage on real block storage systems. I may either extend fio or create a tool specifically tailored to the requirements needed to run this workload. Neat! integration with fio could be interesting. We could then use any of the engines include the librbd one (I think there is some kind of gluster engine as well?) Good point. On github, I have an example IO pattern for a simulation running 50 mil IOs using HRRW_V2. The simulation ran with an ASU1 (Data Store) size of 45GB, ASU2 (User Store) size of 45GB, and ASU3 (Log) size of 10GB. Out of curiosity have you looked at how fast you can generate IOs before CPU is a bottleneck? Good question, I will check that when i have the io tool generation. - Luis -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
