Re: freebsd perf testing
On Nov 10, 2013, at 19:22 , Tim Kientzle t...@kientzle.com wrote: On Nov 10, 2013, at 1:05 PM, Erik Cederstrand erik+li...@cederstrand.dk wrote: Imagine being able to fetch a VirtualBox disk image for a random SVN commit, booting it and start debugging right away. I’ve been working on Crochet’s support for building VMWare images recently and have started using that approach to iterate my dev environment (using one VM to build a new VM instead of upgrading in place). Sorry to come in late. All this sounds good, and I’d like to point out that the project has network testing hardware in place, if people want to use it for these types of experiments. In the absence of a lab just for regression testing (which is also in the works) I’d suggest that prototyping be done here: https://wiki.freebsd.org/TestClusterOnePointers Anyone who is a FreeBSD committer can get access, and those who want access but are not yet committers should contact me so we can try to work something out. Best, George signature.asc Description: Message signed with OpenPGP using GPGMail
Re: freebsd perf testing
On 11/9/13, 1:24 PM, Erik Cederstrand wrote: Hi Julian, Den 08/11/2013 kl. 03.02 skrev Julian Elischer jul...@elischer.org: Some time ago someone showed some freebsd performance graphs graphed against time. He had them up on a website that was updated each day or so. I think they were network perf tests but I'm not sure. He indicated that he was going to continue the daily testing but I've not seen any mention of them since. If you know who that was or how to find him let me (or gnn) know... I did a master’s thesis on this some years ago. I haven’t kept the project up-to-date, due to lack of time and hardware. it would be interesting to know what you did. and what conclusions you came to.. the actual web page I was thinkng of was: http://lists.freebsd.org/pipermail/freebsd-current/2013-April/041323.html but the more players we have thinking about this the better.. it would be good if we could have some project supported way to follow this. it may be that we could make a 'contributor' image that has all the tools and framework on it that would allow people to submit daily reports (from the same hardware each time) to some central aggregator.. or maybe ot would all be project resources. I see that Mr Symbolics (is that his real name?) has some suggestions in another mail in this thread. I wasn't really going to be doing much in this space myself though I think it is important, I just volunteered in the meeting to try find some examples. Julian Erik ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: freebsd perf testing
Den 10/11/2013 kl. 09.45 skrev Julian Elischer jul...@freebsd.org: it would be interesting to know what you did. and what conclusions you came to.. I created a system with a build server, a set of slaves, and a website to collect and publish the benchmark data in graphs and raw data and highlight significant changes. The build server built FreeBSD and any required ports and benchmark software, and bundled it in installation images complete with a script to run the benchmarks on the slave and push data to the website. The slaves were dedicated machines that booted via PXE, wiped the hard drive, fetched and installed an image and proceeded to run the specified benchmarks. The website published graphs, and the graphs were linked to useful related data like raw measurements, slave hardware, commit messages, changed source files and binaries etc. It actually worked pretty well. I only ran benchmarks over a couple of months of commits, but I did identify some significant regressions in MySQL and some of the unixbench microbenchmarks. I did not spend much time designing the benchmarking strategy, and “symbolics” has many good points there. I did find out that all the current continuous integration / performance benchmark frameworks (Jenkins etc) did not play well with a situation where the entire operating system is reinstalled. A wide range of useful benchmarks can be collected in just 10-20 minutes, but building FreeBSD and all the ports takes much longer than that (1-2 hours using the default procedure from the handbook). Most of my work went into optimizing build times, installation image sizes and slave installation times, so I could build an image in just 10-15 minutes and use ca. 10MB disk space amortized, and install an image on a slave in just 2 minutes (cheap 2008-era hardware). But having a complete archive of ready-to-install images for each commit is a real advantage, not just for performance benchmarking. Imagine being able to fetch a VirtualBox disk image for a random SVN commit, booting it and start debugging right away. Or you could write a script to test if a bug is present in a FreeBSD installation, and then let an automated process do a binary search to find the offending commit in maybe less than an hour. My work was completed in 2008 and would (at least) require updating the scripts to handle the upgrade from GCC to Clang, and from CVS to SVN. Erik ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: freebsd perf testing
On Nov 10, 2013, at 1:05 PM, Erik Cederstrand erik+li...@cederstrand.dk wrote: Imagine being able to fetch a VirtualBox disk image for a random SVN commit, booting it and start debugging right away. I’ve been working on Crochet’s support for building VMWare images recently and have started using that approach to iterate my dev environment (using one VM to build a new VM instead of upgrading in place). Tim ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: freebsd perf testing
On Fri, Nov 08, 2013 at 12:28:02PM -0800, Julian Elischer wrote: On 11/8/13, 1:54 AM, Olivier Cochard-Labbé wrote: On Fri, Nov 8, 2013 at 3:02 AM, Julian Elischer jul...@elischer.org mailto:jul...@elischer.orgwrote: Some time ago someone showed some freebsd performance graphs graphed against time. He had them up on a website that was updated each day or so. I think they were network perf tests but I'm not sure. He indicated that he was going to continue the daily testing but I've not seen any mention of them since. If you know who that was or how to find him let me (or gnn) know... Hi Julian, Perhaps you are referring to my network performance graphs on this thread: http://lists.freebsd.org/pipermail/freebsd-current/2013-April/041323.html yes, you are the person we are looking for. In yesterday's 'vendor summit' we were discussing performance monitoring and your methodology was cited as one worth looking at. The idea of graphing the output of various performance tests against svn commit number is a very good one. I thonk it migh teven be worth doing these tests daily, and putting the output onto a web site, showing the last month, the last year and the whole range. it would even be interesting to put out 'xplot' files so that people can zoom in and out using xplot to see exactly which revision was responsinble for reversions or problems. George.. this is what we mentioned at the meeting yesterday. Julian As it happens I've been thinking over a design for something along these lines recently. It's just some ideas at the moment but it might be of interest to others. Forgive me; it's a long E-mail and it gets a bit pie-in-the-sky too. I was prompted to think about the problem in the first place because I read commit mail and I see performance related changes going into the tree from time to time. These changes often do not come with any specific data and when they do its normally quite narrow in focus. For instance, an organisation contributes performance improvements specific to their workloads and without interest in anyone elses (fair enough). Initially, what I wanted was a way of viewing how performance changed for a number of workloads on a commit by commit basis. This sounds very much like what you are after. Anyway, after thinking about this for sometime it occurred to me that much of the infrastructure required to do performance testing could be generalised to all sorts of software experiments. E.g. software builds, regression tests, and so on. So, my first conclusion was: build an experimentation framework within which performance is one aspect. Having decided this, I thought about the scope of experiments I wanted to make. For instance, it would be good to test at least every supported platform. On top of that I would like to be able to vary the relevant configuration options too. Taking the product of commit, platform, n-configuration options (not to mention compilers, etc...) you start to get some pretty big numbers. The numbers grow far too fast and no person or even organisation could feasibly cover the hardware resources required to test every permutation. This led me to my next conclusion: build a distributed system that allows for anyone to contribute their hardware to the cause. Collectively the project, vendors, and users could tackle a big chunk of this. My rough sketch for how this would work is as follows. A bootable USB image would be made for all platforms. This would boot up, connect to the network and checkout a repository. The first phase of the process would be to profile what the host can offer. For example, we might have experiments that require four identical hard drives, or a particular CPU type, and so on. Shell scripts or short programmes would be written, e.g. has-atom-cpu, with these returning either 1 or 0. The results of this profiling would be submitted to a service. The service matches the host with available experiments based on its particular capabilities and current experimental priorities laid down by the developers. A priority system would allow for the system to be controlled precisely. If, for instance, major work is done to the VM subsystem, relevant experiments could be prioritised over others for a period. Once a decision on the experiment to conduct has been made, the relevant image must be deployed to the system. Free space on the USB device would be used a staging area, with a scripted installation occurring after reboot. The images would need to be built somewhere, since it doesn't make sense to rebuild the system endlessly, especially if we're including low-powered embedded devices (which we should be). One possible solution to this would be to use more powerful contributed hosts to cross-build images and make them available for download. Finally, the experiments would be conducted. Data produced would be submitted back to the project using
Re: freebsd perf testing
Hi Julian, Den 08/11/2013 kl. 03.02 skrev Julian Elischer jul...@elischer.org: Some time ago someone showed some freebsd performance graphs graphed against time. He had them up on a website that was updated each day or so. I think they were network perf tests but I'm not sure. He indicated that he was going to continue the daily testing but I've not seen any mention of them since. If you know who that was or how to find him let me (or gnn) know... I did a master’s thesis on this some years ago. I haven’t kept the project up-to-date, due to lack of time and hardware. Erik ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: freebsd perf testing
On Sat, Nov 9, 2013 at 6:37 AM, symbol...@gmx.com wrote: On Fri, Nov 08, 2013 at 12:28:02PM -0800, Julian Elischer wrote: On 11/8/13, 1:54 AM, Olivier Cochard-Labbé wrote: On Fri, Nov 8, 2013 at 3:02 AM, Julian Elischer jul...@elischer.org mailto:jul...@elischer.orgwrote: Some time ago someone showed some freebsd performance graphs graphed against time. He had them up on a website that was updated each day or so. I think they were network perf tests but I'm not sure. He indicated that he was going to continue the daily testing but I've not seen any mention of them since. If you know who that was or how to find him let me (or gnn) know... Hi Julian, Perhaps you are referring to my network performance graphs on this thread: http://lists.freebsd.org/pipermail/freebsd-current/2013-April/041323.html yes, you are the person we are looking for. In yesterday's 'vendor summit' we were discussing performance monitoring and your methodology was cited as one worth looking at. The idea of graphing the output of various performance tests against svn commit number is a very good one. I thonk it migh teven be worth doing these tests daily, and putting the output onto a web site, showing the last month, the last year and the whole range. it would even be interesting to put out 'xplot' files so that people can zoom in and out using xplot to see exactly which revision was responsinble for reversions or problems. George.. this is what we mentioned at the meeting yesterday. Julian As it happens I've been thinking over a design for something along these lines recently. It's just some ideas at the moment but it might be of interest to others. Forgive me; it's a long E-mail and it gets a bit pie-in-the-sky too. I was prompted to think about the problem in the first place because I read commit mail and I see performance related changes going into the tree from time to time. These changes often do not come with any specific data and when they do its normally quite narrow in focus. For instance, an organisation contributes performance improvements specific to their workloads and without interest in anyone elses (fair enough). Initially, what I wanted was a way of viewing how performance changed for a number of workloads on a commit by commit basis. This sounds very much like what you are after. Anyway, after thinking about this for sometime it occurred to me that much of the infrastructure required to do performance testing could be generalised to all sorts of software experiments. E.g. software builds, regression tests, and so on. So, my first conclusion was: build an experimentation framework within which performance is one aspect. Having decided this, I thought about the scope of experiments I wanted to make. For instance, it would be good to test at least every supported platform. On top of that I would like to be able to vary the relevant configuration options too. Taking the product of commit, platform, n-configuration options (not to mention compilers, etc...) you start to get some pretty big numbers. The numbers grow far too fast and no person or even organisation could feasibly cover the hardware resources required to test every permutation. This led me to my next conclusion: build a distributed system that allows for anyone to contribute their hardware to the cause. Collectively the project, vendors, and users could tackle a big chunk of this. My rough sketch for how this would work is as follows. A bootable USB image would be made for all platforms. This would boot up, connect to the network and checkout a repository. The first phase of the process would be to profile what the host can offer. For example, we might have experiments that require four identical hard drives, or a particular CPU type, and so on. Shell scripts or short programmes would be written, e.g. has-atom-cpu, with these returning either 1 or 0. The results of this profiling would be submitted to a service. The service matches the host with available experiments based on its particular capabilities and current experimental priorities laid down by the developers. A priority system would allow for the system to be controlled precisely. If, for instance, major work is done to the VM subsystem, relevant experiments could be prioritised over others for a period. Once a decision on the experiment to conduct has been made, the relevant image must be deployed to the system. Free space on the USB device would be used a staging area, with a scripted installation occurring after reboot. The images would need to be built somewhere, since it doesn't make sense to rebuild the system endlessly, especially if we're including low-powered embedded devices (which we should be). One possible solution to this would be to use more powerful contributed hosts to cross-build images and make them available for download. Finally,
Re: freebsd perf testing
Hi Julian, Den 08/11/2013 kl. 03.02 skrev Julian Elischer jul...@elischer.org: Some time ago someone showed some freebsd performance graphs graphed against time. He had them up on a website that was updated each day or so. I think they were network perf tests but I'm not sure. He indicated that he was going to continue the daily testing but I've not seen any mention of them since. If you know who that was or how to find him let me (or gnn) know... I did a master’s thesis on this some years ago. I haven’t kept the project up-to-date, due to lack of time and hardware. Erik ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: freebsd perf testing
On Fri, Nov 8, 2013 at 3:02 AM, Julian Elischer jul...@elischer.org wrote: Some time ago someone showed some freebsd performance graphs graphed against time. He had them up on a website that was updated each day or so. I think they were network perf tests but I'm not sure. He indicated that he was going to continue the daily testing but I've not seen any mention of them since. If you know who that was or how to find him let me (or gnn) know... Hi Julian, Perhaps you are referring to my network performance graphs on this thread: http://lists.freebsd.org/pipermail/freebsd-current/2013-April/041323.html I didn't generate other tests since because I need to fix the test used before continuing. I only generate one IP-flow (same IP src/dst, same UDP port): there is no IP distribution. And by generating only one flow, we didn't use the multi-queue NIC capability, neither the multi-threaded features (like with the SMP-pf) of FreeBSD forwarding/firewalling stack. I plan to add the support of generating multiple IP source/destination to the netmap pkt-gen, but the current status of this project is: I'm at page 42 of the book C programming, a modern approach, then you have to wait :-) By the way, I've documented a little more my benching lab and scripts used here: http://bsdrp.net/documentation/examples/freebsd_performance_regression_lab Regards, Olivier ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: freebsd perf testing
On 11/8/13, 1:54 AM, Olivier Cochard-Labbé wrote: On Fri, Nov 8, 2013 at 3:02 AM, Julian Elischer jul...@elischer.org mailto:jul...@elischer.orgwrote: Some time ago someone showed some freebsd performance graphs graphed against time. He had them up on a website that was updated each day or so. I think they were network perf tests but I'm not sure. He indicated that he was going to continue the daily testing but I've not seen any mention of them since. If you know who that was or how to find him let me (or gnn) know... Hi Julian, Perhaps you are referring to my network performance graphs on this thread: http://lists.freebsd.org/pipermail/freebsd-current/2013-April/041323.html yes, you are the person we are looking for. In yesterday's 'vendor summit' we were discussing performance monitoring and your methodology was cited as one worth looking at. The idea of graphing the output of various performance tests against svn commit number is a very good one. I thonk it migh teven be worth doing these tests daily, and putting the output onto a web site, showing the last month, the last year and the whole range. it would even be interesting to put out 'xplot' files so that people can zoom in and out using xplot to see exactly which revision was responsinble for reversions or problems. George.. this is what we mentioned at the meeting yesterday. Julian I didn't generate other tests since because I need to fix the test used before continuing. I only generate one IP-flow (same IP src/dst, same UDP port): there is no IP distribution. And by generating only one flow, we didn't use the multi-queue NIC capability, neither the multi-threaded features (like with the SMP-pf) of FreeBSD forwarding/firewalling stack. I plan to add the support of generating multiple IP source/destination to the netmap pkt-gen, but the current status of this project is: I'm at page 42 of the book C programming, a modern approach, then you have to wait :-) By the way, I've documented a little more my benching lab and scripts used here: http://bsdrp.net/documentation/examples/freebsd_performance_regression_lab Regards, Olivier ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
freebsd perf testing
Some time ago someone showed some freebsd performance graphs graphed against time. He had them up on a website that was updated each day or so. I think they were network perf tests but I'm not sure. He indicated that he was going to continue the daily testing but I've not seen any mention of them since. If you know who that was or how to find him let me (or gnn) know... Julian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
