Let me suggest an alternative. = Within the Phoronix Test Suite ecosystem, we have a continious = integration/validation system called Phoromatic
([1]http://www.phoromatic.c= om/). We have a brief theory of operation on it captured = [2]https://docs.google.com/present/view?id=ds439pg_42hpg57m86 . You can see a Linux oriented tracker available at [3]http://ph oromatic.com/kernel-tracker.php (the nice graphs are when you sel ect 180 days and the ION 330 systems). We've re-assigned the sy stems to other projects so it is no longer being updated, but it serv= ed our purposes. This system will allow you to just de= dicate a machine to be updateable and pick up the directions for whic= h test to be run. The FreeBSD project (or contributors) would m= aintain the slave or test machines. The test suite would be sel= ected and managed by the FreeBSD project or contributors. The g= lue code to emit triggers (possibly SVN, git or other submission, or = even just daily) and the scripts to update the systems would also be maintained by the FreeBSD community. Phoronix Me= dia would be happy to host it on Phoromatic.com (we've played with ho= sted with project branding) and provide the data store and the analyt= ics. We'd also be willing to make enhancements to support the F= reeBSD project. This should solve the "I don't have ti= me to maintain a automated test infrastructure". You don't need= to, just write the glue scripts, and dedicate a couple of machines.&= nbsp; I believe FreeBSD vendors like ixSystems may be able to support= this effort with a dedicated machine. You can have as many mac= hines as you like demonstrating AMD/Intel/32/64/large mem/low mem,etc= . The comments around interactivi= ty can also be measured to some extent. We have the model of a = "monitor". This can be configured to determine jitter around a = number of system variables and to possibly inject actions to measure = impacts. We're more than happy to work with you guys, = and are willing to help do a lot of the infrastructure and automation= lifting. Regards, Matthew On 12/22/11 8:56 AM, Adrian Chadd wrot= e: Guys, girls, fuzzy creatures, This is by far the best example of a constructive email in this entire thread. If people would like to help, Erik here is exactly the kind of person with exactly the kind of software that needs a hand. I think enough philosophizing has been done - now we have questions that need answering; theories that need testing. And that requires, you know, coding. :) The best thing right now would be for *BSD people to pick up the Phronix test suite, try to compile/run it, and provide feedback. Do your own benchmarks on your own hardware and report back the results. That's how we fix the "benchmarking problem." We don't fix it by armchair philosophy, we fix it by getting our hands dirty. :) 2c, Adrian On 22 December 2011 03:21, Erik Cederstrand [4]<erik@ cederstrand.dk> wrote: = Den 21/12/2011 kl. 19.48 skrev Alexander Leidinger: And related to the subject: wasn't it you who developed the automatic benchmarking stuff? If yes, why not make it available? If you don't have he resources, I offer my help to make it available somewhere. Yes, that's me. I'm mostly out of time right now, but I'd like to offer help if someone wants to pick up the project. For those who haven't heard about it, it's a system designed specifically to track performance of FreeBSD over time by comparing revisions of FreeBSD, everything else being equal. It consists of a tinderbox-like build script for a build server, a script to install FreeBSD and run benchmarks on at least one slave, and a database-backed website to aggregate and visualize results. The framework does work as-is, but it really needs to be updated: convert the scripts to use the SVN repo instead of CVS, improve visualization and search on the web fronted, and improve the benchmarking script so it's easier to extend. I don't have hardware available to run the benchmarks, but I think there's hardware available in the FreeBSD cluster. Here's a link to the source code: [5]http://dev.affect-it.dk/tracker.tgz And to my thesis describing how it works: [6]http://dev.affect-it.dk/tracker.pdf Just send me a mail if you're interested. Thanks, Erik_______________________________________________ [7]free bsd-performa...@freebsd.org mailing list [8]http://lists.freebsd.org/mailman/ listinfo/freebsd-performance To unsubscribe, send any mail to [9]"freebsd-performance-unsubscr...@freebsd.org" _______________________________________________ [10]freebsd-performance@freeb sd.org mailing list [11]http://lists.freebsd.org/mailman/listinfo/freebsd-per formance To unsubscribe, send any mail to [12]"freebsd-performance-unsubscr...@freebsd.org" = References 1. 3D"http://www.phoromatic.com/"; 2. 3D"https://docs.google.com/present/view?id=ds439pg_42hpg57m86 3. 3D"http://phoromatic.com/kernel-tracker.php"; 4. 3D"mailto:e...@cederstrand.dk"; 5. 3D"http://dev.affect-it.dk/tracker.tgz"; 6. 3D"http://dev.affect-it.dk/tracker.pdf"; 7. 3D"mailto:freebsd-performance@freeb 8. 3D"http://lists.freebsd.org/mailman/listinfo/freebs 9. 3D"mailto:freebsd-performance-unsubscr...@freebsd.org"; 10. 3D"mailto:freebsd-performa...@freebsd.or 11. 3D"http://lists.freebsd.org/mailman/listinfo/freebsd-per 12. ="mailto:freebsd-performance-unsubscr...@freebsd.org"; _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscr...@freebsd.org"
