----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28474/#review63130 -----------------------------------------------------------
Thanks a lot for breaking ground on this! I've been thinking about this for a while, and i feel there's some additional structure needed to get it right: a. for timing information to be valuable, we really need a high number of iterations (leading to b.). Azul posted a great presentation [1] on java benchmarking gotchas that i highly recommend to all members of the team (i'm going to re-read it, myself). There's a similar two-part IBM article [2] that also includes sample code and more statistical analysis. b. we need to look into existing microbenchmark frameworks, such as caliper [3] (gradle example [4]), and jmh [5] (plus jmh-gradle-plugin [6]), and others i'm sure. c. a benchmark producing useful data will likely be too long-running to include in our normal unit tests, so some gradle work will be needed to split them out. Perhaps we capture '*Benchmark.java' as a separate gradle test task? See [7] and section 23.7.3 in [8] for examples. d. the suite we put together here would greatly benefit from different input data sets to serve as performance smoke tests - for example, a very large task group should not starve small task groups, multiple large task groups should be ~equally served, and high numbers of task groups. we don't need to figure out all of these right now, but building the scaffolding this will be really useful for building future data sets to serve as regression tests e. more a gut instinct than anything else, but we should probably use fakes instead of mocks in performance tests to (hopefully) make them easier to write, but also to avoid unexpected synchronization tainting timing measurements [1] http://www.azulsystems.com/presentations/art-of-java-benchmarking [2] http://www.ibm.com/developerworks/library/j-benchmark1/index.html [3] https://code.google.com/p/caliper/ [4] http://stackoverflow.com/questions/10099804/defining-a-gradle-task-to-run-caliper-microbenchmark [5] http://openjdk.java.net/projects/code-tools/jmh/ [6] https://github.com/melix/jmh-gradle-plugin [7] http://forums.gradle.org/gradle/topics/gradle_does_not_support_the_old_way_of_defining_junit_test_suites [8] http://www.gradle.org/docs/current/userguide/java_plugin.html - Bill Farner On Nov. 26, 2014, 6:15 p.m., Maxim Khutornenko wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28474/ > ----------------------------------------------------------- > > (Updated Nov. 26, 2014, 6:15 p.m.) > > > Review request for Aurora and Bill Farner. > > > Bugs: AURORA-909 > https://issues.apache.org/jira/browse/AURORA-909 > > > Repository: aurora > > > Description > ------- > > A manual testbed for the upcoming AURORA-909 work. > > > Diffs > ----- > > src/test/java/org/apache/aurora/scheduler/async/Offers.java > 8293dd181b0d062e89776fdc1205c1c227d6bb6c > src/test/java/org/apache/aurora/scheduler/async/SchedulerPerfIT.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/28474/diff/ > > > Testing > ------- > > ./gradlew -Pq build > > IDE manual run: > > Nov 26, 2014 10:12:21 AM > org.apache.aurora.scheduler.async.preemptor.PreemptorModule configure > INFO: Preemptor Enabled. > Nov 26, 2014 10:12:21 AM com.twitter.common.util.BuildInfo fetchProperties > INFO: Fetching build properties from build.properties > Nov 26, 2014 10:12:21 AM com.twitter.common.util.BuildInfo fetchProperties > WARNING: Failed to fetch build properties from build.properties > Nov 26, 2014 10:12:21 AM > com.twitter.common.application.modules.StatsModule$StartStatPoller execute > INFO: Build information: {} > Nov 26, 2014 10:12:26 AM org.apache.aurora.scheduler.async.SchedulerPerfIT > runTest > INFO: Results for: INSUFFICIENT_RESOURCES > Nov 26, 2014 10:12:26 AM org.apache.aurora.scheduler.async.SchedulerPerfIT > logDuration > INFO: Mean task_schedule_attempt duration: 737660.0796871068 ns > Nov 26, 2014 10:12:26 AM org.apache.aurora.scheduler.async.SchedulerPerfIT > logDuration > INFO: Mean offer_queue_launch_first duration: 332330.25185686833 ns > > > Thanks, > > Maxim Khutornenko > >
