On Thu, May 5, 2011 at 6:37 PM, Hadrian Zbarcea <hzbar...@gmail.com> wrote: > Improving testing is a long time pet peeve of mine. I experimented with both > junit parallel tests (works awesome) and with maven 3 multithreaded builds > (using -T). Combining both on a box with one quad core i7 and 16G ram gave me > an under 25 min full build and test (except assembly), but with quite a few > issues that can be fixed, but probably only for 3.0 because they are not > backwards compatible. > > Are the camel unit tests unit? Unfortunately no. There is however a historic > reason for that (not that it makes it right). Back in the baby days of camel, > when there was a focus on EIPs and new ones were added every week, it seemed > like a good idea to automate that. That technique, because of its simplicity > spilled into being used in testing components (and there is some value in > that too, but mostly as an integration test). In the meantime camel holds the > record of a hard to justify 4+ hours build/test. > > I still have hopes to get that improved. Some of the things that need to be > done: use a lighter CamelContext (something like a TestCamelContext) for > testing components (that doesn't load TypeConverters for instance and is much > faster), cleanup around jmx (one major reason why parallel tests are > impossible now) and a few other smaller things. Christian is right, the sheer > number of components we have makes the cleanup task quite complicated. There > were discussions about splitting the core and that would help with dividing > and conquering this task too. I believe you will see all these (major) > improvements in 3.0, but I don't see it likely to happen until then. >
If we take a look at a recent report from Hudson https://hudson.apache.org/hudson/job/Camel.trunk.fulltest/272/ You can see some of the test time numbers. At first glance you can notice that most of the Camel components is tested in seconds (< 1 min). However there are some which takes longer time. Obviously there is - camel-core - camel-spring - camel-jms As the big spenders. So why do they take "long time" to run? Well camel-core has 3400 unit tests and Hudson says it took 22 min to test that. Thats roughly 150 tests/min. So now we approach my take on why Camel tests may be slow. Due Spring Framework. Spring 2.5 onwards became much slower to load and startup its XML files. And camel-spring is mostly tests that are a replicate of camel-core, but using the XML DSL instead. camel-spring takes 16 min to run 800 tests, which is roughly 50 tests/min In terms of JMX, then we disabled that by default for the test kit. So we only enable that when we test JMX specially. It used to be enabled by default back then. And we discovered it was added about 5-10% as overhead. So yeah Hadrian is right that JMX influence the speed. But we have disabled JMX by default in camel-test so there shouldnt be any quick win there anymore. Then the JUnit jvm fork-mode plays a huge role. There are a few components which has it pertest. We may find a way around resolving that so we can have the JVM forked only once. That makes it actually much faster. I think camel-jms (and maybe camel-cxf) is pertest. We have made some recent improvements to camel-jms. So it may be possible to try to experiment with a pertest. But the risk could be we could start see subtle tests errors due AMQ not properly cleanup itself in between tests. And thus it could have side effects running the next tests. And the last big spender is OSGi tests. They take a long long time. However Pax Exam 2 (we use Pax Exam 1), when its released. We can upgrade to it. It allows to run tests in the same JVM as the test. And it should thus be much faster than Pax Exam 1, which is painfully slow. The idea of having the type converter lazy scan for converters may help as well. However that said. We do have a INFO logging in Camel that says how long time it took for it to scan and load all the type converters. So it may be worthwhile to check the logs after a test run to see for any "slow". There is a ticket in JIRA to add an option so you can tell Camel to not lazy load on startup. That would require we still load the type converters from camel-core so Camel has the most common converters. But we could use an alternative way of doing that instead of classpath scanning. The downside is that if we do "lazy" then we need locked code to ensure for concurrency that the lazy is only loaded once, and that any concurrent thread is waiting for it to complete. So there is a little penalty to pay at runtime. So from a runtime perspective it would be faster to load all up-front. But still we could just enable the lazy by default in unit tests. And see if that improves the test times. Okay my coffee is ready. And sorry for this long post. But Hadrian started a great conversation about his endeavor to make testing the 7000 tests of Camel faster. PS: Hudson don't test the OSGi tests. If it did that would take 30+ min or more. We should find a way to enable that again. Maybe when Pax Exam 2 is out, that may ensure CI servers can better test with OSGi. In the past we have seen the CI servers "lock up" during OSGi tests, and hence we had to skip those tests. > Cheers, > Hadrian > > > On May 5, 2011, at 10:49 AM, Christian Schneider wrote: > >> Theoretically yes and I know Hadrian was already experimenting with that but >> we are quite conservative with that. You can easily have occasional test >> failures then that take a lot of time to find and fix. >> >> Christian >> >> >> Am 05.05.2011 15:26, schrieb David Karlsen: >>> Can they be run in parallel - or can modules be run in parallel - that could >>> speed things up. >>> >>> 2011/5/5 Christian Schneider<christian.schnei...@sopera.com> >>> >>>> I think it is only named wrong. The camel test support is written to >>>> support integration >>>> tests not unit tests. >>>> >>>> I think it is a very good idea to write real unit tests and integration >>>> tests. The unit tests are extremly fast and give you a first security. >>>> Still >>>> the integration tests are very important as many errors can only be spotted >>>> this way. >>>> >>>> I think at the moment camel uses a quite pragmatic test concept. Each >>>> component comes with unit tests and integration tests. As long as the tests >>>> are fast I think there is no big problem with that. >>>> >>>> All in all the problem is though that a whole camel build takes aroung 4 >>>> hours. So we might really be able to do that better. A problem here is >>>> though the big number of components we have. >>>> >>>> Christian >>>> >>>> >>>> Am 05.05.2011 10:56, schrieb Gert Villemos: >>>> >>>> Some would argue that a unit test per definition tests the unit completely >>>>> standalone. Using JUnit this is easy, especially when combined with >>>>> Spring, >>>>> i.e. you can isolate your bean and test each method directly. >>>>> >>>>> The Camel Junit on the other hand test the unit as part of a camle route. >>>>> Even if the route is very simple such as >>>>> >>>>> <route> >>>>> <from uri="direct:in"/> >>>>> <to uri="bean:myBean"/> >>>>> <to uri="mock:out"/> >>>>> </route> >>>>> >>>>> Still, It's testing my bean in a context that is more than just using the >>>>> bean methods. >>>>> >>>>> The Camel in Action book 'only' list on benefit of using Camel Junit >>>>> tests, >>>>> namely simplification of the unit tests. I would like to hear what the >>>>> rest >>>>> of you do / think. >>>>> >>>>> - Do you use only Camel Junit tests using routes? >>>>> >>>>> - Do you use 'normal' method oriented JUnit tests for low level tests of >>>>> individual methods combined with Camel Unit tests for >>>>> 'component/application' level testing? >>>>> >>>>> - Do you see a conceptual problem in unit testing using Camel JUnit? >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://camel.465427.n5.nabble.com/Conceptual-correctness-of-using-Camel-Unit-Tests-tp4372286p4372286.html >>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>> >>>> >>>> -- >>>> Christian Schneider >>>> http://www.liquid-reality.de >>>> >>>> CXF and Camel Architect >>>> SOPERA - The Application Integration Division of Talend >>>> http://www.talend.com >>>> >>>> >>> >> >> >> -- >> Christian Schneider >> http://www.liquid-reality.de >> >> CXF and Camel Architect >> SOPERA - The Application Integration Division of Talend >> http://www.talend.com >> > > -- Claus Ibsen ----------------- FuseSource Email: cib...@fusesource.com Web: http://fusesource.com CamelOne 2011: http://fusesource.com/camelone2011/ Twitter: davsclaus Blog: http://davsclaus.blogspot.com/ Author of Camel in Action: http://www.manning.com/ibsen/
