Re: Thoughts about unit testing and mock objects
Gentlements, since there is this nice little guy ([1]), I'd like to come back to this discussion. Adam and I spoke about starting to solve this problem. For JSF infrastructure things we'd like to give Shale a try. For stuff like UIComponent / Rendere there might be JMock useful. Btw. I checked the legal-discuss list and the jMock license (BSD style) fit's fine into the ASF world. -Matthias [1] http://issues.apache.org/jira/browse/ADFFACES-4 On 5/14/06, John Fallows [EMAIL PROTECTED] wrote: On 5/14/06, Adam Winer [EMAIL PROTECTED] wrote: On 5/14/06, John Fallows [EMAIL PROTECTED] wrote: On 5/13/06, Craig McClanahan [EMAIL PROTECTED] wrote: In an example that ought to be relevant for this codebase :-), consider what happens when you want to test the encodeBegin(), encodeChildren(), and encodeEnd() methods of a Renderer implementation. The methods return void, so the output isn't particularly interesting :-). What is interesting, however, is that the methods themselves are going to assume that FacesContext.getResponseWriter() returns somethng usable, even in a unit test scenario. Shale's MockFacesContext, for example, gives you back a mock writer that can write out to a buffer, which you can later examine to determine whether the component created correct markup or not (based on the properties/attributes you set on the component). And, this functionality is needed in all renderer tests, so it makes sense to encapsulate into a separate library that can be debugged once. How do you approach this sort of scenario with dynamic mocks? Nice example, thanks Craig! With dynamic mock objects, you would setup each method call made on the ResponseWriter which is functionality equivalent, although much more verbose. Far more so, to the point that I can hardly imagine anyone writing a test like that. Also, the order of invocations becomes partially important, and partially irrelevant - which attributes are written on which elements are important, which order the attributes are written is irrelevant. A dynamic mock is never going to have enough semantics to capture this requirement. Surprisingly, jMock does have sufficient semantics, but the ResponseWriter buffer approach makes it easier to specify the expected output. That's why the ADF Faces codebase has a handwritten mock for a response writer that very intentionally *is not* the same response writer we use at runtime: http://tinyurl.com/m4zth ... and there's no way that the default mock implementation would capture any of the semantics that this test captures. Since you mention this example, the TestResponseWriter seems to include some implementation-specific stuff, such as optimized removal of empty span elements in the final markup. For isolated unit testing, we probably ought to be unit testing the Renderers to verify that they interact as designed with the mock ResponseWriter, and then separately unit testing our ResponseWriter implementation to make sure it optimizes correctly. Your point is to simplify the effort involved in verifying the result, right? That's pretty useful, although this example doesn't appear to address my question about the behavior that might become incorrect. Simplifying the effort to write a test is crucial, not just pretty useful. Developers are habitually lazy at writing tests, and putting obstacles in their path ensures they won't. Sure. Like I said above, buffering the Renderer output is a good idea that makes it easier to verify the output. For your question about behavior that might be incorrect, say I was mocking UIComponent behaviors for a piece of code that will need to invoke findComponent(). How do I mock findComponent()? Odds are, if I'm writing the test, I mock it to produce exactly the result I want - not necessarily implement the correct algorithm. Do I know that my test is right? Thanks for the example, this is getting closer to what I'm trying to figure out. :-) Suppose there are two codepaths in such a unit test, one that expects a non-null component from findComponent, and another codepath that handles the null case. As a unit test writer, I don't need to be concerned about how findComponent is actually implemented, just that there are only two possible results affecting the codepath, null and non-null. If the unit tests are isolated, then there would be a separate unit test for UIComponentBase to verify the implementation of findComponent. In general, I *think* the only behavior that is relevant to each unit test is the implementation of the method under test. All other participants in that implementation are mocked to control the codepath during execution, giving an opportunity to provide 100% codepath coverage over several unit tests. tc, -john. -- http://apress.com/book/bookDisplay.html?bID=10044 Author: Pro JSF and Ajax: Building Rich Internet Components, Apress
Re: Thoughts about unit testing and mock objects
I do believe as well that jMock doesn't provide enough context for writing proper tests of JSF-behaviour. At leasts whatever tests I have written so far needed some more than just a method to call and the information on whether this method has been called. regards, Martin On 5/13/06, Adam Winer [EMAIL PROTECTED] wrote: On 5/12/06, John Fallows [EMAIL PROTECTED] wrote: On 5/12/06, Craig McClanahan [EMAIL PROTECTED] wrote: On 5/12/06, John Fallows [EMAIL PROTECTED] wrote: On 5/10/06, Adam Winer [EMAIL PROTECTED] wrote: I thought more about this over the last few hours. I think my basic gripe with easymock and mockobject approaches to JSF API objects is that most of the JSF tests I write rarely are concerned specifically with testing how my code is interacting with the JSF API; it's how my code is itself behaving. The former is where mock object suites pay dividends, but when your main concern is in your own code, mock suites seem to get in the way for more than they help. Basically, the Shale test framework seems like a better fit (yeah, handcoded, but that work's done and released...). I don't understand your point. Shale test framework is founded on mock objects. In addition, code we write in things like ADF Faces is going to assume certain behavior on the part of the JSF runtime -- I like to use a mock object framework that behaves enough like the real thing so that I can test the parts of my code that depend on that behavior too, not just static unit tests on individual methods. Also, no one so far seems to have addressed the second question in the original post about how we should provide mock objects for our own code in our unit tests. It probably blurs the line between unit tests and system integration tests a bit, but I like to use the real objects, rather than mocks, whenever I can feasibly do so. Besides having to do less work (not building mocks for classes that can be used both at test time and runtime), this also reduces the risk that I will mistakenly program my own class to the possibly botched behavior of a badly written mock object, giving me a false sense of security when the tests pass ... Well I definitely agree about not wanting to hand-craft mock object implementations! Dynamic mock creation also seems to address that by not requiring an implementation and keeping the recipe for the behavior isolated to the unit test itself. This provides self-documenting tests which are especially useful for anyone trying to learn the semantics of the code. IMHO, unit testing is about verifying the semantics of each codepath in each method, aiming for 100% coverage. This is most easily verified if each codepath is unit tested in isolation, preventing cascading failures and minimizing initialization overhead (only create the objects used by the method codepath). The point about false positives is interesting. Let's assume that we don't have any reusable mock object implementations to possibly botch, instead we have a specification of expected behavior (to possibly botch!) inside each unit test method definition (using jMock). This is, I guess, Objection #1 for me. Why put the burden of defining proper JSF behavior into every test, instead of into a reusable mock implementation? It's not just a question of effort - it's one of correctness, since if that behavior can be independently defined in each test, it can be defined differently in each test. Then, the unit test reads as setup expectations, execute real object method, and verify actual behavior. And here's Objection #2: setup expectations is not a natural programming model. It requires defining criteria in a meta-programming language so that verify actual behavior becomes simply a call to a mock verify() method. Anything that isn't in this meta-language is either impossible or requires writing custom critierion objects which is tedious and verbose. And, my main Objection #3: most tests (not all, but most) that I write in JSF are not concerned with testing the behavior of the JSF objects. That is, I rarely care about testing how I invoked FacesContext or UIComponent or ViewHandler. So the assertions that the mock frameworks give me - method foo() was called once with certain parameters - don't actually help me very much! -- Adam The unit test verifies that the method behaves according to the expected semantics defined in the same method. With a minimal amount of expected behavior to specify in each unit test method, the general approach does not seem that error prone to me. Perhaps some of this setup just needs to be hoisted out of the unit test method in special cases, like Shale does for JSF specification objects. Lastly, I think that the best way to verify in-container behavior is to run the code in the actual container and update the isolated behavior specifications in
Re: Thoughts about unit testing and mock objects
On 5/13/06, Adam Winer [EMAIL PROTECTED] wrote: On 5/12/06, John Fallows [EMAIL PROTECTED] wrote: On 5/12/06, Craig McClanahan [EMAIL PROTECTED] wrote: On 5/12/06, John Fallows [EMAIL PROTECTED] wrote: On 5/10/06, Adam Winer [EMAIL PROTECTED] wrote: I thought more about this over the last few hours. I think my basic gripe with easymock and mockobject approaches to JSF API objects is that most of the JSF tests I write rarely are concerned specifically with testing how my code is interacting with the JSF API; it's how my code is itself behaving. The former is where mock object suites pay dividends, but when your main concern is in your own code, mock suites seem to get in the way for more than they help. Basically, the Shale test framework seems like a better fit (yeah, handcoded, but that work's done and released...). I don't understand your point. Shale test framework is founded on mock objects. In addition, code we write in things like ADF Faces is going to assume certain behavior on the part of the JSF runtime -- I like to use a mock object framework that behaves enough like the real thing so that I can test the parts of my code that depend on that behavior too, not just static unit tests on individual methods. Also, no one so far seems to have addressed the second question in the original post about how we should provide mock objects for our own code in our unit tests. It probably blurs the line between unit tests and system integration tests a bit, but I like to use the real objects, rather than mocks, whenever I can feasibly do so. Besides having to do less work (not building mocks for classes that can be used both at test time and runtime), this also reduces the risk that I will mistakenly program my own class to the possibly botched behavior of a badly written mock object, giving me a false sense of security when the tests pass ... Well I definitely agree about not wanting to hand-craft mock object implementations! Dynamic mock creation also seems to address that by not requiring an implementation and keeping the recipe for the behavior isolated to the unit test itself. This provides self-documenting tests which are especially useful for anyone trying to learn the semantics of the code. IMHO, unit testing is about verifying the semantics of each codepath in each method, aiming for 100% coverage. This is most easily verified if each codepath is unit tested in isolation, preventing cascading failures and minimizing initialization overhead (only create the objects used by the method codepath). The point about false positives is interesting. Let's assume that we don't have any reusable mock object implementations to possibly botch, instead we have a specification of expected behavior (to possibly botch!) inside each unit test method definition (using jMock). This is, I guess, Objection #1 for me. Why put the burden of defining proper JSF behavior into every test, instead of into a reusable mock implementation? It's not just a question of effort - it's one of correctness, since if that behavior can be independently defined in each test, it can be defined differently in each test. The intent is to specify a set of inputs to the method invocation to force it down a particular codepath, then verify the outputs either as a return value, or that methods interaction with passed parameters. Maybe someone can provide an example of the behavior you are talking about that might become incorrect. Craig? Martin? I'd appreciate some insight here. Then, the unit test reads as setup expectations, execute real object method, and verify actual behavior. And here's Objection #2: setup expectations is not a natural programming model. It requires defining criteria in a meta-programming language so that verify actual behavior becomes simply a call to a mock verify() method. Anything that isn't in this meta-language is either impossible or requires writing custom critierion objects which is tedious and verbose. Yeah, you're specifying how the objects should behave during this test. That's the same as Shale Test Framework, except that common top level concepts are pre-initialized during TestCase.setUp(). Interestingly, the jMock mock objects are verified implicitly, so there's no need to call (or forget to call) mock.verify(). And, my main Objection #3: most tests (not all, but most) that I write in JSF are not concerned with testing the behavior of the JSF objects. That is, I rarely care about testing how I invoked FacesContext or UIComponent or ViewHandler. So the assertions that the mock frameworks give me - method foo() was called once with certain parameters - don't actually help me very much! The assertions that jMock gives you are more complete than what you have used before with EasyMock. For example, one can say that the parameters don't
Re: Thoughts about unit testing and mock objects
On 5/13/06, John Fallows [EMAIL PROTECTED] wrote: On 5/13/06, Adam Winer [EMAIL PROTECTED] wrote: On 5/12/06, John Fallows [EMAIL PROTECTED] wrote: On 5/12/06, Craig McClanahan [EMAIL PROTECTED] wrote: On 5/12/06, John Fallows [EMAIL PROTECTED] wrote: On 5/10/06, Adam Winer [EMAIL PROTECTED] wrote: I thought more about this over the last few hours. I think my basic gripe with easymock and mockobject approaches to JSF API objects is that most of the JSF tests I write rarely are concerned specifically with testing how my code is interacting with the JSF API; it's how my code is itself behaving. The former is where mock object suites pay dividends, but when your main concern is in your own code, mock suites seem to get in the way for more than they help. Basically, the Shale test framework seems like a better fit (yeah, handcoded, but that work's done and released...). I don't understand your point. Shale test framework is founded on mock objects. In addition, code we write in things like ADF Faces is going to assume certain behavior on the part of the JSF runtime -- I like to use a mock object framework that behaves enough like the real thing so that I can test the parts of my code that depend on that behavior too, not just static unit tests on individual methods. Also, no one so far seems to have addressed the second question in the original post about how we should provide mock objects for our own code in our unit tests. It probably blurs the line between unit tests and system integration tests a bit, but I like to use the real objects, rather than mocks, whenever I can feasibly do so. Besides having to do less work (not building mocks for classes that can be used both at test time and runtime), this also reduces the risk that I will mistakenly program my own class to the possibly botched behavior of a badly written mock object, giving me a false sense of security when the tests pass ... Well I definitely agree about not wanting to hand-craft mock object implementations! Dynamic mock creation also seems to address that by not requiring an implementation and keeping the recipe for the behavior isolated to the unit test itself. This provides self-documenting tests which are especially useful for anyone trying to learn the semantics of the code. IMHO, unit testing is about verifying the semantics of each codepath in each method, aiming for 100% coverage. This is most easily verified if each codepath is unit tested in isolation, preventing cascading failures and minimizing initialization overhead (only create the objects used by the method codepath). The point about false positives is interesting. Let's assume that we don't have any reusable mock object implementations to possibly botch, instead we have a specification of expected behavior (to possibly botch!) inside each unit test method definition (using jMock). This is, I guess, Objection #1 for me. Why put the burden of defining proper JSF behavior into every test, instead of into a reusable mock implementation? It's not just a question of effort - it's one of correctness, since if that behavior can be independently defined in each test, it can be defined differently in each test. The intent is to specify a set of inputs to the method invocation to force it down a particular codepath, then verify the outputs either as a return value, or that methods interaction with passed parameters. Maybe someone can provide an example of the behavior you are talking about that might become incorrect. Craig? Martin? I'd appreciate some insight here. In an example that ought to be relevant for this codebase :-), consider what happens when you want to test the encodeBegin(), encodeChildren(), and encodeEnd() methods of a Renderer implementation. The methods return void, so the output isn't particularly interesting :-). What is interesting, however, is that the methods themselves are going to assume that FacesContext.getResponseWriter() returns somethng usable, even in a unit test scenario. Shale's MockFacesContext, for example, gives you back a mock writer that can write out to a buffer, which you can later examine to determine whether the component created correct markup or not (based on the properties/attributes you set on the component). And, this functionality is needed in all renderer tests, so it makes sense to encapsulate into a separate library that can be debugged once. How do you approach this sort of scenario with dynamic mocks? Craig
Re: Thoughts about unit testing and mock objects
On 5/12/06, John Fallows [EMAIL PROTECTED] wrote: On 5/12/06, Craig McClanahan [EMAIL PROTECTED] wrote: On the surface, it would appear so, but IANAL and never played one on TV. I had to read that several times to figure it out - does IANAL mean I Am Not A Laywer ? :-) Right the first time :-). It's a pretty common acronym when discussing arcane subjects like licensing on open source lists, because nobody wants to have someone come back later and try to sue you for giving legal advice. Of course, for me it does evoke a small personal smile when *I* use this acronym, because my father is, in fact, a lawyer :-). Craig
Re: Thoughts about unit testing and mock objects
The test suites for myfaces core and tomahawk were refactored for the shale test framework. A *lot* of easymock and mockobject code, some aspect dependencies, and dozens of incomplete api/pojo classes were removed. By the time I was done I had actually removed more code than I had ever committed to the project. There is no learning curve for this stuff and it's in ibiblio. Dennis Byrne -Original Message- From: Adam Winer [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 10, 2006 08:43 PM To: adffaces-dev@incubator.apache.org Subject: Re: Thoughts about unit testing and mock objects JMock does look interesting, but a couple of basic questions: - Is it available in a maven repository? - Is its license compatible? ... and one more general one. The thing that drives me up the wall with the current mock codebase in the ADF Faces tests is that it forces you to say I expect method foo() to be called at least N times, even though with JSF there's rarely any such assurances whether a method will be called or not, and if so how many times. E.g., how often is FacesContext.getViewRoot() called? Once? Twice? 20 times? This leads to extremely brittle tests that are a mess to write and understand. So, to be more brief: does JMock fix this awfulness, or perpetuate it? :) -- Adam On 5/10/06, John Fallows [EMAIL PROTECTED] wrote: The recent delivery ADF Faces codebase with the missing jsf-mock dependency got me thinking... - Do we really need hand-written (or possibly generated) mock implementation classes as a dependency? - What strategy should we follow for our own classes that require mock implementations? For future test development, we might benefit from using jMock which doesn't require a separate set of mock class implementations to mock the interfaces / abstract classes. http://jmock.codehaus.org/getting-started.html Interestingly, jMock can leverage CGLib to manage dynamic proxy creation of abstract classes, and produces extremely readable unit tests that capture the semantics of the code being tested. Does anyone have jMock or mock object experiences to share? tc, -john. -- http://apress.com/book/bookDisplay.html?bID=10044 Author: Pro JSF and Ajax: Building Rich Internet Components, Apress
