Re: How to handle refactoring with TDD and mocking/stubbing

[Allen Rohner]

On Wednesday, January 7, 2015 at 2:49:50 PM UTC-6, Sean Corfield wrote:
>
> On Jan 7, 2015, at 10:21 AM, Allen Rohner > 
> wrote: 
> > As a design rule, I prefer making I/O fns (things that hit the DB or a 
> REST API), 'dumb', and perform all logic/processing in fns that just 
> receive plain data, and return plain data. 
>
>
In my current project, I'm using component + datomic, so I'll talk about 
that here.

 

> I’m curious: do you have wrapper functions for the DB reads/writes (i.e., 
> a domain layer that just wraps the calls to your persistence library as 
> well as managing the connections etc)? 
>
>
Yes. Finding a user looks like

(defn get-user-by-login [db login]
  (d/q '[:find (pull ?u [*]) :in $ ?login :where [?u :user/login ?login]] 
db login))

I'm passing the datomic DB value into the find function. 



Currently we primarily run Clojure as the embedded "Model" in a legacy MVC 
> app so we can’t really use Stuart’s Component (at least, I don’t think we 
> can, because we have a lot of entry points into the Clojure code) but we 
> would like to figure out a cleaner way to separate DB access from our 
> business logic on a per request basis... 
>
>
The important part of component isn't using the actual library, it's the 
philosophy. The most important thing is the signatures on your fns. You 
"just" need to change all of your code to take the DB connection as an 
argument, rather than doing something like:

(defn get-user-by-login [login]
  (d/q '[:find...] db login))

The only difference from the first example is that the DB is no longer an 
argument, so now it's implicit global state. Fixing that is 90% of the 
benefit of component, the rest of component is designed to make starting 
the system simpler. Component may or may not be a good fit for you, but you 
can certainly get most of the benefit on your own. 

Thanks,
Allen


>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Sean Corfield]

On Jan 7, 2015, at 10:21 AM, Allen Rohner  wrote:
> As a design rule, I prefer making I/O fns (things that hit the DB or a REST 
> API), 'dumb', and perform all logic/processing in fns that just receive plain 
> data, and return plain data.

I’m curious: do you have wrapper functions for the DB reads/writes (i.e., a 
domain layer that just wraps the calls to your persistence library as well as 
managing the connections etc)?

Currently we primarily run Clojure as the embedded "Model" in a legacy MVC app 
so we can’t really use Stuart’s Component (at least, I don’t think we can, 
because we have a lot of entry points into the Clojure code) but we would like 
to figure out a cleaner way to separate DB access from our business logic on a 
per request basis...

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Allen Rohner]

On Wednesday, December 31, 2014 8:48:27 AM UTC-6, Jonathon McKitrick wrote:
>
> I use TDD and mocking/stubbing (conjure) to test each layer of my code.  
> The problem is when I change the function signature and the tests do not 
> break, because the mocks/stubs do not know when their argument lists no 
> longer agree with the underlying function they are mocking.  Is there a way 
> to catch this?  Short of a test suite that eschews stubbing in favor of 
> full setup/teardown of DB data for each test?
>

I'll second the avoiding mocks, and avoiding midje votes.  

As for practical advice, I noticed that many times when I had 
mocking/stubbing problems in my tests, it was because my app was too 
tightly complected or stateful. I highly highly recommend using 
https://github.com/stuartsierra/component . Component strongly encourages 
you to program in a style where it's easy to e.g. replace your DB with an 
in-memory instance, and make it obvious which fns need DB access, 
encouraging you to split fns into pure and impure. Also, use core.async to 
communicate between separate components, where it makes sense. 

As a design rule, I prefer making I/O fns (things that hit the DB or a REST 
API), 'dumb', and perform all logic/processing in fns that just receive 
plain data, and return plain data. 

When you follow those two practices, your need for stubbing & mocking will 
drop significantly. As a bonus, your tests usually get faster!

Thanks,
Allen





-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Brian Marick]

Akos Gyimesi wrote:

every function call has a
corresponding mock call in the test, so whenever you modify the
implementation (without even changing the top-level result) you have to
modify the tests as well, and vica-versa. I once encountered a codebase
that had thousands of lines of such "tests", and it was a nightmare to
modify anything in it.


That used to bother me too, until I had an Experience. It was a browser 
front end written in, heaven help me, Objective-J. (Not -C. -J. 
http://en.wikipedia.org/wiki/Objective-J)


Because I am lousy at GUI design, I at some point had to make a big 
revision. I remember distinctly looking at a file of maybe 20 tests and 
realizing I was going to have to rewrite them all. I was discouraged 
until I suddenly realized that every one of those tests represented 
something that had once been important about the UI. They could be seen 
as a record of design choices/decisions. Some of them were relevant to 
the new UI, and I ended up being glad I could look at each test and ask 
myself "Self, what would be an analogous test for the new UI?"


http://atulgawande.com/book/the-checklist-manifesto/

Since then, modifying tests hasn't bothered me as much. (But note: if a 
simple change means modifying a whole lot of tests, you're probably 
doing something wrong.)


In fact, I sometimes point out that I've lived through the transition 
from a time when almost all programmers scorned and despised writing 
tests and the few who did were oddballs, to a time when writing tests is 
- if not as common as NOT writing tests used to be - not at all oddball. 
BUT the same programmers who love writing tests despise REwriting tests. 
Except for a few oddballs. Like me. The vanguard of a new revolution!


Or perhaps not.


Maybe the "big trick" is to use the mocking features for unfinished code
(they are indeed convenient in Midje), but remove them immediately after
the lower-level components are implemented.


I believe some people do that. I don't. The main reason is that doing so 
will very often require spending time setting up data. My coding is very 
hashmap heavy. My test for a function will pass typically pass in a very 
sparse map, containing only the keys referenced by that function. If I 
now replace a call to a mock with a call to the real function (which 
itself calls another real function, etc.), chances are I'll have to 
construct a big old map like the ones that flow through it in production.


I have as strong an aversion to maintaining setup code as other people 
have to maintaining mocking tests.



BTW my current approach is to test mostly without mocks, from as
high-level as reasonably possible:
- If possible, test a module from its public interface
- If this is too much pain (e.g. the execution has too many branches and
it is hard to exercise all of them), descend one level: take the
high-level components of the module and test these parts individually.
Descend another level if it's still too painful, (recurse). Always keep
a few integration tests that exercises the components together.


Interestingly (to me, at least), that's pretty much the approach I took 
in my first book, /The Craft of Software Testing/, 1994. It took some 
ferocious arguments on the C2 wiki around 2000 + buckling down and 
really trying XP-style unit testing for a few thousand lines of Java to 
change my mind. Then it took me many more years to finally figure out 
what the London mocking people were talking about. (I was actually a 
reviewer on the very first paper on mock objects, and I *completely* 
missed the point. I apologized to Steve Freeman for that the first time 
I met him, and he replied, "That's OK - everyone did.")


I don't claim my progression is inevitable for all Right-Thinking 
People. I value some things more than other people, and I worry less 
about some drawbacks than other people. Thus we might make different 
choices.


--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Brian Marick]

Akos Gyimesi wrote:


Now, later that day I decide that I pass the whole user object to the
check-pw function. Maybe I want to use the user ID as a salt, or maybe I
just want to leave the possibility for checking password expiration,
etc. So I modify the test and the implementation of check-pw so that the
first parameter is the user object, not the password hash.

Suddenly my co-worker comes to me saying "hey, I need you on a meeting
right now!" I close my laptop, and an hour later I think "where were
we?..." I run all the tests, and they all pass, so I commit.


What I'll often do in this case is start the change by marking every 
test that contains `check-pw` "pending" (using Midje's `future-fact`). 
Thereafter, running the test suite nags you about unfinished work.


The pending tests might include some set of mock tests, but it will also 
contain direct tests of `check-pw`. (I might also find that some uses of 
`check-pw` are in untested functions.)


Now to coding.

I would typically change the mock tests first. In that case, after you 
return from the interruption, the direct test(s) of `check-pw` will 
remind you that you're not finished.


You might instead work on the direct tests. In that case, the pending 
mock tests will remind you that you're not done.




Things get more complicated with more code. For example, we right now 
have nine (I think) Clojure services, plus a big library ("clojure 
commons") of shared code. When I have to change commons code, I'll often 
start by making a separate function (maybe called something obnoxious 
like check-pw-change-in-progress, maybe something different than 
`check-pw` but still sensible). I gradually migrate services over to the 
new version in the way described above. Eventually the original function 
can be deleted.


I'm sure this strategy is covered in Fowler's book on refactoring.

It's not always that simple. Monger 2.X was a big change from Monger 1.X 
(every function in the API grew an extra argument). Rather than make a 
minimal change to our getset.services.mongo, I did a 
function-generates-many-functions thing. That was more "big bang", and 
there was one problem that only got caught on the staging server by 
manual testing.




In the past, I would sometimes deliberately break the function being 
changed:


 (defn check-pw [])

... and run all the tests. That could tell me two things:

1. "You have an awful lot of "unit" tests that have nothing to do with 
passwords that nevertheless fail when `check-pw` [never mentioned in 
these tests] is changed." That's bad, if only because debugging is 
easier when only one test fails and it's a test that's "close to" the 
code that changed.


2. "None of your end-to-end tests test check the password form." As 
other people have noted, that means there's a class of bug that won't be 
caught by the test suite. On the other hand, end-to-end tests are harder 
to maintain than isolated unit tests. You have a tradeoff-type decision 
to make. In my case, I would almost always decide to forego the 
end-to-end test in this case. Instead, I'd make sure to exercise that 
form a little more thoroughly before I counted the story Done.


(More generally: the closer the code is to the "surface" of the app, the 
less need for end-to-end tests. It's the down deep code that makes me 
nervous. Way back in 1998, I wrote a paper explaining that thinking (in 
a different context): "When Should a Test Be Automated?" 
http://www.exampler.com/testing-com/writings/automate.pdf)


I seem to have drifted out of the habit of doing this. Not sure why. 
Maybe it's because doing in Clojure leads to multiple tests spewing huge 
swaths of stack trace garbage at me, and seeing that depresses me.



Hope this is of use.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Akos Gyimesi]

Thank you for all the responses! To be honest, I hoped that someone
would explain why this mocking style is a good thing, and I just
misunderstand something about the "top-down development" that the Midje
wiki suggests:
  https://github.com/marick/Midje/wiki/The-idea-behind-top-down-development

While this idea sounds perfectly reasonable when you write the code for
the first time, it seems to me that it will make your life very hard
when you try to modify that code later.

I agree with Timothy that the main problem with my example is that these
tests are checking almost nothing. In essence, the tests are just a
"shadow implementation" of the real code: every function call has a
corresponding mock call in the test, so whenever you modify the
implementation (without even changing the top-level result) you have to
modify the tests as well, and vica-versa. I once encountered a codebase
that had thousands of lines of such "tests", and it was a nightmare to
modify anything in it.

The thing that I like in the Midje article though is the process how the
code is written: I usually design things top-down, but implement it
bottom-up because that's what works easily with the tools. It would be
great (at least for me) if top-down thinking could be applied during
implementation as well.

Maybe the "big trick" is to use the mocking features for unfinished code
(they are indeed convenient in Midje), but remove them immediately after
the lower-level components are implemented. In some way it is similar to
the classic TDD cycle when you first write a fake method that satisfies
your test, and then "refactor" it to have a real implementation.

BTW my current approach is to test mostly without mocks, from as
high-level as reasonably possible:
- If possible, test a module from its public interface
- If this is too much pain (e.g. the execution has too many branches and
it is hard to exercise all of them), descend one level: take the
high-level components of the module and test these parts individually.
Descend another level if it's still too painful, (recurse). Always keep
a few integration tests that exercises the components together.
- Try to find balance between not going too deep vs being able to test
effectively. I usually end up with unit-testing a few complex low-level
functions, and have much higher-level tests for the rest of the code.
- Use mocks for undeterministic functions like (rand) or (current-time)
only

Although this approach is maybe far from classic TDD, I found that it
usually results in a test suite that is easy to maintain: the public
interfaces rarely change, so you can refactor most of the implementation
details without touching the tests. It also helps you think in
"contracts", as your focus is on the interface that is used between
components.

Regards,
Akos

On Tue, Jan 6, 2015, at 03:41 PM, Colin Yates wrote:
> +1 - I think we are saying the same thing (not sure if you meant to
> reply to me?)
> 
> On 6 January 2015 at 14:35, Timothy Baldridge 
> wrote:
> > I think the answer to questions like this is that you are testing the wrong
> > thing, or more correctly you are writing incomplete tests.
> >
> > In your example, you stubbed out check-pw. But calling check-pw has a
> > contract, a contract that (at the moment) only exists in your head, but a
> > contract none-the-less. That contract needs to be tested, from both sides.
> > Tests should invoke all instances of check-pw. In addition you should have a
> > test that pairs a login form with a check-pw and runs tests against this
> > system.
> >
> > Some people call these tests "integration tests" or "system tests". But I
> > think of them as contract tests.
> >
> > Here's a diagram of the problem:
> >
> > login -> check-pw
> >
> > I've found that most code that uses mocking will test the login and the
> > check-pw bits, but completely neglect testing the "arrow" between them, and
> > when that happens, you get exactly the experience you described.
> >
> > The other thing I'd like to mention is that I have found it very valuable to
> > sit down and think about what code is actually being hit by a test. In your
> > example, if check-pw and the db are both mocked, what is actually being
> > tested? In your example all you are testing with those functions mocked is
> > that Clojure is capable of compiling a function that calls two other
> > functions. I can't tell you how many times I've looked at mocked tests and
> > realized that the only thing being tested is something like read-string,
> > get, or destructuring.
> >
> > So my personal approach is this: write very coarse tests that exercise the
> > entire system. These will catch the protocol mis-matches. Then if you want
> > more detail for when tests do fail, write more specific tests. In short:
> >
> > System (integration) tests: so I feel good about my codebase
> > Unit (smaller) tests: so I can figure out what went wrong when the larger
> > tests fail.
> >
> > Timothy
> >
> > On Tue, Jan 6,

Re: How to handle refactoring with TDD and mocking/stubbing

[Colin Yates]

+1 - I think we are saying the same thing (not sure if you meant to
reply to me?)

On 6 January 2015 at 14:35, Timothy Baldridge  wrote:
> I think the answer to questions like this is that you are testing the wrong
> thing, or more correctly you are writing incomplete tests.
>
> In your example, you stubbed out check-pw. But calling check-pw has a
> contract, a contract that (at the moment) only exists in your head, but a
> contract none-the-less. That contract needs to be tested, from both sides.
> Tests should invoke all instances of check-pw. In addition you should have a
> test that pairs a login form with a check-pw and runs tests against this
> system.
>
> Some people call these tests "integration tests" or "system tests". But I
> think of them as contract tests.
>
> Here's a diagram of the problem:
>
> login -> check-pw
>
> I've found that most code that uses mocking will test the login and the
> check-pw bits, but completely neglect testing the "arrow" between them, and
> when that happens, you get exactly the experience you described.
>
> The other thing I'd like to mention is that I have found it very valuable to
> sit down and think about what code is actually being hit by a test. In your
> example, if check-pw and the db are both mocked, what is actually being
> tested? In your example all you are testing with those functions mocked is
> that Clojure is capable of compiling a function that calls two other
> functions. I can't tell you how many times I've looked at mocked tests and
> realized that the only thing being tested is something like read-string,
> get, or destructuring.
>
> So my personal approach is this: write very coarse tests that exercise the
> entire system. These will catch the protocol mis-matches. Then if you want
> more detail for when tests do fail, write more specific tests. In short:
>
> System (integration) tests: so I feel good about my codebase
> Unit (smaller) tests: so I can figure out what went wrong when the larger
> tests fail.
>
> Timothy
>
> On Tue, Jan 6, 2015 at 6:26 AM, Colin Yates  wrote:
>>
>> I don't think there is an easy answer here, and note that this is a
>> problem generic to mocking (i.e. not clojure or midje specific).
>>
>> The usual advice applies though:
>>  - do you really need to mock? Unit testing is about the coarseness of
>> granularity which is defined more by cohesion and abstractions than "one
>> function and only this function" (e.g. remove the problem by not overly
>> mocking)
>>  - make everything fail then fix rather than fix and then upgrade (i.e.
>> update every instance of the call/mock to check-pw before the implementation
>> of check-pw).
>>  - Clojure's lack of types means the compiler can't help. Schema or
>> core.typed can. This isn't *the* answer, but I have found it very helpful.
>>
>> As mentioned elsewhere, mocking in general is a very powerful tool, but it
>> is does need wielding carefully. These problems are easier to swallow in
>> strongly typed languages because of IDE support (changing parameters around
>> in Java with IntelliJ is a matter of a few key presses for example).
>>
>> Hope this helps.
>>
>>
>> On Tuesday, 6 January 2015 08:22:36 UTC, Akos Gyimesi wrote:
>>>
>>>
>>> On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote:
>>> >
>>> > > I use TDD and mocking/stubbing (conjure) to test each layer of my
>>> > > code.
>>> > > The problem is when I change the function signature and the tests do
>>> > > not
>>> > > break, because the mocks/stubs do not know when their argument lists
>>> > > no
>>> > > longer agree with the underlying function they are mocking.  Is there
>>> > > a
>>> > > way to catch this?  Short of a test suite that eschews stubbing in
>>> > > favor
>>> > > of full setup/teardown of DB data for each test?
>>> >
>>> > Could you give an example? I use mocks fairly heavily, and I don't seem
>>> > to have this problem. Perhaps it's because I change the tests before
>>> > the
>>> > code?
>>>
>>> Although the subject changed a little bit, I would be still interested
>>> in your approach to refactoring if there is heavy use of mocking. Let me
>>> give you an example:
>>>
>>> Let's say I am writing a login form, trying to use the top-down approach
>>> you described. My approach could be the following:
>>>
>>> (unfinished check-pw)
>>>
>>> (fact "login-form succeeds if user enters the correct password"
>>>   (login-form-success? {:username "admin" :password "secret"}) => true
>>>   (provided
>>> (db/get-user "admin") => (contains (:password "my-secret-hash"))
>>> (check-pw "my-secret-hash" "secret") => true))
>>>
>>> (defn login-form-success? [user-input]
>>>   (let [user (db/get-user (:username user-input))]
>>> (check-pw (:password user) (:password user-input
>>>
>>> Then I finish the check-pw function and everything works.
>>>
>>> Now, later that day I decide that I pass the whole user object to the
>>> check-pw function. Maybe I want to use the user ID as a salt, or maybe I
>>> just 

Re: How to handle refactoring with TDD and mocking/stubbing

[Timothy Baldridge]

I think the answer to questions like this is that you are testing the wrong
thing, or more correctly you are writing incomplete tests.

In your example, you stubbed out check-pw. But calling check-pw has a
contract, a contract that (at the moment) only exists in your head, but a
contract none-the-less. That contract needs to be tested, from both sides.
Tests should invoke all instances of check-pw. In addition you should have
a test that pairs a login form with a check-pw and runs tests against this
system.

Some people call these tests "integration tests" or "system tests". But I
think of them as contract tests.

Here's a diagram of the problem:

login -> check-pw

I've found that most code that uses mocking will test the login and the
check-pw bits, but completely neglect testing the "arrow" between them, and
when that happens, you get exactly the experience you described.

The other thing I'd like to mention is that I have found it very valuable
to sit down and think about what code is actually being hit by a test. In
your example, if check-pw and the db are both mocked, what is actually
being tested? In your example all you are testing with those functions
mocked is that Clojure is capable of compiling a function that calls two
other functions. I can't tell you how many times I've looked at mocked
tests and realized that the only thing being tested is something like
read-string, get, or destructuring.

So my personal approach is this: write very coarse tests that exercise the
entire system. These will catch the protocol mis-matches. Then if you want
more detail for when tests do fail, write more specific tests. In short:

System (integration) tests: so I feel good about my codebase
Unit (smaller) tests: so I can figure out what went wrong when the larger
tests fail.

Timothy

On Tue, Jan 6, 2015 at 6:26 AM, Colin Yates  wrote:

> I don't think there is an easy answer here, and note that this is a
> problem generic to mocking (i.e. not clojure or midje specific).
>
> The usual advice applies though:
>  - do you really need to mock? Unit testing is about the coarseness of
> granularity which is defined more by cohesion and abstractions than "one
> function and only this function" (e.g. remove the problem by not overly
> mocking)
>  - make everything fail then fix rather than fix and then upgrade (i.e.
> update every instance of the call/mock to check-pw before the
> implementation of check-pw).
>  - Clojure's lack of types means the compiler can't help. Schema or
> core.typed can. This isn't *the* answer, but I have found it very helpful.
>
> As mentioned elsewhere, mocking in general is a very powerful tool, but it
> is does need wielding carefully. These problems are easier to swallow in
> strongly typed languages because of IDE support (changing parameters around
> in Java with IntelliJ is a matter of a few key presses for example).
>
> Hope this helps.
>
>
> On Tuesday, 6 January 2015 08:22:36 UTC, Akos Gyimesi wrote:
>>
>>
>> On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote:
>> >
>> > > I use TDD and mocking/stubbing (conjure) to test each layer of my
>> code.
>> > > The problem is when I change the function signature and the tests do
>> not
>> > > break, because the mocks/stubs do not know when their argument lists
>> no
>> > > longer agree with the underlying function they are mocking.  Is there
>> a
>> > > way to catch this?  Short of a test suite that eschews stubbing in
>> favor
>> > > of full setup/teardown of DB data for each test?
>> >
>> > Could you give an example? I use mocks fairly heavily, and I don't seem
>> > to have this problem. Perhaps it's because I change the tests before
>> the
>> > code?
>>
>> Although the subject changed a little bit, I would be still interested
>> in your approach to refactoring if there is heavy use of mocking. Let me
>> give you an example:
>>
>> Let's say I am writing a login form, trying to use the top-down approach
>> you described. My approach could be the following:
>>
>> (unfinished check-pw)
>>
>> (fact "login-form succeeds if user enters the correct password"
>>   (login-form-success? {:username "admin" :password "secret"}) => true
>>   (provided
>> (db/get-user "admin") => (contains (:password "my-secret-hash"))
>> (check-pw "my-secret-hash" "secret") => true))
>>
>> (defn login-form-success? [user-input]
>>   (let [user (db/get-user (:username user-input))]
>> (check-pw (:password user) (:password user-input
>>
>> Then I finish the check-pw function and everything works.
>>
>> Now, later that day I decide that I pass the whole user object to the
>> check-pw function. Maybe I want to use the user ID as a salt, or maybe I
>> just want to leave the possibility for checking password expiration,
>> etc. So I modify the test and the implementation of check-pw so that the
>> first parameter is the user object, not the password hash.
>>
>> Suddenly my co-worker comes to me saying "hey, I need you on a meeting
>> right now!" I close

Re: How to handle refactoring with TDD and mocking/stubbing

[Colin Yates]

I don't think there is an easy answer here, and note that this is a problem 
generic to mocking (i.e. not clojure or midje specific). 

The usual advice applies though:
 - do you really need to mock? Unit testing is about the coarseness of 
granularity which is defined more by cohesion and abstractions than "one 
function and only this function" (e.g. remove the problem by not overly 
mocking)
 - make everything fail then fix rather than fix and then upgrade (i.e. 
update every instance of the call/mock to check-pw before the 
implementation of check-pw).
 - Clojure's lack of types means the compiler can't help. Schema or 
core.typed can. This isn't *the* answer, but I have found it very helpful.

As mentioned elsewhere, mocking in general is a very powerful tool, but it 
is does need wielding carefully. These problems are easier to swallow in 
strongly typed languages because of IDE support (changing parameters around 
in Java with IntelliJ is a matter of a few key presses for example).

Hope this helps.

On Tuesday, 6 January 2015 08:22:36 UTC, Akos Gyimesi wrote:
>
>
> On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote: 
> > 
> > > I use TDD and mocking/stubbing (conjure) to test each layer of my 
> code. 
> > > The problem is when I change the function signature and the tests do 
> not 
> > > break, because the mocks/stubs do not know when their argument lists 
> no 
> > > longer agree with the underlying function they are mocking.  Is there 
> a 
> > > way to catch this?  Short of a test suite that eschews stubbing in 
> favor 
> > > of full setup/teardown of DB data for each test? 
> > 
> > Could you give an example? I use mocks fairly heavily, and I don't seem 
> > to have this problem. Perhaps it's because I change the tests before the 
> > code? 
>
> Although the subject changed a little bit, I would be still interested 
> in your approach to refactoring if there is heavy use of mocking. Let me 
> give you an example: 
>
> Let's say I am writing a login form, trying to use the top-down approach 
> you described. My approach could be the following: 
>
> (unfinished check-pw) 
>
> (fact "login-form succeeds if user enters the correct password" 
>   (login-form-success? {:username "admin" :password "secret"}) => true 
>   (provided 
> (db/get-user "admin") => (contains (:password "my-secret-hash")) 
> (check-pw "my-secret-hash" "secret") => true)) 
>
> (defn login-form-success? [user-input] 
>   (let [user (db/get-user (:username user-input))] 
> (check-pw (:password user) (:password user-input 
>
> Then I finish the check-pw function and everything works. 
>
> Now, later that day I decide that I pass the whole user object to the 
> check-pw function. Maybe I want to use the user ID as a salt, or maybe I 
> just want to leave the possibility for checking password expiration, 
> etc. So I modify the test and the implementation of check-pw so that the 
> first parameter is the user object, not the password hash. 
>
> Suddenly my co-worker comes to me saying "hey, I need you on a meeting 
> right now!" I close my laptop, and an hour later I think "where were 
> we?..." I run all the tests, and they all pass, so I commit. 
>
> Except... I forgot to modify all the invocations of check-pw in both the 
> test and the implementation. Every test pass, so I have no way of 
> finding out the problem without careful code review or by examining the 
> stack traces from the live code. 
>
> While this bug is easy to catch, what if my function is mocked in 
> several places, and I fail to rewrite all of them properly? 
>
> Do you have any advice on what you would have done differently here to 
> avoid this bug? 
>
> Regards, 
> Akos 
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Jonathon McKitrick]

Akos,

that is exactly the kind of problem I'm talking about!  Right down to the
detail about stopping work and returning to the project later, and seeing
all the tests pass!


--
Jonathon McKitrick

On Tue, Jan 6, 2015 at 3:22 AM, Akos Gyimesi  wrote:

>
> On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote:
> >
> > > I use TDD and mocking/stubbing (conjure) to test each layer of my code.
> > > The problem is when I change the function signature and the tests do
> not
> > > break, because the mocks/stubs do not know when their argument lists no
> > > longer agree with the underlying function they are mocking.  Is there a
> > > way to catch this?  Short of a test suite that eschews stubbing in
> favor
> > > of full setup/teardown of DB data for each test?
> >
> > Could you give an example? I use mocks fairly heavily, and I don't seem
> > to have this problem. Perhaps it's because I change the tests before the
> > code?
>
> Although the subject changed a little bit, I would be still interested
> in your approach to refactoring if there is heavy use of mocking. Let me
> give you an example:
>
> Let's say I am writing a login form, trying to use the top-down approach
> you described. My approach could be the following:
>
> (unfinished check-pw)
>
> (fact "login-form succeeds if user enters the correct password"
>   (login-form-success? {:username "admin" :password "secret"}) => true
>   (provided
> (db/get-user "admin") => (contains (:password "my-secret-hash"))
> (check-pw "my-secret-hash" "secret") => true))
>
> (defn login-form-success? [user-input]
>   (let [user (db/get-user (:username user-input))]
> (check-pw (:password user) (:password user-input
>
> Then I finish the check-pw function and everything works.
>
> Now, later that day I decide that I pass the whole user object to the
> check-pw function. Maybe I want to use the user ID as a salt, or maybe I
> just want to leave the possibility for checking password expiration,
> etc. So I modify the test and the implementation of check-pw so that the
> first parameter is the user object, not the password hash.
>
> Suddenly my co-worker comes to me saying "hey, I need you on a meeting
> right now!" I close my laptop, and an hour later I think "where were
> we?..." I run all the tests, and they all pass, so I commit.
>
> Except... I forgot to modify all the invocations of check-pw in both the
> test and the implementation. Every test pass, so I have no way of
> finding out the problem without careful code review or by examining the
> stack traces from the live code.
>
> While this bug is easy to catch, what if my function is mocked in
> several places, and I fail to rewrite all of them properly?
>
> Do you have any advice on what you would have done differently here to
> avoid this bug?
>
> Regards,
> Akos
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/T8fIW27kDYE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Akos Gyimesi]

On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote:
> 
> > I use TDD and mocking/stubbing (conjure) to test each layer of my code.
> > The problem is when I change the function signature and the tests do not
> > break, because the mocks/stubs do not know when their argument lists no
> > longer agree with the underlying function they are mocking.  Is there a
> > way to catch this?  Short of a test suite that eschews stubbing in favor
> > of full setup/teardown of DB data for each test?
> 
> Could you give an example? I use mocks fairly heavily, and I don't seem 
> to have this problem. Perhaps it's because I change the tests before the 
> code?

Although the subject changed a little bit, I would be still interested
in your approach to refactoring if there is heavy use of mocking. Let me
give you an example:

Let's say I am writing a login form, trying to use the top-down approach
you described. My approach could be the following:

(unfinished check-pw)

(fact "login-form succeeds if user enters the correct password"
  (login-form-success? {:username "admin" :password "secret"}) => true
  (provided
(db/get-user "admin") => (contains (:password "my-secret-hash"))
(check-pw "my-secret-hash" "secret") => true))

(defn login-form-success? [user-input]
  (let [user (db/get-user (:username user-input))]
(check-pw (:password user) (:password user-input

Then I finish the check-pw function and everything works.

Now, later that day I decide that I pass the whole user object to the
check-pw function. Maybe I want to use the user ID as a salt, or maybe I
just want to leave the possibility for checking password expiration,
etc. So I modify the test and the implementation of check-pw so that the
first parameter is the user object, not the password hash.

Suddenly my co-worker comes to me saying "hey, I need you on a meeting
right now!" I close my laptop, and an hour later I think "where were
we?..." I run all the tests, and they all pass, so I commit.

Except... I forgot to modify all the invocations of check-pw in both the
test and the implementation. Every test pass, so I have no way of
finding out the problem without careful code review or by examining the
stack traces from the live code.

While this bug is easy to catch, what if my function is mocked in
several places, and I fail to rewrite all of them properly?

Do you have any advice on what you would have done differently here to
avoid this bug?

Regards,
Akos

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Timothy Baldridge]

>>  I think you are coming across a bit strong

That's probably true. At the end of the day I believe that tests should be
written in the same language with the same semantics as the code they are
testing. Midje does not recommend this. It is a multi-thousand line
compiler that transforms a DSL into Clojure, introducing new semantics, and
syntax. Simplicity always wins in my book. I'll take more verbose tests any
day over a custom compiler wrapped in a macro.

Timothy

On Mon, Jan 5, 2015 at 8:36 AM, Sam Ritchie  wrote:

> Agreed, Timothy - obviously the mental model gets more tangled when state
> mocking comes into play, but the fact is, sometimes you don't have the
> option (right away) of rewriting the code you're testing.
>
> Midje has been great for the Cascalog community:
> http://www.samritchie.io/testing-cascalog-with-midje/
> http://www.samritchie.io/cascalog-testing-2-0/
>
> The state mocking is just one piece of Midje. A bunch of its other
> features, such as its collection checkers and chatty checkers, are
> excellent.
>
> Colin Yates wrote:
>
> I have thousands of lines of tests written using Midje and it was the
> second one I turned to when I started using Clojure full-time a couple of
> years ago. I think it would be fairer to say that Midje is powerful enough
> to hang yourself, but that doesn't make that power wrong. This is the good
> old power/not power dilema and caution should dfefinitely be used by
> newbies using Midje, particularly established OO developers to ensure they
> don't mis-use Midje's tools as a bridge to stay in the OO paradigm.
>
>
> --
> Sam Ritchie (@sritchie)
> Paddleguru Co-Founder
> 703.863.8561
> www.paddleguru.com
> Twitter  // Facebook
> 
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Sam Ritchie]

Agreed, Timothy - obviously the mental model gets more tangled when 
state mocking comes into play, but the fact is, sometimes you don't have 
the option (right away) of rewriting the code you're testing.


Midje has been great for the Cascalog community:
http://www.samritchie.io/testing-cascalog-with-midje/
http://www.samritchie.io/cascalog-testing-2-0/

The state mocking is just one piece of Midje. A bunch of its other 
features, such as its collection checkers and chatty checkers, are 
excellent.


Colin Yates wrote:
I have thousands of lines of tests written using Midje and it was the 
second one I turned to when I started using Clojure full-time a couple 
of years ago. I think it would be fairer to say that Midje is powerful 
enough to hang yourself, but that doesn't make that power wrong. This 
is the good old power/not power dilema and caution should dfefinitely 
be used by newbies using Midje, particularly established OO developers 
to ensure they don't mis-use Midje's tools as a bridge to stay in the 
OO paradigm.


--
Sam Ritchie (@sritchie)
Paddleguru Co-Founder
703.863.8561
www.paddleguru.com 
Twitter // Facebook 



--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Colin Yates]

(Happy new year all!)

I have thousands of lines of tests written using Midje and it was the 
second one I turned to when I started using Clojure full-time a couple of 
years ago. I think it would be fairer to say that Midje is powerful enough 
to hang yourself, but that doesn't make that power wrong. This is the good 
old power/not power dilema and caution should dfefinitely be used by 
newbies using Midje, particularly established OO developers to ensure they 
don't mis-use Midje's tools as a bridge to stay in the OO paradigm.

Having said that, I don't use midje anymore simply because of the IDE 
support. I did encounter some minor issues (like gettings facts working in 
macros for example) but that was almost certainly my issue and if I stayed 
on the "beaten track" as it were it was a delight to use and rock solid.

I gotta say Timothy, I think you are coming across a bit strong - it might 
not float your boat, but you need to back up "abomination". At the moment 
you risk people writing off your experience as "mis-used the tools, got in 
a mess and then blamed the tools" which would be a shame as your "paradigm 
shift/mental model" point is valid.

Brian - Midje rocks - keep up the good work ;).

On Saturday, 3 January 2015 15:39:08 UTC, tbc++ wrote:
>
> >> Hacker News notwithstanding, "idiosyncratic interface" is not a 
> synonym for abomination.
>
> True, however, reaching into code, and re-deffing a function's definition 
> is. Not only is it not thread-safe (removing the possibility of ever 
> running two tests in parallel), but it also hides main problem. If you need 
> to use Midje's "providing" you wrote your code wrong. If your only way to 
> test a given bit of code is to re-deff a var, then that var should be 
> abstracted into a different component. Because at the end of the day, I 
> don't care if 'foo' makes a call to 'bar', all I really care about is that 
> 'foo' saves data to 'baz'.
>
> Stuff like with-redefs and providing muck with a developer's mental model 
> of the source code. So instead of being able to say "well foo calls baz 
> here, so this should work". They have to think "well foo calls baz unless 
> someone re-deffs it, in which case I haven't a clue what's going to happen, 
> so let me go check every test that calls foo, or calls something that calls 
> foo to make sure it isn't overriding baz somehow". 
>
> >> Lots of people use Midje for production code without your problems.
>
> Not in my experience. I've used Midje in two fairly large codebases (not 
> including mine, on these I was brought in as a consultant). And not only 
> were the tests brittle (doing simple refactoring to the codebase would 
> break many tests unrelated to that code, and that weren't really bugs at 
> all), but the developers themselves mentioned more than once that they wish 
> they had gone with a simpler testing framework. 
>
> So perhaps that is the problem. Midje is "easy", but not "simple". New 
> Clojure developers pick up Midje because it has all the bells and whistles, 
> but they lack the experience to use it properly, and so they end up with 
> unmaintainable code. And so I assert that a simpler library, something that 
> only provides deftest, assert and run-tests forces developers to think 
> about the best way to test something, and to write their own macros (as 
> patterns emerge). 
>
> But hey, don't take my word for it, run a poll, see what users of Midje 
> think and would like to see. Include the factors I mentioned above (# of 
> years using Clojure, # of years using Midje, do they still use it, etc). 
>
> Timothy
>
> On Fri, Jan 2, 2015 at 6:55 PM, Brian Marick  > wrote:
>
>>
>>
>> Timothy Baldridge wrote:
>>
>>> I don't recommend Midje at all. Many of the framework's mocking
>>> facilities (such as providing) are abominations.
>>>
>>
>> Hacker News notwithstanding, "idiosyncratic interface" is not a synonym 
>> for abomination.
>>
>>  It may look cute,
>>> but I've lost countless hours to bugs and unexpected behavior related to
>>> Midje.
>>>
>>
>> An example would be handy right about now.
>>
>> As the author of Midje, I protest. Lots of people use Midje for 
>> production code without your problems. I've been using on production code 
>> for coming on two years now.
>>
>> C'mon: all of the main Clojure testing frameworks are opinionated, each 
>> in different ways. They're all perfectly competent at what they do. They 
>> don't hide bugs. They all have deterministic behavior. None of them are all 
>> that complicated.
>>
>>  
>>> The pattern I do recommend is to break your system up into components.
>>> Each component should be testable by mocking its dependencies and then
>>> testing only that component. In a final test, stitch together all your
>>> components and run system tests against all components at once.
>>>
>>
>> I think this is good advice.
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post t

Re: How to handle refactoring with TDD and mocking/stubbing

[Brian Marick]

Timothy Baldridge wrote:

Stuff like with-redefs and providing muck with a developer's mental
model of the source code. So instead of being able to say "well foo
calls baz here, so this should work". They have to think "well foo calls
baz unless someone re-deffs it, in which case I haven't a clue what's
going to happen, so let me go check every test that calls foo, or calls
something that calls foo to make sure it isn't overriding baz somehow".


Here I have to disagree. Technically, I think the description is wrong, 
but the bit about "mental model" is more important. Midje's mocking 
behavior is directly inspired by /Structure and Interpretation of 
Computer Programs/' idea of "programming by wishful thinking". You write 
your code assuming that whatever convenience functions you need already 
exist. Then you "discover" they don't and go and write them.


Midje (and similar tools) add two things to that idea:
1. they let you run the function you're working on and see it do 
something right (or wrong) before you've finished all the functions it 
uses.
2. the test cases for the current function contain examples of how its 
missing functions are supposed to behave.


At the end of this, your program is a lattice of functions that use each 
other. You also have a collection of tests that document the assumptions 
functions make about the functions they use. See 
https://github.com/marick/Midje/wiki/The-idea-behind-top-down-development


It turns out to be useful when the functions so described are - and this 
is fuzzy - "meaningful ideas in the domain of the program". That is, 
they are "about" some part of the problem the program is supposed to 
solve. When they're not - when they're just "about" the implementation 
of the function that calls them, you get the sort of fragile tests I 
think you're complaining about.


It shouldn't be surprising that mocks require some skill, experience, 
and study to use correctly. Just like, oh, actors or logic programming 
or lazy sequences, they're a simple idea that has some subtleties and 
lore and non-obvious consequences.


I recommend reading /Growing Object-Oriented Software: Driven by Tests/. 
The ideas are readily translated to functional languages; I actually 
think they work better there.


--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Brian Marick]

Timothy Baldridge wrote:

I assert that a simpler library, something that only provides deftest,
assert and run-tests forces developers to think about the best way to
test something, and to write their own macros (as patterns emerge)


A not-unreasonable idea. Midje supports that by also running 
clojure.test tests, making it easy to migrate when you get tired of 
writing your own functions to compare expected to actual results in 
different ways.


It's also reasonable to simply not use the mocking parts of Midje (or 
other frameworks). Then you'll have clojure.test with a syntax you may 
hate or like, plus more checker functions. The documentation since 1.5 
should make that easy.


--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Timothy Baldridge]

>> Hacker News notwithstanding, "idiosyncratic interface" is not a synonym
for abomination.

True, however, reaching into code, and re-deffing a function's definition
is. Not only is it not thread-safe (removing the possibility of ever
running two tests in parallel), but it also hides main problem. If you need
to use Midje's "providing" you wrote your code wrong. If your only way to
test a given bit of code is to re-deff a var, then that var should be
abstracted into a different component. Because at the end of the day, I
don't care if 'foo' makes a call to 'bar', all I really care about is that
'foo' saves data to 'baz'.

Stuff like with-redefs and providing muck with a developer's mental model
of the source code. So instead of being able to say "well foo calls baz
here, so this should work". They have to think "well foo calls baz unless
someone re-deffs it, in which case I haven't a clue what's going to happen,
so let me go check every test that calls foo, or calls something that calls
foo to make sure it isn't overriding baz somehow".

>> Lots of people use Midje for production code without your problems.

Not in my experience. I've used Midje in two fairly large codebases (not
including mine, on these I was brought in as a consultant). And not only
were the tests brittle (doing simple refactoring to the codebase would
break many tests unrelated to that code, and that weren't really bugs at
all), but the developers themselves mentioned more than once that they wish
they had gone with a simpler testing framework.

So perhaps that is the problem. Midje is "easy", but not "simple". New
Clojure developers pick up Midje because it has all the bells and whistles,
but they lack the experience to use it properly, and so they end up with
unmaintainable code. And so I assert that a simpler library, something that
only provides deftest, assert and run-tests forces developers to think
about the best way to test something, and to write their own macros (as
patterns emerge).

But hey, don't take my word for it, run a poll, see what users of Midje
think and would like to see. Include the factors I mentioned above (# of
years using Clojure, # of years using Midje, do they still use it, etc).

Timothy

On Fri, Jan 2, 2015 at 6:55 PM, Brian Marick  wrote:

>
>
> Timothy Baldridge wrote:
>
>> I don't recommend Midje at all. Many of the framework's mocking
>> facilities (such as providing) are abominations.
>>
>
> Hacker News notwithstanding, "idiosyncratic interface" is not a synonym
> for abomination.
>
>  It may look cute,
>> but I've lost countless hours to bugs and unexpected behavior related to
>> Midje.
>>
>
> An example would be handy right about now.
>
> As the author of Midje, I protest. Lots of people use Midje for production
> code without your problems. I've been using on production code for coming
> on two years now.
>
> C'mon: all of the main Clojure testing frameworks are opinionated, each in
> different ways. They're all perfectly competent at what they do. They don't
> hide bugs. They all have deterministic behavior. None of them are all that
> complicated.
>
>
>> The pattern I do recommend is to break your system up into components.
>> Each component should be testable by mocking its dependencies and then
>> testing only that component. In a final test, stitch together all your
>> components and run system tests against all components at once.
>>
>
> I think this is good advice.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit ht

Re: How to handle refactoring with TDD and mocking/stubbing

[Brian Marick]

Timothy Baldridge wrote:

I don't recommend Midje at all. Many of the framework's mocking
facilities (such as providing) are abominations.


Hacker News notwithstanding, "idiosyncratic interface" is not a synonym 
for abomination.



It may look cute,
but I've lost countless hours to bugs and unexpected behavior related to
Midje.


An example would be handy right about now.

As the author of Midje, I protest. Lots of people use Midje for 
production code without your problems. I've been using on production 
code for coming on two years now.


C'mon: all of the main Clojure testing frameworks are opinionated, each 
in different ways. They're all perfectly competent at what they do. They 
don't hide bugs. They all have deterministic behavior. None of them are 
all that complicated.




The pattern I do recommend is to break your system up into components.
Each component should be testable by mocking its dependencies and then
testing only that component. In a final test, stitch together all your
components and run system tests against all components at once.


I think this is good advice.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Brian Marick]

I use TDD and mocking/stubbing (conjure) to test each layer of my code.
The problem is when I change the function signature and the tests do not
break, because the mocks/stubs do not know when their argument lists no
longer agree with the underlying function they are mocking.  Is there a
way to catch this?  Short of a test suite that eschews stubbing in favor
of full setup/teardown of DB data for each test?


Could you give an example? I use mocks fairly heavily, and I don't seem 
to have this problem. Perhaps it's because I change the tests before the 
code?


That said, I don't mock out our persistent storage tests. That is, we 
have facade namespaces over Monger (Mongo), Korma (Postgres), and Redis 
(Carmine). The facades' tests go directly to the database. This is 
plenty fast enough, and I rarely find writing the setup code much of a 
burden. Part of the reason is that we explicitly allow "partially 
constructed data" (as opposed to the OO advice that all constructors 
should produce objects that satisfy all the invariants). The benefits 
exceed the dangers, I believe.


Note: this is consistent with the advice of some of the "mockish TDD" 
old-timers: "only mock objects you own".


A fair number of tests for the code that uses the facades goes to the 
database too. Whether I use mocks or not is really a matter of my own 
convenience.


--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Sean Corfield]

On Dec 31, 2014, at 8:24 AM, Timothy Baldridge  wrote:
> This is one of the main reasons why I try to stay clear of heavy use of 
> invasive mocks. I can't tell you the amount of times I've looked at code and 
> realized that because of mocks nothing was really being tested at all. 
> Instead, think of mocks as the terminator in a test chain. That is to say, 
> they only exist to allow the program to continue executing. Tests should only 
> validate the behavior up to the call to the mock'd function, no further. 
> Otherwise you're testing your mock, and that causes all sorts of problems (as 
> you discovered).

I agree with Timothy here: we have about 6kloc test code (for a production code 
base of about 24kloc) and only have 25 with-redefs to stub just 8 functions in 
our Expectations code. In addition we use Expectations’ side-effects to mock 
just one function in four tests. Whenever we find ourselves needing to stub or 
mock in tests, we usually try to refactor the code so it’s easier to test fully 
without that.

As for DB tear down & rebuild: we do a full tear down & rebuild at the start of 
running our suite - to create a baseline of test data - and have a few tests 
that tear down & rebuild their own specific test data but we try to keep this 
to a minimum (since it slows testing down). For some of the DB-heavy sections 
of our application, this is inevitable since we need to ensure all tests can be 
run multiple times without affecting their result, but we think carefully about 
any tests that have to do this.

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Timothy Baldridge]

This is one of the main reasons why I try to stay clear of heavy use of
invasive mocks. I can't tell you the amount of times I've looked at code
and realized that because of mocks nothing was really being tested at all.
Instead, think of mocks as the terminator in a test chain. That is to say,
they only exist to allow the program to continue executing. Tests should
only validate the behavior up to the call to the mock'd function, no
further. Otherwise you're testing your mock, and that causes all sorts of
problems (as you discovered).

I don't recommend Midje at all. Many of the framework's mocking facilities
(such as providing) are abominations. You shouldn't go around mucking with
functions and re-deffing them on the fly. It may look cute, but I've lost
countless hours to bugs and unexpected behavior related to Midje. IMO, stay
clear of that.

The pattern I do recommend is to break your system up into components. Each
component should be testable by mocking its dependencies and then testing
only that component. In a final test, stitch together all your components
and run system tests against all components at once. If you do what Ashton
suggests and use temporary databases, this will work quite well. This way
you get the detailed errors of unit tests when things fail inside a
component, as well as contractual tests between components.

I highly recommend Stuart Sierra's Component Library for all of this:
https://github.com/stuartsierra/component It forces you to define the
dependencies of each component, and so mocking or stubbing out a component
is almost trivial.

Timothy Baldridge

On Wed, Dec 31, 2014 at 8:13 AM, Ashton Kemerling  wrote:

> I've always done the full database setup and tear down thing, but that's
> made sufficiently performant with datomics in memory store. Consider using
> transactions to isolate tests, or use Midje, which is more designed for
> this kind of usage.
>
> --Ashton
>
> Sent from my iPhone
>
> On Dec 31, 2014, at 9:48 AM, Jonathon McKitrick 
> wrote:
>
> I use TDD and mocking/stubbing (conjure) to test each layer of my code.
> The problem is when I change the function signature and the tests do not
> break, because the mocks/stubs do not know when their argument lists no
> longer agree with the underlying function they are mocking.  Is there a way
> to catch this?  Short of a test suite that eschews stubbing in favor of
> full setup/teardown of DB data for each test?
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>  --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to handle refactoring with TDD and mocking/stubbing

[Ashton Kemerling]

I've always done the full database setup and tear down thing, but that's made 
sufficiently performant with datomics in memory store. Consider using 
transactions to isolate tests, or use Midje, which is more designed for this 
kind of usage. 

--Ashton

Sent from my iPhone

> On Dec 31, 2014, at 9:48 AM, Jonathon McKitrick  wrote:
> 
> I use TDD and mocking/stubbing (conjure) to test each layer of my code.  The 
> problem is when I change the function signature and the tests do not break, 
> because the mocks/stubs do not know when their argument lists no longer agree 
> with the underlying function they are mocking.  Is there a way to catch this? 
>  Short of a test suite that eschews stubbing in favor of full setup/teardown 
> of DB data for each test?
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to handle refactoring with TDD and mocking/stubbing

[Jonathon McKitrick]

I use TDD and mocking/stubbing (conjure) to test each layer of my code.  
The problem is when I change the function signature and the tests do not 
break, because the mocks/stubs do not know when their argument lists no 
longer agree with the underlying function they are mocking.  Is there a way 
to catch this?  Short of a test suite that eschews stubbing in favor of 
full setup/teardown of DB data for each test?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.