[Tutor] Unit testing infinite loops
Hello tutors
I've constructed an example which shows a problem I'm having testing a real
world program and would like to run it past you.
tutor_question.py
--
# -*- coding: utf-8 -*-
import sys
import threading
import time
class Time_Printer(threading.Thread):
def run(self):
for i in range(60):
print('%s - %s' % (self, time.ctime(time.time(
time.sleep(1)
class Infinite_Loop_Tutor_Question(object):
def start_A(self):
thread = Time_Printer()
thread.daemon = True
thread.start()
print('Started %s' % (thread))
def start_B(self):
thread = Time_Printer()
thread.daemon = True
thread.start()
print('Started %s' % (thread))
def run_forever(self):
self.start_A()
time.sleep(0.5)
self.start_B()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
print(Caught Keyboard Interrupt...)
sys.exit(0)
if __name__ == '__main__':
infinite_loop = Infinite_Loop_Tutor_Question()
infinite_loop.run_forever()
--
In my example above, testing the everything but the run_forever method is
trivial.
So on to my question... The run_forever method essentially just fires up a
bunch of threads to serve various purposes and then waits for CTRL-C to
terminate the entire program. Testing this at the moment is very difficult
because the unit test ends up in the infinite loop. So, would a better idea
be to create an attribute, set it to True and then do
try:
while self.attribute:
time.sleep(1)
except KeyboardInterrupt:
...
My unit test could then set the attribute. However I'd still have the
problem of how I get from the unit test line that fires up the method to
the next line to change the attribute.
So how should the run_forever method be written so that it's testable, or
if it's testable as is, how would I test it?
And please, no comments about syntax, clean exits of threads, thread
communication, resources, or even the need for testing the run_forever
method. In my test I want to test that it makes the relevant calls and then
enters the infinite loop at which point I want to terminate it.
Thanks in advance, and hopefully there are no formatting issues this time.
--
James
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unit testing infinite loops
On 01/31/2014 12:31 PM, James Chapman wrote: try: while self.attribute: time.sleep(1) except KeyboardInterrupt: Maybe I'm missing apoint or reasoning wrongly, but I'd rather do: while self.attribute: try: time.sleep(1) except KeyboardInterrupt: ... or something like that however, (I don't know whether one can interrupt while sleeping) d ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unit testing infinite loops
On Fri, Jan 31, 2014 at 01:10:03PM +0100, spir wrote: I don't know whether one can interrupt while sleeping py from time import sleep py sleep(60*60*24*365) # 1 year Traceback (most recent call last): File stdin, line 1, in module KeyboardInterrupt Yes you can. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unit testing infinite loops
On Fri, Jan 31, 2014 at 6:31 AM, James Chapman ja...@uplinkzero.com wrote: try: while self.attribute: time.sleep(1) except KeyboardInterrupt: ... My unit test could then set the attribute. However I'd still have the problem of how I get from the unit test line that fires up the method to the next line to change the attribute. You could add a method that toggles the attribute, and use a threading.Timer to run it after a set interval. if it's testable as is, how would I test it? CPython 2.3+ can interrupt the main thread from another thread using the built-in function `_thread.interrupt_main`: http://docs.python.org/3/library/_thread#_thread.interrupt_main import _thread _thread.interrupt_main() Traceback (most recent call last): File stdin, line 1, in module KeyboardInterrupt It's also implemented in PyPy, but not in Jython. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unit testing infinite loops
Hmm...
Here is an example of how I'm currently trying to test it:
test_tutor_question.py
-
# -*- coding: utf-8 -*-
import unittest
import mock
from tutor_question import Infinite_Loop_Tutor_Question
class Test_Infinite_Loop_Tutor_Question(unittest.TestCase):
def test_run_forever(self):
with mock.patch('tutor_question.Infinite_Loop_Tutor_Question.start_A')
as start_A:
with
mock.patch('tutor_question.Infinite_Loop_Tutor_Question.start_B') as
start_B:
inf_loop = Infinite_Loop_Tutor_Question()
print start_A.call_count
print start_B.call_count
inf_loop.run_forever()
inf_loop.interrupt_main()
print start_A.call_count
print start_B.call_count
if __name__ == __main__:
unittest.main()
-
As you can see if you run this, the test doesn't reach the lines below
inf_loop.run_forever().
So ideally, I'd need a way of injecting a keyboard interrupt into the
method, I could then check that the exception was handled and that the
start_A and start_B calls were made.
** Obviously the print lines will be substituted for some kind of
assert lines **
FYI I'm using CPython 2.7.something
--
James
On 31 January 2014 12:57, eryksun eryk...@gmail.com wrote:
On Fri, Jan 31, 2014 at 6:31 AM, James Chapman ja...@uplinkzero.com wrote:
try:
while self.attribute:
time.sleep(1)
except KeyboardInterrupt:
...
My unit test could then set the attribute. However I'd still have the
problem of how I get from the unit test line that fires up the method to the
next line to change the attribute.
You could add a method that toggles the attribute, and use a
threading.Timer to run it after a set interval.
if it's testable as is, how would I test it?
CPython 2.3+ can interrupt the main thread from another thread using
the built-in function `_thread.interrupt_main`:
http://docs.python.org/3/library/_thread#_thread.interrupt_main
import _thread
_thread.interrupt_main()
Traceback (most recent call last):
File stdin, line 1, in module
KeyboardInterrupt
It's also implemented in PyPy, but not in Jython.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unit testing infinite loops
On Fri, Jan 31, 2014 at 11:31:49AM +, James Chapman wrote: Hello tutors I've constructed an example which shows a problem I'm having testing a real world program and would like to run it past you. [...] class Infinite_Loop_Tutor_Question(object): def run_forever(self): self.start_A() time.sleep(0.5) self.start_B() try: while True: time.sleep(1) except KeyboardInterrupt: print(Caught Keyboard Interrupt...) sys.exit(0) [...] In my example above, testing the everything but the run_forever method is trivial. So on to my question... The run_forever method essentially just fires up a bunch of threads to serve various purposes and then waits for CTRL-C to terminate the entire program. Testing this at the moment is very difficult because the unit test ends up in the infinite loop. So, would a better idea be to create an attribute, set it to True and then do try: while self.attribute: time.sleep(1) except KeyboardInterrupt: ... That probably won't hurt. My unit test could then set the attribute. However I'd still have the problem of how I get from the unit test line that fires up the method to the next line to change the attribute. So how should the run_forever method be written so that it's testable, or if it's testable as is, how would I test it? What are you trying to test? You don't just test a method, you test *something specific* about the method. So what specifically are you trying to test? And please, no comments about syntax, clean exits of threads, thread communication, resources, or even the need for testing the run_forever method. In my test I want to test that it makes the relevant calls and then enters the infinite loop at which point I want to terminate it. Ah, you see, now you have a problem. Consider this function: def long_calc(): time.sleep(60*60*24*365) return 1 How do I test that the function returns 1? As given, I can't really, not unless I wait a whole year for the sleep() to return. So what I can do is split the function into two pieces: def _sleep_a_year(): time.sleep(60*60*24*365) def _do_calculation(): return 1 def long_calc(): _sleep_a_year() return _do_calculation() Now I can unit-test the _do_calculation function, and long_calc() is now simple enough that I don't really need to unit-test it. (Unit testing should not be treated as a religion. You test what you can. Any testing is better than nothing, and if there are some parts of the program which are too hard to test automatically, don't test them automatically.) Or, I can monkey-patch the time.sleep function. Before running my test_long_calc unit-test, I do this: import time time.sleep = lambda n: None and then restore it when I'm done. But like all monkey-patching, that's risky -- what if the calculation relies on time.sleep somewhere else? (Perhaps it calls a function, which calls another function in a module somewhere, which calls a third module, which needs time.sleep.) So monkey-patching should be a last resort. Another alternative is to write the function so it can be tested using a mock: def long_calc(sleeper=time.sleep): sleeper(60*60*24*365) return 1 Then I can test it like this: assert stupid(lambda n: None) == 1 where the lambda acts as a mock-up for the real sleep function. Let's look at your method. As given, it's too hard to test. Maybe you could write a unit test which fires off another thread, which then sleeps for a few seconds before (somehow!) sending a KeyboardInterrupt to the main thread. But that's hard to explain and harder to do, and I really wouldn't want to rely on something so fiddly. So let's re-design the method with testing in mind. First, pull out the part that does the infinite loop: def do_infinite_loop(): # Loop forever. Sleep a bit to avoid hogging the CPU. while True: time.sleep(1) That's *so simple* that it doesn't need a test. The body of the method is two short, easy lines, plus a comment. If somebody can read that and be unsure whether or not it works correctly, they're in trouble. But, if you like, you can make it more complicated. Have the method check for a magic global variable, or a instance attribute, or something: def do_infinite_loop(): # Loop forever. Sleep a bit to avoid hogging the CPU. # Perhaps not forever. if hasattr(self, 'DONT_LOOP_FOREVER'): x = 10 while x 0: time.sleep(1) x -= 1 else: while True: time.sleep(1) Yuck. Now you have added enough complication that it is no longer obvious that the method works, and while you can test the non-infinite loop part, you still can't test the infinite loop part. No, better to stick with the simplest thing that works. Now for the rest of
Re: [Tutor] Unit testing infinite loops
Thanks Steven! You've raised a few valid points, mostly that the run_forever method should be broken up. I like the principle of a method doing just one thing and for whatever reason I didn't apply that thinking to this method as it's the master loop (even though it does nothing). So for starters I'll fix that. Breaking everything up makes testing easier, which in turn makes development easier. On the note of what doesn't need a test... The question of coverage always comes up when unit testing is mentioned and I read an interesting blog article once about it. It basically said: Assume you have 85% coverage on a program that consists of 1,000 lines. That's 150 lines which are not tested. If those lines are print lines, sleep lines, getters etc it's not really a problem. But what happens when you scale that up. 1,000,000 lines of code lets say (not unheard of, although in python that would be out of this world big). You now end up with 150,000 lines of untested code. While the percentage of code covered is high, there is _a_lot_ of code there that isn't tested and a lot of room for mistakes to creep in. A mistake on one of those 150,000 lines could break the build and possibly cost you hours or even days tracking it down. If those lines were tested however, your continuous integration build system would hopefully highlight the fault. In my experience testing works, saves time down the line, and makes code easier to come back to. -- James On 31 January 2014 13:21, Steven D'Aprano st...@pearwood.info wrote: On Fri, Jan 31, 2014 at 11:31:49AM +, James Chapman wrote: Hello tutors I've constructed an example which shows a problem I'm having testing a real world program and would like to run it past you. [...] class Infinite_Loop_Tutor_Question(object): def run_forever(self): self.start_A() time.sleep(0.5) self.start_B() try: while True: time.sleep(1) except KeyboardInterrupt: print(Caught Keyboard Interrupt...) sys.exit(0) [...] In my example above, testing the everything but the run_forever method is trivial. So on to my question... The run_forever method essentially just fires up a bunch of threads to serve various purposes and then waits for CTRL-C to terminate the entire program. Testing this at the moment is very difficult because the unit test ends up in the infinite loop. So, would a better idea be to create an attribute, set it to True and then do try: while self.attribute: time.sleep(1) except KeyboardInterrupt: ... That probably won't hurt. My unit test could then set the attribute. However I'd still have the problem of how I get from the unit test line that fires up the method to the next line to change the attribute. So how should the run_forever method be written so that it's testable, or if it's testable as is, how would I test it? What are you trying to test? You don't just test a method, you test *something specific* about the method. So what specifically are you trying to test? And please, no comments about syntax, clean exits of threads, thread communication, resources, or even the need for testing the run_forever method. In my test I want to test that it makes the relevant calls and then enters the infinite loop at which point I want to terminate it. Ah, you see, now you have a problem. Consider this function: def long_calc(): time.sleep(60*60*24*365) return 1 How do I test that the function returns 1? As given, I can't really, not unless I wait a whole year for the sleep() to return. So what I can do is split the function into two pieces: def _sleep_a_year(): time.sleep(60*60*24*365) def _do_calculation(): return 1 def long_calc(): _sleep_a_year() return _do_calculation() Now I can unit-test the _do_calculation function, and long_calc() is now simple enough that I don't really need to unit-test it. (Unit testing should not be treated as a religion. You test what you can. Any testing is better than nothing, and if there are some parts of the program which are too hard to test automatically, don't test them automatically.) Or, I can monkey-patch the time.sleep function. Before running my test_long_calc unit-test, I do this: import time time.sleep = lambda n: None and then restore it when I'm done. But like all monkey-patching, that's risky -- what if the calculation relies on time.sleep somewhere else? (Perhaps it calls a function, which calls another function in a module somewhere, which calls a third module, which needs time.sleep.) So monkey-patching should be a last resort. Another alternative is to write the function so it can be tested using a mock: def long_calc(sleeper=time.sleep): sleeper(60*60*24*365) return 1 Then I can test it like this: assert stupid(lambda n: None) == 1 where the lambda acts as a mock-up for the real sleep
Re: [Tutor] Unit testing infinite loops
On Fri, Jan 31, 2014 at 02:03:13PM +, James Chapman wrote: On the note of what doesn't need a test... The question of coverage always comes up when unit testing is mentioned and I read an interesting blog article once about it. It basically said: Assume you have 85% coverage on a program that consists of 1,000 lines. That's 150 lines which are not tested. If those lines are print lines, sleep lines, getters etc it's not really a problem. But what happens when you scale that up. 1,000,000 lines of code lets say (not unheard of, although in python that would be out of this world big). You now end up with 150,000 lines of untested code. While the percentage of code covered is high, there is _a_lot_ of code there that isn't tested and a lot of room for mistakes to creep in. A mistake on one of those 150,000 lines could break the build and possibly cost you hours or even days tracking it down. Or weeks, or months... You're right of course. But look at it this way. How much time, effort, money, and lost opportunities to be doing other things, are you willing to spend to asymptotically approach 100% coverage? It might take a week of development writing nothing but tests to get to 80% coverage, another week to get to 85%, a further week to get to 87%, another week again to get to 88%... At some point you say, I have better things to do. Or your customers start questioning why the bills keep coming but the features aren't being delivered. (Customers want features, not tests.) I'm not trying to talk you out of testing this. I agree completely with this: If those lines were tested however, your continuous integration build system would hopefully highlight the fault. In my experience testing works, saves time down the line, and makes code easier to come back to. but life is short, and even if you like writing tests, there comes a point of diminishing returns. And the ideal of programming is to write code which is obviously correct (as opposed to code which merely contains no obvious bugs). And of course: better 85% coverage than 50%. Better 50% coverage than 10%. Better 10% coverage than no tests at all. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
