[Twisted-Python] automated tests for a server application
Hi All, (copying in Moshe as he expressed an interest!) I've inherited a twisted app for which I'd like to add some changes but want to improve the automated test coverage before I do. The app itself listens on a web socket (autobahn), an rpc port (RPyC) and also connects to a mysql database. So, for the automated tests, I want to check what happens when I send various messages to the websocket or rpc port, and then check what ends up in the db and what other state changes happen in the app itself. What's the best/correct way to write automated tests for this kind of situation? I'm using pytest and would happily use pytest-twisted too. Not keen to use trial... Any help is good help, pointers to nice existing examples would be fantastic! cheers, Chris ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] automated tests for a server application
On 06/11/2018 05:43, Moshe Zadka wrote: Some of the best advice depends on details of the application. One trick that is sometimes useful is passing in a "fake" reactor object. This, of course, is only useful if the application is structured in a way that functions/classes expect to get a reactor, instead of reaching for the global one. However, usually *that's* not a complicated refactoring to do. Cool, do you have any example tests that do this? Interesting, looks like pytest-twisted does away for the need for this by showing how to install a fake reactor globally: https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L129-L142 You can look at https://twistedmatrix.com/documents/current/api/twisted.test.proto_helpers.MemoryReactor.html for inspiration although, as all code under `twisted.test`, it is not intended as end-user functionality, so using it directly is problematic. Not sure I fully understand this, why is the MemoryReactor bad to use? Where is it used? It would be nice to have externally-useful reactors along the lines of Clock (https://twistedmatrix.com/documents/current/api/twisted.internet.task.Clock.html ) but, unfortunately, we do not have that yet. So, clock is just a clock, right? How would that get worked into a "real" reactor for testing? You can, of course, use a real reactor and a real client to pump data. However, in that case, you probably do want to switch to trial so that you can return a deferred from a test function and have the reactor run until the deferred fires. This is not great, but can work in a pinch. pytest-twisted looks like it supports this pattern too, allowing test functions to return deferreds... I guess I'm still really looking to see examples of all of this... Chris ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] automated tests for a server application
On 2018-11-06 11:28, Chris Withers wrote: On 06/11/2018 12:14, Kyle Altendorf wrote: On November 6, 2018 6:41:23 AM EST, Chris Withers wrote: Cool, do you have any example tests that do this? Interesting, looks like pytest-twisted does away for the need for this by showing how to install a fake reactor globally: https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L129-L142 What is 'fake' about this globally installed normally-the-default reactor? (Otherwise the qt5reactor if chosen) I guess I'm still not clear on what the point of using a 'fake' reactor over a 'real' one is. Not that I'm an expert here... Nothing, but the technique could be used to install a fake reactor rather than having to change all the existing code to accept an optional reactor parameter. I use @pytest_twisted.inlineCallbacks anyways, yes. Overall I'm not clear what was recommended here. Why fake the reactor? Even if not using a 'real client' wouldn't you just fake the data going through the connections rather than faking the entire reactor? I'd love to see some good example of faking the data going through the connections, can you point me at some? I would assume you would just write a 'client' in the test of whatever complexity (could just write hardcoded byte sequences) which opens a connection to the server and transmits the bytes and then asserts things about the response. But no, I don't have any code. I can't say I have a good test suite myself and I also don't actually use Twisted for internet stuff (canbus and serial). Sorry. I would expect the servers provided by Twisted would have their own tests you could look at though. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] automated tests for a server application
On Tue, Nov 6, 2018, at 8:43 AM, Kyle Altendorf wrote: > On 2018-11-06 11:28, Chris Withers wrote: > I guess I'm still not clear on what the point of using a 'fake' reactor > over a 'real' one is. Not that I'm an expert here... There are a bunch of advantages! Including: 1. Your tests become more deterministic --- even *fully* deterministic. 2. Fake I/O is way faster --- no syscalls! 3. Fake I/O is more reliable --- no worries about failing to allocate a port. 4. You gain full control over time. This makes it easy to test behaviors like timeouts and backoff timers. 5. You can run multiple fake reactors. Great for testing distributed systems! At an extreme, you can make the whole suite synchronous (meaning, Deferreds fire immediately). treq.testing[1] can be used like this. [2] is an example of a suite in this style (this suite is actually run under the Django test runner, which knows nothing of reactors or Deferreds, though I don't recommend doing that). There are some potential pitfalls: 1. As with any test double, you can trick yourself. I find faking I/O in this way is less prone to these problems than tools like unittest.mock, though. 2. Manually controlling time can harm composability. Application-level Twisted tends to eventually require some sort of time-related behavior, if only the humble reactor.callLater(0, ...) to avoid stack overflow. (Use of the cooperator from twisted.internet.task is another common one.) 3. Some (old) bits of Twisted don't allow passing a reactor or use the one passed in all code paths[3]. You can combine fake I/O with techniques like generative testing, too, though I don't have any public examples to point at. And of course you can also combine with other forms of dependency injection, like passing in an endpoint. FWIW twisted.test.proto_helpers and twisted.test.iosim *are* public API, despite their presence in the test package. They could certainly use docs, and there is a ticket to move them somewhere more appropriate. For the moment the best way to learn to read them is to look at Twisted's own test suite. I found reading the tests for HostnameEndpoint helpful [4]. ---Tom [1]: https://treq.readthedocs.io/en/latest/testing.html [2]: https://github.com/twm/yarrharr/blob/e31a488f8884fb82f1dba733585bef4a7ad86968/yarrharr/tests/test_fetch.py#L282 [3]: https://github.com/twm/yarrharr/pull/277/commits/1a5b5832edbf1fb1b1f45e9d99b65dad51ada566#diff-29ffe25b52ad7bddee9f2f08544e899cR98 [4]: https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/test/test_endpoints.py ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
