On 01/12/2015 02:20 PM, Chris Dent wrote:
After some discussion with Sean Dague and a few others it became clear that it would be a good idea to introduce a new tool I've been working on to the list to get a sense of its usefulness generally, work towards getting it into global requirements, and get the documentation fleshed out so that people can actually figure out how to use it well. tl;dr: Help me make this interesting tool useful to you and your HTTP testing by reading this message and following some of the links and asking any questions that come up. The tool is called gabbi https://github.com/cdent/gabbi http://gabbi.readthedocs.org/ https://pypi.python.org/pypi/gabbi It describes itself as a tool for running HTTP tests where requests and responses are represented in a declarative form. Its main purpose is to allow testing of APIs where the focus of test writing (and reading!) is on the HTTP requests and responses, not on a bunch of Python (that obscures the HTTP). The tests are written in YAML and the simplest test file has this form: ``` tests: - name: a test url: / ``` This test will pass if the response status code is '200'. The test file is loaded by a small amount of python code which transforms the file into an ordered sequence of TestCases in a TestSuite[1]. ``` def load_tests(loader, tests, pattern): """Provide a TestSuite to the discovery process.""" test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR) return driver.build_tests(test_dir, loader, host=None, intercept=SimpleWsgi, fixture_module=sys.modules[__name__]) ``` The loader provides either: * a host to which real over-the-network requests are made * a WSGI app which is wsgi-intercept-ed[2] If an individual TestCase is asked to be run by the testrunner, those tests that are prior to it in the same file are run first, as prerequisites. Each test file can declare a sequence of nested fixtures to be loaded from a configured (in the loader) module. Fixtures are context managers (they establish the fixture upon __enter__ and destroy it upon __exit__). With a proper group_regex setting in .testr.conf each YAML file can run in its own process in a concurrent test runner. The docs contain information on the format of the test files: http://gabbi.readthedocs.org/en/latest/format.html Each test can state request headers and bodies and evaluate both response headers and response bodies. Request bodies can be strings in the YAML, files read from disk, or JSON created from YAML structures. Response verifcation can use JSONPath[3] to inspect the details of response bodies. Response header validation may use regular expressions. There is limited support for refering to the previous request to construct URIs, potentially allowing traversal of a full HATEOAS compliant API. At the moment the most complete examples of how things work are: * Ceilometer's pending use of gabbi: https://review.openstack.org/#/c/146187/ * Gabbi's testing of gabbi: https://github.com/cdent/gabbi/tree/master/gabbi/gabbits_intercept (the loader and faked WSGI app for those yaml files is in: https://github.com/cdent/gabbi/blob/master/gabbi/test_intercept.py) One obvious thing that will need to happen is a suite of concrete examples on how to use the various features. I'm hoping that feedback will help drive that. In my own experimentation with gabbi I've found it very useful. It's helped me explore and learn the ceilometer API in a way that existing test code has completely failed to do. It's also helped reveal several warts that will be very useful to fix. And it is fast. To run and to write. I hope that with some work it can be useful to you too.
Very impressive, Chris, thanks very much for bringing Gabbi into the OpenStack ecosystem. I very much look forward to replacing the API samples code in Nova with Gabbi, which looks very clean and easily-understandable for anyone.
Best, -jay __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev