On 8/16/15 8:48 AM, Greg Stark wrote:
On Sun, Aug 16, 2015 at 7:33 AM, Noah Misch <n...@leadboat.com> wrote:
When I've just spent awhile implementing a behavior change, the test diffs are
a comforting sight. They confirm that the test suite exercises the topic I
just changed. Furthermore, most tests today do not qualify under this
stringent metric you suggest. The nature of pg_regress opposes it.
It's not a comforting sight for me. It means I need to change the test
results and then of course the tests will pass. So they didn't really
test anything and I don't really know whether the changes broke
anything. Any test I had to adjust for my change was effectively
worthless.
This is precisely my point about pg_regress and why it's the reason
there's pressure not to have extensive tests. It's useful to have some
end-to-end black box tests like this but it's self-limiting. You can't
test many details without tying the tests to specific behaviour.
I have worked with insanely extensive testing suites that didn't have
this problem. But they were unit tests for code that was structured
around testability. When the API is designed to be testable and you
have good infrastructure for mocking up the environment and comparing
function results in a much narrower way than just looking at text
output you can test the correctness of each function without tying the
test to the specific behaviour.
*That* gives a real comfort. When you change a function to behave
entirely differently and can still run all the tests and see that all
the code that depends on it still operate correctly then you know you
haven't broken anything. In that case it actually *speeds up*
development rather than slowing it down. It lets newbie developers
hack on code where they don't necessarily know all the downstream
dependent code and not be paralyzed by fear that they'll break
something.
As someone who oversaw a database test suite with almost 500 test files
(each testing multiple cases), I completely agree. Our early framework
was actually similar to pg_regress and that worked "OK" up to about 200
test files, but it got very unwieldy very quickly. We switched to pgTap
and it was far easier to work with.
I suspect any effort to significantly improve Postgres test coverage is
doomed until there's an alternative to pg_regress.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
