Hi, Robert Haas <robertmh...@gmail.com> writes: > On Thu, Aug 27, 2009 at 2:54 PM, Peter Eisentraut<pete...@gmx.net> wrote: >> We have regression tests. They could and should be expanded. That's a >> developer job, and we can start working on that now. But this >> discussion was about what to do during beta. And I think during beta >> you want to test PostgreSQL against a large set of real applications. >> But we could try to clarify how to actually do that in an organized >> way.
Exactly, and I think that what we're missing here is a simple tool for our users to check a new PostgreSQL release against their existing application. We already know how to either log all queries and analyze the log files (CSV makes it easier, pgfouine parses them too) or to have a fe/be protocol proxy to record application SQL traffic (tsung recorder does that). What we miss is a tool to run the captured queries through both versions of PG and report any resultset mismatch, of course with a way to account for ordering issues (but we've seen people rely on the ordering when they don't give an order by clause, then bug the lists about it if a new release changes it). > What I want to do is address the concern about too much of any given > year being consumed by beta and CommitFest. I'm not sure I know how > to do that though. To do this I think we *need* to provide beta tester a validator kind of tool which they can use even without having an application unit test suite. And a way to easily report success or failure, and in case of success, a notion of their tests coverage (which is reduced to the list of PG features the application exercices, but an automated way to extract the information while running the tool would allow for hackers friendly categories). Anyone interrested into starting a project and coding the tool we so much need? Regards, -- dim PS: of course provided with such a tool, I'd run it for several applications as soon as possible, which might include every alpha release. PPS: I'll be happy to participate into such a project, but I seem to have a rather full plate already and a shrinking OpenSource free time, so waiting for me to get there won't help until it's about too late. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers