On Mon, Nov 9, 2015 at 9:57 PM, Craig Ringer <cr...@2ndquadrant.com> wrote: > On 10 November 2015 at 02:26, Robert Haas <robertmh...@gmail.com> wrote: >>> I'd like to see, say, python and the 'unittest' module added, and >>> to see acceptance of tests using psycopg2 to stream and decode WAL >>> from a slot. >> >> I actually kind of hate this sort of thing. > > For what it's worth, I don't actually like it either. However, I > suspect that the current restricted tools are a significant impediment > to test tooling and infrastructure improvement.
But that's exactly the argument that was made to justify adding the TAP testing framework. It didn't work? >> Now I'm not dead set against changing anything at all here, but I want >> to point out that we just added the TAP framework and already there >> are proposals (like Kevin's snapshot too old patch) to require DBD::Pg >> for some tests, which a lot of people aren't going to have > > The alternative seems to be driving psql -qAtz as a coprocess over a > pipeline as a poor man's database driver. I really doubt that's any > kind of improvement when it comes to the complexity of maintaining > tests, fixing them when they break, etc. I'd rather just write tests > in C against libpq despite the verbosity, annoyance of maintaining > them, etc. I thought about an idea like this: have some sort of dummy program, and then have a module written in Perl that provides an interface to it, so that we don't have to depend on DBD::Pg. I'm not at all convinced that's a bad plan. > Part of why I'm somewhat interested in seeing python and psycopg2 > added is that I'd personally quite like to see widely used client > drivers covered by the buildfarm to some extent specifically to see if > we break them. Though really, it'd make sense to add the most > important drivers like psqlODBC and PgJDBC first if doing that.... and > that'd be a whole new barrel of fun. EnterpriseDB has something like this, and it does catch some bugs, but it is an unbelievable amount of work to keep it working. When something fails, it's unlikely that any individual developer has exactly the right configuration on their machine to reproduce the failure. Now maybe we could come up with something that provides similar test coverage but is less work to maintain - I'm certainly not going to rule out the possibility that it can be done better than EnterpriseDB has done it. However, I bet it will still be a lot of work, and unlike at EntepriseDB where we've got entire teams of people who spend major amounts of time on this kind of stuff, PostgreSQL relies on its committers to keep the buildfarm green. And if I commit something and it breaks the Win32 version of some test that is written in Python and relies on pgsqlODBC, there is no way in heck I'm committing myself to reproducing that environment. Even if it's Linux64 and JDBC I'm not doing it. If you want to build such a thing and maintain it in perpetuity, that is fine, and you don't need my permission, and I and I'm sure many others will greatly appreciate both the effort and any bugs that get found and fixed that way. But when you talk about putting this kind of thing to core you are asking the committers to be responsible for it, and I'm still not very confident we've got the TAP tests in a state where maintaining them is going to be a reasonable level of effort. Those tests failed on my machine for months, and the only reason they eventually got fixed is because I knew enough Perl to track the problem down to some screwy coding in one of the Perl modules MacPorts installs. If the same thing happens with Python, it would be extremely difficult for me to find it, because I don't actually know Python. >> Accepting more tools also increases the breadth of knowledge expected >> of committers and patch authors. > > That's the real problem IMO. The cognitive and knowledge load. That's definitely a big part of it, but I think you're selling short the amount of havoc that build dependencies cause. On a Linux system where you can install whatever you need via 'yum install', sure, it's easy. It's not so easy on Windows. It's less easy on other UNIX systems without modern package managers. But that's just the tip of the iceberg. Installing your operating system's DBD::Pg package is going to install a version of DBD::Pg compiled against the system's libpq, not the latest PostgreSQL libpq that you just built. Now perhaps, you will say, that's OK: that way we test cross-version libpq compatibility, too! And on a certain level I can't argue with that. But now you could have a situation where the tests pass for one user and fail for another user even though both are compiling from the same PostgreSQL commit and the same DBD::Pg version, and that kind of stuff is a lot of work to track down. And it may not really be a bug: maybe the person who added the test intended the test to exercise some new libpq feature which is indeed supported by DBD::Pg, but your build of DBD::Pg doesn't have that because it was compiled against an older version of libpq that didn't have that feature, so in your environment, the test fails. >> Similarly with testing frameworks >> - except those are even worse, because nobody's saying that any of the >> stuff we have now will ever go away. > > Yeah. I guess that's the issue, isn't it. When something gets added > we're generally stuck with it forever. We still have PL/TCL! > > Even if it were desirable (and ignoring the horrific backpatching pain > that'd result), nobody's going to leap up and volunteer to rewrite all > the Perl like the Windows build system so we could remove Perl. It's > there essentially forever. The concern that if we added > python+psycopg2 we'd be stuck with it too is more than valid. > > Drat. Yes, it sucks. I think no matter how we try to improve testing, it's likely to be a lot of work. But personally I think it's better to put that work into tools that live inside our tree and that we have full control over, rather than adding dependencies. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
