Hi,
Kevin Grittner wrote:
I just got to the point of having what appears to be a working but
poorly optimized version of serializable transactions,
Cool.
so it is
critical that I create a good set of tests to confirm correct
behavior and monitor for regressions as I optimize. I see that
you
I wrote:
> [off-list to avoid distracting from the 9.0 wrap-up effort]
Arg. I really didn't mean that to go to the list. :-(
-Kevin
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[off-list to avoid distracting from the 9.0 wrap-up effort]
Markus Wanner wrote:
> Quoting "Kevin Grittner" :
>> I strongly encourage you to set that up on git.postgresql.org.
>
> I'm about to provide git repositories for Postgres-R anyway, so
> I've setup two projects on git.postgres-r.org:
>
Hi,
Jan Urbański wrote:
> sorry to butt in to the conversation, but I have spent some time
> wrapping/refining the concepts in dtester, and the results are here:
>
> http://git.wulczer.org/?p=twisted-psql.git;a=summary
That seems to cover the concurrent psql part of dtester. But I don't see
how
Hi,
Quoting "Kevin Grittner" :
I strongly encourage you to set that up on git.postgresql.org.
I'm about to provide git repositories for Postgres-R anyway, so I've
setup two projects on git.postgres-r.org:
dtester: that's the driver/harness code
postgres-dtests: a Postgres clone with the dtest
Markus Wanner wrote:
Kevin Grittner wrote:
>> args=['psql', '-A', '--pset=pager=off',
> That looks like a correct fix for psql, yes.
> Other processes might be confused by (or at least act differently
> with) a PAGER env variable, so that still needs to be cleared in
> general.
Markus Wanner wrote:
Kevin Grittner wrote:
> I differentiate tests and test suites. Tests mainly have a run
> method, while test suites have setUp and tearDown ones.
I hadn't caught on to that distinction yet. That should help.
>> "uses" means that the referenced task has complimentary setU
Markus Wanner wrote:
>> I do want to expand the tests quite a bit -- do I work them all into
>> this same file, or how would I proceed? I think I'll need about 20
>> more tests, but I don't want to get in the way of your work on the
>> framework which runs them.
>
> Well, first of all, another pi
Hi,
Kevin Grittner wrote:
Based on Andrew's suggestion, I changed line 276 to:
args=['psql', '-A', '--pset=pager=off',
That looks like a correct fix for psql, yes. Thanks for pointing that
out Andrew.
Other processes might be confused by (or at least act differently with)
a P
Hi,
Kevin Grittner wrote:
I'm a little unclear about the differences between "uses",
"depends", and "onlyAfter". Here's what they *sound* like they
mean, to me; although I don't think the code isn't entirely
consistent with this interpretation.
Wow, you are way ahead of me. I intended to writ
Markus Wanner wrote:
> Go try it, read the code and simply ask, if you get stuck. I'll
> try to come up with some more documentation and such...
I'm a little unclear about the differences between "uses",
"depends", and "onlyAfter". Here's what they *sound* like they
mean, to me; although I do
Markus Wanner wrote:
> Please recheck with the StreamReporter and try to grep the lines
> starting with "[psql0]", "[psql1]" and "[psql2]". Dtester simply
> logs all and any output of all 3rd party processes started.
For me, all psql output seems to be [psql0]; no [psql1] or [psql2].
Bug?
"Kevin Grittner" wrote:
> Also, in looking closer at how you have the tests defined, it
> doesn't look to me like you're carefully interleaving specific
> sequences of statements on specific connections so much as opening
> multiple connections and then for each statement saying "run this
> on a
Markus Wanner wrote:
> So, the solution probably lies in adjusting the environment,
> before starting psql. (Maybe even dropping all existing
> environment variables for better control of the situation). Will
> add that for dtester 0.1.
Based on Andrew's suggestion, I changed line 276 to:
Markus Wanner wrote:
Hi,
Kevin Grittner wrote:
My pager is "less"; could that cause it? Could the twisted
environment look like one where the pager should kick in?
Yes, that could be it. At least it fails here, too, if I set
PAGER=less. Try:
PAGER=more make dcheck
Surely for automa
Hi,
Kevin Grittner wrote:
My pager is "less"; could that cause it? Could the twisted
environment look like one where the pager should kick in?
Yes, that could be it. At least it fails here, too, if I set PAGER=less.
Try:
PAGER=more make dcheck
So, the solution probably lies in adjusting
Markus Wanner wrote:
> Strangely, your log has escape codes in it, which I'm assuming
> makes the parsing choke. Is that something special to your
> installation?
My pager is "less"; could that cause it? Could the twisted
environment look like one where the pager should kick in?
-Kevin
--
Hi,
Kevin Grittner wrote:
I haven't configured anything like that intentionally. I don't
*see* any colors when I use psql. Can you think of anywhere I
should check something which might be causing this?
No idea ATM.
However, just to make sure that has absolutely nothing to do with the
curs
Markus Wanner wrote:
> Strangely, your log has escape codes in it, which I'm assuming
> makes the parsing choke. Is that something special to your
> installation? My psql never colors its outputs...
I haven't configured anything like that intentionally. I don't
*see* any colors when I use psq
Markus Wanner wrote:
> I wasn't aware I had non-ascii characters in there. Inserting an
> encoding line seems fine. I'll fix that for the upcoming version
> 0.1.
Yeah, I couldn't find any, either. I just tried creating a minimal
python file in Kate, and it gave me that even though I *know* i
Hi,
Kevin Grittner wrote:
You are trying to save a python file as non ASCII, without
specifiying a correct source encoding line for encoding "utf-8"
I wasn't aware I had non-ascii characters in there. Inserting an
encoding line seems fine. I'll fix that for the upcoming version 0.1.
Regards
Hi,
Kevin Grittner wrote:
Not sure what's relevant there. Entire file tarball attached.
Due to reasons mentioned in this thread as well, I've decided to use
psql to connect to the database. dtester is parsing its output and
checks that against expectations. Hawever, that has its own pitfall
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
> Dead or not, it still works, even against 8.4. I have many programs
> that use it. It's simply a wrapper around the libpq interface and as
> long as the libpq interface remains stable (which we go to great pains
>
Markus Wanner wrote:
> Second: at the very end of pg_dtester.py, you find the line:
> reporter = StreamReporter()
>
> Try a CursesReporter() instead, it gives much nicer output!
When I try to do that, Kate complains (I'm even copying their typo):
You are trying to save a python file as no
"Markus Wanner" wrote:
> Quoting "Kevin Grittner" :
>> I haven't quite gotten it to work yet; I'll start over with 3.0
>> and see how it goes.
>
> Let's stick to 2.x versions, first...
OK
> It does: "temp_install: creating temporary installation" means
> it's running make install in the backgr
Hi,
Quoting "Kevin Grittner" :
I haven't quite gotten it to work yet; I'll start over with 3.0 and
see how it goes.
Let's stick to 2.x versions, first...
I'll also attach the results of the 2.6 attempt.
Thanks, that looks already pretty promising. ;-)
A few other issues in testing so far
I wrote:
> I'll also attach the results of the 2.6 attempt.
Let's try that again.
-Kevin
kgri...@kgrittn-desktop:~/git/postgresql/kgrittn$ make dcheck
make -C src/test dcheck
make[1]: Entering directory `/home/kgrittn/git/postgresql/kgrittn/src/test'
make
Markus Wanner wrote:
> I must admit that I haven't ever tested on python 2.6 before. I'll
> try that (especially as it's the staircase to 3.0, IIUC).
I don't use python much, so I can't comment on that. I do see that
my system has these two versions on it, with a symlink that makes
2.6 the de
Hi,
Kevin Grittner wrote:
C'mon, you could have tried to inspire a *bit* more confidence by
calling it version 0.1 or something! ;-)
LOL
As David used to say: JFDI
> I found that I just needed to ask for python-twisted.
Oh, sorry, yes, requirements: python, twisted.
I must admit that I ha
Markus Wanner wrote:
> Okay, here we go: dtester version 0.0.
C'mon, you could have tried to inspire a *bit* more confidence by
calling it version 0.1 or something! ;-)
> It's certainly missing lots of things, mainly documentation.
> However, I've attached a patch which integrates nicely in
Hi,
Markus Wanner wrote:
Sorry, if that didn't get clear. I'm trying to put together something I
can release real soon now (tm). I'll keep you informed.
Okay, here we go: dtester version 0.0.
This emerged out of Postgres-R, where I don't just need to test multiple
client connections, but mul
On Mon, Jan 11, 2010 at 05:42:08PM -, Greg Sabino Mullane wrote:
> > Is there a reason why you're suggesting using DBI? There is also the Pg
> > perl module which works as well and is one tenth of the size. It also
> > doesn't have external dependancies. It's just a plain wrapper around
> > lib
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
> Is there a reason why you're suggesting using DBI? There is also the Pg
> perl module which works as well and is one tenth of the size. It also
> doesn't have external dependancies. It's just a plain wrapper around
> libpq, which for the purpo
On Mon, Jan 11, 2010 at 04:17:42AM -, Greg Sabino Mullane wrote:
> > Because you'd have to build DBD::Pg against the new libpq, as you do
> > psql. That means you need DBD::Pg sources and the build environment for
> > Perl (headers etc) not just a working Perl runtime. Big difference.
>
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
>> Using DBI/DBD::Pg would raise another issue - what version of libpq
>> would it be using? Not the one in the build being tested, that's for
On 8/01/2010 1:39 AM, Greg Sabino Mullane wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
Using DBI/DBD::Pg would raise another issue - what version of libpq
would it be using? Not the one in the build being tested, that's for
sure.
Er...why not? That's what psql uses.
Because yo
A.M. wrote:
On Jan 7, 2010, at 12:39 PM, Greg Sabino Mullane wrote:
I'm still *very* interested in making a libpq-less pure perl driver,
if anyone feels like funding it, let me know! :)
You mean this one:
http://search.cpan.org/~arc/DBD-PgPP-0.07/lib/DBD/PgPP.pm
?
It has a l
On Jan 7, 2010, at 12:39 PM, Greg Sabino Mullane wrote:
> I'm still *very* interested in making a libpq-less pure perl driver,
> if anyone feels like funding it, let me know! :)
You mean this one:
http://search.cpan.org/~arc/DBD-PgPP-0.07/lib/DBD/PgPP.pm
?
Cheers,
M
--
Sent via pgsql-hackers
Alvaro Herrera wrote:
> Kevin Grittner escribió:
>
>> Are we anywhere close to an agreement on what the multi-session
>> psql implementation would look like?
> See
>
http://archives.postgresql.org/message-id/8204.1207689...@sss.pgh.pa.us
> and followups.
Thanks, I had missed or forgotten this
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
>> We could even bundle DBI and DBD::Pg to ensure that the minimum versions
>> are there.
> As a packager, my reaction to that is "over my dead body". We have
> enough trouble keeping our own software up to date, and pretty much
> every extern
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
> Using DBI/DBD::Pg would raise another issue - what version of libpq
> would it be using? Not the one in the build being tested, that's for
> sure.
Er...why not? That's what psql uses. As for those advocating using a
custom C program written u
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
> Unless we are prepared to define concurrency testing as something
> separate from the basic regression tests. Which is kind of annoying but
> perhaps less so than the alternatives. It certainly seems to me to
> be the kind of thing you would
On Jan 7, 2010, at 9:19 AM, Tom Lane wrote:
>> In that case, there's nothing for it except concurrent psql.
>
> Unless we are prepared to define concurrency testing as something
> separate from the basic regression tests. Which is kind of annoying but
> perhaps less so than the alternatives. It
"David E. Wheeler" writes:
> On Jan 7, 2010, at 9:08 AM, Tom Lane wrote:
>> Right, but to my mind "building from a tarball" needs to include the
>> ability to run the regression tests on what you built. So injecting
>> Perl into that is moving the goalposts on build requirements.
> In that case,
On Jan 7, 2010, at 9:08 AM, Tom Lane wrote:
> Right, but to my mind "building from a tarball" needs to include the
> ability to run the regression tests on what you built. So injecting
> Perl into that is moving the goalposts on build requirements.
In that case, there's nothing for it except con
Andrew Dunstan writes:
> Unless I am mistaken, Perl is required in any case to build from CVS,
> although not from a tarball.
Right, but to my mind "building from a tarball" needs to include the
ability to run the regression tests on what you built. So injecting
Perl into that is moving the goa
On Jan 6, 2010, at 6:31 PM, Kevin Grittner wrote:
> As far as I've been able to determine so far, to call psql in a
> relatively portable way would require something like this:
>
> http://perldoc.perl.org/perlfork.html
Here's an example using IPC::Open3:
#!/usr/local/bin/perl -w
use st
On Wed, Jan 06, 2010 at 08:40:28PM -0500, Andrew Dunstan wrote:
> A parallel psql seems to me a better way to go. We talked about that
> a while ago, but I don't recall what happened to it.
Greg Stark had a patch a couple of years ago. Dunno what happened to
it since then.
Cheers,
David.
--
Dav
On Thu, Jan 7, 2010 at 11:46 AM, Andrew Dunstan wrote:
> Using DBI/DBD::Pg would raise another issue - what version of libpq would it
> be using? Not the one in the build being tested, that's for sure. If you
> really want to use Perl then either a Pure Perl DBI driver (which Greg has
> talked abo
David E. Wheeler wrote:
On Jan 6, 2010, at 6:26 PM, Tom Lane wrote:
We have not yet fully accepted the notion that you must have Perl to
build (and, in fact, I am right now trying to verify that you don't).
I don't think that requiring Perl to test is going to fly.
I believe that th
"Greg Sabino Mullane" writes:
> We could even bundle DBI and DBD::Pg to ensure that the minimum versions
> are there.
As a packager, my reaction to that is "over my dead body". We have
enough trouble keeping our own software up to date, and pretty much
every external component that we've started
On Jan 6, 2010, at 6:26 PM, Tom Lane wrote:
> We have not yet fully accepted the notion that you must have Perl to
> build (and, in fact, I am right now trying to verify that you don't).
> I don't think that requiring Perl to test is going to fly.
I believe that the build farm already requires Pe
On 2010-01-07 18:13 +0200, Marko Tiikkaja wrote:
I had a similar syntax in mind, but instead of using threads, just
execute the file in order using asynchronous connections.
I completely failed to make the point here which was to somehow mark
which statements will (or, should) block. So here
On 2010-01-07 11:50 +0200, Craig Ringer wrote:
On 7/01/2010 9:15 AM, Robert Haas wrote:
Doing this without DBI is going to be ten times harder than doing it
with DBI. Are we really sure that's not a viable option?
At this point, I'm personally wondering if it's worth putting together a
simple
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
>> Doing this without DBI is going to be ten times harder than doing it
>> with DBI. Are we really sure that's not a viable option?
> In the buildfarm? Yes, I think so. The philosophy of the buildfarm is
> that it should do what you would do y
On 7/01/2010 9:15 AM, Robert Haas wrote:
On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner
wrote:
"David E. Wheeler" wrote:
Last I heard, Andrew was willing to require Test::More for
testing, so that a Perl script could handle multiple psql
connections (perhaps forked) and output test results
On ons, 2010-01-06 at 20:49 -0500, Robert Haas wrote:
> Another idea would be to make a set of Perl libpq bindings that is
> simpler than DBD::Pg and don't go through DBI. If we put those in the
> main source tree (perhaps as a contrib module) they would be available
> wherever we need them.
htt
On ons, 2010-01-06 at 14:10 -0800, David E. Wheeler wrote:
> On Jan 6, 2010, at 2:08 PM, Peter Eisentraut wrote:
>
> > Then I don't see much of a point in using Perl. You might as well fire
> > up a few psqls from a shell script
>
> If you're more comfortable with shell, then yes. Although then
Robert Haas wrote:
It just seems crazy to me to try to test anything without proper
language bindings. Opening a psql session and parsing the results
seems extraordinarily painful.
I've written a Python based program that spawns a captive psql and talks
to it--twice for different people--that
On Wed, Jan 6, 2010 at 10:00 PM, Kevin Grittner
wrote:
>> For basic regression tests, yeah, we'd probably like to keep that
>> Perl-free.
>
> OK. Is parallel psql the only reasonable option?
It seems so, assuming you're willing to concede that it is a
reasonable option in the first place.
>> Fo
Kevin Grittner escribió:
> Well, if that's the consensus, I have to choose between trying to
> implement multi-session psql and using testing which can't carry over
> to long-term regular use. Are we anywhere close to an agreement on
> what the multi-session psql implementation would look like?
Robert Haas wrote:
> I'm not exactly sure what Kevin's goal is here.
I think it would be insane to try to do something like serializable
isolation mode without having regression tests. I would want more
tests than could reasonably go into the 'make check' suite to support
development, but it
On Wed, Jan 6, 2010 at 9:26 PM, Tom Lane wrote:
> Robert Haas writes:
>> It just seems crazy to me to try to test anything without proper
>> language bindings. Opening a psql session and parsing the results
>> seems extraordinarily painful. I wonder if it would make sense write
>> a small wrapp
Tom Lane wrote:
> We have not yet fully accepted the notion that you must have Perl
> to build (and, in fact, I am right now trying to verify that you
> don't). I don't think that requiring Perl to test is going to fly.
Well, if that's the consensus, I have to choose between trying to
impleme
Andrew Dunstan wrote:
> Robert Haas wrote:
>> Doing this without DBI is going to be ten times harder than doing
>> it with DBI. Are we really sure that's not a viable option?
> In the buildfarm? Yes, I think so. The philosophy of the buildfarm
> is that it should do what you would do yourself
Robert Haas writes:
> It just seems crazy to me to try to test anything without proper
> language bindings. Opening a psql session and parsing the results
> seems extraordinarily painful. I wonder if it would make sense write
> a small wrapper program that uses libpq and dumps out the results in
On Wed, Jan 6, 2010 at 8:40 PM, Andrew Dunstan wrote:
>> Doing this without DBI is going to be ten times harder than doing it
>> with DBI. Are we really sure that's not a viable option?
>
> In the buildfarm? Yes, I think so. The philosophy of the buildfarm is that
> it should do what you would do
Robert Haas wrote:
On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner
wrote:
"David E. Wheeler" wrote:
Last I heard, Andrew was willing to require Test::More for
testing, so that a Perl script could handle multiple psql
connections (perhaps forked) and output test results based on
them
On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner
wrote:
> "David E. Wheeler" wrote:
>
>> Last I heard, Andrew was willing to require Test::More for
>> testing, so that a Perl script could handle multiple psql
>> connections (perhaps forked) and output test results based on
>> them. But he wasn't as
Alvaro Herrera wrote:
> Open a few psql with -f pointing to a pipe, and from the shell
> write into the pipe? I don't think it's straightforward, but it
> should be possible.
I'll play with it and see what I can do.
Thanks,
-Kevin
--
Sent via pgsql-hackers mailing list (pgsql-hackers@po
Marko Tiikkaja wrote:
> On 2010-01-07 00:08 +0200, Peter Eisentraut wrote:
> >Then I don't see much of a point in using Perl. You might as well fire
> >up a few psqls from a shell script.
>
> I don't see how that would work, but I might have misunderstood what
> we're reaching for here. What I
Marko Tiikkaja wrote:
> On 2010-01-07 00:08 +0200, Peter Eisentraut wrote:
>> Then I don't see much of a point in using Perl. You might as
>> well fire up a few psqls from a shell script.
>
> I don't see how that would work, but I might have misunderstood
> what we're reaching for here. What
On 2010-01-07 00:08 +0200, Peter Eisentraut wrote:
On ons, 2010-01-06 at 15:52 -0600, Kevin Grittner wrote:
"David E. Wheeler" wrote:
Last I heard, Andrew was willing to require Test::More for
testing, so that a Perl script could handle multiple psql
connections (perhaps forked) and output te
On Jan 6, 2010, at 2:08 PM, Peter Eisentraut wrote:
> Then I don't see much of a point in using Perl. You might as well fire
> up a few psqls from a shell script
If you're more comfortable with shell, then yes. Although then it won't run on
Windows, will it?
Best,
David
--
Sent via pgsql-hac
On Jan 6, 2010, at 1:52 PM, Kevin Grittner wrote:
>> Last I heard, Andrew was willing to require Test::More for
>> testing, so that a Perl script could handle multiple psql
>> connections (perhaps forked) and output test results based on
>> them. But he wasn't as interested in requiring DBI and DB
On ons, 2010-01-06 at 15:52 -0600, Kevin Grittner wrote:
> "David E. Wheeler" wrote:
>
> > Last I heard, Andrew was willing to require Test::More for
> > testing, so that a Perl script could handle multiple psql
> > connections (perhaps forked) and output test results based on
> > them. But he w
"David E. Wheeler" wrote:
> Last I heard, Andrew was willing to require Test::More for
> testing, so that a Perl script could handle multiple psql
> connections (perhaps forked) and output test results based on
> them. But he wasn't as interested in requiring DBI and DBD::Pg,
> neither of which
Hi,
Kevin Grittner wrote:
Where would I find this (and any related documentation)?
Sorry, if that didn't get clear. I'm trying to put together something I
can release real soon now (tm). I'll keep you informed.
Regards
Markus Wanner
--
Sent via pgsql-hackers mailing list (pgsql-hackers@p
Markus Wanner wrote:
> Kevin Grittner wrote:
>> It's very soon going to be critical that I be able to test
>> particular interleavings of statements in particular concurrent
>> transaction sets to be able to make meaningful progress on the
>> serializable transaction work.
>
> I've something in
Hi,
Kevin Grittner wrote:
> It's very soon going to be critical that I be able to test particular
> interleavings of statements in particular concurrent transaction sets
> to be able to make meaningful progress on the serializable
> transaction work.
I've something in place for Postgres-R, as I a
On Jan 4, 2010, at 3:29 PM, Peter Eisentraut wrote:
> If you're not deep into Perl, perhaps ignore the Test::More comment for
> the moment and just use DBI to connect to several database sessions,
> execute your queries and check if the results are what you want. Once
> you have gotten somewhere
On mån, 2010-01-04 at 17:10 -0600, Kevin Grittner wrote:
> "David E. Wheeler" wrote:
>
> > I think the consensus was, failing support for concurrent sessions
> > in psql, to use a Perl script to control multiple psql sessions
> > and perhaps use Test::More to do the testing.
>
> Are there any
"David E. Wheeler" wrote:
> I think the consensus was, failing support for concurrent sessions
> in psql, to use a Perl script to control multiple psql sessions
> and perhaps use Test::More to do the testing.
Are there any examples of that? While I can hack my way through
regular expressions
On Jan 1, 2010, at 6:01 PM, Kevin Grittner wrote:
> It's very soon going to be critical that I be able to test particular
> interleavings of statements in particular concurrent transaction sets
> to be able to make meaningful progress on the serializable
> transaction work. It would be wonderful
There has been periodic discussion here about allowing psql to deal
with multiple sessions, or possibly creating another tool to allow
this sort of test. Is anyone working on this?
It's very soon going to be critical that I be able to test particular
interleavings of statements in particular con
85 matches
Mail list logo
