Re: Query execution in Perl TAP tests needs work

2024-03-30 Thread Noah Misch
On Wed, Aug 30, 2023 at 09:48:42AM +1200, Thomas Munro wrote: > On Wed, Aug 30, 2023 at 1:49 AM Noah Misch wrote: > > On Tue, Aug 29, 2023 at 04:25:24PM +1200, Thomas Munro wrote: > > > On Tue, Aug 29, 2023 at 1:48 PM Noah Misch wrote: > > > >

Re: Query execution in Perl TAP tests needs work

2023-11-06 Thread Jehan-Guillaume de Rorthais
On Wed, 18 Oct 2023 18:25:01 +0200 Alvaro Herrera wrote: > On 2023-Oct-18, Robert Haas wrote: > > > Without FFI::Platypus, we have to write Perl code that can speak the > > wire protocol directly. Basically, we're writing our own PostgreSQL > > driver for Perl, though we might need only a

Re: Query execution in Perl TAP tests needs work

2023-10-24 Thread Andrew Dunstan
On 2023-10-18 We 11:47, Tom Lane wrote: Robert Haas writes: On Wed, Oct 18, 2023 at 10:28 AM Tom Lane wrote: I did a bit of research on this on my favorite platforms, and did not like the results: Hmm. That's unfortunate. Is perl -MCPAN -e 'install Platypus::FFI' a viable alternative?

Re: Query execution in Perl TAP tests needs work

2023-10-18 Thread Alvaro Herrera
On 2023-Oct-18, Robert Haas wrote: > Without FFI::Platypus, we have to write Perl code that can speak the > wire protocol directly. Basically, we're writing our own PostgreSQL > driver for Perl, though we might need only a subset of the things a > real driver would need to handle, and we might

Re: Query execution in Perl TAP tests needs work

2023-10-18 Thread Robert Haas
On Wed, Oct 18, 2023 at 11:43 AM Tom Lane wrote: > Having said that ... I read the man page for FFI::Platypus, > and I'm failing to draw a straight line between what it can do > and what we need. Aren't we going to need a big chunk of new > Perl code anyway? If so, why not write a big chunk of

Re: Query execution in Perl TAP tests needs work

2023-10-18 Thread Tom Lane
Robert Haas writes: > On Wed, Oct 18, 2023 at 10:28 AM Tom Lane wrote: >> I did a bit of research on this on my favorite platforms, and did >> not like the results: > Hmm. That's unfortunate. Is perl -MCPAN -e 'install Platypus::FFI' a > viable alternative? Probably, see my followup.

Re: Query execution in Perl TAP tests needs work

2023-10-18 Thread Tom Lane
=?utf-8?Q?Dagfinn_Ilmari_Manns=C3=A5ker?= writes: > Tom Lane writes: >> gfortran? Really?? I mean, I don't care about the disk space, >> but this is not promising for anyone who has to build it themselves. > The Fortran support for FFI::Platypus is in a separate CPAN distribution >

Re: Query execution in Perl TAP tests needs work

2023-10-18 Thread Robert Haas
On Wed, Oct 18, 2023 at 10:28 AM Tom Lane wrote: > I did a bit of research on this on my favorite platforms, and did > not like the results: Hmm. That's unfortunate. Is perl -MCPAN -e 'install Platypus::FFI' a viable alternative? -- Robert Haas EDB: http://www.enterprisedb.com

Re: Query execution in Perl TAP tests needs work

2023-10-18 Thread Dagfinn Ilmari Mannsåker
Tom Lane writes: > Robert Haas writes: >> On Sat, Sep 2, 2023 at 2:42 PM Andrew Dunstan wrote: How much burden is it? Would anyone actually mind? > >> ... At the same time, fallbacks can be a problem too, >> because then you can end up with things that work one way on one >> developer's

Re: Query execution in Perl TAP tests needs work

2023-10-18 Thread Tom Lane
Robert Haas writes: > On Sat, Sep 2, 2023 at 2:42 PM Andrew Dunstan wrote: >>> How much burden is it? Would anyone actually mind? > ... At the same time, fallbacks can be a problem too, > because then you can end up with things that work one way on one > developer's machine (or BF machine) and

Re: Query execution in Perl TAP tests needs work

2023-10-18 Thread Robert Haas
On Sat, Sep 2, 2023 at 2:42 PM Andrew Dunstan wrote: > I confess I'm a little reluctant to impose this burden on buildfarm owners. > We should think about some sort of fallback in case this isn't supported on > some platform, either due to technological barriers or buildfarm owner >

Re: Query execution in Perl TAP tests needs work

2023-09-12 Thread Andres Freund
Hi, On 2023-08-28 17:29:56 +1200, Thomas Munro wrote: > As an experiment, I hacked up a not-good-enough-to-share experiment > where $node->safe_psql() would automatically cache a BackgroundPsql > object and reuse it, and the times for that test dropped ~51 -> ~9s on > Windows, and ~7 -> ~2s on

Re: Query execution in Perl TAP tests needs work

2023-09-11 Thread Thomas Munro
On Sun, Sep 3, 2023 at 12:17 PM Thomas Munro wrote: > (Huh, while contemplating trying that, I just noticed that the GCC > build farm's AIX 7.2 system seems to have given up the ghost a few > weeks ago. I wonder if it'll come back online with the current > release, or if that's the end... There

Re: Query execution in Perl TAP tests needs work

2023-09-04 Thread Andrew Dunstan
On 2023-09-02 Sa 20:17, Thomas Munro wrote: On Sun, Sep 3, 2023 at 6:42 AM Andrew Dunstan wrote: I confess I'm a little reluctant to impose this burden on buildfarm owners. We should think about some sort of fallback in case this isn't supported on some platform, either due to

Re: Query execution in Perl TAP tests needs work

2023-09-02 Thread Thomas Munro
On Sun, Sep 3, 2023 at 6:42 AM Andrew Dunstan wrote: > I guess the next thing would be to test it on a few more platforms and also > to see if we need to expand the coverage of libpq for the intended uses. Nice. It works fine on my FreeBSD battlestation after "sudo pkg install p5-FFI-Platypus"

Re: Query execution in Perl TAP tests needs work

2023-09-02 Thread Andrew Dunstan
On 2023-08-30 We 21:29, Thomas Munro wrote: On Thu, Aug 31, 2023 at 10:32 AM Andrew Dunstan wrote: #!/usr/bin/perl use strict; use warnings; use FFI::Platypus; my $ffi = FFI::Platypus->new(api=>1); $ffi->lib("inst/lib/libpq.so"); $ffi->type('opaque' => 'PGconn'); $ffi->attach(PQconnectdb

Re: Query execution in Perl TAP tests needs work

2023-08-30 Thread Thomas Munro
On Thu, Aug 31, 2023 at 10:32 AM Andrew Dunstan wrote: > #!/usr/bin/perl > > use strict; use warnings; > > use FFI::Platypus; > > my $ffi = FFI::Platypus->new(api=>1); > $ffi->lib("inst/lib/libpq.so"); > > > $ffi->type('opaque' => 'PGconn'); > $ffi->attach(PQconnectdb => [ 'string' ] =>

Re: Query execution in Perl TAP tests needs work

2023-08-30 Thread Andrew Dunstan
On 2023-08-28 Mo 09:23, Andrew Dunstan wrote: On 2023-08-28 Mo 01:29, Thomas Munro wrote: Hi, Every time we run a SQL query, we fork a new psql process and a new cold backend process. It's not free on Unix, and quite a lot worse on Windows, at around 70ms per query. Take

Re: Query execution in Perl TAP tests needs work

2023-08-30 Thread Andres Freund
Hi, On 2023-08-28 17:29:56 +1200, Thomas Munro wrote: > Every time we run a SQL query, we fork a new psql process and a new > cold backend process. It's not free on Unix, and quite a lot worse on > Windows, at around 70ms per query. Take amcheck/001_verify_heapam for > example. It runs 272

Re: Query execution in Perl TAP tests needs work

2023-08-29 Thread Thomas Munro
On Wed, Aug 30, 2023 at 1:49 AM Noah Misch wrote: > On Tue, Aug 29, 2023 at 04:25:24PM +1200, Thomas Munro wrote: > > On Tue, Aug 29, 2023 at 1:48 PM Noah Misch wrote: > > > https://github.com/cpan-authors/IPC-Run/issues/166#issuecomment-1288190929 > > > > Interesting. But that shows a case

Re: Query execution in Perl TAP tests needs work

2023-08-29 Thread Noah Misch
On Tue, Aug 29, 2023 at 04:25:24PM +1200, Thomas Munro wrote: > On Tue, Aug 29, 2023 at 1:48 PM Noah Misch wrote: > > https://github.com/cpan-authors/IPC-Run/issues/166#issuecomment-1288190929 > > Interesting. But that shows a case with no pipes connected, using > select() as a dumb sleep and

Re: Query execution in Perl TAP tests needs work

2023-08-29 Thread Thomas Munro
On Tue, Aug 29, 2023 at 1:48 PM Noah Misch wrote: > On Mon, Aug 28, 2023 at 05:29:56PM +1200, Thomas Munro wrote: > > 1. Don't fork anything at all: open (and cache) a connection directly > > from Perl. > > 1a. Write xsub or ffi bindings for libpq. Or vendor (parts) of the > > popular Perl

Re: Query execution in Perl TAP tests needs work

2023-08-28 Thread Thomas Munro
On Tue, Aug 29, 2023 at 1:23 AM Andrew Dunstan wrote: > I like the idea of using a pure perl pq implementation, not least because it > could expand our ability to test things at the protocol level. Not sure how > much work it would be. I'm willing to help if we want to go that way. Cool.

Re: Query execution in Perl TAP tests needs work

2023-08-28 Thread Thomas Munro
On Tue, Aug 29, 2023 at 1:48 PM Noah Misch wrote: > https://github.com/cpan-authors/IPC-Run/issues/166#issuecomment-1288190929 Interesting. But that shows a case with no pipes connected, using select() as a dumb sleep and ignoring SIGCHLD. In our usage we have pipes connected, and I think

Re: Query execution in Perl TAP tests needs work

2023-08-28 Thread Noah Misch
On Mon, Aug 28, 2023 at 05:29:56PM +1200, Thomas Munro wrote: > CI shows Windows > consuming 4 CPUs at 100% for a full 10 minutes to run a test suite > that finishes in 2-3 minutes everywhere else with the same number of > CPUs. Could there be an event handling snafu in IPC::Run or elsewhere >

Re: Query execution in Perl TAP tests needs work

2023-08-28 Thread Andrew Dunstan
On 2023-08-28 Mo 01:29, Thomas Munro wrote: Hi, Every time we run a SQL query, we fork a new psql process and a new cold backend process. It's not free on Unix, and quite a lot worse on Windows, at around 70ms per query. Take amcheck/001_verify_heapam for example. It runs 272 subtests

Query execution in Perl TAP tests needs work

2023-08-27 Thread Thomas Munro
Hi, Every time we run a SQL query, we fork a new psql process and a new cold backend process. It's not free on Unix, and quite a lot worse on Windows, at around 70ms per query. Take amcheck/001_verify_heapam for example. It runs 272 subtests firing off a stream of queries, and completes in