Re: Recording test runtimes with the buildfarm

2020-06-16 Thread David Rowley
On Fri, 12 Jun 2020 at 04:32, Tom Lane wrote: > > Magnus Hagander writes: > > On Thu, Jun 11, 2020 at 4:56 PM Andrew Dunstan < > > It seems pretty trivial to for example get all the steps out of check.log > > and their timing with a regexp. > > Yeah, I don't see why we can't scrape this data from

Re: Recording test runtimes with the buildfarm

2020-06-14 Thread Fabien COELHO
Hello Tom, I have in the past scraped the latter results and tried to make sense of them. They are *mighty* noisy, even when considering just one animal that I know to be running on a machine with little else to do. Maybe averaging across the whole buildfarm could reduce the noise level, b

Re: Recording test runtimes with the buildfarm

2020-06-11 Thread Tom Lane
Andrew Dunstan writes: > On 6/11/20 12:32 PM, Tom Lane wrote: >> Yeah, I don't see why we can't scrape this data from the existing >> buildfarm output, at least for the core regression tests. New >> infrastructure could make it easier/cheaper, but I don't think we >> should invest in that until w

Re: Recording test runtimes with the buildfarm

2020-06-11 Thread Andrew Dunstan
On 6/11/20 12:32 PM, Tom Lane wrote: > Magnus Hagander writes: >> On Thu, Jun 11, 2020 at 4:56 PM Andrew Dunstan < >> andrew.duns...@2ndquadrant.com> wrote: >>> Yeah, we'll need to work out where to stash the file. The client will >>> pick up anything in src/regress/log for "make check", but wou

Re: Recording test runtimes with the buildfarm

2020-06-11 Thread Magnus Hagander
On Thu, Jun 11, 2020 at 6:32 PM Tom Lane wrote: > Magnus Hagander writes: > > On Thu, Jun 11, 2020 at 4:56 PM Andrew Dunstan < > > andrew.duns...@2ndquadrant.com> wrote: > >> Yeah, we'll need to work out where to stash the file. The client will > >> pick up anything in src/regress/log for "make

Re: Recording test runtimes with the buildfarm

2020-06-11 Thread Tom Lane
Magnus Hagander writes: > On Thu, Jun 11, 2020 at 4:56 PM Andrew Dunstan < > andrew.duns...@2ndquadrant.com> wrote: >> Yeah, we'll need to work out where to stash the file. The client will >> pick up anything in src/regress/log for "make check", but would need >> adjusting for other steps that inv

Re: Recording test runtimes with the buildfarm

2020-06-11 Thread Magnus Hagander
On Thu, Jun 11, 2020 at 4:56 PM Andrew Dunstan < andrew.duns...@2ndquadrant.com> wrote: > > On 6/11/20 10:21 AM, Stephen Frost wrote: > > Greetings, > > > > * David Rowley (dgrowle...@gmail.com) wrote: > >> On Thu, 11 Jun 2020 at 10:02, Tom Lane wrote: > >>> Thomas Munro writes: > I've been

Re: Recording test runtimes with the buildfarm

2020-06-11 Thread Andrew Dunstan
On 6/11/20 10:21 AM, Stephen Frost wrote: > Greetings, > > * David Rowley (dgrowle...@gmail.com) wrote: >> On Thu, 11 Jun 2020 at 10:02, Tom Lane wrote: >>> Thomas Munro writes: I've been doing that in a little database that pulls down the results and analyses them with primitive rege

Re: Recording test runtimes with the buildfarm

2020-06-11 Thread Stephen Frost
Greetings, * David Rowley (dgrowle...@gmail.com) wrote: > On Thu, 11 Jun 2020 at 10:02, Tom Lane wrote: > > Thomas Munro writes: > > > I've been doing that in a little database that pulls down the results > > > and analyses them with primitive regexes. First I wanted to know the > > > pass/fail

Re: Recording test runtimes with the buildfarm

2020-06-11 Thread Joe Conway
On 6/10/20 6:00 PM, Andres Freund wrote: > On June 10, 2020 2:13:51 PM PDT, David Rowley wrote: >>On Thu, 11 Jun 2020 at 02:13, Tom Lane wrote: >>> I have in the past scraped the latter results and tried to make sense >>of >>> them. They are *mighty* noisy, even when considering just one animal

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread David Rowley
On Thu, 11 Jun 2020 at 10:02, Tom Lane wrote: > > Thomas Munro writes: > > I've been doing that in a little database that pulls down the results > > and analyses them with primitive regexes. First I wanted to know the > > pass/fail history for each individual regression, isolation and TAP > > sc

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Tom Lane
Thomas Munro writes: > I've been doing that in a little database that pulls down the results > and analyses them with primitive regexes. First I wanted to know the > pass/fail history for each individual regression, isolation and TAP > script, then I wanted to build something that could identify

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Andres Freund
Hi, On June 10, 2020 2:13:51 PM PDT, David Rowley wrote: >On Thu, 11 Jun 2020 at 02:13, Tom Lane wrote: >> I have in the past scraped the latter results and tried to make sense >of >> them. They are *mighty* noisy, even when considering just one animal >> that I know to be running on a machine

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Thomas Munro
On Thu, Jun 11, 2020 at 9:43 AM Thomas Munro wrote: > On Thu, Jun 11, 2020 at 2:13 AM Tom Lane wrote: > > I have in the past scraped the latter results and tried to make sense of > > them. They are *mighty* noisy, even when considering just one animal > > that I know to be running on a machine w

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Tom Lane
David Rowley writes: > On Thu, 11 Jun 2020 at 02:13, Tom Lane wrote: >> I have in the past scraped the latter results and tried to make sense of >> them. They are *mighty* noisy, even when considering just one animal >> that I know to be running on a machine with little else to do. > Do you rec

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Thomas Munro
On Thu, Jun 11, 2020 at 2:13 AM Tom Lane wrote: > I have in the past scraped the latter results and tried to make sense of > them. They are *mighty* noisy, even when considering just one animal > that I know to be running on a machine with little else to do. Maybe > averaging across the whole bu

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread David Rowley
On Thu, 11 Jun 2020 at 02:13, Tom Lane wrote: > I have in the past scraped the latter results and tried to make sense of > them. They are *mighty* noisy, even when considering just one animal > that I know to be running on a machine with little else to do. Do you recall if you looked at the para

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Stephen Frost
Greetings, * Andrew Dunstan (andrew.duns...@2ndquadrant.com) wrote: > On 6/10/20 10:13 AM, Tom Lane wrote: > > Andrew Dunstan writes: > >> Alternatively, people with access to the database could extract the logs > >> and post-process them using perl or python. That would involve no work > >> on m

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Andrew Dunstan
On 6/10/20 10:13 AM, Tom Lane wrote: > Andrew Dunstan writes: >> Alternatively, people with access to the database could extract the logs >> and post-process them using perl or python. That would involve no work >> on my part :-) But it would not be automated. > Yeah, we could easily extract per

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Tom Lane
Andrew Dunstan writes: > Alternatively, people with access to the database could extract the logs > and post-process them using perl or python. That would involve no work > on my part :-) But it would not be automated. Yeah, we could easily extract per-test-script runtimes, since pg_regress start

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Andrew Dunstan
On 6/10/20 8:58 AM, David Rowley wrote: > Hi, > > I've just had some thoughts about the possible usefulness of having > the buildfarm record the run-time of each regression test to allow us > to have some sort of ability to track the run-time history of each > test. > > I thought the usefulness m

Re: Recording test runtimes with the buildfarm

2020-06-10 Thread Stephen Frost
Greetings, * David Rowley (dgrowle...@gmail.com) wrote: > 1. We could quickly identify when someone adds some overly complex > test and slows down the regression tests too much. Sure, makes sense to me. We do track the individual 'stage_duration' but we don't track things down to a per-regressio

Recording test runtimes with the buildfarm

2020-06-10 Thread David Rowley
Hi, I've just had some thoughts about the possible usefulness of having the buildfarm record the run-time of each regression test to allow us to have some sort of ability to track the run-time history of each test. I thought the usefulness might be two-fold: 1. We could quickly identify when som