Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Sat, 28 Jun 2003, Jan Wieck wrote:

> Bruce Momjian wrote:
> > See my recent commit of src/tools/pgtest.  It might be a good start.
>
> I was wondering if some existing framework, like from the Apache Xalan
> package, would be a better point to start from? I hate to say it, Bruce,
> but you try to reinvent the wheel by starting with a sled.

Hey, I take offence at that ... up here in Canada, that sled is faster,
dontcha know?  especially if we throw those dogs in front of them :)


---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

Bruce Momjian wrote:
See my recent commit of src/tools/pgtest.  It might be a good start.
I was wondering if some existing framework, like from the Apache Xalan 
package, would be a better point to start from? I hate to say it, Bruce, 
but you try to reinvent the wheel by starting with a sled.

Jan

---

Gavin Sherry wrote:
On Thu, 26 Jun 2003, Kevin Brown wrote:

> The Hermit Hacker wrote:
> > On Wed, 25 Jun 2003, Kevin Brown wrote:
> > 
> > > So...would it make sense to create a gborg project to which people who
> > > have written their own test suites can contribute whatever code and data
> > > they feel comfortable releasing?  As a gborg project, it would be
> > > separate from the main PG distribution and would thus have no impact on
> > > the build process or anything like that.  But at the same time, if there
> > > are any ideas on testing that people have had, they could be shared with
> > > others through that mechanism.
> > >
> > > And any tests which prove to be particularly useful could make their way
> > > into the PG distribution if people here wish.
> > >
> > > Of course, like anything else this could be a bad (or perhaps redundant)
> > > idea.  :-)
> > 
> > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > original thread: are you willing to step up and maintain such a project?
> 
> Yes, I am ("how hard can it be?", he asks himself, knowing all the
> while that it's a really bad idea to be asking that question.  :-).
> But I haven't the faintest idea of how or where to even start, so
> pointers would be appreciated.

Create/modify a script to automate some kind of download/sync, test and
send failure results somewhere. Make it extensible, so that other tests
can be easily added -- preferable in a self contained way. It should grow
from there.
Gavin

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
   http://www.postgresql.org/docs/faqs/FAQ.html




--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
  http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

Wow, I am impressed by 'gmake check'.  Who did all that work?  It is
great.

I modified tools/pgtest to use 'gmake check'.  Thanks.

---

Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > Amazing you find 688 bytes worth discussing.  I know you said "what
> > happens if everyone adds their scripts", but something that would be a
> > mess if everyone did it isn't always a proper way to judge if something
> > is appropriate.
> 
> I said, if everyone adds their test methodologies.  That leads to
> discrepancies, more of them down the road if one method changes and the
> other doesn't catch up.  For instance, your method just calls pg_ctl,
> createdb, etc. from the path.  If people already have a stable
> installation of PostgreSQL on their machine, then this will test the wrong
> installation.  So, from now on, if someone submits a test result I have to
> ask, "which method did you use" -- "don't use that method, because it's
> wrong".  That is one instance, and I'm sure you'll fix it, but there might
> be more.  What I'm saying is, we were in a discussion about improving the
> testing of PostgreSQL, and this is not a step forward.  If we need to
> improve the testing mechanisms for various purposes -- patch application,
> automated testing, etc. -- let's look at it and see how we can improve the
> current infrastructure without inventing a parallel one.  At this point,
> I'm not sure why "make check" doesn't serve you.  Perhaps you are not
> fully aware of what it does (I guess so, from looking at your script).
> 
> -- 
> Peter Eisentraut   [EMAIL PROTECTED]
> 
> 

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Thomas Swan]

Peter Eisentraut wrote:

>Thomas Swan writes:
>
>  
>
>>I just am really concerned about the uninstall/clean up phase and how
>>that can be done in an orderly fashion.  Unless the process can start
>>from a clean state again, then it won't be valid.
>>
>>
>
>The only clean state is if you remove the entire source tree and check it
>out again.  (Of course to save bandwidth, you copy the checked out source
>tree to a temporary location, do your testing, and then remove that
>temporary tree.)  Relying on make clean or make uninstall is flawed,
>because those are among the things you want to test.
>
That sounds plausible.   Should we let everything stay in the compilers
directory.   Something like the configure --prefix=$TEST_ROOT and that
way we can have the whole thing run as one user in one directory so that
system wide impact is minimal.I guess what I'm concerned with is
running this on a clean system, and then leaving unknown artifacts
behind.   Can/does make install output each file it's copying and where
to.   Capturing that output would make life easier for clean up of
things installed outside of the work directory, and provide a more
controlled environment.


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[Peter Eisentraut]

Bruce Momjian writes:

> Amazing you find 688 bytes worth discussing.  I know you said "what
> happens if everyone adds their scripts", but something that would be a
> mess if everyone did it isn't always a proper way to judge if something
> is appropriate.

I said, if everyone adds their test methodologies.  That leads to
discrepancies, more of them down the road if one method changes and the
other doesn't catch up.  For instance, your method just calls pg_ctl,
createdb, etc. from the path.  If people already have a stable
installation of PostgreSQL on their machine, then this will test the wrong
installation.  So, from now on, if someone submits a test result I have to
ask, "which method did you use" -- "don't use that method, because it's
wrong".  That is one instance, and I'm sure you'll fix it, but there might
be more.  What I'm saying is, we were in a discussion about improving the
testing of PostgreSQL, and this is not a step forward.  If we need to
improve the testing mechanisms for various purposes -- patch application,
automated testing, etc. -- let's look at it and see how we can improve the
current infrastructure without inventing a parallel one.  At this point,
I'm not sure why "make check" doesn't serve you.  Perhaps you are not
fully aware of what it does (I guess so, from looking at your script).

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[Peter Eisentraut]

Thomas Swan writes:

> I just am really concerned about the uninstall/clean up phase and how
> that can be done in an orderly fashion.  Unless the process can start
> from a clean state again, then it won't be valid.

The only clean state is if you remove the entire source tree and check it
out again.  (Of course to save bandwidth, you copy the checked out source
tree to a temporary location, do your testing, and then remove that
temporary tree.)  Relying on make clean or make uninstall is flawed,
because those are among the things you want to test.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[Peter Eisentraut]

Tom Lane writes:

> This is something that breaks regularly because few of the key
> developers use it :-(.  If there were automatic tests that used that
> build setup, it would be a good thing 'cause it'd keep us honest.

This should be included in 'make distcheck'.  I'm quite puzzled right now
why it doesn't to it already.  A DESTDIR installation should also be
tested there.  Once that is done, ./configure [options] && make distcheck
&& make maintainer-clean is a quite extensive test.  The only drawback is
that it is hard to isolate the log of the failing part (config.log, build
log, regression test log).

For those who don't know, 'make distcheck' builds a distribution, unpacks
it, configures, builds, and installs the temporary tree, runs the
regression tests, uninstalls, checks whether all files were correctly
uninstalled, and builds another distribution out of the temporary tree to
check whether the distribution is self-contained.

> > That said, poor cleanup would be a result of
> > a broken make clean.
>
> Not to worry, the developers will notice that case fast enough.

This is not my experience, but a check for this could be included in make
distcheck as well.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[Gavin Sherry]

On Thu, 26 Jun 2003, Bruce Momjian wrote:

> 
> See my recent commit of src/tools/pgtest.  It might be a good start.

Yes this is a good start. This is a little concerning though:

pg_ctl stop
rm -rf "$PGDATA"

Perhaps a warning is warranted (ie, $PGDATA shouldn't point to your
production data cluster)?

On another point, I have some ideas for Kevin and others interested in
automated testing. Dann, Tom and others have voiced concern about the
nature of testing itself: progammers testing for bugs they've solved; the
difficulty/impossibility of testing for conditions you are unaware of,
etc.

ISTM that most of the bugs which aren't caught by the programmer, peer
review and regresion test are revealed by users throwing data into a new
version or a version different to that they are running in production and
then running their existing code against it. That is, bugs are revealed by
usage which developers did not foresee or did not think to test.

So, if we had an automated testing framework which was extensible,
postgres users could create testing scripts which simultate their
application, with their application data (real or created specifically
for the test). The win for users is that they can have their data/SQL
tested on a variety of platforms, on new versions of postgres and the win
for developers/testers is exposure of the code to unexpected usage.

There will need to be checks and balances involved (select 1; is a pretty
ordinary test), size limits/configurable thresholds for run times, and a
repository of test results.

Naturally, managing this could be quite time consuming if data formats
change etc. but if people are keen...

Gavin

> 
> ---
> 
> Gavin Sherry wrote:
> > On Thu, 26 Jun 2003, Kevin Brown wrote:
> > 
> > > The Hermit Hacker wrote:
> > > > On Wed, 25 Jun 2003, Kevin Brown wrote:
> > > > 
> > > > > So...would it make sense to create a gborg project to which people who
> > > > > have written their own test suites can contribute whatever code and data
> > > > > they feel comfortable releasing?  As a gborg project, it would be
> > > > > separate from the main PG distribution and would thus have no impact on
> > > > > the build process or anything like that.  But at the same time, if there
> > > > > are any ideas on testing that people have had, they could be shared with
> > > > > others through that mechanism.
> > > > >
> > > > > And any tests which prove to be particularly useful could make their way
> > > > > into the PG distribution if people here wish.
> > > > >
> > > > > Of course, like anything else this could be a bad (or perhaps redundant)
> > > > > idea.  :-)
> > > > 
> > > > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > > > original thread: are you willing to step up and maintain such a project?
> > > 
> > > Yes, I am ("how hard can it be?", he asks himself, knowing all the
> > > while that it's a really bad idea to be asking that question.  :-).
> > > But I haven't the faintest idea of how or where to even start, so
> > > pointers would be appreciated.
> > 
> > Create/modify a script to automate some kind of download/sync, test and
> > send failure results somewhere. Make it extensible, so that other tests
> > can be easily added -- preferable in a self contained way. It should grow
> > from there.
> > 
> > Gavin
> > 
> > 
> > ---(end of broadcast)---
> > TIP 5: Have you checked our extensive FAQ?
> > 
> >http://www.postgresql.org/docs/faqs/FAQ.html
> > 
> 
> 





---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[Tom Lane]

Rod Taylor <[EMAIL PROTECTED]> writes:
> I've not tried, but if PostgreSQL can do an 'out of tree' compile it
> could make it much easier.

Yes it can, following the usual procedure for autoconfiscated trees:
just invoke configure from an empty directory, eg
mkdir build
cd build
../pgsql/configure

This is something that breaks regularly because few of the key
developers use it :-(.  If there were automatic tests that used that
build setup, it would be a good thing 'cause it'd keep us honest.

> That said, poor cleanup would be a result of
> a broken make clean.

Not to worry, the developers will notice that case fast enough.
I think the auto test script should just rm -rf build and then
proceed as above.

regards, tom lane

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

See my recent commit of src/tools/pgtest.  It might be a good start.

---

Gavin Sherry wrote:
> On Thu, 26 Jun 2003, Kevin Brown wrote:
> 
> > The Hermit Hacker wrote:
> > > On Wed, 25 Jun 2003, Kevin Brown wrote:
> > > 
> > > > So...would it make sense to create a gborg project to which people who
> > > > have written their own test suites can contribute whatever code and data
> > > > they feel comfortable releasing?  As a gborg project, it would be
> > > > separate from the main PG distribution and would thus have no impact on
> > > > the build process or anything like that.  But at the same time, if there
> > > > are any ideas on testing that people have had, they could be shared with
> > > > others through that mechanism.
> > > >
> > > > And any tests which prove to be particularly useful could make their way
> > > > into the PG distribution if people here wish.
> > > >
> > > > Of course, like anything else this could be a bad (or perhaps redundant)
> > > > idea.  :-)
> > > 
> > > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > > original thread: are you willing to step up and maintain such a project?
> > 
> > Yes, I am ("how hard can it be?", he asks himself, knowing all the
> > while that it's a really bad idea to be asking that question.  :-).
> > But I haven't the faintest idea of how or where to even start, so
> > pointers would be appreciated.
> 
> Create/modify a script to automate some kind of download/sync, test and
> send failure results somewhere. Make it extensible, so that other tests
> can be easily added -- preferable in a self contained way. It should grow
> from there.
> 
> Gavin
> 
> 
> ---(end of broadcast)---
> TIP 5: Have you checked our extensive FAQ?
> 
>http://www.postgresql.org/docs/faqs/FAQ.html
> 

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Gavin Sherry]

On Thu, 26 Jun 2003, Kevin Brown wrote:

> The Hermit Hacker wrote:
> > On Wed, 25 Jun 2003, Kevin Brown wrote:
> > 
> > > So...would it make sense to create a gborg project to which people who
> > > have written their own test suites can contribute whatever code and data
> > > they feel comfortable releasing?  As a gborg project, it would be
> > > separate from the main PG distribution and would thus have no impact on
> > > the build process or anything like that.  But at the same time, if there
> > > are any ideas on testing that people have had, they could be shared with
> > > others through that mechanism.
> > >
> > > And any tests which prove to be particularly useful could make their way
> > > into the PG distribution if people here wish.
> > >
> > > Of course, like anything else this could be a bad (or perhaps redundant)
> > > idea.  :-)
> > 
> > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > original thread: are you willing to step up and maintain such a project?
> 
> Yes, I am ("how hard can it be?", he asks himself, knowing all the
> while that it's a really bad idea to be asking that question.  :-).
> But I haven't the faintest idea of how or where to even start, so
> pointers would be appreciated.

Create/modify a script to automate some kind of download/sync, test and
send failure results somewhere. Make it extensible, so that other tests
can be easily added -- preferable in a self contained way. It should grow
from there.

Gavin


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Thu, 26 Jun 2003, Kevin Brown wrote:

> > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > original thread: are you willing to step up and maintain such a project?
>
> Yes, I am ("how hard can it be?", he asks himself, knowing all the
> while that it's a really bad idea to be asking that question.  :-).
> But I haven't the faintest idea of how or where to even start, so
> pointers would be appreciated.

Which, of course, is always the fun part ...

I believe Thomas is going to be starting to work on test scripts, so you
might want to co-ordinate with him ...


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[Kevin Brown]

The Hermit Hacker wrote:
> On Wed, 25 Jun 2003, Kevin Brown wrote:
> 
> > So...would it make sense to create a gborg project to which people who
> > have written their own test suites can contribute whatever code and data
> > they feel comfortable releasing?  As a gborg project, it would be
> > separate from the main PG distribution and would thus have no impact on
> > the build process or anything like that.  But at the same time, if there
> > are any ideas on testing that people have had, they could be shared with
> > others through that mechanism.
> >
> > And any tests which prove to be particularly useful could make their way
> > into the PG distribution if people here wish.
> >
> > Of course, like anything else this could be a bad (or perhaps redundant)
> > idea.  :-)
> 
> It doesn't sound like a bad idea ... but, it pretty much comes down to the
> original thread: are you willing to step up and maintain such a project?

Yes, I am ("how hard can it be?", he asks himself, knowing all the
while that it's a really bad idea to be asking that question.  :-).
But I haven't the faintest idea of how or where to even start, so
pointers would be appreciated.


-- 
Kevin Brown   [EMAIL PROTECTED]

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[Thomas Swan]

The Hermit Hacker wrote:

On Thu, 26 Jun 2003, Thomas Swan wrote:

 

Of course, these are just ideas and I'm not sure how practical it is to
do any of them.  I just am really concerned about the uninstall/clean up
phase and how that can be done in an orderly fashion.  Unless the
process can start from a clean state again, then it won't be valid.  At
one point I had even given thought, vainly, to purchasing VMWare for
such an occasion.  Suggestions?
   

Personally ... if you could build up the test script, I think there are
enough ppl with more platforms on these lists that would be willing ot run
it ... the problem isn't getting the "farm" together, its coming up with
the automated (or even semi-automated) tests :(
I'll see what I can do... my shell script skills are pretty good, but 
I'm not sure how to handle the noting changes in the gcc output.  My 
best guess is to just do it a couple of times and force something to 
change (make an intentional mistake) and see if it can catch it, or at 
least what changes.



---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Thu, 26 Jun 2003, Rod Taylor wrote:

> I think we should replace Bruce's pgtest script with this one -- with an
> argument to accept the email address to report to for FAILING cases.
> Success isn't very interesting if it runs regularly.

that was why I suggested getting it into the tree ... to at least give a
starting point to work from ...

and I have at least one machine right now that I can run such tests on ...
Dual PII with FreeBSD 5.x-CURRENT on her ...

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Thu, 26 Jun 2003, Thomas Swan wrote:

> Of course, these are just ideas and I'm not sure how practical it is to
> do any of them.  I just am really concerned about the uninstall/clean up
> phase and how that can be done in an orderly fashion.  Unless the
> process can start from a clean state again, then it won't be valid.  At
> one point I had even given thought, vainly, to purchasing VMWare for
> such an occasion.  Suggestions?

Personally ... if you could build up the test script, I think there are
enough ppl with more platforms on these lists that would be willing ot run
it ... the problem isn't getting the "farm" together, its coming up with
the automated (or even semi-automated) tests :(


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Gonyou, Austin]

-Original Message-
From: Rod Taylor [mailto:[EMAIL PROTECTED]
Sent: Thursday, June 26, 2003 3:33 PM
To: Gonyou, Austin
Cc: Thomas Swan; Nigel J. Andrews; Tom Lane; PostgreSQL Development
Subject: RE: Two weeks to feature freeze


On Thu, 2003-06-26 at 16:09, Gonyou, Austin wrote:
> DOH!. YesYou're right I totally forgot about that. My apologies. I
> believe though, that there is a HP testing lab that is somewhat like OSDL,
> in that they offer OSS free services and many platforms to run on. (used
to
> be compaq's developer testdrive sort of program) I believe it still
exists.

>>I've been on Compaq's testdrive boxes before -- very nice.  But I'm not
>>sure we want to use those for a testing platform.

Agreed, but the point was, there is a developer side where developers can
test their code for a longer period of time in a more lab-like environment,
much the same way as OSDL schedules their testing, etc, but you can try
stuff on Alpha and x86, run windows, solaris, Tru64, etc.




>>More to the point, if OSDL benchmarks keep coming along we should have a
>>variety of PostgreSQL benchmarks to choose from.  Adding a regression
>>test for performance would be a wise thing to do, but will require
>>dedicated hardware for good numbers (especially if run nightly).

Understood and I concurr, I just didn't think about the testing scope of
OSDL. 

>>Thomas had a good start in mind.  Hopefully we can extend it a little
>>once it has been started.

I agree, just trying to offer something I thought was more valuable than it
may have been. :) 


>>-- 
>>Rod Taylor <[EMAIL PROTECTED]>
>>
>>PGP Key: http://www.rbt.ca/rbtpub.asc

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Rod Taylor]

On Thu, 2003-06-26 at 16:09, Gonyou, Austin wrote:
> DOH!. YesYou're right I totally forgot about that. My apologies. I
> believe though, that there is a HP testing lab that is somewhat like OSDL,
> in that they offer OSS free services and many platforms to run on. (used to
> be compaq's developer testdrive sort of program) I believe it still exists.

I've been on Compaq's testdrive boxes before -- very nice.  But I'm not
sure we want to use those for a testing platform.

More to the point, if OSDL benchmarks keep coming along we should have a
variety of PostgreSQL benchmarks to choose from.  Adding a regression
test for performance would be a wise thing to do, but will require
dedicated hardware for good numbers (especially if run nightly).

Thomas had a good start in mind.  Hopefully we can extend it a little
once it has been started.

-- 
Rod Taylor <[EMAIL PROTECTED]>

PGP Key: http://www.rbt.ca/rbtpub.asc


signature.asc
Description: This is a digitally signed message part

Re: [HACKERS] Two weeks to feature freeze

[Rod Taylor]

> * clean the source, destination directories
> * pull latest CVS tip down.

Why tip?  Lets simply update the current source tree to the most current
of whatever branch they had checked out initially.

Running it on older stable branches is just as useful.

> * record environment / installed packages

Virtually impossible, especially if people take to installing some items
by hand (we want to test them as well).  Recording the configure output
on the other hand would be quite useful.

> * loop - on different options ( w/ or w/o krb5, w/ or w/o ssl, etc. )
>   o make clean
>   o configure with sets of options
>   o compile
> + log messages
> + analyze errors ( perhaps gather statitistics:
>   warnings, failures, notices, etc.)

Any warnings, failures, etc. aside from what is in the 'known' file
should be reported -- makes sense.

>   o (run / install) if successful
>   o run tests
> + output results (perhaps to HTML)
> + compare results with expected
> + record differences if any | gather aggregate information

Standard regression tests should suffice.  Any non-ignored errors
reported.

>   o uninstall  / clean up

Skip this.  Cleanup prior to the run.  If something is wrong, we may
need additional information from the environment.

> * end loop
> 
> Perhaps there could be an occasion where the test would be able to put
> in a corrupt WAL or a corrupt table to do regression tests for recovery
> of errors.

These would be interesting extensions to the standard regression suite
-- but perhaps require a flag

> Of course, these are just ideas and I'm not sure how practical it is to
> do any of them.  I just am really concerned about the uninstall/clean up
> phase and how that can be done in an orderly fashion.   Unless the
> process can start from a clean state again, then it won't be valid.   At

I've not tried, but if PostgreSQL can do an 'out of tree' compile it
could make it much easier.  That said, poor cleanup would be a result of
a broken make clean.

> one point I had even given thought, vainly,  to purchasing VMWare for
> such an occasion.  Suggestions?

Skip VMWare.  Find a few volunteers to cron the script (I'll volunteer).

I think we should replace Bruce's pgtest script with this one -- with an
argument to accept the email address to report to for FAILING cases. 
Success isn't very interesting if it runs regularly.

 
-- 
Rod Taylor <[EMAIL PROTECTED]>

PGP Key: http://www.rbt.ca/rbtpub.asc


signature.asc
Description: This is a digitally signed message part

Re: [HACKERS] Two weeks to feature freeze

[Gonyou, Austin]

DOH!. YesYou're right I totally forgot about that. My apologies. I
believe though, that there is a HP testing lab that is somewhat like OSDL,
in that they offer OSS free services and many platforms to run on. (used to
be compaq's developer testdrive sort of program) I believe it still exists.

-Original Message-
From: Rod Taylor [mailto:[EMAIL PROTECTED]
Sent: Thursday, June 26, 2003 3:06 PM
To: Austin Gonyou
Cc: Thomas Swan; Nigel J. Andrews; Tom Lane; PostgreSQL Development
Subject: Re: Two weeks to feature freeze


On Thu, 2003-06-26 at 15:00, Austin Gonyou wrote:
> I know I'm new to this list, but is OSDL's testing capabilities out of
> the question?

>From what I've seen, OSDL is only concerned with a very very small set
of platforms (Linux in a couple of configurations).

-- 
Rod Taylor <[EMAIL PROTECTED]>

PGP Key: http://www.rbt.ca/rbtpub.asc

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Rod Taylor]

On Thu, 2003-06-26 at 15:00, Austin Gonyou wrote:
> I know I'm new to this list, but is OSDL's testing capabilities out of
> the question?

From what I've seen, OSDL is only concerned with a very very small set
of platforms (Linux in a couple of configurations).

-- 
Rod Taylor <[EMAIL PROTECTED]>

PGP Key: http://www.rbt.ca/rbtpub.asc


signature.asc
Description: This is a digitally signed message part

Re: [HACKERS] Two weeks to feature freeze

[Austin Gonyou]

I know I'm new to this list, but is OSDL's testing capabilities out of
the question?


On Thu, 2003-06-26 at 13:48, Thomas Swan wrote:
> Nigel J. Andrews wrote:
> 
> >On Thu, 26 Jun 2003, Thomas Swan wrote:
> >  
> >
> >>Is it possible the sourceforge compile farms could be used for some of 
> >>the automated testing?  I'm not sure how that system works, but it could 
> >>be worth looking into.
> >>
> >>
> >
> >Isn't the sourceforge license very scary and along the lines of "whatever you
> >put on here we own it's just we tend not to persue that at the moment as
> >there's not much money in it for us but that doesn't stop us from claiming it 
> >at some indeterminate time in the future"?
> >
> If it's that intrusive, then it was a bad idea.  But, I didn't find
> anything like that on their Terms of Use
> 
> page.  The compiler farm has a relatively small number of platforms, but
> perhaps it would be enough to get started with at least verifying an
> automated test would work. See Guide to the Sourceforge Compile Farm
> .
...
-- 
Austin Gonyou <[EMAIL PROTECTED]>
Coremetrics, Inc.

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Thomas Swan]

Nigel J. Andrews wrote:

>On Thu, 26 Jun 2003, Thomas Swan wrote:
>  
>
>>Is it possible the sourceforge compile farms could be used for some of 
>>the automated testing?  I'm not sure how that system works, but it could 
>>be worth looking into.
>>
>>
>
>Isn't the sourceforge license very scary and along the lines of "whatever you
>put on here we own it's just we tend not to persue that at the moment as
>there's not much money in it for us but that doesn't stop us from claiming it 
>at some indeterminate time in the future"?
>
If it's that intrusive, then it was a bad idea.  But, I didn't find
anything like that on their Terms of Use

page.  The compiler farm has a relatively small number of platforms, but
perhaps it would be enough to get started with at least verifying an
automated test would work. See Guide to the Sourceforge Compile Farm
.

In terms of implementation, I was thinking of something like the following.

* clean the source, destination directories
* pull latest CVS tip down.
* record environment / installed packages
* loop - on different options ( w/ or w/o krb5, w/ or w/o ssl, etc. )
  o make clean
  o configure with sets of options
  o compile
+ log messages
+ analyze errors ( perhaps gather statitistics:
  warnings, failures, notices, etc.)
  o (run / install) if successful
  o run tests
+ output results (perhaps to HTML)
+ compare results with expected
+ record differences if any | gather aggregate information
  o uninstall  / clean up
* end loop

Perhaps there could be an occasion where the test would be able to put
in a corrupt WAL or a corrupt table to do regression tests for recovery
of errors.

Of course, these are just ideas and I'm not sure how practical it is to
do any of them.  I just am really concerned about the uninstall/clean up
phase and how that can be done in an orderly fashion.   Unless the
process can start from a clean state again, then it won't be valid.   At
one point I had even given thought, vainly,  to purchasing VMWare for
such an occasion.  Suggestions?



---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Nigel J. Andrews]

On Thu, 26 Jun 2003, Thomas Swan wrote:
> >
> Is it possible the sourceforge compile farms could be used for some of 
> the automated testing?  I'm not sure how that system works, but it could 
> be worth looking into.

Isn't the sourceforge license very scary and along the lines of "whatever you
put on here we own it's just we tend not to persue that at the moment as
there's not much money in it for us but that doesn't stop us from claiming it 
at some indeterminate time in the future"?


-- 
Nigel J. Andrews


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[Tom Lane]

Thomas Swan <[EMAIL PROTECTED]> writes:
> Is it possible the sourceforge compile farms could be used for some of 
> the automated testing?  I'm not sure how that system works, but it could 
> be worth looking into.

The last time I used it (which admittedly was a year or two back), they
didn't really have a very wide range of platforms available.  And I
don't think they were offering cron access on the farm machines, so this
would be much more painful to automate than work done locally on
contributors' machines.

regards, tom lane

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[Thomas Swan]

Tom Lane wrote:

Peter Eisentraut <[EMAIL PROTECTED]> writes:
 

Thomas Swan writes:
   

Have you considered something similar to the Mozilla tinderbox approach
where you have a daemon checkout the cvs, compile, run regression tests,
and report a status or be able to report a status?
 

 

Even if you could achieve near complete coverage of the platforms,
platform versions, and auxilliary software versions and combinations that
PostgreSQL runs with, in most cases, something breaks on a new
version or combination of these things.
   

Still, whenever we're doing something that interacts at all with the OS,
it seems we get breakages that don't show in the original author's
testing, but only pop up days to months later when some beta tester
tries the code on platform P or using option Q.  The current
difficulties with the IPv6 patches are a fine case in point.
If we could get feedback more easily about whether a proposed patch
compiles and passes regression on a variety of platforms, we could
reduce the pain involved by a great deal, simply because the problems
could be fixed while the code is still fresh in mind.
I don't think there is any company involved with Postgres that is
willing to commit the resources to run a Mozilla-style tinderbox setup
singlehanded.  But I wonder whether we couldn't set up something that is
community-based: get a few dozen people with different platforms to
volunteer to check the code regularly on their own machines.  I'm
imagining a cron job that fires daily in the wee hours, pulls the latest
CVS tip, does "make distclean; configure; make; make check", and mails
the results to someplace that puts 'em up on our website.
It's possible that we could adapt the tinderbox software to work this
way, but even if we had to write our own, it seems like a fairly simple
task.  And it'd give *much* better feedback on porting problems than we
have now.  Sure, there will always be corner cases you don't catch,
but the first rule of testing is the sooner you find a bug the cheaper
it is to fix.	
 

Is it possible the sourceforge compile farms could be used for some of 
the automated testing?  I'm not sure how that system works, but it could 
be worth looking into.

			regards, tom lane

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
   (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
 



---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
   (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

Doesn't matter, this entire approach has a fundamental flaw. If the 
lungs are "empty" ... that means that the guy has an open thorax, the 
lungs are collapsed, and you'll probably have problems catching his 
attention to make your claim.

Jan

The Hermit Hacker wrote:
On Wed, 25 Jun 2003, Shridhar Daithankar wrote:

On 25 Jun 2003 at 14:50, Andreas Pflug wrote:
> Jan Wieck wrote:
> > Change that into "* Remove bugs from source code" and get a patent on
> > it. Should be a nobrainer (as in those guy's have no brains)
> > considering that NetFlix even got a patent on DVD subscription rentals:
> >
> > http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99
>
> I'm applying for a patent on breathing now.
> The trick I found is reversing the direction of airflow in a regular way.
>
> The algorithm seems apparently simple, but it really makes the deal:
>
> Step 1.
> If your lungs are empty, let air flow into them through some air intake.
> This airflow might be ducted by some bronchial or additional tubing.
>
> Step 2 (optional)
> As soon as there's enough air in the lungs, you may decide to hold it
> there for a while. Some time limits might apply, please consult some
> specialist for details.
>
> Step 3
> Press the air out of the lungs, using some muscles or externally applied
> force on the chest. The air will eventually escape through some body
> opening. Another patent I'll be applying for covers the use of nostrils
> for this purpose.
>
> Step 4
> Restart the cycle at step 1.
>
>
> Regards and don't dare to try this without royalty fees!
Oops... I challenge it for prior art..:-)
Why, are you older then he is?  I think that is about the only way that
'prior art' would apply here, no? :)
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
   http://www.postgresql.org/docs/faqs/FAQ.html


--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 6: Have you searched our list archives?
  http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Wed, 25 Jun 2003, Kevin Brown wrote:

> So...would it make sense to create a gborg project to which people who
> have written their own test suites can contribute whatever code and data
> they feel comfortable releasing?  As a gborg project, it would be
> separate from the main PG distribution and would thus have no impact on
> the build process or anything like that.  But at the same time, if there
> are any ideas on testing that people have had, they could be shared with
> others through that mechanism.
>
> And any tests which prove to be particularly useful could make their way
> into the PG distribution if people here wish.
>
> Of course, like anything else this could be a bad (or perhaps redundant)
> idea.  :-)

It doesn't sound like a bad idea ... but, it pretty much comes down to the
original thread: are you willing to step up and maintain such a project?


---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Wed, 25 Jun 2003, Shridhar Daithankar wrote:

> On 25 Jun 2003 at 14:50, Andreas Pflug wrote:
> > Jan Wieck wrote:
> > > Change that into "* Remove bugs from source code" and get a patent on
> > > it. Should be a nobrainer (as in those guy's have no brains)
> > > considering that NetFlix even got a patent on DVD subscription rentals:
> > >
> > > http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99
> >
> > I'm applying for a patent on breathing now.
> > The trick I found is reversing the direction of airflow in a regular way.
> >
> > The algorithm seems apparently simple, but it really makes the deal:
> >
> > Step 1.
> > If your lungs are empty, let air flow into them through some air intake.
> > This airflow might be ducted by some bronchial or additional tubing.
> >
> > Step 2 (optional)
> > As soon as there's enough air in the lungs, you may decide to hold it
> > there for a while. Some time limits might apply, please consult some
> > specialist for details.
> >
> > Step 3
> > Press the air out of the lungs, using some muscles or externally applied
> > force on the chest. The air will eventually escape through some body
> > opening. Another patent I'll be applying for covers the use of nostrils
> > for this purpose.
> >
> > Step 4
> > Restart the cycle at step 1.
> >
> >
> > Regards and don't dare to try this without royalty fees!
>
> Oops... I challenge it for prior art..:-)

Why, are you older then he is?  I think that is about the only way that
'prior art' would apply here, no? :)


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[Josh Berkus]

Kevin,

> So...would it make sense to create a gborg project to which people who
> have written their own test suites can contribute whatever code and
> data they feel comfortable releasing?  As a gborg project, it would be
> separate from the main PG distribution and would thus have no impact on
> the build process or anything like that.  But at the same time, if there
> are any ideas on testing that people have had, they could be shared with
> others through that mechanism.

+1

-Josh

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Kevin Brown]

Tom Lane wrote:
> Josh Berkus <[EMAIL PROTECTED]> writes:
> > 3) Dann is proposing not just a feature but sweeping changes to the way our 
> > commmunity works, despite having been a member of this community for about 3 
> > weeks total.
> 
> In Dann's defense, I didn't think I heard him proposing that we get rid
> of our existing testing methods, but rather that we see if we can't
> supplement them with something more formal.  This strikes me as a
> perfectly reasonable proposal.  However, he hasn't succeeded in
> convincing anyone else to put their time into it (for reasons that
> are also perfectly reasonable, namely that we find that our existing
> methods do pretty well, and we don't have the manpower to create a large
> formal testing structure ... even if we thought it would repay the effort,
> which many of us doubt).  So it's his itch to scratch, or not.
> 
> Unless there's something more profitable to be said on the subject,
> could we drop the thread?

One thing that came out of the thread is the fact that many people who
use PostgreSQL do testing in many different ways, and that much of the
stability of PostgreSQL can be attributed to that.

It occurs to me that there may be (perhaps) a lot of duplication of
effort there that could be reduced a bit.

So...would it make sense to create a gborg project to which people who
have written their own test suites can contribute whatever code and
data they feel comfortable releasing?  As a gborg project, it would be
separate from the main PG distribution and would thus have no impact on
the build process or anything like that.  But at the same time, if there
are any ideas on testing that people have had, they could be shared with
others through that mechanism.

And any tests which prove to be particularly useful could make their
way into the PG distribution if people here wish.


Of course, like anything else this could be a bad (or perhaps redundant)
idea.  :-)


-- 
Kevin Brown   [EMAIL PROTECTED]

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Tom Lane]

Josh Berkus <[EMAIL PROTECTED]> writes:
> 3) Dann is proposing not just a feature but sweeping changes to the way our 
> commmunity works, despite having been a member of this community for about 3 
> weeks total.

In Dann's defense, I didn't think I heard him proposing that we get rid
of our existing testing methods, but rather that we see if we can't
supplement them with something more formal.  This strikes me as a
perfectly reasonable proposal.  However, he hasn't succeeded in
convincing anyone else to put their time into it (for reasons that
are also perfectly reasonable, namely that we find that our existing
methods do pretty well, and we don't have the manpower to create a large
formal testing structure ... even if we thought it would repay the effort,
which many of us doubt).  So it's his itch to scratch, or not.

Unless there's something more profitable to be said on the subject,
could we drop the thread?

regards, tom lane

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Josh Berkus]

Jan,

> There have been a good number of examples where the one who raised an
> issue isn't just of the format to implement it. So someone else jumped
> in and did it instead. I don't need to pick any particular samples, you
> know that it happened a few times.

Sure.  But in those cases, the fix/feature was something that the consensus on 
-Hackers agreed needed to be done, someday.   Putting those items on the TODO 
was an acknowledgement of that decision, even though they won't be 
implemented until we aquire more contributors.  As an example, the various 
PL/pgSQL enhancements which Patrick and I pushed onto the TODO list, which 
are still features in search of a programmer. 

 That took, as I recall, about a month of lobbying on this list, which 
required me to prove a) how our current PL/pgSQL was weak, and b) how the 
improvements would benefit the project overall, and c) how many people were 
interested in the improvements.

There are 3 problems with Dann's proposal that have caused it to be shot down 
in flames instead of being put on the TODO list:
1) Most people on this list ... particularly, most contributors ... do not 
agree with Dann's proposal.  This is in no little part due to Dann's lack of 
material evidence for his case.
2) Dann has a particularly abrasive and insulting communication style that 
hasn't helped his case any, either.
3) Dann is proposing not just a feature but sweeping changes to the way our 
commmunity works, despite having been a member of this community for about 3 
weeks total.

As such, this discussion is an example of the "Open Source Process" (tm) 
working correctly.  Someone brought up a proposal, it was challenged, they 
were not able to defend the proposal, and it was voted down.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[Kaare Rasmussen]

> Change that into "* Remove bugs from source code" and get a patent on
> it. Should be a nobrainer (as in those guy's have no brains) considering
> that NetFlix even got a patent on DVD subscription rentals:

And for all the nice royalty money*, think about what can be done to 
PostgreSQL. Maybe even a test suite :-)

* Or maybe the best step is to sue IBM or Microsoft. Then again, maybe not. Do 
they care about bug removal? ;-)

-- 
Kaare Rasmussen--Linux, spil,--Tlf:3816 2582
Kaki Datatshirts, merchandize  Fax:3816 2501
Howitzvej 75   Åben 12.00-18.00Email: [EMAIL PROTECTED]
2000 FrederiksbergLørdag 12.00-16.00   Web:  www.suse.dk

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[Shridhar Daithankar]

On 25 Jun 2003 at 14:50, Andreas Pflug wrote:
> Jan Wieck wrote:
> > Change that into "* Remove bugs from source code" and get a patent on 
> > it. Should be a nobrainer (as in those guy's have no brains) 
> > considering that NetFlix even got a patent on DVD subscription rentals:
> >
> > http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99
> 
> I'm applying for a patent on breathing now.
> The trick I found is reversing the direction of airflow in a regular way.
> 
> The algorithm seems apparently simple, but it really makes the deal:
> 
> Step 1.
> If your lungs are empty, let air flow into them through some air intake. 
> This airflow might be ducted by some bronchial or additional tubing.
> 
> Step 2 (optional)
> As soon as there's enough air in the lungs, you may decide to hold it 
> there for a while. Some time limits might apply, please consult some 
> specialist for details.
> 
> Step 3
> Press the air out of the lungs, using some muscles or externally applied 
> force on the chest. The air will eventually escape through some body 
> opening. Another patent I'll be applying for covers the use of nostrils 
> for this purpose.
> 
> Step 4
> Restart the cycle at step 1.
> 
> 
> Regards and don't dare to try this without royalty fees!

Oops... I challenge it for prior art..:-)

Bye
 Shridhar

--
Gomme's Laws:   (1) A backscratcher will always find new itches.(2) Time 
accelerates.(3) The weather at home improves as soon as you go away.


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[Andreas Pflug]

Jan Wieck wrote:

Change that into "* Remove bugs from source code" and get a patent on 
it. Should be a nobrainer (as in those guy's have no brains) 
considering that NetFlix even got a patent on DVD subscription rentals:

http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99
I'm applying for a patent on breathing now.
The trick I found is reversing the direction of airflow in a regular way.
The algorithm seems apparently simple, but it really makes the deal:

Step 1.
If your lungs are empty, let air flow into them through some air intake. 
This airflow might be ducted by some bronchial or additional tubing.

Step 2 (optional)
As soon as there's enough air in the lungs, you may decide to hold it 
there for a while. Some time limits might apply, please consult some 
specialist for details.

Step 3
Press the air out of the lungs, using some muscles or externally applied 
force on the chest. The air will eventually escape through some body 
opening. Another patent I'll be applying for covers the use of nostrils 
for this purpose.

Step 4
Restart the cycle at step 1.
Regards and don't dare to try this without royalty fees!

Andreas



---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
   (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

Kaare Rasmussen wrote:
> That seems too vague for TODO.  We might as well add "Make PostgreSQL
> faster".  :-)
'K, can you add that one too? :)
How about "* Remove all bugs from the source". Can you put that in TODO ?

:-)

Change that into "* Remove bugs from source code" and get a patent on 
it. Should be a nobrainer (as in those guy's have no brains) considering 
that NetFlix even got a patent on DVD subscription rentals:

http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99

Jan

--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Kaare Rasmussen]

> > That seems too vague for TODO.  We might as well add "Make PostgreSQL
> > faster".  :-)
> 'K, can you add that one too? :)

How about "* Remove all bugs from the source". Can you put that in TODO ?

:-)

-- 
Kaare Rasmussen--Linux, spil,--Tlf:3816 2582
Kaki Datatshirts, merchandize  Fax:3816 2501
Howitzvej 75   Åben 12.00-18.00Email: [EMAIL PROTECTED]
2000 FrederiksbergLørdag 12.00-16.00   Web:  www.suse.dk

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[scott.marlowe]

On Tue, 24 Jun 2003, Dann Corbit wrote:

> > -Original Message-
> > From: The Hermit Hacker [mailto:[EMAIL PROTECTED] 
> > Sent: Tuesday, June 24, 2003 6:10 PM
> > To: Dann Corbit
> > Cc: The Hermit Hacker; Jan Wieck; scott.marlowe; Bruce 
> > Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> > Subject: RE: [HACKERS] Two weeks to feature freeze
> > 
> > 
> > On Tue, 24 Jun 2003, Dann Corbit wrote:
> > 
> > > I did something about it.  I raised the issue.
> > > Is it really so that whoever it is that raises a question 
> > is also the 
> > > one who must fix the issue raised? A strange model indeed.
> > 
> > Its worked for us ...
> > 
> > Wait, I know what should make you happy ... it won't get 
> > anytihng done, but ...
> > 
> > Bruce, can you add "* Improve Testing" to the TODO list
> 
> It is surely a titanic mistake to bring up any issue on this list if you
> do not plan to fix it yourself.

No, it's not.  But you have to realize that when you bring up a "problem" 
without a solution, you are, in effect, asking someone else to solve your 
problem.  I do that all the time.  I don't code postgresql (well, I've 
hacked around in it a bit for fun, but most of it is over my head.)

So, then I bring up something (like the screwy date mangling / parsing 
misfeature we've been discussing this last week) I KNOW I'm asking the 
programmers for a favor to fix it, and I know that if they all said, no, 
we don't have time to fix it, if you want it done you'd better get 
hacking, then I can either get hacking or wait until someone has time to 
fix it.

I.e. I have to be civil and present my case in a convincing enough manner 
to get someone else to do it.

The bigger the change, the more likely it is that it'll get thrown back at 
the person suggesting it.

A prime example is using a threaded model.  About every three months or so 
someone shows up waving their hands about how Postgresql just can't be 
fast without threads.  No code, no benchmarks, no proof, just a rough 
feeling they got from reading a magazine article.  Invariably, the issue 
is far more complex than just "make Postgresql threaded" and the person 
suggesting it can "justify" why someone else should spend months doing it, 
but they can't be bothered themselves.

In the Open Source universe, if you want a feature, you have to be willing 
to "pay for it."  Whether that's with code or a checkbook, either way, 
TANSTAAFL.

In the case of better testing, if you're willing to get out your checkbook 
and start pumping out cash to a few of the developers here on the list, 
I'd sure someone would jump right up and start writing your test suite.

Or, you can code it yourself.

Or, you can convince someone that the testing is necessary and they'll 
volunteer their time.

For me, more testing would gain me nothing.  I already test postgresql on 
a test box before throwing it into production.  I test it for load, 
response, correctness, etc... and if I miss something, it's usually a 
pretty small issue, and it gets fixed fast.

I'd much rather see time and effort go into the things they go into, like 
optimizing the query planner, in() with subselects, full text indexing, 
and many other things.

There is a finite pool of hacker skill poured into Postgresql, and pouring 
more of it into testing means less gets poured into development, and right 
now, I've got all the testing I need, as do most of the developers and 
users I know.

Open Source works because some scratches and itch.  right now, you're the 
only one with an itch for more testing.  If you don't scratch it, it 
likely won't get scratched any time soon.

who knows, in a year, maybe someone else will get the itch and scratch it.  
But your arguments have been unconvincing, since the reality of Postgresql 
is that it is the absolutely most reliable database I know of, and has one 
of the fastest turnaround times for bug fixes when they do pop up.

Does apache have a comprehensive test suite?  Not that I know of.  Neither 
does the Linux kernel or the BSD kernel.  But, they all get the job done 
better than their commercial counterparts with far less headache.

This discussion hasn't been a complete waste of time, but your arguments 
have bordered on insulting to those of us who currently DO the testing on 
our own machines.  You've dismissed our testing out of hand on several 
occasions as insufficient, when at least we ARE testing postgresql, even 
if it doesn't meat up to your higher standards.  

If you want to raise the bar, then YOU get to raise it.


---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Tue, 24 Jun 2003, Jan Wieck wrote:

> The Hermit Hacker wrote:
> > On Tue, 24 Jun 2003, Bruce Momjian wrote:
> >
> >> The Hermit Hacker wrote:
> >> > On Tue, 24 Jun 2003, Dann Corbit wrote:
> >> >
> >> > > I did something about it.  I raised the issue.
> >> > > Is it really so that whoever it is that raises a question is also the
> >> > > one who must fix the issue raised?
> >> > > A strange model indeed.
> >> >
> >> > Its worked for us ...
> >> >
> >> > Wait, I know what should make you happy ... it won't get anytihng done,
> >> > but ...
> >> >
> >> > Bruce, can you add "* Improve Testing" to the TODO list
> >>
> >> That seems too vague for TODO.  We might as well add "Make PostgreSQL
> >> faster".  :-)
> >
> > 'K, can you add that one too? :)
>
> Make his life easier:
>
>  Replace the entire TODO with "Make PostgreSQL better"
>
> That pretty much summs it up, no?

Oh, I like that ... definitely leaves it open to the interpretation of the
reader as to what would make it better :)


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

The Hermit Hacker wrote:
On Tue, 24 Jun 2003, Bruce Momjian wrote:

The Hermit Hacker wrote:
> On Tue, 24 Jun 2003, Dann Corbit wrote:
>
> > I did something about it.  I raised the issue.
> > Is it really so that whoever it is that raises a question is also the
> > one who must fix the issue raised?
> > A strange model indeed.
>
> Its worked for us ...
>
> Wait, I know what should make you happy ... it won't get anytihng done,
> but ...
>
> Bruce, can you add "* Improve Testing" to the TODO list
That seems too vague for TODO.  We might as well add "Make PostgreSQL
faster".  :-)
'K, can you add that one too? :)
Make his life easier:

Replace the entire TODO with "Make PostgreSQL better"

That pretty much summs it up, no?

Jan

--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

The Hermit Hacker wrote:
On Tue, 24 Jun 2003, Dann Corbit wrote:

I did something about it.  I raised the issue.
Is it really so that whoever it is that raises a question is also the
one who must fix the issue raised?
A strange model indeed.
Its worked for us ...
Sorry Marc, but that ain't entirely true.

There have been a good number of examples where the one who raised an 
issue isn't just of the format to implement it. So someone else jumped 
in and did it instead. I don't need to pick any particular samples, you 
know that it happened a few times.

And don't get the wrong picture. Yes, Dann is "just talking" here on the 
testing methodology front. But as much as it looks like we two hate each 
other on this thread, we actually start working together on a totally 
different issue. And to say the least, I like his version better than 
Katie's ... 'nuff said.

Jan


Wait, I know what should make you happy ... it won't get anytihng done,
but ...
Bruce, can you add "* Improve Testing" to the TODO list

> Someone has to spearhead it
> ... you seem to be passionate about seeing it happen, but
> don't care enough to do anything about it ...
Don't care and won't do are not the same thing.
Well, actually, they are ... if someone doesn't care, they aren't going to
do, are they?


--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
 subscribe-nomail command to [EMAIL PROTECTED] so that your
 message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Tue, 24 Jun 2003, Dann Corbit wrote:

> > > Don't care and won't do are not the same thing.
> >
> > Well, actually, they are ... if someone doesn't care, they
> > aren't going to do, are they?
>
> You have had the time to do everything you ever cared about?

No no, that isn't what he is arguing (or I'm miss reading) ... he said
that "not caring about something *and* not doing it aren't the same thing"
... which isnt' the same as caring but not having the time ... is it?

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Tue, 24 Jun 2003, Bruce Momjian wrote:

> The Hermit Hacker wrote:
> > On Tue, 24 Jun 2003, Dann Corbit wrote:
> >
> > > I did something about it.  I raised the issue.
> > > Is it really so that whoever it is that raises a question is also the
> > > one who must fix the issue raised?
> > > A strange model indeed.
> >
> > Its worked for us ...
> >
> > Wait, I know what should make you happy ... it won't get anytihng done,
> > but ...
> >
> > Bruce, can you add "* Improve Testing" to the TODO list
>
> That seems too vague for TODO.  We might as well add "Make PostgreSQL
> faster".  :-)

'K, can you add that one too? :)


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

I think it was a useful discussion.  I find it interesting to compare
our clearly ad-hock testing methods to traditional commercial testing
strategies.  I think our results are very good, but it does look awful
"ad-hock" and it is easy to see how someone would question its
effectiveness.

Of course, the whole open source development model seems ad-hock too,
and on the surface seems inferior to a commercial software development
model, but there again, the proof is in the result.

---

Dann Corbit wrote:
> > -Original Message-
> > From: The Hermit Hacker [mailto:[EMAIL PROTECTED] 
> > Sent: Tuesday, June 24, 2003 6:10 PM
> > To: Dann Corbit
> > Cc: The Hermit Hacker; Jan Wieck; scott.marlowe; Bruce 
> > Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> > Subject: RE: [HACKERS] Two weeks to feature freeze
> > 
> > 
> > On Tue, 24 Jun 2003, Dann Corbit wrote:
> > 
> > > I did something about it.  I raised the issue.
> > > Is it really so that whoever it is that raises a question 
> > is also the 
> > > one who must fix the issue raised? A strange model indeed.
> > 
> > Its worked for us ...
> > 
> > Wait, I know what should make you happy ... it won't get 
> > anytihng done, but ...
> > 
> > Bruce, can you add "* Improve Testing" to the TODO list
> 
> It is surely a titanic mistake to bring up any issue on this list if you
> do not plan to fix it yourself.
>  
> > > > Someone has to spearhead it
> > > > ... you seem to be passionate about seeing it happen, but 
> > don't care 
> > > > enough to do anything about it ...
> > >
> > > Don't care and won't do are not the same thing.
> > 
> > Well, actually, they are ... if someone doesn't care, they 
> > aren't going to do, are they?
> 
> You have had the time to do everything you ever cared about?
> 
> It is really true that I have made a titanic waste of time in an effort
> to get someone else to do something about it or at least get them
> interested in it.  I won't waste my time in that way again.  I deeply
> rue the time I have wasted already.
> 
> 

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: The Hermit Hacker [mailto:[EMAIL PROTECTED] 
> Sent: Tuesday, June 24, 2003 6:10 PM
> To: Dann Corbit
> Cc: The Hermit Hacker; Jan Wieck; scott.marlowe; Bruce 
> Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> Subject: RE: [HACKERS] Two weeks to feature freeze
> 
> 
> On Tue, 24 Jun 2003, Dann Corbit wrote:
> 
> > I did something about it.  I raised the issue.
> > Is it really so that whoever it is that raises a question 
> is also the 
> > one who must fix the issue raised? A strange model indeed.
> 
> Its worked for us ...
> 
> Wait, I know what should make you happy ... it won't get 
> anytihng done, but ...
> 
> Bruce, can you add "* Improve Testing" to the TODO list

It is surely a titanic mistake to bring up any issue on this list if you
do not plan to fix it yourself.
 
> > > Someone has to spearhead it
> > > ... you seem to be passionate about seeing it happen, but 
> don't care 
> > > enough to do anything about it ...
> >
> > Don't care and won't do are not the same thing.
> 
> Well, actually, they are ... if someone doesn't care, they 
> aren't going to do, are they?

You have had the time to do everything you ever cared about?

It is really true that I have made a titanic waste of time in an effort
to get someone else to do something about it or at least get them
interested in it.  I won't waste my time in that way again.  I deeply
rue the time I have wasted already.


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

The Hermit Hacker wrote:
> On Tue, 24 Jun 2003, Dann Corbit wrote:
> 
> > I did something about it.  I raised the issue.
> > Is it really so that whoever it is that raises a question is also the
> > one who must fix the issue raised?
> > A strange model indeed.
> 
> Its worked for us ...
> 
> Wait, I know what should make you happy ... it won't get anytihng done,
> but ...
> 
> Bruce, can you add "* Improve Testing" to the TODO list
 
That seems too vague for TODO.  We might as well add "Make PostgreSQL
faster".  :-)

--
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Tue, 24 Jun 2003, Dann Corbit wrote:

> I did something about it.  I raised the issue.
> Is it really so that whoever it is that raises a question is also the
> one who must fix the issue raised?
> A strange model indeed.

Its worked for us ...

Wait, I know what should make you happy ... it won't get anytihng done,
but ...

Bruce, can you add "* Improve Testing" to the TODO list

> > Someone has to spearhead it
> > ... you seem to be passionate about seeing it happen, but
> > don't care enough to do anything about it ...
>
> Don't care and won't do are not the same thing.

Well, actually, they are ... if someone doesn't care, they aren't going to
do, are they?


---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: The Hermit Hacker [mailto:[EMAIL PROTECTED] 
> Sent: Tuesday, June 24, 2003 5:26 PM
> To: Dann Corbit
> Cc: Jan Wieck; scott.marlowe; Bruce Momjian; Tom Lane; Jason 
> Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> On Mon, 23 Jun 2003, Dann Corbit wrote:
> 
> > > -Original Message-
> > > From: Jan Wieck [mailto:[EMAIL PROTECTED]
> > > Sent: Monday, June 23, 2003 10:30 PM
> > > To: Dann Corbit
> > > Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; 
> > > PostgreSQL-development
> > > Subject: Re: [HACKERS] Two weeks to feature freeze
> > [snip]
> > > I personally think you don't actually ever did any 
> software testing 
> > > yourself. You are not really talking from experience, are you? So 
> > > please, show me what you have now, or get one more plus on my 
> > > minus-list.
> >
> > I have already posted enough information to show my qualitications.
> >
> > Whether I am qualified or not is irrelevant to whether the argument 
> > has merit or not.
> >
> > I have raised what I consider to be an important issue.
> 
> You have raised a point that you are not prepared to do 
> anything about though 

I did something about it.  I raised the issue.
Is it really so that whoever it is that raises a question is also the
one who must fix the issue raised?
A strange model indeed.

> ... nobody disagrees with you about 
> adding more testing, 

Actually some do, but that's neither here nor there.

> but if you aren't willing to do the 
> work, why should anyone else be? 

Perhaps they have the time and care about the result.

> Someone has to spearhead it 
> ... you seem to be passionate about seeing it happen, but 
> don't care enough to do anything about it ...

Don't care and won't do are not the same thing.
 

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Tue, 24 Jun 2003, Robert Treat wrote:

> On Mon, 2003-06-23 at 21:36, The Hermit Hacker wrote:
> > On Mon, 23 Jun 2003, Robert Treat wrote:
> >
> > > > The target-date-based approach we've taken in the last couple of
> > > > releases seems much more productive.
> > > >
> > >
> > > productive on a small scale; for sure. productive for large scale
> > > features...  well, that's why it's being discussed.
> >
> > 'K, but if we extend the dev cycle in order to get 'foo' in, how is that
> > better then having 'foo' continue to be developed thru the release and
> > committed in the next cycle?
> >
>
> I think it makes it easier to code 'foo' since you're not coding against
> (quite as much of) a moving target.

I *soo* disagree with this one ... the only way that postgresql is
going to stop being a moving target so that someone can code 'foo' is if
everyone else goes to sleep until that happens ...

> It could also help people plan better since they would know that foo is
> coming in the next release.

'K, that helps the end users, but that doesn't do anything for the person
developing 'foo' ...

> i'm sure everyone who doesn't want foo would agree with that position.
> The counter though is those folks who did want foo but now have to wait
> 4-6 months for the next release since foo took a month longer to develop
> than was initially planned.

Ya, but, based on past experience (hell, this release has been a good
example) ... you delay the release because developer of 'foo' figures "oh,
it will be ready in another month", and then something comes up that
causes that "commitment" not to happen, so we delay it "yet another
month"?  And I'm not saying that the second delay was due to
mis-estimating the time needed ... suddenly getting really busy on a
contract, a day job, a death in the family, etc ... you cannot predict
what could cause a delay, or how long such a delay would take ...

> the whole discussion is based on how do we get big projects done when no
> one is motivated to work on 'foo' until there faced with a deadline;
> this idea puts the pressure on 'foo' developers from the get go. i'm not
> saying this a guaranteed way to solve that problem but i think it is a
> possible solution. i'm sure there will be big projects most people don't
> care about (win32) and big projects most people would like (replication)
> so the amount of like or dislike of the idea would be relative in
> practice, the key question is would this actually motivate folks more to
> get big projects finished faster? since there are only a handful of
> folks who fall into that category they can decide for themselves, and if
> it wouldn't motivate them, then the question can be asked again, what
> would?

First, we already seen that such a 'pressure' doesn't matter, especially
if when push comes to shove, they know we'll postpone to accommodate them
...

Second ... I don't believe the problem *is* the release cycles ... I think
the problem that needs a solution is how to "open up" these large projects
so that the deadline(s) don't fall on ones person's shoulders to get done
.. how do we encourage/promote "work groups" for the large projects?


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Mon, 23 Jun 2003, Dann Corbit wrote:

> > Would it be nice if we had more tests?  Yes.  In fact, one of
> > the items on my
> > personal todo list is to devise a more versatile performance
> > test than
> > pgbench for testing postgresql parameters, builds, and
> > installations.  But
> > it's not getting done by me carping at people on the Hackers
> > list.  It'll get
> > done when I spend a long weekend writing Perl.
> >
> > Put up or shut up time, Dann.
>
> It's shut up, then.  I'm not willing to commit to this effort.

Woo hoo ... now *that* was the longest, useless thread I think we've had
here so far .. we almost need to start a 'hall of shameful threads' ...


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Mon, 23 Jun 2003, Dann Corbit wrote:

> > -Original Message-
> > From: Jan Wieck [mailto:[EMAIL PROTECTED]
> > Sent: Monday, June 23, 2003 10:30 PM
> > To: Dann Corbit
> > Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl;
> > PostgreSQL-development
> > Subject: Re: [HACKERS] Two weeks to feature freeze
> [snip]
> > I personally think you don't actually ever did any software testing
> > yourself. You are not really talking from experience, are you? So
> > please, show me what you have now, or get one more plus on my
> > minus-list.
>
> I have already posted enough information to show my qualitications.
>
> Whether I am qualified or not is irrelevant to whether the argument has
> merit or not.
>
> I have raised what I consider to be an important issue.

You have raised a point that you are not prepared to do anything about
though ... nobody disagrees with you about adding more testing, but if you
aren't willing to do the work, why should anyone else be?  Someone has to
spearhead it ... you seem to be passionate about seeing it happen, but
don't care enough to do anything about it ...

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

'K, and do you have any ETA on when you'll have this translated into some
useful tests that we can incorporate?

On Mon, 23 Jun 2003, Dann Corbit wrote:

> Here is a list of a small sample of the citations available from the ACM
> on software testing:
>
> http://portal.acm.org/citation.cfm?id=581358&coll=portal&dl=ACM&CFID=657
> 0092&CFTOKEN=81653602
> http://portal.acm.org/citation.cfm?id=376180&coll=portal&dl=ACM&CFID=657
> 0092&CFTOKEN=81653602
> http://portal.acm.org/citation.cfm?id=367020&coll=portal&dl=ACM&CFID=657
> 0092&CFTOKEN=81653602
> http://portal.acm.org/citation.cfm?id=308790&coll=portal&dl=ACM&CFID=657
> 0092&CFTOKEN=81653602
> http://portal.acm.org/citation.cfm?id=257668&coll=portal&dl=ACM&CFID=657
> 0092&CFTOKEN=81653602
> http://portal.acm.org/citation.cfm?id=248262&coll=portal&dl=ACM&CFID=657
> 0092&CFTOKEN=81653602
> http://portal.acm.org/citation.cfm?id=227759&coll=portal&dl=ACM&CFID=657
> 0092&CFTOKEN=81653602
>
> These articles are careful, scientific studies that have been
> extensively peer reviewed.
>
> These articles indicate that testing is a good idea.
>
> ---(end of broadcast)---
> TIP 6: Have you searched our list archives?
>
>http://archives.postgresql.org
>

Marc G. Fournier   ICQ#7615664   IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: [EMAIL PROTECTED]   secondary: [EMAIL PROTECTED]|postgresql}.org

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

Robert Treat wrote:
> the whole discussion is based on how do we get big projects done when no
> one is motivated to work on 'foo' until there faced with a deadline; 
> this idea puts the pressure on 'foo' developers from the get go. i'm not
> saying this a guaranteed way to solve that problem but i think it is a
> possible solution. i'm sure there will be big projects most people don't
> care about (win32) and big projects most people would like (replication)
> so the amount of like or dislike of the idea would be relative in
> practice, the key question is would this actually motivate folks more to
> get big projects finished faster? since there are only a handful of
> folks who fall into that category they can decide for themselves, and if
> it wouldn't motivate them, then the question can be asked again, what
> would?

I can confirm that there are several people working on Win32/PITR right
now, maybe four, that wouldn't if we hadn't set the beta freeze at July
1 --- so such pressure is a real motivator --- though certainly this
isn't the way we want to motivate developers.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Robert Treat]

On Mon, 2003-06-23 at 21:36, The Hermit Hacker wrote:
> On Mon, 23 Jun 2003, Robert Treat wrote:
> 
> > > The target-date-based approach we've taken in the last couple of
> > > releases seems much more productive.
> > >
> >
> > productive on a small scale; for sure. productive for large scale
> > features...  well, that's why it's being discussed.
> 
> 'K, but if we extend the dev cycle in order to get 'foo' in, how is that
> better then having 'foo' continue to be developed thru the release and
> committed in the next cycle?
> 

I think it makes it easier to code 'foo' since you're not coding against
(quite as much of) a moving target. I could also help developers stay
focused on particular projects since they are the "hot potato" for a
given release. It could also help people plan better since they would
know that foo is coming in the next release.

> If it takes foo 6 months to develop, I'd rather have the release happen
> after 4 months as per normal (or close to it) and have 'foo' brought in
> part way into dev cycle 2 ... at least then, if 'foo' ends up taking 7
> months, we aren't delaying even further ...
> 

i'm sure everyone who doesn't want foo would agree with that position.
The counter though is those folks who did want foo but now have to wait
4-6 months for the next release since foo took a month longer to develop
than was initially planned.

> Its not like our dev cycles have 'idle periods' where nothing is happening
> and we're waiting for a feature to come along ... there is *alot* of
> changes going on that affect alot of ppl that don't really care about
> feature 'foo' coming along ...
> 

this doesn't really change anything for those folks, since the only
rational is "every 6 months we do a release." ie. there are *alot* of
changes going on that affect alot of ppl that don't really care about
waiting n more months... 

the whole discussion is based on how do we get big projects done when no
one is motivated to work on 'foo' until there faced with a deadline; 
this idea puts the pressure on 'foo' developers from the get go. i'm not
saying this a guaranteed way to solve that problem but i think it is a
possible solution. i'm sure there will be big projects most people don't
care about (win32) and big projects most people would like (replication)
so the amount of like or dislike of the idea would be relative in
practice, the key question is would this actually motivate folks more to
get big projects finished faster? since there are only a handful of
folks who fall into that category they can decide for themselves, and if
it wouldn't motivate them, then the question can be asked again, what
would?

Robert Treat
-- 
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

Dann Corbit wrote:
-Original Message-
From: Jan Wieck [mailto:[EMAIL PROTECTED] 
Sent: Monday, June 23, 2003 10:30 PM
To: Dann Corbit
Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; 
PostgreSQL-development
Subject: Re: [HACKERS] Two weeks to feature freeze
[snip]
I personally think you don't actually ever did any software testing 
yourself. You are not really talking from experience, are you? So 
please, show me what you have now, or get one more plus on my 
minus-list.
I have already posted enough information to show my qualitications.

Whether I am qualified or not is irrelevant to whether the argument has
merit or not.
I have raised what I consider to be an important issue.

Astute members of the list have noticed that I have not volunteered to
perform the work.  I may or may not produce some efforts towards testing
PostgreSQL.  Whether I decide to help or not is irrelevant towards the
concept of what needs to be done.
Right! Whatever you decide is totally irrelevant towards the concept of 
what needs to be done. But that wasn't the question and you wheren't in 
the position or asked for making any decisions.

And after this mail I doubt more than before that the input you can 
provide will lead to any meaningful improvement of PostgreSQL. Then 
again, you still have the chance to surprise me. I know by now that 
you're not a programmer and don't know SQL very well. But do you at 
least have some concept of your own, other than URL's pointing to 
someone elses work?

Jan

--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: Josh Berkus [mailto:[EMAIL PROTECTED] 
> Sent: Monday, June 23, 2003 10:50 PM
> To: Dann Corbit; Jan Wieck
> Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; 
> PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> Dann,
> 
> > Astute members of the list have noticed that I have not 
> volunteered to 
> > perform the work.  I may or may not produce some efforts towards 
> > testing PostgreSQL.  Whether I decide to help or not is irrelevant 
> > towards the concept of what needs to be done.
> 
> It is not irrelevant.  This is an Open Source project, not 
> some Dot-Com where 
> you can float good ideas until you go bankrupt.   If there's 
> no possibility 
> of us getting a major 3rd-party certified battery of QA tests 
> donated, why 
> bother putting it on the TODO list?
> 
> Would it be nice if we had more tests?  Yes.  In fact, one of 
> the items on my 
> personal todo list is to devise a more versatile performance 
> test than 
> pgbench for testing postgresql parameters, builds, and 
> installations.  But 
> it's not getting done by me carping at people on the Hackers 
> list.  It'll get 
> done when I spend a long weekend writing Perl.
> 
> Put up or shut up time, Dann.

It's shut up, then.  I'm not willing to commit to this effort.

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])

Re: [HACKERS] Two weeks to feature freeze

[Josh Berkus]

Dann,

> Astute members of the list have noticed that I have not volunteered to
> perform the work.  I may or may not produce some efforts towards testing
> PostgreSQL.  Whether I decide to help or not is irrelevant towards the
> concept of what needs to be done.

It is not irrelevant.  This is an Open Source project, not some Dot-Com where 
you can float good ideas until you go bankrupt.   If there's no possibility 
of us getting a major 3rd-party certified battery of QA tests donated, why 
bother putting it on the TODO list?

Would it be nice if we had more tests?  Yes.  In fact, one of the items on my 
personal todo list is to devise a more versatile performance test than 
pgbench for testing postgresql parameters, builds, and installations.  But 
it's not getting done by me carping at people on the Hackers list.  It'll get 
done when I spend a long weekend writing Perl.

Put up or shut up time, Dann.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: Jan Wieck [mailto:[EMAIL PROTECTED] 
> Sent: Monday, June 23, 2003 10:30 PM
> To: Dann Corbit
> Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; 
> PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
[snip]
> I personally think you don't actually ever did any software testing 
> yourself. You are not really talking from experience, are you? So 
> please, show me what you have now, or get one more plus on my 
> minus-list.

I have already posted enough information to show my qualitications.

Whether I am qualified or not is irrelevant to whether the argument has
merit or not.

I have raised what I consider to be an important issue.

Astute members of the list have noticed that I have not volunteered to
perform the work.  I may or may not produce some efforts towards testing
PostgreSQL.  Whether I decide to help or not is irrelevant towards the
concept of what needs to be done.

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

Dann Corbit wrote:
-Original Message-
From: Jan Wieck [mailto:[EMAIL PROTECTED] 
Sent: Monday, June 23, 2003 10:10 PM
To: scott.marlowe
Cc: Dann Corbit; Bruce Momjian; Tom Lane; Jason Earl; 
PostgreSQL-development
Subject: Re: [HACKERS] Two weeks to feature freeze

scott.marlowe wrote:
> On Mon, 23 Jun 2003, Dann Corbit wrote:
 > [Dann Corbit wrote a lot]
 > [...]
> It may be reassuring to think your product is very well 
tested before 
> it
> goes out the door, but it's a false security, proven over 
and over by 
> commercial products that simply don't work in the field because of 
> problems that the original designers never envisioned, and 
now that they 
> have a thorough and long drawn out testing cycle, it simply 
takes longer 
> and longer to get fixes, while providing little, if any, 
improvement in 
> quality.

Scott, it's worse.

It's been back in the early 90's, when we had WfW-3.11 
systems with some 
MS-Word dinosaur, and we just lost 14 days of work because it simply 
crashed on loading the document. The Microsoft support solution was 
something that lost all the formatting, indexing and cross 
references of 
a structured 250 page concept. I don't remember the exact 
procedure as 
my brain cells did overcharge, but the dummy on the hotline really 
believed that their thoroughly tested software wasn't the problem and 
that the error lies within our document. That that was a 
file, written 
by their thoroughly tested software was a point she really 
didn't catch.

This dumb hotline girl is the type of people, Dann Corbit's test 
strategy will reassure. Plus maybe a few (unfortunately important but 
otherwise useless) managers. Other than that, it'll not make 
the life of 
the average DBA any better. Big amounts of useless tests just give 
otherwise clueless people the false impression, the error must be 
somewhere else. MySQL's crash-me is a perfect example for that.
Do you really believe that such disasters were the result of careful
testing before release?
Everyone who thinks a careful test plan and implementation is a bad idea
is very, very wrong.
IMO-YMMV.
No, I do not.

But again, where is your "careful test plan" please? All I have seen 
from you so far is asking us to provide you with a careful test plan 
while dancing carefully around every single attempt to get a look at 
what you got so far.

I have written PostgreSQL regression tests. I have done consistency 
checks of entire ERP systems prior to data conversion attempts. I know 
the value of tests, whether they are against software or data. May I ask 
what you've done so far?

I personally think you don't actually ever did any software testing 
yourself. You are not really talking from experience, are you? So 
please, show me what you have now, or get one more plus on my minus-list.

Jan

--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

Here is a list of a small sample of the citations available from the ACM
on software testing:

http://portal.acm.org/citation.cfm?id=581358&coll=portal&dl=ACM&CFID=657
0092&CFTOKEN=81653602
http://portal.acm.org/citation.cfm?id=376180&coll=portal&dl=ACM&CFID=657
0092&CFTOKEN=81653602
http://portal.acm.org/citation.cfm?id=367020&coll=portal&dl=ACM&CFID=657
0092&CFTOKEN=81653602
http://portal.acm.org/citation.cfm?id=308790&coll=portal&dl=ACM&CFID=657
0092&CFTOKEN=81653602
http://portal.acm.org/citation.cfm?id=257668&coll=portal&dl=ACM&CFID=657
0092&CFTOKEN=81653602
http://portal.acm.org/citation.cfm?id=248262&coll=portal&dl=ACM&CFID=657
0092&CFTOKEN=81653602
http://portal.acm.org/citation.cfm?id=227759&coll=portal&dl=ACM&CFID=657
0092&CFTOKEN=81653602

These articles are careful, scientific studies that have been
extensively peer reviewed.

These articles indicate that testing is a good idea.

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: Jan Wieck [mailto:[EMAIL PROTECTED] 
> Sent: Monday, June 23, 2003 10:10 PM
> To: scott.marlowe
> Cc: Dann Corbit; Bruce Momjian; Tom Lane; Jason Earl; 
> PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> scott.marlowe wrote:
> > On Mon, 23 Jun 2003, Dann Corbit wrote:
>  > [Dann Corbit wrote a lot]
>  > [...]
> > It may be reassuring to think your product is very well 
> tested before 
> > it
> > goes out the door, but it's a false security, proven over 
> and over by 
> > commercial products that simply don't work in the field because of 
> > problems that the original designers never envisioned, and 
> now that they 
> > have a thorough and long drawn out testing cycle, it simply 
> takes longer 
> > and longer to get fixes, while providing little, if any, 
> improvement in 
> > quality.
> 
> Scott, it's worse.
> 
> It's been back in the early 90's, when we had WfW-3.11 
> systems with some 
> MS-Word dinosaur, and we just lost 14 days of work because it simply 
> crashed on loading the document. The Microsoft support solution was 
> something that lost all the formatting, indexing and cross 
> references of 
> a structured 250 page concept. I don't remember the exact 
> procedure as 
> my brain cells did overcharge, but the dummy on the hotline really 
> believed that their thoroughly tested software wasn't the problem and 
> that the error lies within our document. That that was a 
> file, written 
> by their thoroughly tested software was a point she really 
> didn't catch.
> 
> This dumb hotline girl is the type of people, Dann Corbit's test 
> strategy will reassure. Plus maybe a few (unfortunately important but 
> otherwise useless) managers. Other than that, it'll not make 
> the life of 
> the average DBA any better. Big amounts of useless tests just give 
> otherwise clueless people the false impression, the error must be 
> somewhere else. MySQL's crash-me is a perfect example for that.

Do you really believe that such disasters were the result of careful
testing before release?

Everyone who thinks a careful test plan and implementation is a bad idea
is very, very wrong.

IMO-YMMV.


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

scott.marlowe wrote:
On Mon, 23 Jun 2003, Dann Corbit wrote:
> [Dann Corbit wrote a lot]
> [...]
It may be reassuring to think your product is very well tested before it 
goes out the door, but it's a false security, proven over and over by 
commercial products that simply don't work in the field because of 
problems that the original designers never envisioned, and now that they 
have a thorough and long drawn out testing cycle, it simply takes longer 
and longer to get fixes, while providing little, if any, improvement in 
quality.
Scott, it's worse.

It's been back in the early 90's, when we had WfW-3.11 systems with some 
MS-Word dinosaur, and we just lost 14 days of work because it simply 
crashed on loading the document. The Microsoft support solution was 
something that lost all the formatting, indexing and cross references of 
a structured 250 page concept. I don't remember the exact procedure as 
my brain cells did overcharge, but the dummy on the hotline really 
believed that their thoroughly tested software wasn't the problem and 
that the error lies within our document. That that was a file, written 
by their thoroughly tested software was a point she really didn't catch.

This dumb hotline girl is the type of people, Dann Corbit's test 
strategy will reassure. Plus maybe a few (unfortunately important but 
otherwise useless) managers. Other than that, it'll not make the life of 
the average DBA any better. Big amounts of useless tests just give 
otherwise clueless people the false impression, the error must be 
somewhere else. MySQL's crash-me is a perfect example for that.

Jan

--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

Dann Corbit wrote:
-Original Message-
From: Jan Wieck [mailto:[EMAIL PROTECTED] 

What do you think is the way to become an insider?
Join the CVS tree and make a large and valuable contribution to the
project.
Go ahead, let's see it.

I have contributed a lot of crap over the years. After some other 
knowingly good and professional programmers layed hand on it, it turned 
out to be usefull. This isn't meant 100% ironically. I really don't 
think the original version of the parallel regression test, just to stay 
with the topic for this example, was worth the bit's needed for storage. 
But it was a starting point, others built on. And crappy as it was, it 
caught a bug - funny, isn't it?

So don't be afraid, let's see what you got so far.

Jan

--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Jan Wieck]

Thank's Robert,

that was probably what Bruce needs to call me every other hour now ...

Jan

Robert Treat wrote:
On Sat, 2003-06-21 at 22:55, Bruce Momjian wrote:
Andrew Dunstan wrote:
> 
> Maybe a better strategy would be to get a release out soon but not wait 6
> months for another release which would contain the Win32 port and the PITR
> stuff (assuming those aren't done in time for this release).

What concerns me is that we thought that after 7.3, and didn't do much
work on either until we got near 7.4 release.  I wonder if this is just
going to be a pattern, where these items are so large, we can't get any
motivation to focus on them until we get near the final release.  I
guess if each final release gets us a little closer, eventually we will
get there, but this process is not ideal.
The big puzzle is how do you get people (including myself) motivated to
work on a feature that takes a _huge_ amount of work to see any payoff? 
I would like to know.  Anyone?

Here's a sure to be wildly unpopular suggestion:

Make the deciding factor for the next release support of "foo" (foo can
be win32, pitr, replication, 2PC, whatever...).  This would put ample
pressure on the developers of "foo" to get it done since the whole
release rides on their shoulders. It also might encourage others to help
out more since they can't get something new until "foo" is completed.
This would also prioritize said developers time on the large project
rather than other non-prioritized tasks, and it also helps to ensure a
"killer feature" for those who want such things,
Robert Treat


--
#==#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.  #
#== [EMAIL PROTECTED] #
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
 subscribe-nomail command to [EMAIL PROTECTED] so that your
 message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Mon, 23 Jun 2003, Dann Corbit wrote:

> The resistance to testing is typical of programmers.  The PostgreSQL
> group is a group of programmers.  I don't think I can change anyone's
> mind, since the most significant people on the list don't think it is
> worth the bother.
>
> Therefore, I am going to stop harping on it.

*rofl* I believe several of us have suggested that we would welcome
submissions for improved testing ... so, what, we feel that the test that
is done is sufficient, but are willing to accept submissions to improve
it, but you aren't willing to spend the time/effort to do so?

We might be a bunch of 'typical programmers', but you definitely sound
like a typical "I want you to do something to make life easier for me, but
am not willing to expend the time or effort to do anyting about it" ...


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Mon, 23 Jun 2003, Robert Treat wrote:

> > The target-date-based approach we've taken in the last couple of
> > releases seems much more productive.
> >
>
> productive on a small scale; for sure. productive for large scale
> features...  well, that's why it's being discussed.

'K, but if we extend the dev cycle in order to get 'foo' in, how is that
better then having 'foo' continue to be developed thru the release and
committed in the next cycle?

If it takes foo 6 months to develop, I'd rather have the release happen
after 4 months as per normal (or close to it) and have 'foo' brought in
part way into dev cycle 2 ... at least then, if 'foo' ends up taking 7
months, we aren't delaying even further ...

Its not like our dev cycles have 'idle periods' where nothing is happening
and we're waiting for a feature to come along ... there is *alot* of
changes going on that affect alot of ppl that don't really care about
feature 'foo' coming along ...

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Lamar Owen]

On Monday 23 June 2003 15:42, Dann Corbit wrote:
> Let me rephrase it:
> "Only a  cohesive, organized testing effort can result in a product that
> is proven reliable."

One can never 100% prove reliability without time in the field with real-world 
data, testing or no testing.  I would dare say that there are numerous 
reliable software packages out there in OSS-land that have never had that 
sort of testing.  But it really hinges on ones definition of proof, no?

Furthermore, the beta testers here in hackers are not 'end-users' per se.  The 
people in hackers who test the betas are very valuable as our QA team.

PostgreSQL is already proven reliable, to various degrees of reliability, 
enough to where a PostgreSQL backend was approved as the new .ORG registry.  
The track record continues, without mathematically rigorous and cohesive 
testing.  Such testing would be useful, of course, by it is not required for 
our purposes.  

Those who want rigorous testing can do the rigorous testing.  You yourself 
said that your company has a separate QA team from the development team; OK, 
organize a rigorous QA team.  While you won't stop releases (unless you find 
showstopper bugs, like those that have been found by our wonderful hackers 
testers), your input into actually testing PostgreSQL (as opposed to trying 
to convince someone else to test for you) would be valuable.  But you're not 
going to get me to do it; I do enough testing of a varied nature already.  I 
do this for fun.

If you find testing fun, more power to you. :-)
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: Nigel J. Andrews [mailto:[EMAIL PROTECTED] 
> Sent: Monday, June 23, 2003 1:30 PM
> To: Dann Corbit
> Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; 
> PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
[snip]
> So you've never worked on a project where the data is of high 
> value, since in those circumstances the customer is always 
> going to apply their own acceptance testing anyway. If you 
> think that doesn't happen you try sitting through 2 solid 
> days of Y2k testing on _one_ system and tell me customers 
> never do their own testing.

Here's an old copy of my resume.  You be the judge:
ftp://cap.connx.com/C.A.P.%20Biographies/DannCorbit.htm#_Toc441048186

The burden of reliablity testing must not rest solely on the end user.
That constitutes negligence on the part of the software vendor
(IMO-YMMV).

> > Therefore, I am going to stop harping on it.
> 
> But there is no need to, as has been mentioned before, if the 
> testing is not upto your level of testing submit something 
> that makes it so. Having said that I do believe you mentioned 
> that you didn't have the time to create something but you 
> would be happy to test it, i.e. test the test.

I may or may not have time to work on a software test system for
PostgreSQL.  I do a lot of PostgreSQL work here, and at some point I
think I will have valuable contributions to the project.

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Nigel J. Andrews]

On Mon, 23 Jun 2003, Dann Corbit wrote:

> > -Original Message-
> > From: scott.marlowe [mailto:[EMAIL PROTECTED] 
> > Sent: Monday, June 23, 2003 12:25 PM
> > To: Dann Corbit
> > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> > Subject: Re: [HACKERS] Two weeks to feature freeze
> > 
> > 
> > On Mon, 23 Jun 2003, Dann Corbit wrote:
> > 
> > > Vendor A: "We think our tool is pretty solid and our end 
> > users hardly 
> > > ever turn up any bugs."
> > > 
> > > Vendor B:" We think our tool is pretty solid and our 8500 tests 
> > > currently show only 3 defects with the released version, 
> > and these are 
> > > low impact issues.  To view our current database of issues, 
> > log onto 
> > > web form ."
> > > 
> > > Which tool would you prefer to install?
> > 
> > The one I've tested and found to meet my needs, both now and 
> > by providing 
> > fixes when I needed it.

How about the one that doesn't run tests in order to show how much better it is
than the competition but to actually test operation? In other words Vendor B
has an interest in having the tests pass, what gives you the confidence it just
hasn't listed the ones that fail and that the tests that do pass are not just
testing something vendor B wants to show it can do?


> > Real world example:  We run Crystal Reports Enterprise 
> > edition where I 
> > work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
> > However, getting it to work right and stay up is a nightmare. 
> >  It's taken 
> > them almost a year to get around to testing against the OpenLDAP LDAP 
> > server we use.  The box said "LDAP V3 compliant" and they 
> > assured us that 
> > it was.  Well, it doesn't work with our LDAP V3 compliant 
> > LDAP server at 
> > all, and the problem is something they can't fix for months 
> > because it 
> > doesn't fit into their test cycle.
> > 
> > 
> > Real world example: Postgresql aggregates in subselects. 
> > Someone found a bug in subselects in Postgresql with inner 
> > references to 
> > outter aggregates.  The postgresql team delivered a patch in 
> > less than a 
> > week.  User tested it and it works.
> > 
> > I'm not against testing and all, but as one of the many beta 
> > testers for 
> > Postgresql, I do feel a bit insulted by your attitude that only a 
> > cohesive, organized testing effort can result in a reliable product.
> 
> Let me rephrase it:
> "Only a  cohesive, organized testing effort can result in a product that
> is proven reliable."
> 
> Without such an effort, it is only an educated guess as to whether the
> product is reliable or not.  The data is the most valuable software
> component in an organization.  It is worth more than the hardware and it
> is worth more than the software.  If you are going to trust one billion
> dollars worth of corporate data on a software system, you ought to
> ensure that the system has been carefully tested.  I don't think that is
> just an opinion.  It's simply common sense.

So you've never worked on a project where the data is of high value, since in
those circumstances the customer is always going to apply their own acceptance
testing anyway. If you think that doesn't happen you try sitting through 2
solid days of Y2k testing on _one_ system and tell me customers never do their
own testing.


> Therefore, I am going to stop harping on it.

But there is no need to, as has been mentioned before, if the testing is not
upto your level of testing submit something that makes it so. Having said that
I do believe you mentioned that you didn't have the time to create something
but you would be happy to test it, i.e. test the test.


-- 
Nigel J. Andrews


---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[scott.marlowe]

On Mon, 23 Jun 2003, Dann Corbit wrote:

> > -Original Message-
> > From: scott.marlowe [mailto:[EMAIL PROTECTED] 
> > Sent: Monday, June 23, 2003 12:25 PM
> > To: Dann Corbit
> > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> > Subject: Re: [HACKERS] Two weeks to feature freeze
> > 
> > 
> > On Mon, 23 Jun 2003, Dann Corbit wrote:
> > 
> > > Vendor A: "We think our tool is pretty solid and our end 
> > users hardly 
> > > ever turn up any bugs."
> > > 
> > > Vendor B:" We think our tool is pretty solid and our 8500 tests 
> > > currently show only 3 defects with the released version, 
> > and these are 
> > > low impact issues.  To view our current database of issues, 
> > log onto 
> > > web form ."
> > > 
> > > Which tool would you prefer to install?
> > 
> > The one I've tested and found to meet my needs, both now and 
> > by providing 
> > fixes when I needed it.
> > 
> > Real world example:  We run Crystal Reports Enterprise 
> > edition where I 
> > work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
> > However, getting it to work right and stay up is a nightmare. 
> >  It's taken 
> > them almost a year to get around to testing against the OpenLDAP LDAP 
> > server we use.  The box said "LDAP V3 compliant" and they 
> > assured us that 
> > it was.  Well, it doesn't work with our LDAP V3 compliant 
> > LDAP server at 
> > all, and the problem is something they can't fix for months 
> > because it 
> > doesn't fit into their test cycle.
> > 
> > 
> > Real world example: Postgresql aggregates in subselects. 
> > Someone found a bug in subselects in Postgresql with inner 
> > references to 
> > outter aggregates.  The postgresql team delivered a patch in 
> > less than a 
> > week.  User tested it and it works.
> > 
> > I'm not against testing and all, but as one of the many beta 
> > testers for 
> > Postgresql, I do feel a bit insulted by your attitude that only a 
> > cohesive, organized testing effort can result in a reliable product.
> 
> Let me rephrase it:
> "Only a  cohesive, organized testing effort can result in a product that
> is proven reliable."

No, it isn't proven reliable PERIOD, it's proven reliable under the exact 
conditions of the testing procedure you've implemented.  And no matter how 
idiot proof we try to make Postgresql and its testing, someone else will 
always make a better idiot.  :-)  Actually, what I'm saying is that the 
corner cases are the ones that are hard to predict, and no amount of 
effort up front is going to find a corner case you haven't thought of yet.

> Without such an effort, it is only an educated guess as to whether the
> product is reliable or not.  The data is the most valuable software
> component in an organization.  It is worth more than the hardware and it
> is worth more than the software.  If you are going to trust one billion
> dollars worth of corporate data on a software system, you ought to
> ensure that the system has been carefully tested.  I don't think that is
> just an opinion.  It's simply common sense.

But if that is true, then Postgresql should cause me no end of problems as 
it crashes down around my feet every other week.  Oddly, the dbas for the 
other systems here at work (Oracle and MSSQL server) have a much higher 
workload keeping their factory tested databases up and running.  In over 
four years of use, we have had exactly ZERO downtime of postgresql.

Carefully testing the system is what I, the DBA of our postgresql servers, 
do.  Only I know how we use the system, so no matter how you or Bruce or 
Tom might test it, I'll always be able to do something you wouldn't think 
of, because you're simply not in my shoes.

It's not an educated guess that postgresql works for us, it's proven over 
and over again every single day by the throrough testing and use of every 
Postgresql user.

> > I take my support of Postgresql seriously, and answer many 
> > questions every 
> > week in the general, sql, and performance mailing lists.  I'm 
> > not paid to 
> > do it, I stay at work an extra hour or so each day to "pay 
> > for it."  I 
> > test every beta and RC release on our dataset at work, and 
> > with a test 
> > load to make sure it works for us, and it does.
> > 
> > I offered to beta test for Crystal Reports and was told they weren't 
> > interested, they can do it in house.  Their

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

Dann Corbit wrote:
> Let me rephrase it:
> "Only a  cohesive, organized testing effort can result in a product that
> is proven reliable."
> 
> Without such an effort, it is only an educated guess as to whether the
> product is reliable or not.  The data is the most valuable software
> component in an organization.  It is worth more than the hardware and it
> is worth more than the software.  If you are going to trust one billion
> dollars worth of corporate data on a software system, you ought to
> ensure that the system has been carefully tested.  I don't think that is
> just an opinion.  It's simply common sense.

True, it is an "educated guess", but it turns out our educated guess
method produces more reliable code than the exhaustive testing method,
so though there isn't as much of a _feeling_ of confidence, there is a
_result_ of more reliability.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

scott.marlowe wrote:
> Peter is coming off awfully paternalistic here.  I'd rather have a few 
> extra scripts to look through to find what I need when I'm trying to 
> figure out something than to have a tool that only the hackers know exists 
> and I can only get by asking nicely to see the pretty code.
> 
> While no one wants to see a contrib or tool directory of a hundred megs, 
> lots of little example files and testing scripts can be nice for nothing 
> else other than the examples they provide.  I learned a lot when I first 
> started using pgsql from the things in the contrib directory.

Having something that just runs and I don't have to fiddle with it is a
big help --- of course, it took me a few years to realize that this is
the best way to test patches --- kind of embarassing that I didn't think
of it in 1997.

I think the patch came out of complaints that my patch application was
letting in broken code --- since I have been using it, I am mostly down
to weird bugs or platform problem, but the obvious patch problems are
pretty much gone, which some people might have noticed.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: scott.marlowe [mailto:[EMAIL PROTECTED] 
> Sent: Monday, June 23, 2003 12:25 PM
> To: Dann Corbit
> Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> On Mon, 23 Jun 2003, Dann Corbit wrote:
> 
> > Vendor A: "We think our tool is pretty solid and our end 
> users hardly 
> > ever turn up any bugs."
> > 
> > Vendor B:" We think our tool is pretty solid and our 8500 tests 
> > currently show only 3 defects with the released version, 
> and these are 
> > low impact issues.  To view our current database of issues, 
> log onto 
> > web form ."
> > 
> > Which tool would you prefer to install?
> 
> The one I've tested and found to meet my needs, both now and 
> by providing 
> fixes when I needed it.
> 
> Real world example:  We run Crystal Reports Enterprise 
> edition where I 
> work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
> However, getting it to work right and stay up is a nightmare. 
>  It's taken 
> them almost a year to get around to testing against the OpenLDAP LDAP 
> server we use.  The box said "LDAP V3 compliant" and they 
> assured us that 
> it was.  Well, it doesn't work with our LDAP V3 compliant 
> LDAP server at 
> all, and the problem is something they can't fix for months 
> because it 
> doesn't fit into their test cycle.
> 
> 
> Real world example: Postgresql aggregates in subselects. 
> Someone found a bug in subselects in Postgresql with inner 
> references to 
> outter aggregates.  The postgresql team delivered a patch in 
> less than a 
> week.  User tested it and it works.
> 
> I'm not against testing and all, but as one of the many beta 
> testers for 
> Postgresql, I do feel a bit insulted by your attitude that only a 
> cohesive, organized testing effort can result in a reliable product.

Let me rephrase it:
"Only a  cohesive, organized testing effort can result in a product that
is proven reliable."

Without such an effort, it is only an educated guess as to whether the
product is reliable or not.  The data is the most valuable software
component in an organization.  It is worth more than the hardware and it
is worth more than the software.  If you are going to trust one billion
dollars worth of corporate data on a software system, you ought to
ensure that the system has been carefully tested.  I don't think that is
just an opinion.  It's simply common sense.
 
> I take my support of Postgresql seriously, and answer many 
> questions every 
> week in the general, sql, and performance mailing lists.  I'm 
> not paid to 
> do it, I stay at work an extra hour or so each day to "pay 
> for it."  I 
> test every beta and RC release on our dataset at work, and 
> with a test 
> load to make sure it works for us, and it does.
> 
> I offered to beta test for Crystal Reports and was told they weren't 
> interested, they can do it in house.  Their support, like many big 
> commercial houses, is designed more to make my boss's boss 
> happy, not me, 
> and it shows every day in how they fail to provide timely support for 
> their product while playing CYA to the higherups.

A long test cycle does result in a slower patch.  But when you get the
patch, it is going to work and not introduce new problems.

The resistance to testing is typical of programmers.  The PostgreSQL
group is a group of programmers.  I don't think I can change anyone's
mind, since the most significant people on the list don't think it is
worth the bother.

Therefore, I am going to stop harping on it.

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > I put stuff in /tools so if something happens to me, you guys can keep
> > going.
> 
> Yes, we keep going with make clean; make check, like everyone else.  Why
> don't you consider using that?

Actually, I used to manually do all those tests to test patches, but I
found a single script was less error prone, and allowed me to more
reliably test patches --- in the old days, I would forget the initdb or
regression runs, only to find out later the patch broke something.

There is a value to that script for me, and if someone else does patch
application, I suggest they use it too.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > I put stuff in /tools so if something happens to me, you guys can keep
> > going.
> 
> Yes, we keep going with make clean; make check, like everyone else.  Why
> don't you consider using that?

The script is automated to run at night, it captures gmake output while
returning the proper error return code, and exits if it finds any
problems.  There was talk others want to automate nightly builds, so
this a start for them.

Amazing you find 688 bytes worth discussing.  I know you said "what
happens if everyone adds their scripts", but something that would be a
mess if everyone did it isn't always a proper way to judge if something
is appropriate.

If someone wants to submit a better test build script, I will merge it
into the existing one or replace mine.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[scott.marlowe]

On Mon, 23 Jun 2003, Bruce Momjian wrote:

> Peter Eisentraut wrote:
> > Bruce Momjian writes:
> > 
> > > Well, it is a nice test template for people who aren't shell script
> > > experts, and I have been in the habit of pushing stuff I use into /tools
> > > so it is available for others.
> > 
> > I know and I'm not happy about it.  The PostgreSQL source tree isn't a
> > repository of everyone's favorite shell scripts.  There's an official
> > method to build and test a PostgreSQL installation.  If that is flawed or
> > incomplete, then let's talk about it.  But everyone pushing out their own
> > little test methodology without further consideration is not going to help
> > this discussion.
> 
> I put stuff in /tools so if something happens to me, you guys can keep
> going.  Do I have to be clearer that that?  I have RELEASE_CHANGES,
> which Tom used for 7.3.X releases, pgindent, stuff for finding
> missing/extraneous includes, static requirements, stuff like that.
> 
> Unless you can find someone else who agrees with you, it stays.

Peter is coming off awfully paternalistic here.  I'd rather have a few 
extra scripts to look through to find what I need when I'm trying to 
figure out something than to have a tool that only the hackers know exists 
and I can only get by asking nicely to see the pretty code.

While no one wants to see a contrib or tool directory of a hundred megs, 
lots of little example files and testing scripts can be nice for nothing 
else other than the examples they provide.  I learned a lot when I first 
started using pgsql from the things in the contrib directory.


---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[scott.marlowe]

On Mon, 23 Jun 2003, Dann Corbit wrote:

> Vendor A: "We think our tool is pretty solid and our end users hardly
> ever turn up any bugs."
> 
> Vendor B:" We think our tool is pretty solid and our 8500 tests
> currently show only 3 defects with the released version, and these are
> low impact issues.  To view our current database of issues, log onto web
> form ."
> 
> Which tool would you prefer to install?

The one I've tested and found to meet my needs, both now and by providing 
fixes when I needed it.

Real world example:  We run Crystal Reports Enterprise edition where I 
work.  It's tested thouroughly (supposedly) and has all kinds of QA.  
However, getting it to work right and stay up is a nightmare.  It's taken 
them almost a year to get around to testing against the OpenLDAP LDAP 
server we use.  The box said "LDAP V3 compliant" and they assured us that 
it was.  Well, it doesn't work with our LDAP V3 compliant LDAP server at 
all, and the problem is something they can't fix for months because it 
doesn't fit into their test cycle.


Real world example: Postgresql aggregates in subselects.
Someone found a bug in subselects in Postgresql with inner references to 
outter aggregates.  The postgresql team delivered a patch in less than a 
week.  User tested it and it works.

I'm not against testing and all, but as one of the many beta testers for 
Postgresql, I do feel a bit insulted by your attitude that only a 
cohesive, organized testing effort can result in a reliable product.

I take my support of Postgresql seriously, and answer many questions every 
week in the general, sql, and performance mailing lists.  I'm not paid to 
do it, I stay at work an extra hour or so each day to "pay for it."  I 
test every beta and RC release on our dataset at work, and with a test 
load to make sure it works for us, and it does.

I offered to beta test for Crystal Reports and was told they weren't 
interested, they can do it in house.  Their support, like many big 
commercial houses, is designed more to make my boss's boss happy, not me, 
and it shows every day in how they fail to provide timely support for 
their product while playing CYA to the higherups.


---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Peter Eisentraut]

Bruce Momjian writes:

> I put stuff in /tools so if something happens to me, you guys can keep
> going.

Yes, we keep going with make clean; make check, like everyone else.  Why
don't you consider using that?

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

Dann Corbit wrote:
> PostgreSQL is a fairly mature product, having been in existence in one
> form or another for many years now.
> 
> I expect that most of the bugs that surface will be in areas of new
> functionality.
> 
> Great Bridge had the right idea though.  Let's suppose that they ran
> 10,000 tests and turned up only one bug.  That would be just as valuable
> (if not more so) than turning up 100 bugs.  A large, carefully designed
> test system is *proof* of software quality, or at least of the effort to
> determine the quality level.  It is also proof of the responsibility of
> the software's originators.

Look at the cost/benefit ratio to that.  If you think we don't have to
care about cost/benefit, well, it would be pretty amazing if we didn't.

> Scenario:
> You are going to install a tool that your organization will invest its
> future in.
> 
> Vendor A: "We think our tool is pretty solid and our end users hardly
> ever turn up any bugs."
> 
> Vendor B:" We think our tool is pretty solid and our 8500 tests
> currently show only 3 defects with the released version, and these are
> low impact issues.  To view our current database of issues, log onto web
> form ."
> 
> Which tool would you prefer to install?

I don't think commerical vendors, with those 8500 test, are are doing
any better in reliability than PostgreSQL, and in fact, I think they are
doing worse, and have to expend much more effort than we do.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

Peter Eisentraut wrote:
> Bruce Momjian writes:
> 
> > Well, it is a nice test template for people who aren't shell script
> > experts, and I have been in the habit of pushing stuff I use into /tools
> > so it is available for others.
> 
> I know and I'm not happy about it.  The PostgreSQL source tree isn't a
> repository of everyone's favorite shell scripts.  There's an official
> method to build and test a PostgreSQL installation.  If that is flawed or
> incomplete, then let's talk about it.  But everyone pushing out their own
> little test methodology without further consideration is not going to help
> this discussion.

I put stuff in /tools so if something happens to me, you guys can keep
going.  Do I have to be clearer that that?  I have RELEASE_CHANGES,
which Tom used for 7.3.X releases, pgindent, stuff for finding
missing/extraneous includes, static requirements, stuff like that.

Unless you can find someone else who agrees with you, it stays.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: Bruce Momjian [mailto:[EMAIL PROTECTED] 
> Sent: Saturday, June 21, 2003 8:50 PM
> To: Dann Corbit
> Cc: Tom Lane; Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> Dann Corbit wrote:
> > That is the worst possible test plan.  It totally lacks 
> organization 
> > and there is no hint to define when the feature set has 
> been covered.  
> > Ad hoc testing is a useful addition, but it cannot replace all the 
> > standard tests that have been used by the industry for decades.
> > 
> > If you run literally hundreds of tests designed to ensure that your 
> > product conforms to ANSI/ISO standards then the bugs that 
> are missed 
> > will be few and far between.  Unless you are bad at designing tests.
> > 
> > Designing tests is busywork.  Desiging tests is boring.  
> Nobody wants 
> > to design tests, let alone interpret the results and define correct 
> > baselines.  But testing is very, very important.
> 
> I remember when I was with Great Bridge they said, "Oh, we 
> are going to have a test setup and do all sorts of testing to 
> improve PostgreSQL."  I told them I doubted their testing was 
> going to shake out many more bugs than our existing testing 
> setup, and you know what, I was pretty much right.  Sure, 
> they found a few, but it wasn't much.

PostgreSQL is a fairly mature product, having been in existence in one
form or another for many years now.

I expect that most of the bugs that surface will be in areas of new
functionality.

Great Bridge had the right idea though.  Let's suppose that they ran
10,000 tests and turned up only one bug.  That would be just as valuable
(if not more so) than turning up 100 bugs.  A large, carefully designed
test system is *proof* of software quality, or at least of the effort to
determine the quality level.  It is also proof of the responsibility of
the software's originators.

Scenario:
You are going to install a tool that your organization will invest its
future in.

Vendor A: "We think our tool is pretty solid and our end users hardly
ever turn up any bugs."

Vendor B:" We think our tool is pretty solid and our 8500 tests
currently show only 3 defects with the released version, and these are
low impact issues.  To view our current database of issues, log onto web
form ."

Which tool would you prefer to install?

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Tom Lane]

"Dann Corbit" <[EMAIL PROTECTED]> writes:
>> What do you think is the way to become an insider?

> Join the CVS tree and make a large and valuable contribution to the
> project.

For instance, developing an industrial-strength test suite?  If you've
got an itch there, scratch it.

regards, tom lane

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Peter Eisentraut]

Bruce Momjian writes:

> Well, it is a nice test template for people who aren't shell script
> experts, and I have been in the habit of pushing stuff I use into /tools
> so it is available for others.

I know and I'm not happy about it.  The PostgreSQL source tree isn't a
repository of everyone's favorite shell scripts.  There's an official
method to build and test a PostgreSQL installation.  If that is flawed or
incomplete, then let's talk about it.  But everyone pushing out their own
little test methodology without further consideration is not going to help
this discussion.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: Jan Wieck [mailto:[EMAIL PROTECTED] 
> Sent: Sunday, June 22, 2003 5:45 AM
> To: Dann Corbit
> Cc: Tom Lane; Jason Earl; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> Dann Corbit wrote:
> >> -Original Message-
> >> From: Tom Lane [mailto:[EMAIL PROTECTED]
> >> 
> >> Are you volunteering to create it?  Step right up.
> > 
> > No.  And as an outsider, I rather doubt if any procedures I 
> developed 
> > would be taken very seriously.  If such procedures are to be 
> > developed, I suspect that they will have to be developed 
> from within 
> > if they are to be successful.
> 
> What do you think is the way to become an insider?

Join the CVS tree and make a large and valuable contribution to the
project.

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Dann Corbit]

> -Original Message-
> From: The Hermit Hacker [mailto:[EMAIL PROTECTED] 
> Sent: Sunday, June 22, 2003 12:30 PM
> To: Jan Wieck
> Cc: The Hermit Hacker; Dann Corbit; Tom Lane; Jason Earl; 
> PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
> 
> 
> On Sun, 22 Jun 2003, Jan Wieck wrote:
> 
> > The Hermit Hacker wrote:
> > > On Fri, 20 Jun 2003, Dann Corbit wrote:
> > >
> > >> Designing tests is busywork.  Desiging tests is boring.  Nobody 
> > >> wants to design tests, let alone interpret the results 
> and define 
> > >> correct baselines.  But testing is very, very important.
> > >
> > > But we do do testing ... we even design testing (in the 
> form of the 
> > > regression tests) ... we just don't do testing that you 
> personally 
> > > approve of ... and, from what I've seen so far, you 
> aren't willing 
> > > to actually put
> > > *your* time where your mouth is ... design some tests and 
> submit them to
> > > us ... if they are valid, they will get used ...
> > >
> > > If you feel that crash-me is a valid starting point, 
> start there and 
> > > see where it takes you ...
> >
> > Not that fast! I didn't take the time to check but it wouldn't 
> > surprise me if MySQL's crash-me is GPL'd and copyright MySQL AB. 
> > That's not an optimal point to start PostgreSQL test code 
> from, is it?
> 
> I didn't say to copy it, but if the format is what Dann feels 
> is required to be taken seriously, it does give a starting 
> point to work from ...
> 
> the thing is, as I think it was Tom that pointed out, the 
> crash-me is more a feature tester then anything ... but Dann 
> appears to be stuck on it as the 'be all, end all of testing 
> suites' ...

No.  I think it covers a broad spectrum of functionality.  It is clear
that there are warts in it, and also that it is slanted in a few
instances to turn "bugs into features."  But I think that a large and
thorough test suite that covers all major areas of functionality will
prove useful.  A test suite that covers just as many features and yet is
aimed at honest evaluation would be a big benefit.

The larger and more complete a functionality test suite is, the better.
If a test suite covers ten times the functionality, it will uncover ten
times as many defects.  I think it is part of the responsibility of a
software vendor to ensure that any released product is as free of
defects as possible (even an open source tool set).  All real software
products larger than a few hundred lines of code have some defects in
them.  If you are going to trust your companies data to a software tool,
you would want it to be as free from defects as is possible to achieve,
without rasing the cost prohibitively.

I think that performance testing is also a good idea.  One of the big
benefits of creating a large performance suite is that you can profile
your code and find out where the effort is needed to get a big speed
increase.

I think that the NIST validation suite will be very valuable.  The
coverage of NIST is pretty good, but that test has warts on it too.  You
will find (for instance) that there is not one single index built by
that test suite.  So the joins are all brute force.  Yetch.

If PostgreSQL can pass all three areas of NIST (SQL, ecpg (the pc
directory), and the mc directory) that would be pretty impressive.

---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Andrew Dunstan]

Scott MArlowe wrote:
> On Sat, 21 Jun 2003, Bruce Momjian wrote:
>
>> The big puzzle is how do you get people (including myself) motivated
>> to work on a feature that takes a _huge_ amount of work to see any
>> payoff?  I would like to know.  Anyone?
>
> Pizza?  :-)

Unfortunately it's off my diet :-(

Seriously, I think an increased willingness to share the work around a bit
would be beneficial. I know that I volunteered to work on the w32 port at a
time when I was, as they say, "between jobs". The response was not
encouraging. Now I am working again and don't have much time available.

I know you can't simply divide tasks with infinite granularity ("nine women
can't make a bay in a month"). Some tasks require one or a few people to get
done and that's all that can be done. But if she/he/they can't get it done,
it's time to send out a call for help, IMNSHO.

Not meaning to criticize - the core team does a great job. I, too, have a
tendency to overcommit and under-delegate. It's very understandable.

andrew



---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[scott.marlowe]

On Mon, 23 Jun 2003, Bruce Momjian wrote:

> Peter Eisentraut wrote:
> > The Hermit Hacker writes:
> > 
> > > Ya, the script looked like it did a bit more then just a 'make clean; make
> > > check' ... doesn't it?
> > 
> > No.
> 
> Well, it is a nice test template for people who aren't shell script
> experts, and I have been in the habit of pushing stuff I use into /tools
> so it is available for others.

Please keep pushing such scripts out.  They're a valuable learning tool 
for many beginners and a cost little in size to be included.


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[Tom Lane]

Robert Treat <[EMAIL PROTECTED]> writes:
> Here's a sure to be wildly unpopular suggestion:

> Make the deciding factor for the next release support of "foo" (foo can
> be win32, pitr, replication, 2PC, whatever...).

We've done that before (see WAL in 7.1), with unhappy results.  The
fundamental problem with it is that towards the end of the cycle,
other developers don't know how to schedule their time, because they
don't know when feature freeze is really going to be.  People end up
twiddling their thumbs while the schedule slips a few days at a time.

The target-date-based approach we've taken in the last couple of
releases seems much more productive.

regards, tom lane

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Robert Treat]

On Monday 23 June 2003 10:43 am, Tom Lane wrote:
> Robert Treat <[EMAIL PROTECTED]> writes:
> > Here's a sure to be wildly unpopular suggestion:
> >
> > Make the deciding factor for the next release support of "foo" (foo can
> > be win32, pitr, replication, 2PC, whatever...).
>
> We've done that before (see WAL in 7.1), with unhappy results. 

well, I did say it would be wildly unpopular ;-)

> The
> fundamental problem with it is that towards the end of the cycle,
> other developers don't know how to schedule their time, because they
> don't know when feature freeze is really going to be.  People end up
> twiddling their thumbs while the schedule slips a few days at a time.
>

Ok, what if feature freeze comes 1 month after completion of "foo" feature. 
This way the release is still feature dependent, but people arn't sitting 
around day by day waiting for foo, and they also don't have to worry about 
getting caught in the middle of something when foo gets done.  (i'm kind of 
picking 1 month arbitraily, this could be two weeks if that works better).

> The target-date-based approach we've taken in the last couple of
> releases seems much more productive.
>

productive on a small scale; for sure. productive for large scale features... 
well, that's why it's being discussed. 

Robert Treat


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] Two weeks to feature freeze

[Robert Treat]

On Sat, 2003-06-21 at 22:55, Bruce Momjian wrote:
> Andrew Dunstan wrote:
> > 
> > Maybe a better strategy would be to get a release out soon but not wait 6
> > months for another release which would contain the Win32 port and the PITR
> > stuff (assuming those aren't done in time for this release).
> 
> What concerns me is that we thought that after 7.3, and didn't do much
> work on either until we got near 7.4 release.  I wonder if this is just
> going to be a pattern, where these items are so large, we can't get any
> motivation to focus on them until we get near the final release.  I
> guess if each final release gets us a little closer, eventually we will
> get there, but this process is not ideal.
> 
> The big puzzle is how do you get people (including myself) motivated to
> work on a feature that takes a _huge_ amount of work to see any payoff? 
> I would like to know.  Anyone?
> 

Here's a sure to be wildly unpopular suggestion:

Make the deciding factor for the next release support of "foo" (foo can
be win32, pitr, replication, 2PC, whatever...).  This would put ample
pressure on the developers of "foo" to get it done since the whole
release rides on their shoulders. It also might encourage others to help
out more since they can't get something new until "foo" is completed.
This would also prioritize said developers time on the large project
rather than other non-prioritized tasks, and it also helps to ensure a
"killer feature" for those who want such things,

Robert Treat
-- 
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL


---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Two weeks to feature freeze

[Bruce Momjian]

Peter Eisentraut wrote:
> The Hermit Hacker writes:
> 
> > Ya, the script looked like it did a bit more then just a 'make clean; make
> > check' ... doesn't it?
> 
> No.

Well, it is a nice test template for people who aren't shell script
experts, and I have been in the habit of pushing stuff I use into /tools
so it is available for others.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 359-1001
  +  If your life is a hard drive, |  13 Roberts Road
  +  Christ can be your backup.|  Newtown Square, Pennsylvania 19073

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Re: [HACKERS] Two weeks to feature freeze

[Peter Eisentraut]

The Hermit Hacker writes:

> Ya, the script looked like it did a bit more then just a 'make clean; make
> check' ... doesn't it?

No.

-- 
Peter Eisentraut   [EMAIL PROTECTED]


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Two weeks to feature freeze

[scott.marlowe]

On Sat, 21 Jun 2003, Bruce Momjian wrote:

> Andrew Dunstan wrote:
> > 
> > Maybe a better strategy would be to get a release out soon but not wait 6
> > months for another release which would contain the Win32 port and the PITR
> > stuff (assuming those aren't done in time for this release).
> 
> What concerns me is that we thought that after 7.3, and didn't do much
> work on either until we got near 7.4 release.  I wonder if this is just
> going to be a pattern, where these items are so large, we can't get any
> motivation to focus on them until we get near the final release.  I
> guess if each final release gets us a little closer, eventually we will
> get there, but this process is not ideal.
> 
> The big puzzle is how do you get people (including myself) motivated to
> work on a feature that takes a _huge_ amount of work to see any payoff? 
> I would like to know.  Anyone?

Pizza?  :-)


---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[scott.marlowe]

On Sat, 21 Jun 2003, Christopher Kings-Lynne wrote:

> Crash-me has nothing to do with testing, it jsut checks to see what
> features a db supports:

An interesting point is that until recently, crashme said that the 
postgresql backend crashed on very large queries.  The actual problem was 
that postgresql has NO LIMIT to query size, and the crashme script would 
keep feeding the postgresql backend larger and larger amounts of query 
until the internal buffer of the crashme script overran.

This failure was attributed to postgresql when it was, in fact a bug in 
the crashme script.

This is not an isolated behaviour of crashme.  It's a quick dirty hack job 
designed to show the differences between MySQL and all the other 
databases.  If it was truly comprehensive (i.e. SQL92 spec testing) there 
would be hundreds of failure points for MySQL.  but it isn't.  It tests 
only those things that are good in MySQL against other databases (for the 
most part, there is some token effort at including a few things MySQL 
doesn't do).




---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings

Re: [HACKERS] Two weeks to feature freeze

[Mike Mascari]

I wrote:

> Tom Lane wrote:
> 
>>Basically, the subordinate must be willing to hold its breath *forever*.  
> 
> 
> Yep. And if the cohort crashes while waiting for the coordinator to
> come back on-line, if I understand the world correctly, it must be
> capable of committing the database changes associated with the
> COMMIT-VOTE response it supplied to the coordinator's PREPARE. It
> seems this would require REDO? And yet there are thousands of
> installed distributed databases running enterprises every day.

Please ignore the REDO remark. It's late where I am...

Mike Mascari
[EMAIL PROTECTED]


---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match

Re: [HACKERS] Two weeks to feature freeze

[Mike Mascari]

Tom Lane wrote:

> The Hermit Hacker <[EMAIL PROTECTED]> writes:
> 
>>Hrmmm, I see Tom's point (I think!) ... but what if, for instance, the
>>co-ordinator crashes?
> 
> Or you just lose the network connection for awhile.  The worst case
> scenario I think is where the co-ordinator got everyone's promise to
> commit, and told some of the subordinates to commit, but your own
> response gets lost due to network failure.  Now what?  If you time
> out and decide to abort, you're inconsistent with the other
> subordinates.  On the other hand, you can't commit after a timeout
> either, because that loses in the other scenario (where the coordinator
> didn't decide to commit).  Basically, the subordinate must be willing
> to hold its breath *forever*.  

Yep. And if the cohort crashes while waiting for the coordinator to
come back on-line, if I understand the world correctly, it must be
capable of committing the database changes associated with the
COMMIT-VOTE response it supplied to the coordinator's PREPARE. It
seems this would require REDO? And yet there are thousands of
installed distributed databases running enterprises every day.

A paper on a "A New Presumed Commit Optimization for Two Phase Commit"
describes the cohort as:

"If a prepared cohort does not receive a transaction outcome message
promptly, or crashes without remembering the outcome, the cohort asks
the coordinator for the outcome. It keeps on asking until it gets an
answer. (This is the blocking aspect of 2PC.)"

I'd just like to point out that:

1) The XA interface defines a 2PC protocol library which allows
transaction managers, such as BEAS Tuxedo (and Oracle, for that
matter) to use the database in a distributed transaction. Lack of an
XA interface for PostgreSQL prohibits its use in major enterprise
applications. BEAS Tuxedo can talk to PostgreSQL, but won't allow it
to participate in a distributed tx.

2) The users of distributed databases will/should/can know that a
cohort will block waiting for the coordinator. We're not talking
asynchronous multi-master replication of 4 databases distributed over
low-speed communication lines across the country. We're talking about
the sales dept. database having a few linked tables to the accounting
dept. database, where inserts into the one result in inserts into the
other.

Mike Mascari
[EMAIL PROTECTED]




---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])

Re: [HACKERS] Two weeks to feature freeze

[Sailesh Krishnamurthy]

> "Tom" == Tom Lane <[EMAIL PROTECTED]> writes:

Tom> Sailesh Krishnamurthy <[EMAIL PROTECTED]> writes:
>> I'm not sure if I understand Tom's beef - I think he is
>> concerned about what happens if a subordinate does not respond
>> to a prepare message. I would assume that the co-ordinator
>> would not let the commit go through until it has received
>> confirmations from every subordinate.

Tom> No.  I want to know what the subordinate does when it's
Tom> promised to commit and the co-ordinator never responds.
Tom> AFAICS the subordinate is screwed --- it can't commit, and it
Tom> can't abort, and it can't expect to make progress
Tom> indefinitely on other work while it's holding locks for the
Tom> not-quite-committed transaction.

Okay I understand what you mean now.

AFAIK the general way things happen is that each site has a "recovery
procedure" that kicks in after a crash. If the co-ordinator crashes
(which could be before or after it sends out COMMIT messages to some
of the subordinates), its recovery manager will bring the system up,
read the log and ready information about all uncommitted transactions
in virtual storage.

If a Xact is in the PREPARE stage it will periodically send a message
to the co-ordinator asking about what happened to the transaction in
question. Once the co-ordinator has come back online it can respond to
the query.

Of course in the case of a co-ordinator going out of action totally
and remaining unconnected this is not a viable solution. 

If you're making the case that 2PC is not viable on very wide area
networks with intermitted connectivity, I agree whole-heartedly. 

That said, 2PC (and its children, PA and PC) have their place, and are
indeed used in many systems. 

For instance, say you are rigging up a message queueing infrastructure
(like MQ-series) to your database (say with NOTIFY), you'd at least
like to have the db act as a co-ordinator with the MQ.

Or the parallel cluster example I gave earlier. Clustered linux boxes
are definitely here although no open-source DBMS offers a parallel
solution.

-- 
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] Two weeks to feature freeze

[Tom Lane]

The Hermit Hacker <[EMAIL PROTECTED]> writes:
> Hrmmm, I see Tom's point (I think!) ... but what if, for instance, the
> co-ordinator crashes?

Or you just lose the network connection for awhile.  The worst case
scenario I think is where the co-ordinator got everyone's promise to
commit, and told some of the subordinates to commit, but your own
response gets lost due to network failure.  Now what?  If you time
out and decide to abort, you're inconsistent with the other
subordinates.  On the other hand, you can't commit after a timeout
either, because that loses in the other scenario (where the coordinator
didn't decide to commit).  Basically, the subordinate must be willing
to hold its breath *forever*.  I don't see how this can work.

regards, tom lane

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] Two weeks to feature freeze

[The Hermit Hacker]

On Mon, 23 Jun 2003, Christopher Kings-Lynne wrote:

> > No.  I want to know what the subordinate does when it's promised to
> > commit and the co-ordinator never responds.  AFAICS the subordinate
> > is screwed --- it can't commit, and it can't abort, and it can't expect
> > to make progress indefinitely on other work while it's holding locks
> > for the not-quite-committed transaction.
>
> It takes itself offline and you need to resync it later??

Hrmmm, I see Tom's point (I think!) ... but what if, for instance, the
co-ordinator crashes?  From the subordinates point of view, it has the
complete transaction to commit, but the co-ordinator has gone down without
telling it to do so ...

---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
  joining column's datatypes do not match