Re: Test suite blocking release
Yes, but it's very noisy, and I wanted to get absolute clarification. Thanks. I'll do it now. On 22 Mar 2010, at 15:08, Jan Lehnardt wrote: > > On 22 Mar 2010, at 04:12, Noah Slater wrote: > >> So, are we officially good to go? >> >> Can I upload the artefact from the current tag, or do I need to retag the >> 0.11.x branch? > > You'll need to re-tag. Are you not on comm...@? :) > > Cheers > Jan > -- > > > >> >> Thanks. >> >> On 21 Mar 2010, at 19:00, Jan Lehnardt wrote: >> >>> >>> On 21 Mar 2010, at 13:38, Robert Dionne wrote: >>> Ok Noah, This is only a test case issue, and not in the changes code as I though. Jan found the issue and it works fine for me now in both FF and CLI. -- Bob >>> >>> As an added bonus, the test suite should now work 100% in WebKit (for me at >>> least :) >>> >>> Cheers >>> Jan >>> -- >>> >>> On Mar 21, 2010, at 1:30 PM, Jan Lehnardt wrote: > > On 21 Mar 2010, at 12:24, Robert Dionne wrote: > >> >> >> >> >> On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote: >> >>> >>> On 21 Mar 2010, at 12:10, Noah Slater wrote: >>> What are the CLI tests, if not the etap tests? Are they integrated into the build system? >>> >>> The CLI tests are the same as the browser tests, just run through our >>> couchjs binary >>> that has custom HTTP extensions to make the xhr work. At this point I >>> don't think it >>> is reliable enough to mimic browser behaviour and that we shouldn't use >>> it for vetting >>> the state of the code. >> >> This is likely true, but in this particular case I think there's a bug >> in the changes code (that I'm trying to dig out). It's nice that it >> works on your machine but on my machine, using FF, it fails often >> enough. Moreover it's been around for a long time now so I figure it's >> worth figuring out. >> >> I don't have a dog in this fight (.ie. a paying customer) so this >> failure doesn't bother me. With respect to policy, given the various >> bogocities of browsers, I'd recommend something like these CLI tests >> plus the etaps ought to be the "official" tests for vetting, and part >> of the build > > Not that I disagree, but part (most?) of the appeal of the browser based > tests are that they run in a real-world client instead of the lab that is > couchjs+http :) > > Cheers > Jan > -- > >> >> >>> >>> It is very useful when developing new code to not have to switch to and >>> reload the >>> browser over and over again. >>> >>> Cheers >>> Jan >>> -- >>> >>> >>> >>> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote: > > On 21 Mar 2010, at 06:04, Robert Dionne wrote: > >> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote: >> >>> >>> On 20 Mar 2010, at 20:06, Paul Davis wrote: >>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote: > I think faulty test case should block the release, if I am to > have any > future sanity preparing releases. I don't want to delay and > longer, so if > you guys are absolutely sure this is a test error and not code > error, then I > propose that the test be commented out. Our tests form a contract > between > us, internally, and our users. If that contract has a bug, it > should be > removed or fixed - or it simply dilutes the importance of > contract. If some > one comments out the test, and we agree it is not indicative of > an important > bug, I can call the vote within hours. > I'd have to agree on this. From the point of view of a release, if a test reports a failure then it should be made to not report a failure. If that's accomplished by disabling it, then there will be a commit with a message that explains why it was disabled and etc and such on and so forth. >>> >>> I'd do that if the test was failing for me :) >> >> it's not failing for you when you run changes.js with the CLI ? >> Fails for me every time. > > I don't consider the CLI tests as part of the official test suite > just yet. > > Cheers > Jan > -- > >> >> Anyway I poked at this a bit yesterday and am not 100% sure the >> issue is in the test. I tried putting a sleep in with no luck. If my >> understanding of the JS is correct, CouchDB is supposed to be
Re: Test suite blocking release
On 22 Mar 2010, at 04:12, Noah Slater wrote:
> So, are we officially good to go?
>
> Can I upload the artefact from the current tag, or do I need to retag the
> 0.11.x branch?
You'll need to re-tag. Are you not on comm...@? :)
Cheers
Jan
--
>
> Thanks.
>
> On 21 Mar 2010, at 19:00, Jan Lehnardt wrote:
>
>>
>> On 21 Mar 2010, at 13:38, Robert Dionne wrote:
>>
>>> Ok Noah, This is only a test case issue, and not in the changes code as I
>>> though. Jan found the issue and it works fine for me now in both FF and
>>> CLI. -- Bob
>>
>> As an added bonus, the test suite should now work 100% in WebKit (for me at
>> least :)
>>
>> Cheers
>> Jan
>> --
>>
>>
>>>
>>>
>>> On Mar 21, 2010, at 1:30 PM, Jan Lehnardt wrote:
>>>
On 21 Mar 2010, at 12:24, Robert Dionne wrote:
>
>
>
>
> On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote:
>
>>
>> On 21 Mar 2010, at 12:10, Noah Slater wrote:
>>
>>> What are the CLI tests, if not the etap tests? Are they integrated into
>>> the build system?
>>
>> The CLI tests are the same as the browser tests, just run through our
>> couchjs binary
>> that has custom HTTP extensions to make the xhr work. At this point I
>> don't think it
>> is reliable enough to mimic browser behaviour and that we shouldn't use
>> it for vetting
>> the state of the code.
>
> This is likely true, but in this particular case I think there's a bug in
> the changes code (that I'm trying to dig out). It's nice that it works on
> your machine but on my machine, using FF, it fails often enough. Moreover
> it's been around for a long time now so I figure it's worth figuring out.
>
> I don't have a dog in this fight (.ie. a paying customer) so this failure
> doesn't bother me. With respect to policy, given the various bogocities
> of browsers, I'd recommend something like these CLI tests plus the etaps
> ought to be the "official" tests for vetting, and part of the build
Not that I disagree, but part (most?) of the appeal of the browser based
tests are that they run in a real-world client instead of the lab that is
couchjs+http :)
Cheers
Jan
--
>
>
>>
>> It is very useful when developing new code to not have to switch to and
>> reload the
>> browser over and over again.
>>
>> Cheers
>> Jan
>> --
>>
>>
>>
>>
>>>
>>> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>>>
On 21 Mar 2010, at 06:04, Robert Dionne wrote:
> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>
>>
>> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>>
>>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
I think faulty test case should block the release, if I am to have
any
future sanity preparing releases. I don't want to delay and
longer, so if
you guys are absolutely sure this is a test error and not code
error, then I
propose that the test be commented out. Our tests form a contract
between
us, internally, and our users. If that contract has a bug, it
should be
removed or fixed - or it simply dilutes the importance of
contract. If some
one comments out the test, and we agree it is not indicative of an
important
bug, I can call the vote within hours.
>>>
>>> I'd have to agree on this. From the point of view of a release, if a
>>> test reports a failure then it should be made to not report a
>>> failure.
>>> If that's accomplished by disabling it, then there will be a commit
>>> with a message that explains why it was disabled and etc and such on
>>> and so forth.
>>
>> I'd do that if the test was failing for me :)
>
> it's not failing for you when you run changes.js with the CLI ?
> Fails for me every time.
I don't consider the CLI tests as part of the official test suite just
yet.
Cheers
Jan
--
>
> Anyway I poked at this a bit yesterday and am not 100% sure the issue
> is in the test. I tried putting a sleep in with no luck. If my
> understanding of the JS is correct, CouchDB is supposed to be
> synchronous so it's not timing.
>
> If someone could comment on the test itself it would be helpful. The
> section of the code that fails:
>
> // changes get all_docs style with deleted docs
> var doc = {a:1};
> db.save(doc);
> db.deleteDoc(doc);
> var req = CouchDB.request(
Re: Test suite blocking release
So, are we officially good to go?
Can I upload the artefact from the current tag, or do I need to retag the
0.11.x branch?
Thanks.
On 21 Mar 2010, at 19:00, Jan Lehnardt wrote:
>
> On 21 Mar 2010, at 13:38, Robert Dionne wrote:
>
>> Ok Noah, This is only a test case issue, and not in the changes code as I
>> though. Jan found the issue and it works fine for me now in both FF and CLI.
>> -- Bob
>
> As an added bonus, the test suite should now work 100% in WebKit (for me at
> least :)
>
> Cheers
> Jan
> --
>
>
>>
>>
>> On Mar 21, 2010, at 1:30 PM, Jan Lehnardt wrote:
>>
>>>
>>> On 21 Mar 2010, at 12:24, Robert Dionne wrote:
>>>
On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote:
>
> On 21 Mar 2010, at 12:10, Noah Slater wrote:
>
>> What are the CLI tests, if not the etap tests? Are they integrated into
>> the build system?
>
> The CLI tests are the same as the browser tests, just run through our
> couchjs binary
> that has custom HTTP extensions to make the xhr work. At this point I
> don't think it
> is reliable enough to mimic browser behaviour and that we shouldn't use
> it for vetting
> the state of the code.
This is likely true, but in this particular case I think there's a bug in
the changes code (that I'm trying to dig out). It's nice that it works on
your machine but on my machine, using FF, it fails often enough. Moreover
it's been around for a long time now so I figure it's worth figuring out.
I don't have a dog in this fight (.ie. a paying customer) so this failure
doesn't bother me. With respect to policy, given the various bogocities of
browsers, I'd recommend something like these CLI tests plus the etaps
ought to be the "official" tests for vetting, and part of the build
>>>
>>> Not that I disagree, but part (most?) of the appeal of the browser based
>>> tests are that they run in a real-world client instead of the lab that is
>>> couchjs+http :)
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>
> It is very useful when developing new code to not have to switch to and
> reload the
> browser over and over again.
>
> Cheers
> Jan
> --
>
>
>
>
>>
>> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>>
>>>
>>> On 21 Mar 2010, at 06:04, Robert Dionne wrote:
>>>
On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>
> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>
>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
>>> I think faulty test case should block the release, if I am to have
>>> any
>>> future sanity preparing releases. I don't want to delay and longer,
>>> so if
>>> you guys are absolutely sure this is a test error and not code
>>> error, then I
>>> propose that the test be commented out. Our tests form a contract
>>> between
>>> us, internally, and our users. If that contract has a bug, it
>>> should be
>>> removed or fixed - or it simply dilutes the importance of contract.
>>> If some
>>> one comments out the test, and we agree it is not indicative of an
>>> important
>>> bug, I can call the vote within hours.
>>>
>>
>> I'd have to agree on this. From the point of view of a release, if a
>> test reports a failure then it should be made to not report a
>> failure.
>> If that's accomplished by disabling it, then there will be a commit
>> with a message that explains why it was disabled and etc and such on
>> and so forth.
>
> I'd do that if the test was failing for me :)
it's not failing for you when you run changes.js with the CLI ? Fails
for me every time.
>>>
>>> I don't consider the CLI tests as part of the official test suite just
>>> yet.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
Anyway I poked at this a bit yesterday and am not 100% sure the issue
is in the test. I tried putting a sleep in with no luck. If my
understanding of the JS is correct, CouchDB is supposed to be
synchronous so it's not timing.
If someone could comment on the test itself it would be helpful. The
section of the code that fails:
// changes get all_docs style with deleted docs
var doc = {a:1};
db.save(doc);
db.deleteDoc(doc);
var req = CouchDB.request("GET",
"/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
var resp = JSON.parse(req.responseText);
TEquals(3, resp.results.length, "should return matching rows");
seems odd to me. all_docs as I re
Re: Test suite blocking release
>> I don't have a dog in this fight (.ie. a paying customer) so this failure >> doesn't bother me. With respect to policy, given the various bogocities of >> browsers, I'd recommend something like these CLI tests plus the etaps ought >> to be the "official" tests for vetting, and part of the build > > Not that I disagree, but part (most?) of the appeal of the browser based > tests are that they run in a real-world client instead of the lab that is > couchjs+http :) > The tests in Futon should be treated just as the tests for any other client library: a way to check the client. Instead we treat them as 75% of the tests for the CouchDB implementation. That's bad. Its like surgery with pruning shears. I don't know if the JS CLI runner is the way forward or not, or some other scripting language. But there should be a clear separation between what goes into Futon and what doesn't based on the "What are we trying to test?" question. On the flip side, I also think that the Futon tests would be a great place to 'document' the entire API. Then they could double as a "this browser is compatible with CouchDB" screen as well as a "this server is Couch-y" regardless of what the actual implementation is. Paul
Re: Test suite blocking release
On Sun, Mar 21, 2010 at 1:16 PM, Jan Lehnardt wrote: > > On 21 Mar 2010, at 12:10, Noah Slater wrote: > > >> Are they integrated into the build system? > > No. > Just in case, to be specific, the CLI tests are just running the Futon tests from the command line. I added the few bits so that I could run the tests from the command line instead of having to switch over to futon and get irritated at how it locks up when running the tests. I've also specifically not included them in the build system because they require a running instance of CouchDB. This plays badly with build bot attempting to run two builds on the same machine. in the future when we get around to making the "couchdb on random port" behavior more workable there'd be more of an incentive to add them. Paul > Cheers > Jan > -- > >
Re: Test suite blocking release
On 21 Mar 2010, at 13:38, Robert Dionne wrote:
> Ok Noah, This is only a test case issue, and not in the changes code as I
> though. Jan found the issue and it works fine for me now in both FF and CLI.
> -- Bob
As an added bonus, the test suite should now work 100% in WebKit (for me at
least :)
Cheers
Jan
--
>
>
> On Mar 21, 2010, at 1:30 PM, Jan Lehnardt wrote:
>
>>
>> On 21 Mar 2010, at 12:24, Robert Dionne wrote:
>>
>>>
>>>
>>>
>>>
>>> On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote:
>>>
On 21 Mar 2010, at 12:10, Noah Slater wrote:
> What are the CLI tests, if not the etap tests? Are they integrated into
> the build system?
The CLI tests are the same as the browser tests, just run through our
couchjs binary
that has custom HTTP extensions to make the xhr work. At this point I
don't think it
is reliable enough to mimic browser behaviour and that we shouldn't use it
for vetting
the state of the code.
>>>
>>> This is likely true, but in this particular case I think there's a bug in
>>> the changes code (that I'm trying to dig out). It's nice that it works on
>>> your machine but on my machine, using FF, it fails often enough. Moreover
>>> it's been around for a long time now so I figure it's worth figuring out.
>>>
>>> I don't have a dog in this fight (.ie. a paying customer) so this failure
>>> doesn't bother me. With respect to policy, given the various bogocities of
>>> browsers, I'd recommend something like these CLI tests plus the etaps ought
>>> to be the "official" tests for vetting, and part of the build
>>
>> Not that I disagree, but part (most?) of the appeal of the browser based
>> tests are that they run in a real-world client instead of the lab that is
>> couchjs+http :)
>>
>> Cheers
>> Jan
>> --
>>
>>>
>>>
It is very useful when developing new code to not have to switch to and
reload the
browser over and over again.
Cheers
Jan
--
>
> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>
>>
>> On 21 Mar 2010, at 06:04, Robert Dionne wrote:
>>
>>> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>>>
On 20 Mar 2010, at 20:06, Paul Davis wrote:
> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
>> I think faulty test case should block the release, if I am to have
>> any
>> future sanity preparing releases. I don't want to delay and longer,
>> so if
>> you guys are absolutely sure this is a test error and not code
>> error, then I
>> propose that the test be commented out. Our tests form a contract
>> between
>> us, internally, and our users. If that contract has a bug, it should
>> be
>> removed or fixed - or it simply dilutes the importance of contract.
>> If some
>> one comments out the test, and we agree it is not indicative of an
>> important
>> bug, I can call the vote within hours.
>>
>
> I'd have to agree on this. From the point of view of a release, if a
> test reports a failure then it should be made to not report a failure.
> If that's accomplished by disabling it, then there will be a commit
> with a message that explains why it was disabled and etc and such on
> and so forth.
I'd do that if the test was failing for me :)
>>>
>>> it's not failing for you when you run changes.js with the CLI ? Fails
>>> for me every time.
>>
>> I don't consider the CLI tests as part of the official test suite just
>> yet.
>>
>> Cheers
>> Jan
>> --
>>
>>>
>>> Anyway I poked at this a bit yesterday and am not 100% sure the issue
>>> is in the test. I tried putting a sleep in with no luck. If my
>>> understanding of the JS is correct, CouchDB is supposed to be
>>> synchronous so it's not timing.
>>>
>>> If someone could comment on the test itself it would be helpful. The
>>> section of the code that fails:
>>>
>>> // changes get all_docs style with deleted docs
>>> var doc = {a:1};
>>> db.save(doc);
>>> db.deleteDoc(doc);
>>> var req = CouchDB.request("GET",
>>> "/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
>>> var resp = JSON.parse(req.responseText);
>>> TEquals(3, resp.results.length, "should return matching rows");
>>>
>>>
>>> seems odd to me. all_docs as I read the code will return docs with
>>> deletes and conflicts but in this call the filter bop will not apply to
>>> the doc {a:1} so I'm not sure what this delete prior to the call is
>>> about. Anyway I can make it fail in the debugger so perhaps I can find
>>> the root cause.
>>>
>>>
>>>
>>>
Re: Test suite blocking release
Ok Noah, This is only a test case issue, and not in the changes code as I
though. Jan found the issue and it works fine for me now in both FF and CLI. --
Bob
On Mar 21, 2010, at 1:30 PM, Jan Lehnardt wrote:
>
> On 21 Mar 2010, at 12:24, Robert Dionne wrote:
>
>>
>>
>>
>>
>> On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote:
>>
>>>
>>> On 21 Mar 2010, at 12:10, Noah Slater wrote:
>>>
What are the CLI tests, if not the etap tests? Are they integrated into
the build system?
>>>
>>> The CLI tests are the same as the browser tests, just run through our
>>> couchjs binary
>>> that has custom HTTP extensions to make the xhr work. At this point I don't
>>> think it
>>> is reliable enough to mimic browser behaviour and that we shouldn't use it
>>> for vetting
>>> the state of the code.
>>
>> This is likely true, but in this particular case I think there's a bug in
>> the changes code (that I'm trying to dig out). It's nice that it works on
>> your machine but on my machine, using FF, it fails often enough. Moreover
>> it's been around for a long time now so I figure it's worth figuring out.
>>
>> I don't have a dog in this fight (.ie. a paying customer) so this failure
>> doesn't bother me. With respect to policy, given the various bogocities of
>> browsers, I'd recommend something like these CLI tests plus the etaps ought
>> to be the "official" tests for vetting, and part of the build
>
> Not that I disagree, but part (most?) of the appeal of the browser based
> tests are that they run in a real-world client instead of the lab that is
> couchjs+http :)
>
> Cheers
> Jan
> --
>
>>
>>
>>>
>>> It is very useful when developing new code to not have to switch to and
>>> reload the
>>> browser over and over again.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>>
>>>
>>>
On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>
> On 21 Mar 2010, at 06:04, Robert Dionne wrote:
>
>> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>>
>>>
>>> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>>>
On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
> I think faulty test case should block the release, if I am to have any
> future sanity preparing releases. I don't want to delay and longer,
> so if
> you guys are absolutely sure this is a test error and not code error,
> then I
> propose that the test be commented out. Our tests form a contract
> between
> us, internally, and our users. If that contract has a bug, it should
> be
> removed or fixed - or it simply dilutes the importance of contract.
> If some
> one comments out the test, and we agree it is not indicative of an
> important
> bug, I can call the vote within hours.
>
I'd have to agree on this. From the point of view of a release, if a
test reports a failure then it should be made to not report a failure.
If that's accomplished by disabling it, then there will be a commit
with a message that explains why it was disabled and etc and such on
and so forth.
>>>
>>> I'd do that if the test was failing for me :)
>>
>> it's not failing for you when you run changes.js with the CLI ? Fails
>> for me every time.
>
> I don't consider the CLI tests as part of the official test suite just
> yet.
>
> Cheers
> Jan
> --
>
>>
>> Anyway I poked at this a bit yesterday and am not 100% sure the issue is
>> in the test. I tried putting a sleep in with no luck. If my
>> understanding of the JS is correct, CouchDB is supposed to be
>> synchronous so it's not timing.
>>
>> If someone could comment on the test itself it would be helpful. The
>> section of the code that fails:
>>
>> // changes get all_docs style with deleted docs
>> var doc = {a:1};
>> db.save(doc);
>> db.deleteDoc(doc);
>> var req = CouchDB.request("GET",
>> "/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
>> var resp = JSON.parse(req.responseText);
>> TEquals(3, resp.results.length, "should return matching rows");
>>
>>
>> seems odd to me. all_docs as I read the code will return docs with
>> deletes and conflicts but in this call the filter bop will not apply to
>> the doc {a:1} so I'm not sure what this delete prior to the call is
>> about. Anyway I can make it fail in the debugger so perhaps I can find
>> the root cause.
>>
>>
>>
>>
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>
>
>>>
>>
>
Re: Test suite blocking release
On 21 Mar 2010, at 12:24, Robert Dionne wrote:
>
>
>
>
> On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote:
>
>>
>> On 21 Mar 2010, at 12:10, Noah Slater wrote:
>>
>>> What are the CLI tests, if not the etap tests? Are they integrated into the
>>> build system?
>>
>> The CLI tests are the same as the browser tests, just run through our
>> couchjs binary
>> that has custom HTTP extensions to make the xhr work. At this point I don't
>> think it
>> is reliable enough to mimic browser behaviour and that we shouldn't use it
>> for vetting
>> the state of the code.
>
> This is likely true, but in this particular case I think there's a bug in the
> changes code (that I'm trying to dig out). It's nice that it works on your
> machine but on my machine, using FF, it fails often enough. Moreover it's
> been around for a long time now so I figure it's worth figuring out.
>
> I don't have a dog in this fight (.ie. a paying customer) so this failure
> doesn't bother me. With respect to policy, given the various bogocities of
> browsers, I'd recommend something like these CLI tests plus the etaps ought
> to be the "official" tests for vetting, and part of the build
Not that I disagree, but part (most?) of the appeal of the browser based tests
are that they run in a real-world client instead of the lab that is
couchjs+http :)
Cheers
Jan
--
>
>
>>
>> It is very useful when developing new code to not have to switch to and
>> reload the
>> browser over and over again.
>>
>> Cheers
>> Jan
>> --
>>
>>
>>
>>
>>>
>>> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>>>
On 21 Mar 2010, at 06:04, Robert Dionne wrote:
> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>
>>
>> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>>
>>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
I think faulty test case should block the release, if I am to have any
future sanity preparing releases. I don't want to delay and longer, so
if
you guys are absolutely sure this is a test error and not code error,
then I
propose that the test be commented out. Our tests form a contract
between
us, internally, and our users. If that contract has a bug, it should be
removed or fixed - or it simply dilutes the importance of contract. If
some
one comments out the test, and we agree it is not indicative of an
important
bug, I can call the vote within hours.
>>>
>>> I'd have to agree on this. From the point of view of a release, if a
>>> test reports a failure then it should be made to not report a failure.
>>> If that's accomplished by disabling it, then there will be a commit
>>> with a message that explains why it was disabled and etc and such on
>>> and so forth.
>>
>> I'd do that if the test was failing for me :)
>
> it's not failing for you when you run changes.js with the CLI ? Fails
> for me every time.
I don't consider the CLI tests as part of the official test suite just yet.
Cheers
Jan
--
>
> Anyway I poked at this a bit yesterday and am not 100% sure the issue is
> in the test. I tried putting a sleep in with no luck. If my understanding
> of the JS is correct, CouchDB is supposed to be synchronous so it's not
> timing.
>
> If someone could comment on the test itself it would be helpful. The
> section of the code that fails:
>
> // changes get all_docs style with deleted docs
> var doc = {a:1};
> db.save(doc);
> db.deleteDoc(doc);
> var req = CouchDB.request("GET",
> "/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
> var resp = JSON.parse(req.responseText);
> TEquals(3, resp.results.length, "should return matching rows");
>
>
> seems odd to me. all_docs as I read the code will return docs with
> deletes and conflicts but in this call the filter bop will not apply to
> the doc {a:1} so I'm not sure what this delete prior to the call is
> about. Anyway I can make it fail in the debugger so perhaps I can find
> the root cause.
>
>
>
>
>>
>> Cheers
>> Jan
>> --
>>
>
>>>
>>
>
Re: Test suite blocking release
On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote:
>
> On 21 Mar 2010, at 12:10, Noah Slater wrote:
>
>> What are the CLI tests, if not the etap tests? Are they integrated into the
>> build system?
>
> The CLI tests are the same as the browser tests, just run through our couchjs
> binary
> that has custom HTTP extensions to make the xhr work. At this point I don't
> think it
> is reliable enough to mimic browser behaviour and that we shouldn't use it
> for vetting
> the state of the code.
This is likely true, but in this particular case I think there's a bug in the
changes code (that I'm trying to dig out). It's nice that it works on your
machine but on my machine, using FF, it fails often enough. Moreover it's been
around for a long time now so I figure it's worth figuring out.
I don't have a dog in this fight (.ie. a paying customer) so this failure
doesn't bother me. With respect to policy, given the various bogocities of
browsers, I'd recommend something like these CLI tests plus the etaps ought to
be the "official" tests for vetting, and part of the build
>
> It is very useful when developing new code to not have to switch to and
> reload the
> browser over and over again.
>
> Cheers
> Jan
> --
>
>
>
>
>>
>> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>>
>>>
>>> On 21 Mar 2010, at 06:04, Robert Dionne wrote:
>>>
On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>
> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>
>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
>>> I think faulty test case should block the release, if I am to have any
>>> future sanity preparing releases. I don't want to delay and longer, so
>>> if
>>> you guys are absolutely sure this is a test error and not code error,
>>> then I
>>> propose that the test be commented out. Our tests form a contract
>>> between
>>> us, internally, and our users. If that contract has a bug, it should be
>>> removed or fixed - or it simply dilutes the importance of contract. If
>>> some
>>> one comments out the test, and we agree it is not indicative of an
>>> important
>>> bug, I can call the vote within hours.
>>>
>>
>> I'd have to agree on this. From the point of view of a release, if a
>> test reports a failure then it should be made to not report a failure.
>> If that's accomplished by disabling it, then there will be a commit
>> with a message that explains why it was disabled and etc and such on
>> and so forth.
>
> I'd do that if the test was failing for me :)
it's not failing for you when you run changes.js with the CLI ? Fails for
me every time.
>>>
>>> I don't consider the CLI tests as part of the official test suite just yet.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
Anyway I poked at this a bit yesterday and am not 100% sure the issue is
in the test. I tried putting a sleep in with no luck. If my understanding
of the JS is correct, CouchDB is supposed to be synchronous so it's not
timing.
If someone could comment on the test itself it would be helpful. The
section of the code that fails:
// changes get all_docs style with deleted docs
var doc = {a:1};
db.save(doc);
db.deleteDoc(doc);
var req = CouchDB.request("GET",
"/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
var resp = JSON.parse(req.responseText);
TEquals(3, resp.results.length, "should return matching rows");
seems odd to me. all_docs as I read the code will return docs with deletes
and conflicts but in this call the filter bop will not apply to the doc
{a:1} so I'm not sure what this delete prior to the call is about. Anyway
I can make it fail in the debugger so perhaps I can find the root cause.
>
> Cheers
> Jan
> --
>
>>>
>>
>
Re: Test suite blocking release
On 21 Mar 2010, at 12:10, Noah Slater wrote: > Are they integrated into the build system? No. Cheers Jan --
Re: Test suite blocking release
On 21 Mar 2010, at 12:10, Noah Slater wrote:
> What are the CLI tests, if not the etap tests? Are they integrated into the
> build system?
The CLI tests are the same as the browser tests, just run through our couchjs
binary
that has custom HTTP extensions to make the xhr work. At this point I don't
think it
is reliable enough to mimic browser behaviour and that we shouldn't use it for
vetting
the state of the code.
It is very useful when developing new code to not have to switch to and reload
the
browser over and over again.
Cheers
Jan
--
>
> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>
>>
>> On 21 Mar 2010, at 06:04, Robert Dionne wrote:
>>
>>> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>>>
On 20 Mar 2010, at 20:06, Paul Davis wrote:
> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
>> I think faulty test case should block the release, if I am to have any
>> future sanity preparing releases. I don't want to delay and longer, so if
>> you guys are absolutely sure this is a test error and not code error,
>> then I
>> propose that the test be commented out. Our tests form a contract between
>> us, internally, and our users. If that contract has a bug, it should be
>> removed or fixed - or it simply dilutes the importance of contract. If
>> some
>> one comments out the test, and we agree it is not indicative of an
>> important
>> bug, I can call the vote within hours.
>>
>
> I'd have to agree on this. From the point of view of a release, if a
> test reports a failure then it should be made to not report a failure.
> If that's accomplished by disabling it, then there will be a commit
> with a message that explains why it was disabled and etc and such on
> and so forth.
I'd do that if the test was failing for me :)
>>>
>>> it's not failing for you when you run changes.js with the CLI ? Fails for
>>> me every time.
>>
>> I don't consider the CLI tests as part of the official test suite just yet.
>>
>> Cheers
>> Jan
>> --
>>
>>>
>>> Anyway I poked at this a bit yesterday and am not 100% sure the issue is in
>>> the test. I tried putting a sleep in with no luck. If my understanding of
>>> the JS is correct, CouchDB is supposed to be synchronous so it's not timing.
>>>
>>> If someone could comment on the test itself it would be helpful. The
>>> section of the code that fails:
>>>
>>> // changes get all_docs style with deleted docs
>>> var doc = {a:1};
>>> db.save(doc);
>>> db.deleteDoc(doc);
>>> var req = CouchDB.request("GET",
>>> "/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
>>> var resp = JSON.parse(req.responseText);
>>> TEquals(3, resp.results.length, "should return matching rows");
>>>
>>>
>>> seems odd to me. all_docs as I read the code will return docs with deletes
>>> and conflicts but in this call the filter bop will not apply to the doc
>>> {a:1} so I'm not sure what this delete prior to the call is about. Anyway I
>>> can make it fail in the debugger so perhaps I can find the root cause.
>>>
>>>
>>>
>>>
Cheers
Jan
--
>>>
>>
>
Re: Test suite blocking release
What are the CLI tests, if not the etap tests? Are they integrated into the
build system?
On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>
> On 21 Mar 2010, at 06:04, Robert Dionne wrote:
>
>> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>>
>>>
>>> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>>>
On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
> I think faulty test case should block the release, if I am to have any
> future sanity preparing releases. I don't want to delay and longer, so if
> you guys are absolutely sure this is a test error and not code error,
> then I
> propose that the test be commented out. Our tests form a contract between
> us, internally, and our users. If that contract has a bug, it should be
> removed or fixed - or it simply dilutes the importance of contract. If
> some
> one comments out the test, and we agree it is not indicative of an
> important
> bug, I can call the vote within hours.
>
I'd have to agree on this. From the point of view of a release, if a
test reports a failure then it should be made to not report a failure.
If that's accomplished by disabling it, then there will be a commit
with a message that explains why it was disabled and etc and such on
and so forth.
>>>
>>> I'd do that if the test was failing for me :)
>>
>> it's not failing for you when you run changes.js with the CLI ? Fails for
>> me every time.
>
> I don't consider the CLI tests as part of the official test suite just yet.
>
> Cheers
> Jan
> --
>
>>
>> Anyway I poked at this a bit yesterday and am not 100% sure the issue is in
>> the test. I tried putting a sleep in with no luck. If my understanding of
>> the JS is correct, CouchDB is supposed to be synchronous so it's not timing.
>>
>> If someone could comment on the test itself it would be helpful. The section
>> of the code that fails:
>>
>> // changes get all_docs style with deleted docs
>> var doc = {a:1};
>> db.save(doc);
>> db.deleteDoc(doc);
>> var req = CouchDB.request("GET",
>> "/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
>> var resp = JSON.parse(req.responseText);
>> TEquals(3, resp.results.length, "should return matching rows");
>>
>>
>> seems odd to me. all_docs as I read the code will return docs with deletes
>> and conflicts but in this call the filter bop will not apply to the doc
>> {a:1} so I'm not sure what this delete prior to the call is about. Anyway I
>> can make it fail in the debugger so perhaps I can find the root cause.
>>
>>
>>
>>
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>
>
Re: Test suite blocking release
On 21 Mar 2010, at 06:04, Robert Dionne wrote:
> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>
>>
>> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>>
>>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
I think faulty test case should block the release, if I am to have any
future sanity preparing releases. I don't want to delay and longer, so if
you guys are absolutely sure this is a test error and not code error, then
I
propose that the test be commented out. Our tests form a contract between
us, internally, and our users. If that contract has a bug, it should be
removed or fixed - or it simply dilutes the importance of contract. If some
one comments out the test, and we agree it is not indicative of an
important
bug, I can call the vote within hours.
>>>
>>> I'd have to agree on this. From the point of view of a release, if a
>>> test reports a failure then it should be made to not report a failure.
>>> If that's accomplished by disabling it, then there will be a commit
>>> with a message that explains why it was disabled and etc and such on
>>> and so forth.
>>
>> I'd do that if the test was failing for me :)
>
> it's not failing for you when you run changes.js with the CLI ? Fails for me
> every time.
I don't consider the CLI tests as part of the official test suite just yet.
Cheers
Jan
--
>
> Anyway I poked at this a bit yesterday and am not 100% sure the issue is in
> the test. I tried putting a sleep in with no luck. If my understanding of the
> JS is correct, CouchDB is supposed to be synchronous so it's not timing.
>
> If someone could comment on the test itself it would be helpful. The section
> of the code that fails:
>
> // changes get all_docs style with deleted docs
> var doc = {a:1};
> db.save(doc);
> db.deleteDoc(doc);
> var req = CouchDB.request("GET",
>"/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
> var resp = JSON.parse(req.responseText);
> TEquals(3, resp.results.length, "should return matching rows");
>
>
> seems odd to me. all_docs as I read the code will return docs with deletes
> and conflicts but in this call the filter bop will not apply to the doc {a:1}
> so I'm not sure what this delete prior to the call is about. Anyway I can
> make it fail in the debugger so perhaps I can find the root cause.
>
>
>
>
>>
>> Cheers
>> Jan
>> --
>>
>
Re: Test suite blocking release
On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>
> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>
>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote:
>>> I think faulty test case should block the release, if I am to have any
>>> future sanity preparing releases. I don't want to delay and longer, so if
>>> you guys are absolutely sure this is a test error and not code error, then I
>>> propose that the test be commented out. Our tests form a contract between
>>> us, internally, and our users. If that contract has a bug, it should be
>>> removed or fixed - or it simply dilutes the importance of contract. If some
>>> one comments out the test, and we agree it is not indicative of an important
>>> bug, I can call the vote within hours.
>>>
>>
>> I'd have to agree on this. From the point of view of a release, if a
>> test reports a failure then it should be made to not report a failure.
>> If that's accomplished by disabling it, then there will be a commit
>> with a message that explains why it was disabled and etc and such on
>> and so forth.
>
> I'd do that if the test was failing for me :)
it's not failing for you when you run changes.js with the CLI ? Fails for me
every time.
Anyway I poked at this a bit yesterday and am not 100% sure the issue is in the
test. I tried putting a sleep in with no luck. If my understanding of the JS is
correct, CouchDB is supposed to be synchronous so it's not timing.
If someone could comment on the test itself it would be helpful. The section of
the code that fails:
// changes get all_docs style with deleted docs
var doc = {a:1};
db.save(doc);
db.deleteDoc(doc);
var req = CouchDB.request("GET",
"/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
var resp = JSON.parse(req.responseText);
TEquals(3, resp.results.length, "should return matching rows");
seems odd to me. all_docs as I read the code will return docs with deletes and
conflicts but in this call the filter bop will not apply to the doc {a:1} so
I'm not sure what this delete prior to the call is about. Anyway I can make it
fail in the debugger so perhaps I can find the root cause.
>
> Cheers
> Jan
> --
>
Re: Test suite blocking release
On 20 Mar 2010, at 20:06, Paul Davis wrote: > On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote: >> I think faulty test case should block the release, if I am to have any >> future sanity preparing releases. I don't want to delay and longer, so if >> you guys are absolutely sure this is a test error and not code error, then I >> propose that the test be commented out. Our tests form a contract between >> us, internally, and our users. If that contract has a bug, it should be >> removed or fixed - or it simply dilutes the importance of contract. If some >> one comments out the test, and we agree it is not indicative of an important >> bug, I can call the vote within hours. >> > > I'd have to agree on this. From the point of view of a release, if a > test reports a failure then it should be made to not report a failure. > If that's accomplished by disabling it, then there will be a commit > with a message that explains why it was disabled and etc and such on > and so forth. I'd do that if the test was failing for me :) Cheers Jan --
Re: Test suite blocking release
On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater wrote: > I think faulty test case should block the release, if I am to have any > future sanity preparing releases. I don't want to delay and longer, so if > you guys are absolutely sure this is a test error and not code error, then I > propose that the test be commented out. Our tests form a contract between > us, internally, and our users. If that contract has a bug, it should be > removed or fixed - or it simply dilutes the importance of contract. If some > one comments out the test, and we agree it is not indicative of an important > bug, I can call the vote within hours. > I'd have to agree on this. From the point of view of a release, if a test reports a failure then it should be made to not report a failure. If that's accomplished by disabling it, then there will be a commit with a message that explains why it was disabled and etc and such on and so forth.
Re: Test suite blocking release
I think faulty test case should block the release, if I am to have any future sanity preparing releases. I don't want to delay and longer, so if you guys are absolutely sure this is a test error and not code error, then I propose that the test be commented out. Our tests form a contract between us, internally, and our users. If that contract has a bug, it should be removed or fixed - or it simply dilutes the importance of contract. If some one comments out the test, and we agree it is not indicative of an important bug, I can call the vote within hours. On 20 Mar 2010, at 18:19, Benoit Chesneau wrote: On Sat, Mar 20, 2010 at 7:17 PM, Jan Lehnardt wrote: On 20 Mar 2010, at 10:26, Noah Slater wrote: Jan, should this block the release? From what I can tell, it should. I don't think a faulty test case should block the release. Cheers Jan are we sure it's a faulty test? For what it worth i reproduce it in python in couchdbkit unittests. - benoit
Re: Test suite blocking release
This is the call in the Futon test that fails consistently:
var req = CouchDB.request("GET",
"/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
On Mar 20, 2010, at 2:19 PM, Benoit Chesneau wrote:
> On Sat, Mar 20, 2010 at 7:17 PM, Jan Lehnardt wrote:
>>
>> On 20 Mar 2010, at 10:26, Noah Slater wrote:
>>
>>> Jan, should this block the release? From what I can tell, it should.
>>
>> I don't think a faulty test case should block the release.
>>
>> Cheers
>> Jan
> are we sure it's a faulty test? For what it worth i reproduce it in
> python in couchdbkit unittests.
>
> - benoit
Re: Test suite blocking release
On Sat, Mar 20, 2010 at 7:17 PM, Jan Lehnardt wrote: > > On 20 Mar 2010, at 10:26, Noah Slater wrote: > >> Jan, should this block the release? From what I can tell, it should. > > I don't think a faulty test case should block the release. > > Cheers > Jan are we sure it's a faulty test? For what it worth i reproduce it in python in couchdbkit unittests. - benoit
Re: Test suite blocking release
On 20 Mar 2010, at 10:26, Noah Slater wrote:
> Jan, should this block the release? From what I can tell, it should.
I don't think a faulty test case should block the release.
Cheers
Jan
--
>
>
>
> On 20 Mar 2010, at 11:32, Robert Dionne wrote:
>
>>
>>
>>
>> On Mar 19, 2010, at 7:25 PM, Jan Lehnardt wrote:
>>
>>>
>>> On 19 Mar 2010, at 18:07, J Chris Anderson wrote:
>>>
On Mar 19, 2010, at 11:43 AM, Paul Davis wrote:
> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt wrote:
>>
>> On 19 Mar 2010, at 12:50, Noah Slater wrote:
>>
>>>
>>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote:
>>>
We want to test the CouchDB code, not the browser's HTTP handling.
>>>
>>> Sure, but as one of CouchDB's primary interfaces is the browser, it
>>> seems to makes sense that we would want to test how this works. Testing
>>> from the browser allows us to test for and catch problems introduced by
>>> caching, etc - which is what our real world users would be running into.
>>>
>>> Unless I'm missing something?
>>
>> I fully agree, but we should have a separate browser interaction
>> suite for that. The test suite is a very untypical browser client and
>> doesn't really test real-world browser use-cases.
>>
>> Cheers
>> Jan
>> --
>
> +a bajillion.
>
I prefer the browser tests because I'm much happier with JavaScript.
>>>
>>> I'm not saying we should get rid of the browser tests. But intermittent
>>> errors
>>> in the current test suite are not to be worried about to block a release.
>>
>> I agree with all the comments about separation of tests and so forth. This
>> particular changes test is not intermittent, it consistently fails (on my
>> machine), enough that it's a pleasant surprise when it succeeds. When
>> running from the CLI I get the following:
>>
>> not ok 10 changes expected '3', got '1'
>>
>> When running in FF I also get the message above and occasionally:
>>
>> • Exception raised:
>> {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":154,"stack":";(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:154\u000arun(-2)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}
>>
>> I haven't looked into it closely to find the root cause, it might just be
>> the test, but it's definitely not intermittent. From the CLI it happens
>> almost always
>>
>>
>>
>>>
>>> If we want proper browser client testing, we'd need an additional test suite
>>> that covers common and uncommon use-cases. I believe the current test
>>> suite is as untypical as a browser client can be.
>>>
>>> Cheers
>>> Jan
>>> --
>>>
>>>
But maybe I'm crazy
> I think its important to maintain *some* tests in the browser to test
> its ability to use CouchDB as a client, but we should put more work
> into separating API tests and core tests.
>
> Also, Zed Shaw has a very informative (and colorful) description of
> confounding factors [1]. Its about two thirds of the way down under a
> heading of "Confounding, Confounding, Confounding."
>
> http://www.zedshaw.com/essays/programmer_stats.html
>>>
>>
Re: Test suite blocking release
On Sat, Mar 20, 2010 at 5:52 PM, Noah Slater wrote: > Historically, test suite failures on Futon were release blockers. I'd like > to get some consensus on if this is still the case. It's imperative we have > a formal rule here, to help me, and to help in-between development. Once > we've cleared that up, I suggest that we make this part of the documented > release procedure. That way, asking for a release signifies that the test > suites pass acceptably in the environments we have agreed the must pass in. > I think this will reduce a significant amount of friction for future > releases. If I get blocked, or stalled, in any way - that should indicate a > process failure. Thoughts? The changes failure may be more important than a simple browser thing. I noticed this temporary failure in couchdbkit tests too. I think that we should make sure all tests pass in ff at least to declare this ok. - benoit
Re: Test suite blocking release
Historically, test suite failures on Futon were release blockers. I'd like to get some consensus on if this is still the case. It's imperative we have a formal rule here, to help me, and to help in- between development. Once we've cleared that up, I suggest that we make this part of the documented release procedure. That way, asking for a release signifies that the test suites pass acceptably in the environments we have agreed the must pass in. I think this will reduce a significant amount of friction for future releases. If I get blocked, or stalled, in any way - that should indicate a process failure. Thoughts? On 19 Mar 2010, at 23:25, Jan Lehnardt wrote: On 19 Mar 2010, at 18:07, J Chris Anderson wrote: On Mar 19, 2010, at 11:43 AM, Paul Davis wrote: On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt wrote: On 19 Mar 2010, at 12:50, Noah Slater wrote: On 19 Mar 2010, at 17:11, Jan Lehnardt wrote: We want to test the CouchDB code, not the browser's HTTP handling. Sure, but as one of CouchDB's primary interfaces is the browser, it seems to makes sense that we would want to test how this works. Testing from the browser allows us to test for and catch problems introduced by caching, etc - which is what our real world users would be running into. Unless I'm missing something? I fully agree, but we should have a separate browser interaction suite for that. The test suite is a very untypical browser client and doesn't really test real-world browser use-cases. Cheers Jan -- +a bajillion. I prefer the browser tests because I'm much happier with JavaScript. I'm not saying we should get rid of the browser tests. But intermittent errors in the current test suite are not to be worried about to block a release. If we want proper browser client testing, we'd need an additional test suite that covers common and uncommon use-cases. I believe the current test suite is as untypical as a browser client can be. Cheers Jan -- But maybe I'm crazy I think its important to maintain *some* tests in the browser to test its ability to use CouchDB as a client, but we should put more work into separating API tests and core tests. Also, Zed Shaw has a very informative (and colorful) description of confounding factors [1]. Its about two thirds of the way down under a heading of "Confounding, Confounding, Confounding." http://www.zedshaw.com/essays/programmer_stats.html
Re: Test suite blocking release
Jan, should this block the release? From what I can tell, it should.
On 20 Mar 2010, at 11:32, Robert Dionne
wrote:
On Mar 19, 2010, at 7:25 PM, Jan Lehnardt wrote:
On 19 Mar 2010, at 18:07, J Chris Anderson wrote:
On Mar 19, 2010, at 11:43 AM, Paul Davis wrote:
On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt
wrote:
On 19 Mar 2010, at 12:50, Noah Slater wrote:
On 19 Mar 2010, at 17:11, Jan Lehnardt wrote:
We want to test the CouchDB code, not the browser's HTTP
handling.
Sure, but as one of CouchDB's primary interfaces is the
browser, it seems to makes sense that we would want to test how
this works. Testing from the browser allows us to test for and
catch problems introduced by caching, etc - which is what our
real world users would be running into.
Unless I'm missing something?
I fully agree, but we should have a separate browser interaction
suite for that. The test suite is a very untypical browser
client and
doesn't really test real-world browser use-cases.
Cheers
Jan
--
+a bajillion.
I prefer the browser tests because I'm much happier with JavaScript.
I'm not saying we should get rid of the browser tests. But
intermittent errors
in the current test suite are not to be worried about to block a
release.
I agree with all the comments about separation of tests and so
forth. This particular changes test is not intermittent, it
consistently fails (on my machine), enough that it's a pleasant
surprise when it succeeds. When running from the CLI I get the
following:
not ok 10 changes expected '3', got '1'
When running in FF I also get the message above and occasionally:
• Exception raised: {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11
.0","lineNumber":154,"stack":"(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:154\u000arun(-2)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a
"}
I haven't looked into it closely to find the root cause, it might
just be the test, but it's definitely not intermittent. From the CLI
it happens almost always
If we want proper browser client testing, we'd need an additional
test suite
that covers common and uncommon use-cases. I believe the current test
suite is as untypical as a browser client can be.
Cheers
Jan
--
But maybe I'm crazy
I think its important to maintain *some* tests in the browser to
test
its ability to use CouchDB as a client, but we should put more work
into separating API tests and core tests.
Also, Zed Shaw has a very informative (and colorful) description of
confounding factors [1]. Its about two thirds of the way down
under a
heading of "Confounding, Confounding, Confounding."
http://www.zedshaw.com/essays/programmer_stats.html
Re: Test suite blocking release
On Mar 19, 2010, at 7:25 PM, Jan Lehnardt wrote:
>
> On 19 Mar 2010, at 18:07, J Chris Anderson wrote:
>
>>
>> On Mar 19, 2010, at 11:43 AM, Paul Davis wrote:
>>
>>> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt wrote:
On 19 Mar 2010, at 12:50, Noah Slater wrote:
>
> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote:
>
>> We want to test the CouchDB code, not the browser's HTTP handling.
>
> Sure, but as one of CouchDB's primary interfaces is the browser, it seems
> to makes sense that we would want to test how this works. Testing from
> the browser allows us to test for and catch problems introduced by
> caching, etc - which is what our real world users would be running into.
>
> Unless I'm missing something?
I fully agree, but we should have a separate browser interaction
suite for that. The test suite is a very untypical browser client and
doesn't really test real-world browser use-cases.
Cheers
Jan
--
>>>
>>> +a bajillion.
>>>
>>
>> I prefer the browser tests because I'm much happier with JavaScript.
>
> I'm not saying we should get rid of the browser tests. But intermittent errors
> in the current test suite are not to be worried about to block a release.
I agree with all the comments about separation of tests and so forth. This
particular changes test is not intermittent, it consistently fails (on my
machine), enough that it's a pleasant surprise when it succeeds. When running
from the CLI I get the following:
not ok 10 changes expected '3', got '1'
When running in FF I also get the message above and occasionally:
• Exception raised:
{"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":154,"stack":";(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:154\u000arun(-2)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}
I haven't looked into it closely to find the root cause, it might just be the
test, but it's definitely not intermittent. From the CLI it happens almost
always
>
> If we want proper browser client testing, we'd need an additional test suite
> that covers common and uncommon use-cases. I believe the current test
> suite is as untypical as a browser client can be.
>
> Cheers
> Jan
> --
>
>
>>
>> But maybe I'm crazy
>>
>>
>>> I think its important to maintain *some* tests in the browser to test
>>> its ability to use CouchDB as a client, but we should put more work
>>> into separating API tests and core tests.
>>>
>>> Also, Zed Shaw has a very informative (and colorful) description of
>>> confounding factors [1]. Its about two thirds of the way down under a
>>> heading of "Confounding, Confounding, Confounding."
>>>
>>> http://www.zedshaw.com/essays/programmer_stats.html
>>
>
Re: Test suite blocking release
On 19 Mar 2010, at 18:07, J Chris Anderson wrote: > > On Mar 19, 2010, at 11:43 AM, Paul Davis wrote: > >> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt wrote: >>> >>> On 19 Mar 2010, at 12:50, Noah Slater wrote: >>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote: > We want to test the CouchDB code, not the browser's HTTP handling. Sure, but as one of CouchDB's primary interfaces is the browser, it seems to makes sense that we would want to test how this works. Testing from the browser allows us to test for and catch problems introduced by caching, etc - which is what our real world users would be running into. Unless I'm missing something? >>> >>> I fully agree, but we should have a separate browser interaction >>> suite for that. The test suite is a very untypical browser client and >>> doesn't really test real-world browser use-cases. >>> >>> Cheers >>> Jan >>> -- >> >> +a bajillion. >> > > I prefer the browser tests because I'm much happier with JavaScript. I'm not saying we should get rid of the browser tests. But intermittent errors in the current test suite are not to be worried about to block a release. If we want proper browser client testing, we'd need an additional test suite that covers common and uncommon use-cases. I believe the current test suite is as untypical as a browser client can be. Cheers Jan -- > > But maybe I'm crazy > > >> I think its important to maintain *some* tests in the browser to test >> its ability to use CouchDB as a client, but we should put more work >> into separating API tests and core tests. >> >> Also, Zed Shaw has a very informative (and colorful) description of >> confounding factors [1]. Its about two thirds of the way down under a >> heading of "Confounding, Confounding, Confounding." >> >> http://www.zedshaw.com/essays/programmer_stats.html >
Re: Test suite blocking release
On Mar 19, 2010, at 11:43 AM, Paul Davis wrote: > On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt wrote: >> >> On 19 Mar 2010, at 12:50, Noah Slater wrote: >> >>> >>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote: >>> We want to test the CouchDB code, not the browser's HTTP handling. >>> >>> Sure, but as one of CouchDB's primary interfaces is the browser, it seems >>> to makes sense that we would want to test how this works. Testing from the >>> browser allows us to test for and catch problems introduced by caching, etc >>> - which is what our real world users would be running into. >>> >>> Unless I'm missing something? >> >> I fully agree, but we should have a separate browser interaction >> suite for that. The test suite is a very untypical browser client and >> doesn't really test real-world browser use-cases. >> >> Cheers >> Jan >> -- > > +a bajillion. > I prefer the browser tests because I'm much happier with JavaScript. But maybe I'm crazy > I think its important to maintain *some* tests in the browser to test > its ability to use CouchDB as a client, but we should put more work > into separating API tests and core tests. > > Also, Zed Shaw has a very informative (and colorful) description of > confounding factors [1]. Its about two thirds of the way down under a > heading of "Confounding, Confounding, Confounding." > > http://www.zedshaw.com/essays/programmer_stats.html
Re: Test suite blocking release
On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt wrote: > > On 19 Mar 2010, at 12:50, Noah Slater wrote: > >> >> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote: >> >>> We want to test the CouchDB code, not the browser's HTTP handling. >> >> Sure, but as one of CouchDB's primary interfaces is the browser, it seems to >> makes sense that we would want to test how this works. Testing from the >> browser allows us to test for and catch problems introduced by caching, etc >> - which is what our real world users would be running into. >> >> Unless I'm missing something? > > I fully agree, but we should have a separate browser interaction > suite for that. The test suite is a very untypical browser client and > doesn't really test real-world browser use-cases. > > Cheers > Jan > -- +a bajillion. I think its important to maintain *some* tests in the browser to test its ability to use CouchDB as a client, but we should put more work into separating API tests and core tests. Also, Zed Shaw has a very informative (and colorful) description of confounding factors [1]. Its about two thirds of the way down under a heading of "Confounding, Confounding, Confounding." http://www.zedshaw.com/essays/programmer_stats.html
Re: Test suite blocking release
On 19 Mar 2010, at 12:50, Noah Slater wrote: > > On 19 Mar 2010, at 17:11, Jan Lehnardt wrote: > >> We want to test the CouchDB code, not the browser's HTTP handling. > > Sure, but as one of CouchDB's primary interfaces is the browser, it seems to > makes sense that we would want to test how this works. Testing from the > browser allows us to test for and catch problems introduced by caching, etc - > which is what our real world users would be running into. > > Unless I'm missing something? I fully agree, but we should have a separate browser interaction suite for that. The test suite is a very untypical browser client and doesn't really test real-world browser use-cases. Cheers Jan --
Re: Test suite blocking release
On 19 Mar 2010, at 17:11, Jan Lehnardt wrote: > We want to test the CouchDB code, not the browser's HTTP handling. Sure, but as one of CouchDB's primary interfaces is the browser, it seems to makes sense that we would want to test how this works. Testing from the browser allows us to test for and catch problems introduced by caching, etc - which is what our real world users would be running into. Unless I'm missing something?
Re: Test suite blocking release
On 19 Mar 2010, at 11:56, Noah Slater wrote: > > On 19 Mar 2010, at 16:49, Jan Lehnardt wrote: > >>> And if so, what can we do to fix the whole situation post-release? >> >> (answering anyway :) Move all JS tests to etap tests to factor out browser >> issues, caching, timing, etc and don't introduce any errors. > > > Aren't these exactly the type of issues we should be testing? :) > > (modulo time-dependant failures) We want to test the CouchDB code, not the browser's HTTP handling. Cheers Jan --
Re: Test suite blocking release
On 19 Mar 2010, at 16:49, Jan Lehnardt wrote: >> And if so, what can we do to fix the whole situation post-release? > > (answering anyway :) Move all JS tests to etap tests to factor out browser > issues, caching, timing, etc and don't introduce any errors. Aren't these exactly the type of issues we should be testing? :) (modulo time-dependant failures)
Re: Test suite blocking release
On 19 Mar 2010, at 11:44, Noah Slater wrote:
> Yep, I checked both browsers out of habit more than anything.
>
> What about the other thread discussing an actual issue brought up by some of
> these failures?
It was an issue in the test case, not core code.
> Is it not worrying that they failed first then worked, or is this something
> ignorable?
The latter.
> And if so, what can we do to fix the whole situation post-release?
(answering anyway :) Move all JS tests to etap tests to factor out browser
issues, caching, timing, etc and don't introduce any errors.
Cheers
Jan
--
>
> On 19 Mar 2010, at 15:53, Jan Lehnardt wrote:
>
>> Hi Noah,
>>
>> The test suite doesn't support Safari. While I agree that the
>> first-failing-then-working tests in Firefox are concerning, I say it doesn't
>> block a release.
>>
>> Cheers
>> Jan
>> --
>>
>>
>> On 19 Mar 2010, at 06:59, Noah Slater wrote:
>>
>>> Hey,
>>>
>>> I was going to call a vote on the release today, but I am being blocked by
>>> test suite errors.
>>>
>>> Running in Safari:
>>>
>>> changes
>>>
>>> • Assertion 'should return matching rows' failed: expected '3', got '1'
>>>
>>> rev_stemming
>>>
>>> • Assertion 'should return a truncated revision list' failed: expected
>>> '5', got '6'
>>>
>>> The first one fails consistently, with debugging and without. No additional
>>> details provided by the test.
>>>
>>> The second one is quite mysterious, however, and needs to be fixed.
>>>
>>> I ran it again with the debugger, and IT PASSED.
>>>
>>> I ran it again as normal, and got:
>>>
>>> • Assertion failed: db.open("bar", {revs:true})._revisions.ids.length
>>> == newLimit + 1
>>> • Assertion failed: docB2._conflicts[0] == docB1._rev) // We having
>>> already updated bar before setting the limit, so it's still got // a long
>>> rev history. compact to stem the revs. T(db.open("bar",
>>> {revs:true})._revisions.ids.length == newLimit + 1
>>>
>>> I ran it again with the debugger:
>>>
>>> • Assertion failed: false
>>> • Assertion failed: false
>>>
>>> I ran the whole thing from scratch, and I got the first error, then the
>>> second error message for the second test.
>>>
>>> Running on Firefox:
>>>
>>> changes
>>>
>>> • Assertion 'JSON.parse(lines[2]).id == "rusty", lines[2]' failed:
>>> {"last_seq":9}
>>> • Exception raised:
>>> {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":167,"stack":";(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:167\u000arun(11)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}
>>> compactsuccess304ms
>>>
>>> list_views
>>>
>>> • Exception raised: {}
>>>
>>> Both tests pass when run a second time.
>>>
>>> A few other people on IRC were able to reproduce some of the errors I have
>>> described here.
>>>
>>> I now consider these tests suspicious too, and have no idea how serious the
>>> problems are.
>>>
>>> Any help or guidance would be fantastic.
>>>
>>> We're so close to the release!
>>>
>>
>
Re: Test suite blocking release
Yep, I checked both browsers out of habit more than anything.
What about the other thread discussing an actual issue brought up by some of
these failures?
Is it not worrying that they failed first then worked, or is this something
ignorable?
And if so, what can we do to fix the whole situation post-release?
On 19 Mar 2010, at 15:53, Jan Lehnardt wrote:
> Hi Noah,
>
> The test suite doesn't support Safari. While I agree that the
> first-failing-then-working tests in Firefox are concerning, I say it doesn't
> block a release.
>
> Cheers
> Jan
> --
>
>
> On 19 Mar 2010, at 06:59, Noah Slater wrote:
>
>> Hey,
>>
>> I was going to call a vote on the release today, but I am being blocked by
>> test suite errors.
>>
>> Running in Safari:
>>
>> changes
>>
>> • Assertion 'should return matching rows' failed: expected '3', got '1'
>>
>> rev_stemming
>>
>> • Assertion 'should return a truncated revision list' failed: expected
>> '5', got '6'
>>
>> The first one fails consistently, with debugging and without. No additional
>> details provided by the test.
>>
>> The second one is quite mysterious, however, and needs to be fixed.
>>
>> I ran it again with the debugger, and IT PASSED.
>>
>> I ran it again as normal, and got:
>>
>> • Assertion failed: db.open("bar", {revs:true})._revisions.ids.length
>> == newLimit + 1
>> • Assertion failed: docB2._conflicts[0] == docB1._rev) // We having
>> already updated bar before setting the limit, so it's still got // a long
>> rev history. compact to stem the revs. T(db.open("bar",
>> {revs:true})._revisions.ids.length == newLimit + 1
>>
>> I ran it again with the debugger:
>>
>> • Assertion failed: false
>> • Assertion failed: false
>>
>> I ran the whole thing from scratch, and I got the first error, then the
>> second error message for the second test.
>>
>> Running on Firefox:
>>
>> changes
>>
>> • Assertion 'JSON.parse(lines[2]).id == "rusty", lines[2]' failed:
>> {"last_seq":9}
>> • Exception raised:
>> {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":167,"stack":";(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:167\u000arun(11)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}
>> compactsuccess304ms
>>
>> list_views
>>
>> • Exception raised: {}
>>
>> Both tests pass when run a second time.
>>
>> A few other people on IRC were able to reproduce some of the errors I have
>> described here.
>>
>> I now consider these tests suspicious too, and have no idea how serious the
>> problems are.
>>
>> Any help or guidance would be fantastic.
>>
>> We're so close to the release!
>>
>
Re: Test suite blocking release
Hi Noah,
The test suite doesn't support Safari. While I agree that the
first-failing-then-working tests in Firefox are concerning, I say it doesn't
block a release.
Cheers
Jan
--
On 19 Mar 2010, at 06:59, Noah Slater wrote:
> Hey,
>
> I was going to call a vote on the release today, but I am being blocked by
> test suite errors.
>
> Running in Safari:
>
> changes
>
> • Assertion 'should return matching rows' failed: expected '3', got '1'
>
> rev_stemming
>
> • Assertion 'should return a truncated revision list' failed: expected
> '5', got '6'
>
> The first one fails consistently, with debugging and without. No additional
> details provided by the test.
>
> The second one is quite mysterious, however, and needs to be fixed.
>
> I ran it again with the debugger, and IT PASSED.
>
> I ran it again as normal, and got:
>
> • Assertion failed: db.open("bar", {revs:true})._revisions.ids.length
> == newLimit + 1
> • Assertion failed: docB2._conflicts[0] == docB1._rev) // We having
> already updated bar before setting the limit, so it's still got // a long rev
> history. compact to stem the revs. T(db.open("bar",
> {revs:true})._revisions.ids.length == newLimit + 1
>
> I ran it again with the debugger:
>
> • Assertion failed: false
> • Assertion failed: false
>
> I ran the whole thing from scratch, and I got the first error, then the
> second error message for the second test.
>
> Running on Firefox:
>
> changes
>
> • Assertion 'JSON.parse(lines[2]).id == "rusty", lines[2]' failed:
> {"last_seq":9}
> • Exception raised:
> {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":167,"stack":";(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:167\u000arun(11)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}
> compactsuccess304ms
>
> list_views
>
> • Exception raised: {}
>
> Both tests pass when run a second time.
>
> A few other people on IRC were able to reproduce some of the errors I have
> described here.
>
> I now consider these tests suspicious too, and have no idea how serious the
> problems are.
>
> Any help or guidance would be fantastic.
>
> We're so close to the release!
>
Test suite blocking release
Hey,
I was going to call a vote on the release today, but I am being blocked by test
suite errors.
Running in Safari:
changes
• Assertion 'should return matching rows' failed: expected '3', got '1'
rev_stemming
• Assertion 'should return a truncated revision list' failed: expected
'5', got '6'
The first one fails consistently, with debugging and without. No additional
details provided by the test.
The second one is quite mysterious, however, and needs to be fixed.
I ran it again with the debugger, and IT PASSED.
I ran it again as normal, and got:
• Assertion failed: db.open("bar", {revs:true})._revisions.ids.length
== newLimit + 1
• Assertion failed: docB2._conflicts[0] == docB1._rev) // We having
already updated bar before setting the limit, so it's still got // a long rev
history. compact to stem the revs. T(db.open("bar",
{revs:true})._revisions.ids.length == newLimit + 1
I ran it again with the debugger:
• Assertion failed: false
• Assertion failed: false
I ran the whole thing from scratch, and I got the first error, then the second
error message for the second test.
Running on Firefox:
changes
• Assertion 'JSON.parse(lines[2]).id == "rusty", lines[2]' failed:
{"last_seq":9}
• Exception raised:
{"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":167,"stack":";(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:167\u000arun(11)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}
compactsuccess304ms
list_views
• Exception raised: {}
Both tests pass when run a second time.
A few other people on IRC were able to reproduce some of the errors I have
described here.
I now consider these tests suspicious too, and have no idea how serious the
problems are.
Any help or guidance would be fantastic.
We're so close to the release!
