Re: TAP Diagnostics

[Aristotle Pagaltzis]

* Andy Armstrong <[EMAIL PROTECTED]> [2008-08-22 00:35]:
> Indeed - but there's a sweet spot somewhere between specifying
> too little and specifying too much. It's entirely proper that
> there should be a debate about where that sweet spot is - and
> parsimony should be a guiding principle - but the value of
> having inline, per-test metadata has already been demonstrated.

Fully and strongly agreed, ++ on all counts.

> IMO we don't need a complete serialisation protocol - just
> somewhere we can make notes and have them pass unmolested
> through the harness. If someone needs arbitrary serialisation
> they can use a protocol that makes sense to them and either
> inline their serialised data as a Base64 encoded string or have
> the diagnostic refer to an external file. The important thing
> is to open up a conduit through which user defined data can
> flow from a test script. […] I favour JSON because it's the
> simplest solution that fits those criteria.

Exactly. We want to make it easy to annotate tests with any sort
of testing-relevant metadata that people might have, but the
purpose of the protocol is supposed to be conveying test results,
not one-way messaging of arbitrary data. Eg. someone who uses TAP
in testing a web app will probably have use for an HTTP response
status key and also for a hash of response headers.

The killer app example for diagnostics that I always come back to
is statical analysis over an archive of your TAP streams over
time. It makes sense to preserve as much test-relevant metadata
as possible to be able to examine trends. For this purpose, the
data does have to be machine readable, and it does have to be
structured at least a little – *just* a flat key-value bag won’t
cut it. However, no intricate structure is necessary.

This is a point where the APIs can help. If the API encourages
people to indiscriminately dump complex data structures into the
TAP stream, that’s what people will do. If the API instead makes
it very easy to use diagnostics in the way we intend for them to
be used, and doesn’t go out of its way to accomodate the potato
bag usage, fewer people will be tempted.

Regards,
-- 
Aristotle Pagaltzis // 

Re: JSON TAP Diagnostics?

[Aristotle Pagaltzis]

* Ricardo SIGNES <[EMAIL PROTECTED]> [2008-08-21 19:00]:
> Ovid (and I) would like it to be JSON, pending any better idea
> (that we agree is better).

FWIW, after some consideration, I find myself drifting toward
the JSON camp as well. It’s a slow drift, but I just can’t help
disliking YAML for making the ultimate mockery of its own initial
design goals. (You have no idea how much regret-on-someone-
else’s-behalf this has caused me. I would love to be able to like
YAML.)
 
> Schwern would like it to be YAML (a superset of JSON), with the
> phrasing "consumers MUST understand JSON and SHOULD understand
> YAML."

I could arrange myself with Schwern’s desire, so long as we could
agree to make that SHOULD a MAY. That would in fact work very
well for me.

Regards,
-- 
Aristotle Pagaltzis // 

Re: JSON TAP Diagnostics?

[Aristotle Pagaltzis]

* Eric Wilhelm <[EMAIL PROTECTED]> [2008-08-21 18:50]:
> Does it have to be just one? Now and forever?

It doesn’t have to be *just* one, but it needs to be *at least*
one, and specifically at least one that *everyone* supports, so
that you can count on having a way to make an emitter and
consumer understand each other.

The question that arises is that since everyone has to support
that particular format, how much value do we gain from letting
people use other ones?

Personally I am not sufficiently convinced that allowing for more
than one format will turn out to be harmful that I would argue
against it, but I am also quite unconvinced of the value, and I
do know that this flexibility will incur costs.

So I am inclined to say that it should probably be just one
format, now and forever. (If worst comes to worst we still have
the very-last-resort option of revving TAP.)

Regards,
-- 
Aristotle Pagaltzis // 

Re: IETF list? (was Re: JSON TAP Diagnostics?)

[Andy Armstrong]

On 21 Aug 2008, at 23:37, Michael G Schwern wrote:

What IETF list?



https://www.ietf.org/mailman/listinfo/tap

--
Andy Armstrong, Hexten

IETF list? (was Re: JSON TAP Diagnostics?)

[Michael G Schwern]

Ovid wrote:
> Folks, this really, really needs to go to the IETF list.

What IETF list?


-- 
Ahh email, my old friend.  Do you know that revenge is a dish that is best
served cold?  And it is very cold on the Internet!

Re: [tap] TAP Diagnostics

[Andy Armstrong]

On 21 Aug 2008, at 22:41, chromatic wrote:
I've written a handful of TAP producers and consumers myself, as  
well as
software which other people have used in ways I never intended.  The  
more
complexity you add to a system to reduce the complexity people have  
to manage
to use your system as you never intended, the more complexity  
everyone has to

manage to use your system as you always intended.



Indeed - but there's a sweet spot somewhere between specifying too  
little and specifying too much. It's entirely proper that there should  
be a debate about where that sweet spot is - and parsimony should be a  
guiding principle - but the value of having inline, per-test metadata  
has already been demonstrated.


IMO we don't need a complete serialisation protocol - just somewhere  
we can make notes and have them pass unmolested through the harness.  
If someone needs arbitrary serialisation they can use a protocol that  
makes sense to them and either inline their serialised data as a  
Base64 encoded string or have the diagnostic refer to an external  
file. The important thing is to open up a conduit through which user  
defined data can flow from a test script.


Now, that could just be an opaque block of inline data - but you'd  
want that to be delimited in some way so the TAP parser knows how to  
step over it. Having to wrap it up as a minimal JSON or YAML is not a  
significantly greater burden than, say, MIME multipart encoding it and  
using a structured representation increases the chances that common  
usage will emerge. Cowpaths to pave.


I favour JSON because it's the simplest solution that fits those  
criteria.


--
Andy Armstrong, Hexten

Re: Silence Command Line in TAP::Harness?

[David E. Wheeler]

On Aug 21, 2008, at 14:48, Andy Armstrong wrote:

There was meant to be a smiley in my response. I was just relieved  
it wasn't some debug /I'd/ left lying around - so thanks :)


:-D

Best,

David

Re: TAP Diagnostics

[Ovid]

--- On Thu, 21/8/08, chromatic <[EMAIL PROTECTED]> wrote:

> I've written a handful of TAP producers and consumers
> myself, as well as 
> software which other people have used in ways I never
> intended.  The more 
> complexity you add to a system to reduce the complexity
> people have to manage 
> to use your system as you never intended, the more
> complexity everyone has to 
> manage to use your system as you always intended.

Except that core TAP is not changing and all older TAP consumers will be just 
fine.

> I can deal with a "simple list of key/value
> pairs", but the "mostly" and "for 
> the most part" bother me.

Fair enough.  We'll just have to see how this shakes out.

Cheers,
Ovid

Re: Silence Command Line in TAP::Harness?

[Andy Armstrong]

On 21 Aug 2008, at 22:05, David E. Wheeler wrote:

  print join ' ', @command, $/;


Oh Jesus, that was stupid. Must've been debugging and completely  
forgot about it. Sorry for the noise, and thanks for the prompt reply.



There was meant to be a smiley in my response. I was just relieved it  
wasn't some debug /I'd/ left lying around - so thanks :)


--
Andy Armstrong, Hexten

Re: TAP Diagnostics

[chromatic]

On Thursday 21 August 2008 14:29:54 Ovid wrote:

> Remember, all of this is OPTIONAL.  If you don't want it, don't use it.

I'm not sure that line of thinking works well for standards, OOXML 
notwithstanding.  Acme::Calendar::Mayan might want to define a vigesimal type 
to encode time intervals; why not chuck a canonical representation of 
alternate base systems in the OPTIONAL section of the spec as well while 
we're talking about it?

I've written a handful of TAP producers and consumers myself, as well as 
software which other people have used in ways I never intended.  The more 
complexity you add to a system to reduce the complexity people have to manage 
to use your system as you never intended, the more complexity everyone has to 
manage to use your system as you always intended.

I can deal with a "simple list of key/value pairs", but the "mostly" and "for 
the most part" bother me.

-- c

Re: TAP Diagnostics

[Ovid]

--- On Thu, 21/8/08, chromatic <[EMAIL PROTECTED]> wrote:

> That part is the wad.

No, for the most part, it's a simple list of key/value pairs.

> (You might find it amusing to note that the SOAP folks
> didn't use the 
> sack/potato/wad metaphor when naming their failure.)

SOAP has the problem that the acronym has nothing to do with the reality, so 
they started with a huge mess o' confusion to start with.
 
For the most part, we're just looking for is taking the information that's 
ALREADY in diagnostics and making it machine readable.  Remember, this is 
mostly just a list of key/value pairs :)

> I wonder why anyone wants a test so complex that its
> diagnostic requires you 
> to serialize and deserialize objects and/or nested data
> structures to and 
> from custom TAP producers and consumers ...

Testing in large scale environments has different needs than CPAN modules.  
People who don't work in those environments often don't see it.  Remember the 
brouhaha over Module::Build?  Many people said it was useless because EUMM does 
everything they need.  Obviously, as consumers, their needs are different from 
authors and they didn't appreciate concerns in domains they don't experience.

Remember, all of this is OPTIONAL.  If you don't want it, don't use it.  And 
don't forget to drop an email to Yahoo! telling them that their silly work with 
tagging TAP streams is over-engineering.

> Me, I'll be over here, trying to make my tests
> sufficiently easy to debug that 
> I don't have to worry about these things.

Wish I had that luxury.

Cheers,
Ovid

Re: TAP Diagnostics

[David E. Wheeler]

On Aug 21, 2008, at 13:58, chromatic wrote:

I wonder why anyone wants a test so complex that its diagnostic  
requires you
to serialize and deserialize objects and/or nested data structures  
to and
from custom TAP producers and consumers, and, if you really need to  
do that,
if you should start with a testing zeitgeist which doesn't consider  
a strict
separation of producer and consumer a necessary and sufficient  
condition.


So this begs the question: What is this serialization thingy for,  
exactly? What will it be used for? Why does anything need to be  
serialized? TAP is a protocol for test results, not an object broker  
or remote execution protocol. Is it?


Best,

David

Re: Silence Command Line in TAP::Harness?

[David E. Wheeler]

On Aug 21, 2008, at 12:09, Andy Armstrong wrote:


That's your code, no?

From https://svn.kineticode.com/pgtap/tags/rel-0.02/pg_prove

   push @command, qw(
   --no-psqlrc
   --no-align
   --tuples-only
   --pset pager=
   --pset null=[NULL]
   --set ON_ERROR_ROLLBACK=1
   --file
   );

   print join ' ', @command, $/;


Oh Jesus, that was stupid. Must've been debugging and completely  
forgot about it. Sorry for the noise, and thanks for the prompt reply.


Best,

David

Re: TAP Diagnostics

[chromatic]

On Thursday 21 August 2008 13:07:22 Eric Wilhelm wrote:

> Well, exactly how are we defining "sack", "potato", and "wad" here, and
> how does it have anything to do with what people want?

Specifying a serialization format in TAP diagnostics says "Here's a 
serialization format you can use to store a complex data structure, perhaps 
an object, perhaps with type information, to encode into a text-based 
protocol intended for humans to read and interpret, hoping that something on 
the other end will understand the precise semantics of the serialized object 
and do something sane with it... oh, and there's something in that stream 
which tells you that the test passed, if you can dig out the important one 
bit of information."

That part is the wad.

The sack is the big arbitrary bag of out-of-band information potentially 
attached to the TAP stream which can contain anything (which tends to 
encourage people to put anything in it).

The potato is the arbitrary anything which people may want to put into the 
sack.

(You might find it amusing to note that the SOAP folks didn't use the 
sack/potato/wad metaphor when naming their failure.)

I wonder why anyone wants a test so complex that its diagnostic requires you 
to serialize and deserialize objects and/or nested data structures to and 
from custom TAP producers and consumers, and, if you really need to do that, 
if you should start with a testing zeitgeist which doesn't consider a strict 
separation of producer and consumer a necessary and sufficient condition.

Me, I'll be over here, trying to make my tests sufficiently easy to debug that 
I don't have to worry about these things.

-- c

Re: TAP Diagnostics

[Ovid]

--- On Thu, 21/8/08, Eric Wilhelm <[EMAIL PROTECTED]> wrote:

> Well, exactly how are we defining "sack",
> "potato", and "wad" here, and 
> how does it have anything to do with what people want?

To be fair, those were chromatic's words, not mine, so I'm not sure exactly 
what he meant by them and I don't want to put words in his mouth.

The rest addresses specific issues for the IETF process so I'll pull that out 
of this discussion and put it on the IETF list.  Time to stop spamming folks.  
If they want the discussion, they can join the list:

  https://www.ietf.org/mailman/listinfo/tap
 
Cheers,
Ovid

Re: TAP Diagnostics

[Eric Wilhelm]

# from Ovid
# on Thursday 21 August 2008 11:53:

>> If TAP goes any further than saying "Yes, you *can*
>> stuff a big wad of custom data in this ganky old potato sack
>> ... there may be a hole in the bottom ...
>> so if there's any other way, we encourage you to do that" ...

What would this "any other way" be?  And why would the sack have a hole 
in it?  Are we delivering a package or just kicking a sack down the 
street?

>That's actually all TAP will be doing.  You *can* stuff that big wad
> of diagnostic data there in a machine-readable format, but there are
> two overriding concerns:
>
>1.  It's entirely optional.
>2.  It must not alter the interpretation of core TAP.
>...
>But at the end of the day, this is a feature people have been wanting
> for a long time, even if (for most cases), it's merely a case of
> "file, line, have, want".

Well, exactly how are we defining "sack", "potato", and "wad" here, and 
how does it have anything to do with what people want?

If diagnostics are a data block, the only thing the TAP stream must do 
is keep the data from getting corrupted and/or leaking into the rest of 
the stream.  This says the consumer has to know where it ends, which 
doesn't work if it's a sack with a hole in it.

But none of that provides anybody with the "file, line, have, want" 
thing.  That would be a special, identified, form of data block with 
parsable, meaningful content.

If I have a GUI harness displaying the 'have' and 'want' graphics 
(pretend they are jpg or svg, or even one of each) for a failed test 
next to each other, would this be a custom consumer only in that it 
does something particular in interpreting the 'have' and 'want' values?

Or, is it a custom consumer in that it takes the $diagnostic block (with 
an $identifier which says it is JSON), deserializes that, and looks 
for 'have', 'want', and etc values in the resultant structure?

If the former, then the TAP spec must define the structure of the 
diagnostics, or at least some part of it, and still needs to provide a 
sack somehow.  If the latter, then TAP just defines a sack of bytes (it 
could still go forward from there to claim some particular sacks as 
reserved and/or recommended usage, right?)

--Eric
-- 
"...our schools have been scientifically designed to
prevent overeducation from happening."
--William Troy Harris
---
http://scratchcomputing.com
---

Re: Silence Command Line in TAP::Harness?

[Andy Armstrong]

On 21 Aug 2008, at 19:54, David E. Wheeler wrote:
Is there some way to eliminate that noisy line that shows the  
command that will be run?


That's your code, no?

From https://svn.kineticode.com/pgtap/tags/rel-0.02/pg_prove

push @command, qw(
--no-psqlrc
--no-align
--tuples-only
--pset pager=
--pset null=[NULL]
--set ON_ERROR_ROLLBACK=1
--file
);

print join ' ', @command, $/;

--
Andy Armstrong, Hexten

Silence Command Line in TAP::Harness?

[David E. Wheeler]

Howdy,

When I run the pgTAP tests through pg_prove, which uses TAP::Harness,  
it looks like this:


% pg_prove -d try sql/*.sql
psql --dbname try --no-psqlrc --no-align --tuples-only --pset  
pager= --pset null=[NULL] --set ON_ERROR_ROLLBACK=1 --set  
ON_ERROR_STOP=1 --set QUIET=1 --file

sql/coltap.ok
sql/hastap.ok
sql/moretapok
sql/pg73...ok
sql/pktap..ok
All tests successful.
Files=5, Tests=216,  1 wallclock secs ( 0.06 usr  0.02 sys +   
0.08 cusr  0.07 csys =  0.23 CPU)

Result: PASS

Is there some way to eliminate that noisy line that shows the command  
that will be run?


Thanks,

David

Re: TAP Diagnostics

[Ovid]

--- On Thu, 21/8/08, chromatic <[EMAIL PROTECTED]> wrote:

> If TAP goes any further than saying "Yes, you *can*
> stuff a big wad of custom 
> data in this ganky old potato sack, but it's ugly and
> covered in spiders and 
> there may be a hole in the bottom, so if there's any
> other way, we encourage 
> you to do that", I fear for the safety of my lawn,
> metaphorically speaking.

That's actually all TAP will be doing.  You *can* stuff that big wad of 
diagnostic data there in a machine-readable format, but there are two 
overriding concerns:

1.  It's entirely optional.
2.  It must not alter the interpretation of core TAP.

In fact, Schwern may very well add a feature to TB2 to disable the structured 
diagnostics on demand (I haven't asked him, but people might want that).

But at the end of the day, this is a feature people have been wanting for a 
long time, even if (for most cases), it's merely a case of "file, line, have, 
want".

Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter  - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

Re: TAP Diagnostics

[chromatic]

On Thursday 21 August 2008 11:04:11 Eric Wilhelm wrote:

> But this raises the question of:  what does it mean for a consumer
> to "understand" the diagnostic?  How far down into the content
> (keys/values) of the diagnostic does the TAP spec want to go?  What is
> the consumer going to *do* with this information?  If the TAP protocol
> is simply a matter of reliably communicating the diagnostic info to the
> consumer (and associating it with a result?), does specifying the
> consumer's usage of the diagnostics over-reach?

Let me answer the final question: yes.

TAP should specify the description of the content and not the contents of the 
content.  Remember that TAP has two nice features, one of which leads to the 
other.

1) Producing and consuming TAP are separate -- they're not in one monolithic 
process.

2) You don't even need a TAP consumer to decide whether any particular test 
passed or failed.

#1 also implies that you can write your own producer and consumer.  Here's 
where people start painting the TAP bicycle shed: the only way to communicate 
between a custom producer and a custom consumer is through the TAP stream, 
thus the desire to duct-tape an empty ten pound potato bag to the side of 
TAP, because TAP doesn't normally allow room for five pounds of potatoes.

We're all lovely people and we're mostly helpful people (I'm not, but humor 
me), and so we imagine that people will struggle with the decoupling of TAP 
Feature Number One, and go off and beg, borrow, or steal a typeful object 
serialization format that's cross-platform, cross-language, and suitable for 
carrying the semantics of information from TAP producer [p0 ... pn) to TAP 
consumer [c0 ... cn), and... well I don't believe it will work.

You're going to have to write a custom TAP consumer to consume the custom TAP 
produced by the custom TAP producer you're going to have to write to produce 
the custom TAP you want to communicate the custom data that's only 
interesting to your TAP chain, because TAP doesn't describe the contents of 
the content, only the content.

That's fine.  The kids in my neighborhood like to set off fireworks, but you 
don't see me out there passing them dynamite.

If TAP goes any further than saying "Yes, you *can* stuff a big wad of custom 
data in this ganky old potato sack, but it's ugly and covered in spiders and 
there may be a hole in the bottom, so if there's any other way, we encourage 
you to do that", I fear for the safety of my lawn, metaphorically speaking.

-- c

Re: JSON TAP Diagnostics?

[David E. Wheeler]

On Aug 21, 2008, at 09:57, Ricardo SIGNES wrote:

Schwern would like it to be YAML (a superset of JSON), with the  
phrasing

"consumers MUST understand JSON and SHOULD understand YAML."


+1

David

Re: Test::Harness Output Change

[Andy Armstrong]

On 21 Aug 2008, at 19:02, David E. Wheeler wrote:


On Aug 21, 2008, at 06:55, Thomas Klausner wrote:


unexpected consequences. It also highlights the issue of
Test::Harness's long-standing practice of stripping the .t extension
from filenames. Why? If we want other extensions, stripping them is
probably bad.


FYI, when I run both .t Perl and .s pgTAP tests, It doesn't strip  
them:



Yeah, there was logic in there so it doesn't strip extensions if you  
have more than one distinct extension.


--
Andy Armstrong, Hexten

Re: TAP Diagnostics

[Eric Wilhelm]

# from Ricardo SIGNES
# on Thursday 21 August 2008 09:57:

>> Does it have to be just one?  Now and forever?
>
>Some people want the spec to say that diagnostics (not free-form
> additional data blocks) are always readable by any TAP consumer,
> which means they need a declared format.  It needs to be at least one
> declared format.

How about if the diagnostic is a free-form data block with a format 
identifier?  Then the JSON/YAML diagnostic is just a special case, 
where support for one or both of those could be required/recommended.

>Schwern would like it to be YAML (a superset of JSON), with the
> phrasing "consumers MUST understand JSON and SHOULD understand YAML."

Does that work?  If a JSON parser is handed linewise YAML, doesn't the 
consumer just trip all over itself?

So, you have to identify the format of the data block anyway, right?

# from Ovid on Thursday 21 August 2008 10:20:
>I originally argued that we should allow more than one, but I was
> wrong.  We don't know the source of our TAP and if everybody and
> their dog is speaking a different diagnostic language, we may as well
> junk diagnostics ...

What I'm suggesting is to require the consumer to support one diagnostic 
format, but define the data block and identifiers in such a way that 
additional formats MAY be supported by the consumer.

  not ok 42
--- diagnostic: JSON/4627
  {"want": 18, "have": 40}
...
  ok 43

  not ok 42
--- diagnostic: YAML/1.2
  want: 18
  have: 40
...
  ok 43

So, non-diagnostic data blocks might be "--- data: $whatever" or 
something?

But this raises the question of:  what does it mean for a consumer 
to "understand" the diagnostic?  How far down into the content 
(keys/values) of the diagnostic does the TAP spec want to go?  What is 
the consumer going to *do* with this information?  If the TAP protocol 
is simply a matter of reliably communicating the diagnostic info to the 
consumer (and associating it with a result?), does specifying the 
consumer's usage of the diagnostics over-reach?

--Eric
-- 
Entia non sunt multiplicanda praeter necessitatem.
--Occam's Razor
---
http://scratchcomputing.com
---

Re: Test::Harness Output Change

[David E . Wheeler]

On Aug 21, 2008, at 06:55, Thomas Klausner wrote:


unexpected consequences. It also highlights the issue of
Test::Harness's long-standing practice of stripping the .t extension
from filenames. Why? If we want other extensions, stripping them is
probably bad.


FYI, when I run both .t Perl and .s pgTAP tests, It doesn't strip them:

% ./Build test
t/01app.tok
t/02pod.tok
t/03podcoverage.tok
t/model_DBI.tok
t/types.sok
All tests successful.
Files=5, Tests=613,  7 wallclock secs ( 0.09 usr  0.03 sys +  1.03  
cusr  0.26 csys =  1.41 CPU)

Result: PASS

Best,

David

Re: Test::Harness Output Change

[David E. Wheeler]

On Aug 21, 2008, at 08:06, Paul Johnson wrote:


On Thu, Aug 21, 2008 at 10:09:32AM -0400, Christopher H. Laco wrote:


I've got one at home now that also has .rb files...

Why .phpt instead of .php?


Why not .t for every language?


Because that's how the harness knows what program to execute: Perl or  
PHP (or psql for my stuff).


Best,

David

Re: JSON TAP Diagnostics?

[Andy Armstrong]

On 21 Aug 2008, at 17:57, Ricardo SIGNES wrote:
Ovid (and I) would like it to be JSON, pending any better idea (that  
we agree

is better).


I'm in the JSON camp too.

--
Andy Armstrong, Hexten

Re: JSON TAP Diagnostics?

[Ovid]

Folks, this really, really needs to go to the IETF list.  I mentioned in here 
because the list wasn't set up yet, but IETF list is the official spot for this 
and we can avoid spamming people here with this.  That being said, on with the 
show! ...

--- On Thu, 21/8/08, Eric Wilhelm <[EMAIL PROTECTED]> wrote:

> >Let's not forget that the debated requirement for
> diagnostics is that
> > the generators and consumers speak the same language
> 
> Does it have to be just one?  Now and forever?

I originally argued that we should allow more than one, but I was wrong.  We 
don't know the source of our TAP and if everybody and their dog is speaking a 
different diagnostic language, we may as well junk diagnostics (that being 
said, even speaking the same language doesn't mean that everything can 
translate cleanly.  You generally can't specify file/line in a stored procedure 
emitting TAP)

> Is the specification for either of these going to be part
> of the TAP spec?

Yes.

Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter  - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

Re: JSON TAP Diagnostics?

[Ricardo SIGNES]

* Eric Wilhelm <[EMAIL PROTECTED]> [2008-08-21T12:46:59]
> # from Ovid
> >1.  YAML is prettier.
> >2.  JSON, unlike YAML, is stable.
> >
> >Let's not forget that the debated requirement for diagnostics is that
> > the generators and consumers speak the same language
> 
> Does it have to be just one?  Now and forever?

Some people want the spec to say that diagnostics (not free-form additional
data blocks) are always readable by any TAP consumer, which means they need a
declared format.  It needs to be at least one declared format.

Ovid (and I) would like it to be JSON, pending any better idea (that we agree
is better).

Schwern would like it to be YAML (a superset of JSON), with the phrasing
"consumers MUST understand JSON and SHOULD understand YAML."

> Is the specification for either of these going to be part of the TAP 
> spec?

It will probably have a reference to the relied-upon spec.  JSON has an RFC.
YAML has published, versioned specifications.

-- 
rjbs

Re: JSON TAP Diagnostics?

[Eric Wilhelm]

# from Ovid
# on Thursday 21 August 2008 09:28:

>> (2) YAML is better suited for complex serialization than JSON.

>1.  YAML is prettier.
>2.  JSON, unlike YAML, is stable.
>
>Let's not forget that the debated requirement for diagnostics is that
> the generators and consumers speak the same language

Does it have to be just one?  Now and forever?

Is the specification for either of these going to be part of the TAP 
spec?

Can the spec simply specify the identifier and start/end sentinels of 
the diagnostic and leave the details of the content up to the two ends?  
The consumer doesn't even need to grok the diagnostic if it's e.g. 
jpeg -- it just needs to reliably preserve it and save it in a file or 
hand it to a viewer.

I guess you could put your xml, jpeg, or svg in a JSON value, but I 
think that would make it harder to specify any communication between 
the emitter and consumer -- starting with "under which key?", a debate 
which has already been shown to be endless.

--Eric
-- 
Speak softly and carry a big carrot.
---
http://scratchcomputing.com
---

Re: JSON TAP Diagnostics?

[Ricardo SIGNES]

* Ovid <[EMAIL PROTECTED]> [2008-08-21T12:28:52]
> Let's not forget that the debated requirement for diagnostics is that the
> generators and consumers speak the same language, not that the *consumer*
> emit anything in particular.  If you prefer the appearance of YAML to JSON,
> have the consumer transform the JSON to YAML and print that.  Problem solved.

Right.  It's very nice that TAP is easy to read, but it's a protocol, and the
harness provides a presentation layer.

Schwern argued (on #perl-qa) that JSON will not be sufficient when more
strongly-typed languages want to use TAP, and cannot serialize to JSON because
it lacks tags and references.  That would make TAP not even machine-*writeable*
for those languages.

When emitting have/want for something that can't be represented as native JSON,
YAML will work... as long as there is a serializer for that language to
serialize that kind of datum to YAML.

I don't think that's a really big win, given the adoption rate of YAML, and
given that in the end it's easy to implement tags and references as semantics
atop JSON.

-- 
rjbs

Re: JSON TAP Diagnostics?

[Ovid]

--- On Thu, 21/8/08, Ricardo SIGNES <[EMAIL PROTECTED]> wrote:

> * chromatic <[EMAIL PROTECTED]> [2008-08-20T13:59:14]
> > Aren't these two separate concerns, human versus
> machine readability?  The
> > latter rarely respects ambiguity.
> 
> Yes.
> 
> Right now, there seem to be two pro-YAML arguments.
> 
> (1)  It's easier to for humans read.
...
> (2) YAML is better suited for complex serialization than
> JSON.

As far as I can see, the (non-unanimous) consensus appears to be:

1.  YAML is prettier.
2.  JSON, unlike YAML, is stable.

(Sounds more like pros and cons for choosing a romantic partner than a 
serialization language)

Since the primary issue with diagnostics today is that it's not 
machine-readable and that's the primary reason we need it, I should think this 
is a slam-dunk issue.  If that's not enough to convince folks, though, I have a 
final argument that I (hope) should put this to rest.

Let's not forget that the debated requirement for diagnostics is that the 
generators and consumers speak the same language, not that the *consumer* emit 
anything in particular.  If you prefer the appearance of YAML to JSON, have the 
consumer transform the JSON to YAML and print that.  Problem solved.

Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter  - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

Re: Test::Harness Output Change

[Ricardo SIGNES]

* Ovid <[EMAIL PROTECTED]> [2008-08-21T05:36:11]
> Both of us found this much cleaner. However, this might have unexpected
> consequences. It also highlights the issue of Test::Harness's long-standing
> practice of stripping the .t extension from filenames. Why? If we want other
> extensions, stripping them is probably bad.

I'd definitely like to always see the .t, as I'm now working on more things
that have multiple extensioned tests.  I'd rather not need to know whether or
not the item is a filename or munged filename based on some logic.

It wasn't clear to me whether this is now the case, from AndyA's post, so I'm
just chiming in.

-- 
rjbs

Re: JSON TAP Diagnostics?

[Ricardo SIGNES]

* chromatic <[EMAIL PROTECTED]> [2008-08-20T13:59:14]
> Aren't these two separate concerns, human versus machine readability?  The
> latter rarely respects ambiguity.

Yes.

Right now, there seem to be two pro-YAML arguments.

(1)  It's easier to for humans read.

Sure.  I will admit that.  It is easier for humans to read in probably every
likely case but one.  It's harder for machines to read, though, so that cost
has to be weighed.

Also, given how few languages have reliable, conformant YAML emitters, I
imagine that it will often be easier to use a JSON emitter.  Even if your
language has a YAML emitter, the JSON library will probably be in heavier use,
you're more likely to be familiar with it, etc.  I admit that this is not a
strong argument, but I think it's pretty likely to be the case for a good long
time yet.  (Even Perl doesn't have much by way of YAML libraries.)  The one
case in which YAML is definitely not easier to read than JSON is when it is
formatted in the subset of YAML that is JSON.

So, how often, in non-Perl TAP emitters, will you see pretty YAML instead of
JSONic YAML?  I predict: plenty.

So, yes, languages that can emit pretty YAML can do so, to make the protocol
stream easier for humans to read.  That will, of course, limit the number of
consumers than can parse it.

So: how often is YAML going to realistically give this benefit outside of
Test::Builder and TAP::Harness?

(2) YAML is better suited for complex serialization than JSON.

This is pretty clearly just true.  The ability to tag structures as
representing data more complex than YAML's native kinds allow, or just has
having greater significance than some raw data, is really great.

Your language won't need (say) pickle if it has YAML and a schema for mapping
complex native types to YAML.  Many languages have their own serialization
system, though; how many will implement another one that stores in YAML,
especially in languages (like Perl) where YAML support for making use of tags
is paltry, at best.

It would be almost as easy, if not as easy or easier, to instead make your
language serialize to JSON.  After all, these two things could easily mean the
same thing:

  ---
  account: !yourcorp.biz!account 1234

..or...

  {
"datatype": "yourcorp.biz/account",
"payload" : 1234
  }

So, definitely it is cool that there is a build in way of saying "datatype" and
"payload" in YAML, and I'm glad it's so compact.  How often will people be able
to reap this benefit if YAML is chosen over JSON?

I am really, really worried by statements like "we'll just keep pushing on YAML
implementors to get it implemented more places and better so that we can rely
on it for TAP."

-- 
rjbs

Re: Test::Harness Output Change

[Paul Johnson]

On Thu, Aug 21, 2008 at 10:09:32AM -0400, Christopher H. Laco wrote:

> I've got one at home now that also has .rb files...
> 
> Why .phpt instead of .php?

Why not .t for every language?

I have a suspicion that I will like the dot change but not the .t
change.

-- 
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net

Re: Test::Harness Output Change

[Christopher H. Laco]

Andy Lester wrote:
> 
> On Aug 21, 2008, at 4:36 AM, Ovid wrote:
> 
>>  Why? If we want other extensions, stripping them is probably bad.
> 
> 
> We definitely want other extensions.  I have a pending project that
> relies on running .t and .phpt next to each other.
> 
> xoa

I've got one at home now that also has .rb files...

Why .phpt instead of .php?

Re: Test::Harness Output Change

[Andy Lester]

On Aug 21, 2008, at 4:36 AM, Ovid wrote:


 Why? If we want other extensions, stripping them is probably bad.



We definitely want other extensions.  I have a pending project that  
relies on running .t and .phpt next to each other.


xoa

--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: Test::Harness Output Change

[Andy Armstrong]

On 21 Aug 2008, at 14:55, Thomas Klausner wrote:

unexpected consequences. It also highlights the issue of
Test::Harness's long-standing practice of stripping the .t extension
from filenames. Why? If we want other extensions, stripping them is
probably bad.


Actually, I'd love to have the extension in the report, so I can just
past the listed failing test and copy either into 'prove -vl ' or  
'vim '

(or both..)



I just committed r1164 of Test::Harness that does just that :)

--
Andy Armstrong, Hexten

Re: Test::Harness Output Change

[Thomas Klausner]

Hi!

On Thu, Aug 21, 2008 at 02:36:11AM -0700, Ovid wrote:

> unexpected consequences. It also highlights the issue of 
> Test::Harness's long-standing practice of stripping the .t extension 
> from filenames. Why? If we want other extensions, stripping them is 
> probably bad.

Actually, I'd love to have the extension in the report, so I can just 
past the listed failing test and copy either into 'prove -vl ' or 'vim ' 
(or both..)

-- 
#!/usr/bin/perl  http://domm.plix.at
for(ref bless{},just'another'perl'hacker){s-:+-$"-g&&print$_.$/}

Re: Test::Harness Output Change

[Andy Armstrong]

On 21 Aug 2008, at 10:36, Ovid wrote:

You'll see this:

   t/proverun  ok
   t/regression .. ok
   t/results . ok
   t/scheduler ... ok
   t/source .. ok
   t/spool ... ok
   t/state ... ok
   t/streams . ok

Both of us found this much cleaner. However, this might have  
unexpected consequences. It also highlights the issue of  
Test::Harness's long-standing practice of stripping the .t extension  
from filenames. Why? If we want other extensions, stripping them is  
probably bad.



There's logic in there now to display the extensions if it finds files  
with different extensions. I did that to preserve legacy behaviour but  
still disambiguate in the case where you have foo.t and foo.phpt.


But I don't like it - I'd rather just see the extensions in all cases.

Aesthetically I'm very happy about the leader dots, thanks :)

--
Andy Armstrong, Hexten

Test::Harness Output Change

[Ovid]

I've just committed a minor change that both Andy and I have wanted for a long 
time, but I do wonder if it will impact others. Instead of seeing this:

t/proverun..ok
t/regressionok
t/results...ok
t/scheduler.ok
t/sourceok
t/spool.ok
t/state.ok
t/streams...ok

You'll see this:

t/proverun  ok
t/regression .. ok
t/results . ok
t/scheduler ... ok
t/source .. ok
t/spool ... ok
t/state ... ok
t/streams . ok 

Both of us found this much cleaner. However, this might have unexpected 
consequences. It also highlights the issue of Test::Harness's long-standing 
practice of stripping the .t extension from filenames. Why? If we want other 
extensions, stripping them is probably bad.

Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter  - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

Re: automated dist publication tools

[Eric Wilhelm]

# from Elliot Shank
# on Wednesday 20 August 2008 22:14:

>>> This all built up over years as I tried to automate away each
>>> stupid distribution packaging mistake I've made in releasing
>>> something to CPAN.
>>
>> This should be a module. I'd use it.
>
>What about foy's Module::Release?

"This is the prototype program for using Module::Release. You should 
modify it to fit your needs."

So, you have to read the code, understand it, then modify it, and figure 
out how to test it without uploading 3 spurious dists.  By the time you 
do that (which is, of course, during the "I should have just uploaded 
this by now" time), you start to think that a simple little script or a 
bit of bash even would be easier.

The thought I had when starting CPDK was that most of the authors with 
several dists already have their own way of doing things, so it has to 
be configurable.  But, if the only way to configure it is with code, 
you now have the problem that you have to test your code (and because 
it uses the internet monad, to talk to the pause monad, testing is a 
bit of a pain.)

--Eric
-- 
Consumers want choice, consumers want openness.
--Rob Glaser
---
http://scratchcomputing.com
---

Re: JSON TAP Diagnostics?

[chromatic]

On Mon, Aug 18, 2008 at 03:36:15PM +0200, Aristotle Pagaltzis wrote:

> * Michael Peters <[EMAIL PROTECTED]> [2008-08-18 15:30]:
> > YAML does support things that JSON does not (types, embedded
> > documents, etc) but I've been in doubt that we'd ever need
> > those things for TAP anyway.
 
> That would be useful if any of the YAML producers were capable of
> serialising tricky data structures correctly. If you care about
> those kinds of things, you probably want to use a language-specific
> serialiser and put its output in the TAP diagnostic as a string.

I fear that this is how human-readability ends; not with a bang, but with a
whimper.

Aren't these two separate concerns, human versus machine readability?  The
latter rarely respects ambiguity.

-- c

Re: Should MANIFEST go in the repository?

[Shawn Boyette ☠]

On Wed, Aug 20, 2008 at 12:57 PM, David Golden <[EMAIL PROTECTED]> wrote:
>> Anyway. I am in the "Keep MANIFEST in repo and manually update" camp.
>
> Me, too, usually.

Me three.

-- 
Shawn Boyette
<[EMAIL PROTECTED]>