Re: anyone want to adopt Test::Tester?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 6/26/14, 3:18 PM, Ricardo Signes wrote: > * David Cantrell [2014-06-26T09:19:00] >> I understand that Test::Builder::Tester is the way to go these >> days - and it's distributed with Test::Builder, so >> incompatibilities should never* happen. > > Test::Tester is really nice and easy to use. > > Test::Builder::Tester is awful and often requires that you predict > the exact TAP that will be printed. > > I always recommend Test::Tester. I concur. Test::Tester's style of looking at the results as data, rather than as formatted stuff, is the way to go for most test testing. -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - https://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCgAGBQJTrbndAAoJEMUdZ0L/y3sDGTgH+gNRIG9x0nkLPQvsDVdcQbZ7 06UnghZb17K2O8Xt8U4KQt+lPFUb5xIk5vy3vGyN8bTrgLy2emWbyH7U4MLCu+Kp P3iK+CaSrPqf/8gZKNcLDhbus9V/58KNVa2QNjg1b2RFfJ7Of+yCGuX607MfRN/u CfGHtQmLzx/dkr/5pfDr0BylqhFGwwAnH/Hpgwu89jghmMLgc4tfN1lsC120dhsV 4g1g21/UNCC99E9/7SzQrmJs9difOWk3Svz2jsLXvlAvRCEIpJ3e/m7svx8pLRDB FI5ccYUNDqXDdB7DPwLDz40LmmGrSQ6i6JAYG+EvVOeD0WXQ/NwTf6dHWf2nnT8= =w0z9 -END PGP SIGNATURE-
Re: TPF Devel::Cover grant report March 2013
Huzzah! On 5/1/13 3:38 PM, Paul Johnson wrote: > In accordance with the terms of my grant from TPF this is the monthly > report for my work on improving Devel::Cover covering March 2013. > > Sorry for the delay in this report. Most of March and all of April has been a > very busy time for me outside of grant work on Devel::Cover. > > This month I released Devel::Cover 1.01. > > March's work started by continuing there February's had left off. David > Golden had politely observed that the coverage reporting for ||= operators in > his Path::Iterator::Rule module was somewhat less than optimal. So I got hold > of the module's source, pared the module down to the minimum required to > reproduce the problem, and fixed it, along with another problem that showed up > too. The problem boiled down to C< $x ||= $y > being in void context, but we > want to show the coverage as if were not. This was one of those problems > where the majority of the effort was spent in locating and defining the > problem, and the solution was relatively simple from that point. In any case, > the coverage looks much healthier now: > > http://cpancover.com/latest/Path-Iterator-Rule-1.005/blib-lib-Path-Iterator-Rule-pm--condition.html > > This month, perls 5.14.4, 5.16.3, 5.17.10 were released. I tested against > these, along with 5.17.9 from February. It's obviously important that > Devel::Cover works with newly released stable versions of perl, so I always > try to test with the release candidates too. I also try to keep up with the > development releases (5.17.x at the moment), which are far more of a moving > target as far as Devel::Cover is concerned. Of course, this should make it > more likely that there won't be any problems when 5.18.0 is released, and it > also means that the cpantesters failure reports I get from those people > running development releases use generally useful. > > Devel::Cover now also reports coverage for more of the files that gcov can > exercise. > > I investigated removing the dependency on B::Deparse. Unfortunately, that > won't be simple, and may not be worth the effort, but I did at least reduce > the number of calls to B::Deparse. > > Closed Github tickets: > > 52 cover ignores .cc file > > Merged pull requests: > > 47 Fix for mod_perl on Debian setting $^X to apache2 > 49 fix: respect to $Devel::Cover::Silent > > You can see the commits at https://github.com/pjcj/Devel--Cover/commits/master > > Hours worked: > > 01.03 7:30 > 29.03 2:15 > > Total 9:45 > > Total hours worked on grant: 257:35 >
Re: Smoke Test::More 0.98_05 please.
On 4/27/13 3:41 PM, Karen Etheridge wrote: > On Sat, Apr 27, 2013 at 03:22:31PM -0700, Michael G. Schwern wrote: >> No, it's not Test::Exception's fault. > > No, I understand -- I just prefer Test::Fatal, so I convert dists when > seeing them and I also have a tuit to spare. Oh, ok then.
Re: Smoke Test::More 0.98_05 please.
On 4/27/13 2:33 PM, Karen Etheridge wrote: > On Sat, Apr 27, 2013 at 10:19:44AM -0400, Ricardo Signes wrote: >> Test::Exception fails, which breaks a lot of my basic toolchain from >> installing. > > Which dists? I'll convert them to Test::Fatal. No, it's not Test::Exception's fault. They have a test which looks for the particular output of either isa_ok() or new_ok which changed in 0.98_05. Could have happened to any module. I have a test for dependents which includes Test::Exception, but I forgot I never backported it to the 0.98 branch so it didn't get run. Test::Exception and Test::Class are failing due to the format change.
Smoke Test::More 0.98_05 please.
0.98_05 is a release candidate for 0.99. I would say this is the last stable release of Test::More before 1.5.0 but hahahaha I won't say that. https://metacpan.org/release/MSCHWERN/Test-Simple-0.98_05/ There hasn't been a stable release in two years, so there's likely to be code which has wrapped itself around this version. If smokers and Test::* module authors would give it a go and report and problems I'd appreciate that. Here's the changes since 0.98 which might conceivably have an impact. * cmp_ok() will error when it gets a non-comparison operator * isa_ok() and new_ok() default names have changed * like() and unlike() no longer warn about undef * subtests will output their name at the start of the subtest
Re: Perl QA Hackathon 2013 / 2014 transition
On 4/18/13 6:42 AM, Karen Etheridge wrote: > On Wed, Apr 17, 2013 at 11:58:55AM +0200, Philippe Bruhat (BooK) wrote: >> I would like to collect your impressions about what worked and what >> didn't, what was missing, what you want for next year > > For those of us participating remotely, some small amount of video > conferencing, perhaps to allow eavesdropping on/participating with some > discussions, would be helpful. Perhaps just throwing a webcam on in the > corner? But being able to conduct multi-way conversations would be even > nicer. +1 to that. A sacrificial laptop or three doing video conferencing which can be carried around the room. Or encourage one on-site person from each project to be in a Google Hangout (or similar technology).
Re: Conflicts in the meta spec
On 4/15/13 7:06 PM, Jens Rehsack wrote: > On 15.04.13 18:56, Michael G. Schwern wrote: >> TL;DR version... >> IMO we only need to clarify what "conflicts" means and what actions CPAN >> tools should take. >> >> We talked in the Consensus Dome about the need for and meaning of the >> "conflicts" relationship in the CPAN::Meta::Spec. >> https://metacpan.org/module/CPAN::Meta::Spec#Prereq-Spec >> >> IIRC the conclusion was... >> * We need it. >> * It's not clear what it means. (I don't recall what the confusion was) > > The difference between "A conflicts with B op VER" and "A breaks B op > VER" and "A superflous B". IIRC those meant... "A conflict B" is A and B cannot be on the machine at the same time. For example, they're both http servers and need the same port. I can't think of an example where CPAN has needed that for modules. "A breaks B" is if A is installed B will break. This is what we need on CPAN for things like Test::Builder breaks Test::Class < 0.39. "A superfluous B" is if A is installed B is no longer necessary. Usually called "obsoletes". This is mostly useful if you rename a package, though I'd rather that was made more explicit in the meta data. Most of the use cases I can think of are better handled by the installed modules database and CPAN distribution meta data. >> * Let's see what Debian does. > > Or MacPorts, HomeBrew or pkgsrc :P Whatever packaging system I may actually use, I go to Debian first because they extensively document this sort of thing, both the meanings and the rationales. Let's have a look at what the others do... MacPorts has a conflicts tag, but I can't find the docs. The only mention of conflicts I can find in the MacPorts manual is "variant X conflicts Y" but there's no explanation what that means. https://www.macports.org/guide/chunked/reference.variants.html It's not mentioned in the dependencies section. https://www.macports.org/guide/chunked/reference.dependencies.html pkgsrc has CONFLICTS=Some-Name-[0-9]* essentially a regex. http://www.netbsd.org/docs/pkgsrc/fixes.html#conflicts All they say is the "package may conflict with other packages a user might already have installed on his system" Homebrew has an undocumented "conflicts_with". https://github.com/mxcl/homebrew/blob/master/Library/Homebrew/formula.rb rpm has "Conflicts" and "BuildConflicts" which is only defined as "where one package conflicts with a capability provided by another" which isn't very useful. rpm also has "Obsoletes" "where one package obsoletes capabilities provided by another, usually used when a package changes name and the new package obsoletes the old name". https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-advanced-packaging.html It doesn't shine much new light on the issue except that package management systems agree, you need a conflict tag.
Re: Perl QA Hackathon 2013 / 2014 transition
I use a hackathon to...
* Sync up with what else is going on
* Make long standing hard/bikeshed/warnock'd decisions
* Remind myself that there are people behind the emails
* Be a resource for Test::*, TAP, MakeMaker, build...
* Hack away at code, particularly long standing issues which
haven't been getting attention.
QA Hackathons are often my most days of the year. What I'm looking for
is the ability to focus on the hackathon without worrying about the
necessaries like food, water, internet, power, tables, chairs,
whiteboards, caffeine, bedding, transport... you get the idea.
That said, none of it has to be fancy. The TravelLodge was just fine.
Clean bed. Good shower. Bus to the venue. I don't expect great food
either, just something to keep me going for the weekend.
To that end, the Lancaster team did a lovely job. From being shuttled
from the train to the hotel (I was a bit embarrassed once I saw how
short a walk it was), to the wonderful spread of food and drinks, to the
packet of maps and information (I liked the potted history) to the
little details like phone numbers on the name badges. Claire, Ian, Mark
and any other Shadowcats were wonderful hosts.
On 4/17/13 11:49 AM, Salve J Nilsen wrote:
> Here are the good bits (please keep):
All that, just s{coffee}{tea}. :)
> Here are the bits that would be nice to improve at the next hackathon:
I generally agree with everything Salve said.
I kind of enjoyed having everyone in the same room. People and groups
weren't isolated. I could lean over or walk over to talk to who I
needed. I could hear my name being taken in vain. Sub-projects seemed
to have their own spaces. I didn't feel cramped and I didn't feel
overwhelmed by the noise. However, I'd understand if others didn't like
it. Perhaps one room, but a larger one.
Definitely would like longer access to the site, or a clear alternative
hacking site. A quiet pub next to the hotel is ok.
One thing I would recommend changing is less lavish dinners. They were
absolutely lovely, but I felt they detracted from hacking time. I don't
think we made it out of the Thai place until after 11pm. Perhaps one
lavish dinner at the start, to get everybody talking, and for the rest
we get take away served at the site. Maybe one on the last night. More
time to hack.
For the international travelers, perhaps we could register who will be
needing a SIM card, and what kind, so we only have to do one run to the
shops. And thank you Ian for sorting that for me.
Finally, as much concrete information about the hackathon as early as
possible. At least the date and city for travel planning purposes. The
details of this one were announced quite late for international travel
or having to take time off work.
Thanks so much to hackathon organizers past, present and future.
Re: Conflicts in the meta spec
On 4/15/13 7:02 PM, Mike Doherty wrote: > This requires Moose maintainers to know about all the things their > releases break. Have you considered reversing the directionality so > MooseX::Blah knows about the things which break it? That seems like a > more likely scenario to me, and lets the right maintainer control the > "conflicts" relationship. It's a good idea to reverse the relationship. In this case it doesn't work out. * I believe that was the confusion at the Lancaster Consensus, that conflict was a way to express "I can use Parent >= 1.23 but not 1.69". That is already covered by the "requires" relationship and version ranges. requires: Parent: >= 1.23, != 1.69 * In order to find out what modules Parent conflicts with, necessary when installing Parent, a CPAN shell would have to iterate through every module's metadata. [1] * Parent can test its dependents before Parent releases, but Child cannot. Parent has the opportunity to test and mark its conflicts at release time so at least CPAN shells can protect users of Child from upgrading Parent and breaking Child. [2] [1] It would be nice if we had a reliable meta API for that, but we don't at the moment. [2] This should contain the assumption "in an automated fashion". Sure, Parent and Child can keep in close communication with each other, but this does not scale. When you're a heavily depended on module such as File::Spec, Test::More, Moose, DBIx::Class, Module::Build, ExtUtils::MakeMaker or version you simply have too many dependents and are too many levels removed.
Conflicts in the meta spec
TL;DR version... IMO we only need to clarify what "conflicts" means and what actions CPAN tools should take. We talked in the Consensus Dome about the need for and meaning of the "conflicts" relationship in the CPAN::Meta::Spec. https://metacpan.org/module/CPAN::Meta::Spec#Prereq-Spec IIRC the conclusion was... * We need it. * It's not clear what it means. (I don't recall what the confusion was) * Let's see what Debian does. I can come up with a few situations where being able to declare conflicts with other packages would have (and will be) very useful. Mostly in the "if you install Foo version 1.23 then your previously installed Bar version 2.34 will break". * Moose vs MooseX * Test::Harness 2 and 3 broke a few modules * Test::Builder 1.5 vs Test::Class, Test::Most, Test::Random... * Module::Install vs its plugins Why we need this, why you can't do it with "requires" is because this is for modules which you do not depend on. Often they depend on you. MooseX::Blah depends on Moose, so Moose declares conflict with versions of MooseX::Blah it breaks. IMO CPAN shells can take several actions on a conflict... * Warn the user * Upgrade the conflicting modules (if possible) * Refuse to install the new version if non-conflicting upgrades are not available Debian has several fields to deal with this, "Breaks", "Conflicts" and "Replaces". http://www.debian.org/doc/debian-policy/ch-relationships.html#s-breaks http://www.debian.org/doc/debian-policy/ch-relationships.html#s-conflicts http://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces We may not need them all. Here's a summary. Breaks -- When one binary package declares that it breaks another, dpkg will refuse to allow the package which declares Breaks to be unpacked unless the broken package is deconfigured first, and it will refuse to allow the broken package to be reconfigured. Conflicts - When one binary package declares a conflict with another using a Conflicts field, dpkg will refuse to allow them to be unpacked on the system at the same time. This is a stronger restriction than Breaks, which prevents the broken package from being configured while the breaking package is in the "Unpacked" state but allows both packages to be unpacked at the same time. If one package is to be unpacked, the other must be removed first. Replaces Packages can declare in their control file that they should overwrite files in certain other packages, or completely replace other packages. It is usually an error for a package to contain files which are on the system in another package. However, if the overwriting package declares that it Replaces the one containing the file being overwritten, then dpkg will replace the file from the old package with that from the new. Second, Replaces allows the packaging system to resolve which package should be removed when there is a conflict. This usage only takes effect when the two packages do conflict, so that the two usages of this field do not interfere with each other. === "Breaks" vs "Conflicts" is only about at what phase they conflict. We have that covered with prereq phases. # Declare that Test-Simple 1.5.0 will break Test::Class <= 0.38 name: Test-Simple version: 1.5.0 prereq: runtime: conflicts: Test::Class: <= 0.38 It seems pretty clear from the Debian manual that when two things conflict it means they cannot be installed at the same time. That to me seems the correct reading and covers our use cases. I don't think "Replaces" yet simply because we don't have a clear concept of a distribution. For example, you'd use Replaces to say "LWP replaces libwwwperl" but we don't understand that yet. It might be more useful when we have the installed distributions database. We also have no concept of virtual packages, which means we can't say "Apache replaces http-server". Maybe we can do it at the package name level, but I don't see there is a concrete need.
Test::Builder, MakeMaker & Consensus QA Hackathon Achievements
I've just updated the wiki with the achievements around the projects I
was associated with. I'll paste them below.
http://2013.qa-hackathon.org/qa2013/wiki?node=Achievements
In addition I fleshed out the Lancaster Consensus conclusions according
to my photographs of the white board and my own recollections because it
was looking a bit sparse. Apologies if I got anything wrong, please
correct and fill in. And Wendy, please consider your own notes canonical.
Test::Builder 1.5
-
A small team worked through the roughly 100 of 700 Test-* distributions
failing with Test::Builder 1.5 alpha but passing with the current 0.98
stable.
Verify it's failing with 1.5 and passing with 0.98
Verify it's because of a Test::Builder 1.5 change
Categorize the failure type
Patch either Test::Builder or the module as appropriate
If the module runtime had to be patched, note it as a conflict.
We did the work primarily on an EtherPad and in the Test-More Github
issue tracker by a small team of remote and local developers.
Karen Ethridge (ether)
James Mastros (theorbtwo)
Colin Newell
Daniel Perrett (pdl)
Michael G Schwern (schwern)
Nearly all the modules got looked at, and a number patched.
Other accomplishments by the team...
Determined that rolling back the TAP formatting changes in 1.5 will
not significantly help
Fixed Test::More stable and 1.5 alpha for 5.17.
Released 0.98_04 and 1.5.0_6
Improved the script for testing CPAN modules with stable vs alpha.
Fixed the 1.5 alpha under taint mode
Fixed the 1.5 alpha on Windows
Misc documentation updates
Got a new release of Test::Class working with 1.5
ExtUtils::MakeMaker stable release
--
Chris Williams (BingOS) stepped up to manage the next stable release of
ExtUtils::MakeMaker to be in time for Perl 5.18.0 and released several
alphas. They will be very well tested. :)
Roll back the Test::Builder 1.5 formatting changes?
I'd like people's opinions on a possible downgrade of Test::Builder 1.5 in an attempt to make getting a stable release out easier. It has to do with whether we keep minor changes to the TAP formatting as the new default or replicate Test::Builder's current quirks. https://github.com/schwern/test-more/issues/351 This effects the upcoming Perl QA Hackathon and I have to make a decision before Friday. Replies here are ok, but I'd prefer replies on the Github issue please to keep the discussion coherent.
Re: Withdrawing from the QA-Hackathon
On 3/21/13 3:39 PM, Adrian Howard wrote: > On 21 March 2013 22:35, Michael G. Schwern wrote: >> > Sorry you can't make it Curtis. It won't be the same without you. :-/ >> > >> > Come to think of it, it might be a whole lot quieter if we're not >> > fighting like a married couple. ;) > > We need the antithesis of rubber-duck debugging. Some kind of stuffed > toy that just continually disagrees with you ;-) Nóirín has a reprogrammable Ken doll, and if Curtis wanted to volunteer a few audio recordings we could make this happen.
Re: Withdrawing from the QA-Hackathon
On 3/21/13 11:23 AM, Mark Keating wrote: > On 19/03/2013 08:20, Ovid wrote: >> This really frustrates me because I've attended every QA-hackathon to >> date and we applied for my visa renewal in November :( > > That is massively frustrating and we have already booked a room in your > name, i think we will get Schwern into that room and just pretend he is > called Ovid - I doubt the hotel will check. Sorry you can't make it Curtis. It won't be the same without you. :-/ Come to think of it, it might be a whole lot quieter if we're not fighting like a married couple. ;)
Re: Test::Builder::Tester considered harmful. Use Test::Tester.
On 2012.8.15 4:24 AM, Karen Etheridge wrote: > On Sat, Aug 04, 2012 at 04:57:42PM -0700, Michael G Schwern wrote: >> Executive Summary: If you're using Test::Builder::Tester to test your Test >> module, switch to Test::Tester. It will make the transition to Test::Builder >> 1.5 smoother and avoid future breakage due to TAP formatting changes. > > That would be quite a list of modules: > https://metacpan.org/requires/module/Test::Builder::Tester Sorry, I wasn't clear. In the short term, TBT will continue to work. In the short term, TBT uses a special TAP formatter which emulates the quirks of the existing behavior. So whatever tests you have with whatever assumptions about the TAP format will still work. In the long term, TBT will become further and further divorced from reality. It will be frozen in its format and behaviors and not reflecting actual outputs. Finally, rather than trying to interpret what happened based on the TAP, its easier to look at the test events directly with Test::Tester or TB2::Tester. -- On error resume stupid
Re: Test::Builder::Tester considered harmful. Use Test::Tester.
On 2012.8.15 2:43 AM, Buddy Burden wrote: > Well, I say that it's a bit of a PITA, but I'll add it to my TODO > list. I recall now that the skip/SKIP thing is what was causing some > CPAN Testers failures for Test::File, so I guess I'l just switch it > over to Test::Tester and that'll solve it, eh? Hopefully it won't be > too awful, but my recollection is that TBT and TT have _completely_ > different approaches when it comes to testing, so it won't be trivial > either. So I guess I'm torn between saying "yay for progress!" and > "bummer for my already overstuffed schedule." ;-> Test::Builder::Tester should smooth that over as of alpha 3. It's using a special TAP formatter which emulates TB1 quirks including the skip/SKIP thing. TBT's future is being sacrificed to work as a Test::Builder 0.x -> 1.5 compatibility layer. The behaviors you're testing with TBT will always be stuck in the past. Test::Tester also spans 0.x -> 1.5 but it does so with more grace. You should be able to leave your Test::Builder::Tester stuff alone, it'll work, but eventually you're going to want to change them. -- Robrt: People can't win Schwern: No, but they can riot after the game.
Re: fork and tests: making it easier
On second thought, this has to be turned on explicitly for implementation reasons, so there's no backwards compat issue. "use Test::SharedFork" would simply turn on that feature in Test::Builder 1.5 providing a way to smoothly transition between 0.9x and 1.5. See https://github.com/schwern/test-more/issues/334 for all the implementation details. -- I'm pale as Formica, social skills stunted small. But I'm accurate like a pica, I know the capital of Nepal. I'm the nemesis of error, dreadful diction fears my skills, more inquisitive than Jim Lehrer, snottier than Beverly Hills. -- I.L.O.P. Secret Rap http://goats.com/archive/020830.html
fork and tests: making it easier
Executive Summary: I propose Test::Builder 1.5 makes writing tests using fork as easy as writing tests using threads is. Test::Builder will handle the coordination for you. Downside: this breaks existing behavior. Rebutal: if you're testing with fork your tests are probably broken with Test::Builder 1.5 anyway. Currently, Test::Builder has little knowledge of forking. This makes forking in a test problematic if you expect to run tests in both forks. As people who have done it know, it involves a lot of fiddling with Test::Builder internals and yuck. Test::SharedFork solves this, but its deeply in bed with Test::Builder and fixing it for Test::Builder 1.5 is difficult. Test::TCP depends on Test::SharedFork. Plack depends on Test::TCP. So it's important to make it work. Turns out its easier to just do it inside Test::Builder 1.5. The state of the test is encapsulated in a single object and its a not-so-SMOP to freeze and restore that between processes. This is what geistteufel and I were working on at the QA Hackathon. So I propose having Test::Builder 1.5 coordinate the state of the test like it does with threads. It'll take care of the test counter and all that for you. There will be a switch to turn coordination off. It won't hijack fork or anything. Upsides: * Makes writing tests with forks and doing things in parallel much easier. * Eliminates a lot of Test::Builder hacking which often relies on undocumented behaviors and is difficult to support. Downsides: * It breaks existing behavior. * ...but tests which fork are probably broken with 1.5 anyway. * It involves file locking, which introduces filesystem complications * ...but it will only trigger when a fork runs a test. So most tests will be unaffected. * ...maybe it can detect if file locking is hosed and error early? * ...file locking will be in the temp directory which is usually a sane filesystem. * You might want to fork and run separate tests. * ...but this is less common than forking and wanting coordination * ...there will be a switch to turn coordination off. What do you think? -- Who invented the eponym?
Test::Builder::Tester considered harmful. Use Test::Tester.
Executive Summary: If you're using Test::Builder::Tester to test your Test module, switch to Test::Tester. It will make the transition to Test::Builder 1.5 smoother and avoid future breakage due to TAP formatting changes. A lot of the work done fixing CPAN Test::* modules to be compatible with Test::Builder 1.5 has been to do with small changes to the TAP format. The TAP version header and things like the casing of "# SKIP". Stupid little things. Testing a test library currently involves comparing the output TAP. This sucks, even with Test::Builder::Tester. The whole idea of testing test modules by looking at the formatted output sucks. Test::Builder::Tester is a gallant effort, Test::Builder 1.5 is event driven, so there's no need to look at the formatted output, you can look at the events directly and get a complete view of what's going on. For example, this test... ok( 1 ); is( 23, 42 ); done_testing; Generates something like this: # TB2::Event::TestStart # TB2::Result ok( 1 ); # TB2::Result # TB2::Event::Log (for the failure diagnostics) is( 23, 42 ); # TB2::SetPlan # TB2::Event::TestEnd done_testing; There's a couple of interfaces to test via events. Test::Tester is an existing one that works with both 0.x and 1.5, so that would be the preferred tool to use now. TB2::Tester is the one specifically written for Test::Builder 1.5. My plan is to mark Test::Builder::Tester as "discouraged" and focus on making it work as a transitional tool. That means emulating as many 0.x quirks as possible to keep existing TBT-based tests working, but that also means sacrificing improving it. IMO its approach is dead, so that's ok. What say? -- s7ank: i want to be one of those guys that types "s/j&jd//.^$ueu*///djsls/sm." and it's a perl script that turns dog crap into gold.
Next Test::Builder 1.5 milestone: fix Test modules
Moving along the trail of getting Test::Builder 1.5 ready (and CPAN ready for Test::Builder 1.5), the next step is to fix the Test modules which are failing. While they fail, large chunks of CPAN cannot be tested. I've set up issues for the known failures. Some of them are probably very easy, some are not so easy, and some are really involved. https://github.com/schwern/test-more/issues?milestone=14&state=open This is only round one. Once the known Test module failures are fixed, we can re-smoke and test all the things that can't be tested because of dependency failures. That would be all those "DISCARD"s here in the last smoke run. https://github.com/schwern/test-more/issues/306 What's also very helpful is doing an analysis, figuring out why it's broken, and posting that to the issue. It will make fixing it that much faster. Also very important is figuring out if it's a problem with their test suite, or if it's a problem with the module itself. If the module is broken, Test::Builder 1.5 can declare that version in conflict and prevent upgrading Test::Builder from breaking existing CPAN installations. Of course, complete fixes are appreciated. -- I have a date with some giant cartoon robots and booze.
Re: version.pm and developer version numbers
On 2012.7.27 12:48 PM, Jeffrey Thalhammer wrote: > I just discovered that version.pm always treats version numbers with an > underscore as less than the equivalent version number without the underscore. > So "6.63_02" is less than "6.6302". Is it it just me, or does that seem > crazy? > Dealing with $VERSION numbers in Perl is hard enough as it is. This doesn't help. IMO it makes sense. 6.6302 and 6.63_02 are not the same thing. One is stable, one is alpha. One is version 6.6302, one is 6.63 alpha 2. The alpha version should always be less than an "equivalent" stable version. What's really going on is this: 6.63_02 => 6.63_2 6.6302 => 6.6302.0 And 6.63 < 6.6302. I know this gets in the way because people like to collapse alpha version numbers to avoid a warning when they're compared as numbers. Ironically the very thing people do which trips up version.pm is unnecessary with version.pm. -- Life is like a sewer - what you get out of it depends on what you put into it. - Tom Lehrer
Re: Proposal Test::TAPv13
On 2012.7.11 2:39 PM, Daniel Perrett wrote:
> I may have missed the point here, but would it be sufficient to have
> two flavours of diagnostic calls, distinguishing between pre-test
> diagnostics (`forewarn()`) and post-test diagnostics (`reflect()`), or
> is the problem that ok() must be a single function call?
Rather than tell you what I think the solution should be, let me frame the
problem. I'm going to use a fictional XML format for illustration, but the
problem applies to a broad class of formats which are different than TAP.
Consider this test...
#line 12
ok( 0, "something" );
and the output in our fictional XML format might look like this...
fail
something
12
The important thing to note is that the result has a clear start and end.
Now, if all information about a result has to go inside the tag, how
does this code...
#line 12
ok( 0, "something" );
info( { have => "this", want => "that" } );
Know to produce this output...
fail
something
12
this
that
...and yet still work when there is no tap_diag() call after the ok()?
There are a number of alternative interfaces to solve the problem. Making
each assert a single function call is the "best" from the point of view of
ease of formatting, but requires every assert to take an extra argument. I
feel this is getting bulky from an interface perspective, but maybe it's ok.
ok( 0, "something", {
have => "this", want => "that"
});
Another one would be to turn results into objects and only format when they go
out of scope...
ok( 0, "something" )->info({ have => "this", want => "that" });
...but that goes pear shaped if somebody does this.
$results{$test} = is( $this, $that );
You could do some clever tricks where the formatter waits until the next
result to display a result, in case extra info comes in.
# store the result.
ok( 0, "something" );
# attach this to the previous result.
info( { have => "this", want => "that" } );
# format and display the previous result and info.
# store this result.
ok( 0, "something else" );
# format and display the previous result with no extra info.
# store this result.
ok( 0, "something more" );
# format and display the previous result.
# end the test.
done_testing();
...but that significantly complicates the formatters and puts streamed output
on a delay. Very confusing if you're stepping through the code.
Right now the "all in one" version seems best. It's explicit. It's simple.
> If that's possible, maybe we can extend it to guess at what the
> various messages mean: most diags can be assumed to be of the
> `reflect()` type, as that's typical behaviour, e.g. Test::More, while
> more perlish warnings is most likely to be of the `forewarn()`
> variety.
I think we don't have to guess at that. Structured diagnostics are only
associated with test results (and possibly start/end of test info). They're
not used like diag() is now as both a warning mechanism and a test result
information system.
Since this is a new interface, we can deal with that by making different
functions for each different intent. One for "this goes with the previous
result". One for "this is general test information". And one for "this is
just a diagnostic dump for and not associated with anything".
--
You know what the chain of command is? It's the chain I go get and beat you
with 'til you understand who's in ruttin' command here.
-- Jayne Cobb, "Firefly"
Re: Proposal Test::TAPv13
On 2012.6.1 5:40 AM, Steffen Schwigon wrote:
> I am about to upload Test::TAPv13. I also did an prepan entry for that.
>
> PrePAN: http://prepan.org/module/429En4oFbn .
> URL: https://github.com/renormalist/Test-TAPv13
> Synopsis:
>
>use Test::TAPv13 ':all'; # must come before Test::More
>use Test::More tests => 2;
>
>my $data = { affe => { tiger => 111,
> birne => "amazing",
> loewe => [ qw( 1 two three) ],
> },
> zomtec => "here's another one",
> "DrWho" => undef,
> };
>
>ok(1, "hot stuff");
>tap13_yaml($data);
>tap13_pragma "+strict";
>ok(1, "more hot stuff");
>
> Does it make sense?
> Did I overlook an existing way to generate TAP v13?
I'm just seeing this now. The output looks correct and I'm happy to see
somebody playing with the structured diagnostics for reals! There's a few
code nits, but I'll note them on github.
Test::Builder1.5 can generate TAP v13, in fact it does so by default, but is
currently lacking the structured diagnostics part. Part of this is just
time/effort, but the larger part is with how tests are written...
The problem with...
ok( ... );
diagnostics( ... );
Is that there's no reliable way to know that diagnostics() goes with the
previous ok(). This is ok for TAP, where the test result and diagnostics are
on separate lines and can be printed separately, but other formats need to
print out the results and diagnostics together. Like anything where the
diagnostics information is inside a tag, for example.
That pattern has to be rethought. This was one of the goals of Test::Builder2
(the new class to replace Test::Builder) but that is on hold. Any thoughts?
Also, when did we add the pragma thing?
--
The interface should be as clean as newly fallen snow and its behavior
as explicit as Japanese eel porn.
Re: Getting "No error" instead of "No such file or directory" on Win32
On 2012.6.24 7:06 AM, Shlomi Fish wrote: > [QUOTE] > t/01basic.t . ok > > # Failed test 'error parsing non-existent does_not_exist.xml' > # at t/02parse.t line 238. > # 'Could not create file parser context for file > "does_not_exist.xml": No error at t/02parse.t line 237. > # ' > # doesn't match '(?-xism:\ACould not create file parser context for file > "does_not_exist\.xml": No\ such\ file\ or\ directory)' > # Looks like you failed 1 test of 531. > t/02parse.t . > Dubious, test returned 1 (wstat 256, 0x100) > Failed 1/531 subtests > > [/QUOTE] > > My question is - why? Some systems stringify an errno value of 0 as "no error". A few things could have caused this: 1) Something might have set errno to 0 between the failure and the error being printed. Many I/O operations will do this, some at the OS level. This could be in the CLEANUP_ERROR_HANDLER or REPORT_ERROR or even inside xmlCreateFileParserCtxt itself. 2) The failure may not have been the result of an I/O failure causing errno to never have been set. Is there a guarantee that xmlCreateFileParserCtxt() sets errno reliably? -- Stabbing you in the face so you don't have to.
Test::Builder 1.5.0 alpha 5 released
The fifth alpha for Test::Builder 1.5 has been released. It contains work from Matthew Horsfall, Peter Rabbitson, geistteufel, Karen Etheridge, Michael Schwern and others. It primarily addresses threading issues and regressions discovered via testing CPAN modules. https://metacpan.org/release/MSCHWERN/Test-Simple-1.005000_005/ https://github.com/schwern/test-more/ Please report any bugs or regressions in your CPAN modules at https://github.com/schwern/test-more/issues We very much want to hear about broken CPAN modules, CPAN is a very good test suite. Smokers, please load your alpha smokers with this new version. 1.005000_005 Thu Apr 26 15:23:25 PDT 2012 New Features * cmp_ok() will error when used with something which is not a comparison operator, including =, += and the like. [github 141] Bug Fixes * Using a reference as a test name works again. [github #264] * Protected against Test::More accidentally overwriting $! [github #268] [github #266] * Restored the behavior of a fork, it will not issue its own plan, making testing with fork easier. * Test::Builder->name() has been restored. [github #220] (Matthew Horsfall) * TB2::BlackHole no longer tries to AUTOLOAD its destructor. [github #280] (Peter Rabbitson) * Threads work on 5.12 and 5.10 again. (Peter Rabbitson) Doc Fixes * use_ok() has been discouraged and de-emphasized as a general replacement for `use` in tests. [github #288] Incompatible Changes with previous alphas * TB2::Counter has been removed [github 119] * The test counter has been moved from the TAP formatter into TB2::History [github 190] * TB2::TestState is no longer a complete subclass of TB2::EventCoordinator, but delegates most EC methods. The isa/can trickery caused threading issues. [github 291] * TB2::TestState->current_coordinator is now TB2::TestState->ec. * use_ok() will no longer apply lexical pragams. The incompatibilities and extra complexity is not worth the marginal use. [github #287] 1.005000_004 Sun Mar 25 15:01:49 BST 2012 Bug Fixes * Fix the thread crashes occurring with 5.12 and down. [github 261] Test Fixes * t/History/child_process.t was failing, the forked processes were improperly coordinated. Incompatible Changes with previous alphas * coordinate_threads() has been removed. It had no use case and was complicating threads which are quite complicated enough thank you. -- 31. Not allowed to let sock puppets take responsibility for any of my actions. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
use_ok tasks
I've cut three tasks related to the use_ok() discussion, should people be interested in doing some small work to that end. Revert use_ok() back to its 0.98 features https://github.com/schwern/test-more/issues/287 Discourage use_ok() in the docs https://github.com/schwern/test-more/issues/288 Add test_history() to Test::More to access the History object (this makes it much easier to bail on fail) https://github.com/schwern/test-more/issues/286 -- 191. Our Humvees cannot be assembled into a giant battle-robot. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.14 4:03 AM, Aristotle Pagaltzis wrote:
>> Which is a bit of a mouthful. Providing a function to get at the
>> history object would make a bunch of test state introspection easier.
>>
>># That looks pretty good.
>>END { BAIL_OUT() unless test_history->test_was_successful }
>
> I like that.
Go at it.
https://github.com/schwern/test-more/issues/286
--
package Outer::Space; use Test::More tests => 9;
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.11 1:01 PM, Aristotle Pagaltzis wrote:
>> Unless I'm mistaken, Test::AutoBailOut is doing to need a global
>> $SIG{__DIE__} handler or override CORE::require or add something to
>> @INC or try to parse exception messages or something like that. Any
>> of those have global side effects which can potentially interfere with
>> or be overwritten by the code being tested.
>
> … what for? Is Ovid’s solution of just using an END block insufficient?
> Why?
You're referring to this?
http://www.nntp.perl.org/group/perl.qa/2012/04/msg13128.html
If all you want is to bail out if the test fails to complete, that's easy(er).
I thought it was going to do something specifically when a `use` fails, thus
all the complexity.
A general "bail out if this test file fails" function might be handy. An even
more general "do X if the test finishes in Y state" would be even more handy.
# There is no test_passed()
END { BAIL_OUT() unless test_passed(); }
I don't want to add a huge bunch of functions to Test::More to cover all the
possible test states, in 1.5 you can get them from the TB2::History object.
# This works in TB 1.5
END { BAIL_OUT() unless Test::More->builder->history->test_was_successful }
Which is a bit of a mouthful. Providing a function to get at the history
object would make a bunch of test state introspection easier.
# That looks pretty good.
END { BAIL_OUT() unless test_history->test_was_successful }
--
Look at me talking when there's science to do.
When I look out there it makes me glad I'm not you.
I've experiments to be run.
There is research to be done
On the people who are still alive.
-- Jonathan Coulton, "Still Alive"
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.11 1:45 PM, Andy Lester wrote: > test_requires is Module::Build only, right? I don't use Module::Build. It was just added to ExtUtils::MakeMaker but I don't know when a stable will be released. -- There will be snacks.
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.11 11:43 AM, Eirik Berg Hanssen wrote: > If this fails, the test script will terminate immediately: > > * I won't get to know if any of the other modules loaded correctly, or how > they fail. Less of the interesting output. > * And there will be no BAIL_OUT, so the rest of the tests will run, burying > the interesting output. More uninteresting output. Like I said, if use_ok() is working for you keep using it. Your example where you want to load a bunch of modules just to make sure they compile is one of those cases where use_ok() is probably the right thing. > > * Why would you? :-\ > > Because it reads like normal Perl and doesn't rely on more code to do > something rather simple: load a module. > > But it fails to DWIW: report clearly on failures. Perhaps what it is doing > is not so simple, after all? Personally I'm a fan of "scroll up and read the first failure". It always works! -- 7. Not allowed to add "In accordance with the prophesy" to the end of answers I give to a question an officer asks me. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.11 9:53 AM, Aristotle Pagaltzis wrote:
> * Michael G Schwern [2012-04-11 18:35]:
>> Nope, too much magic for too small a use case.
>
> And faithfully duplicating `use` would be less so? :-)
This discussion is about backing away from that. The most magical bits of
use_ok() have not yet gone out in a stable and can be reverted.
> I don’t see how it is any more magic than `done_testing`.
done_testing() has no global side effects, it's just a function.
Unless I'm mistaken, Test::AutoBailOut is doing to need a global $SIG{__DIE__}
handler or override CORE::require or add something to @INC or try to parse
exception messages or something like that. Any of those have global side
effects which can potentially interfere with or be overwritten by the code
being tested.
> What exactly makes you uneasy? Maybe there is a way to address that
> if you can be more specific.
Mostly that it's cramming yet more complexity into core test library for a
fairly narrow use case and functionality that lives quite happily in its own
module. It's already difficult enough to keep stable.
To reverse it, why should it be in Test::More? "People don't like to have
extra test dependencies" is not accepted. If you want to bundle it all
together into a sort of universal set of testing extensions, Test::Most does
that.
While it could be argued that every distribution should be testing that their
modules load and bailing out if they don't, the bailing out part we can live
without. But it is handy.
>> It works fine if what you want is a runtime require + import + assert,
>> and sometimes you want that. The problem is it's been overused and
>> has come to replace a simple `use` in test scripts. To that end the
>> question is whether to *discourage* its use in the documentation: to
>> scale it back from "use this when you load a module" to "use this for
>> some special cases".
>
> *Which* special cases? I would rather not recommend it in any case ever.
In my reply to Andy I outlined some cases where you need a runtime require +
import + assert and its better to have a function which does that than write
it all out by hand.
> My suggestion to ship AutoBailOut was so you would be able to suggest it
> as a replacement in the docs, as it covers the one case where `use_ok`
> is even of interest (though still not the right solution).
>
> But I guess you could do that even if it ships outside of Test::More.
Precisely. For example, it already suggests Test::Differences and Test::Deep.
--
185. My name is not a killing word.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.11 9:39 AM, Andy Lester wrote:
> In this example:
>
> BEGIN {
> use_ok( 'App::Ack' );
> use_ok( 'App::Ack::Repository' );
> use_ok( 'App::Ack::Resource' );
> use_ok( 'File::Next' );
> }
> diag( "Testing App::Ack $App::Ack::VERSION, File::Next $File::Next::VERSION,
> Perl $], $^X" );
>
> it sounds like we're saying that the use_ok() doesn't help at all, and I
> might as well write
>
> use App::Ack;
> use App::Ack::Repository;
> use App::Ack::Resource;
> use File::Next;
> diag( "Testing App::Ack $App::Ack::VERSION, File::Next $File::Next::VERSION,
> Perl $], $^X" );
>
> Agreed?
Yes, exactly.
--
I do have a cause though. It's obscenity. I'm for it.
- Tom Lehrer
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.11 7:35 AM, Andy Lester wrote:
> So is there ANY legit use for use_ok()?
Yes. Sometimes you want to conditionally test if a module can be loaded and
its import does not blow up.
It's a convenience function so it can be more easily understood what's going
on and we don't each write it a million different ways. require_ok() solves a
big chunk of the problem.
if( something something ) {
use_ok 'Foo';
# Here's one long hand way...
# require_ok 'Foo';
# ok eval { Foo->import; 1; } or diag $@;
# or maybe this, hope you remember that 1;
# eval 'use Foo; 1;' or diag $@;
}
Similar to how like() is basically a convenience for...
my $regex = qr/bar/;
ok $foo =~ $regex or diag "$foo does not match $regex";
If it didn't already exist, it could probably be argued that it's not needed.
But it's there and can be left alone.
--
87. If the thought of something makes me giggle for longer than 15
seconds, I am to assume that I am not allowed to do it.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.11 8:45 AM, Aristotle Pagaltzis wrote:
> Then you simply do this:
>
> use Test::More tests => 1;
> use Test::AutoBailOut reason => 'Could not load prerequisites';
> use Config;
>
> use Test::Trap::Builder::TempFile;
> use Test::Trap::Builder::SystemSafe;
> use Test::Trap::Builder;
> use Test::Trap;
> use if $Config{'useperlio'}, 'Test::Trap::Builder::PerlIO';
>
> no_bailout 'All prerequisites loaded';
>
> Dead simple. Huge swathes of complexity, *puff* – vanished.
>
> Schwern, care to ship that in Test::More to deprecate `use_ok`? :)
Nope, too much magic for too small a use case. You should make it a module
and talk to Ovid about Test::Most.
Let me reiterate, I have no plans to *deprecate* `use_ok`. Even if I wanted
to there are simply too many users to make deprecation worth while.
It works fine if what you want is a runtime require + import + assert, and
sometimes you want that. The problem is it's been overused and has come to
replace a simple `use` in test scripts. To that end the question is whether
to *discourage* its use in the documentation: to scale it back from "use this
when you load a module" to "use this for some special cases".
--
The interface should be as clean as newly fallen snow and its behavior
as explicit as Japanese eel porn.
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.10 6:21 PM, The Sidhekin wrote:
> * How would you rewrite a test script such as my own
> http://cpansearch.perl.org/src/EBHANSSEN/Test-Trap-v0.2.2/t/00-load.t so
> that it does not use use_ok()?
use Test::More tests => 1;
use Test::Trap::Builder::TempFile;
use Test::Trap::Builder::SystemSafe;
SKIP: {
skip 'Lacking PerlIO', 1 unless eval "use PerlIO; 1";
require Test::Trap::Builder::PerlIO;
Test::Trap::Builder::PerlIO->import;
}
use Test::Trap::Builder;
require Test::Trap
or BAIL_OUT( "Nothing to test without the Test::Trap class" );
diag( "Testing Test::Trap $Test::Trap::VERSION, Perl $], $^X" );
pass("Test::Trap loaded successfully");
Or a slimmed down version, recognizing that Test::Trap loads all the above
already and will tell you just which module failed to load. It also makes the
test black box.
eval {
require Test::Trap;
Test::Trap->import;
1;
} or BAIL_OUT("Test::Trap failed to load");
diag( "Testing Test::Trap $Test::Trap::VERSION, Perl $], $^X" );
pass("Test::Trap loaded successfully");
This can be boxed up in a module. There's already Test::Requires. It skips
instead of bails, it would be a good target for test_must_have() or something
that bails.
> * Why would you? :-\
Because it reads like normal Perl and doesn't rely on more code to do
something rather simple: load a module.
TAP (the ok 1 stuff) has us very much convinced that we need to output
something for every assert. The rest of the testing universe relies on
exceptions to tell them something's failed and seems to get along just fine
with that.
But quite honestly, if use_ok() works for you, if all you need is require +
import + test, continue to use it. I don't plan on eliminating it.
--
Hating the web since 1994.
Re: Revert use_ok() change to allow lexical effects?
On 2012.4.10 2:52 PM, Mike Doherty wrote:
> On 12-04-10 05:20 PM, Paul Johnson wrote:
>> On Tue, Apr 10, 2012 at 12:20:20PM -0700, Michael G Schwern wrote:
>>> 2. Should use_ok() be discouraged in the documentation?
>>
>> I'm very much in favour of this.
>
> I don't see any discouragement in the documentation...
We're discussing adding that discouragement.
> and what's wrong with use_ok to begin with?
That was the second half of my post. It's a lot of extra work for a version
of this that doesn't work quite right...
use Foo::Bar;
pass "Loaded Foo::Bar";
And the pass() is just informational. If Foo::Bar fails to load the test dies
which is a failure.
> I typically use_ok(...) or BAIL_OUT. If that's the only way to use
> use_ok safely, then maybe it should do that for you automatically.
It cannot be a bail out, because there's nothing which says the module being
loaded is THE CRITICAL MODULE on which the whole test suite hangs.
It could die, but that introduces even more compatibility issues than the
lexical change. And it still leaves the problem of having to remember to wrap
it in a BEGIN block for it to truly emulate use properly.
Aristotle Pagaltzis said pagalt...@gmx.de
> I think the ideal way to handle `use` would be a `bail_on_use_failure`
> switch that intercepts `use` errors and turns them into BAIL_OUT,
Implementing this involves putting in a $SIG{__DIE__} and all the fun that
entails. Test::Builder got rid of its $SIG{__DIE__} to avoid interfering with
other code.
And it cannot be a bail out for the reasons above.
> Then `use_ok` would just completely deprecated.
I don't think it needs to be deprecated in the "issue a warning" sense. It's
not actively harmful to use, if its working for you go ahead and use it, it's
just kinda useless and not worth the effort of maintaining. I don't think
it's worth the effort to change all the hojillions of test suites that use it.
Documentation and a decision not to add more features should be enough.
--
ROCKS FALL! EVERYONE DIES!
http://www.somethingpositive.net/sp05032002.shtml
Revert use_ok() change to allow lexical effects?
In a series of patches, Father Chrysostomos and I enhanced use_ok() so that it
can apply lexical effects to more closely emulate the real `use`. For example,
use_ok('strict');
Previously this would just load strict and call import, but strictures would
not actually be applied to your scope. Now strictures *will* be applied
making it work more like a regular use.
Testing has shown it's caused just a handful of compatibility problems, and it
hasn't gone out in a stable release yet. Because the new use_ok() is so much
more complex than before I'm having second thoughts about whether it's worth
it and would like opinions.
There are two questions:
1. Should the lexical effect patches to use_ok() be rolled back?
2. Should use_ok() be discouraged in the documentation?
The new use_ok(), while it works remarkably well, is significantly more
complex than previously and it touches magical variables $^H and %^H. There
is a danger of invoking bugs by touching those variables.
It continues to have compatibility issues, for example...
BEGIN { use_ok 'Some::Module' }
this will apply lexical effect while...
BEGIN { use strict }
will not but
BEGIN { require strict; strict->import }
will. Confusing.
use_ok() has low utility to begin with. If you want exports and lexical
effects to be properly applied you have to wrap it in a BEGIN block. And as
has been pointed out before, if use_ok() fails you probably want the program
to halt. So the full invocation should be...
BEGIN {
use_ok("Some::Module") or die;
}
At which point you might as well write...
use Some::Module;
And if you really want the load to be a passing test, you can add...
pass("Loaded Some::Module");
If all you want to do is check that a module can compile, there is the much
simpler require_ok().
Apologies to Father Chrysostomos if his work is reverted, it has been top
notch. If it is reverted, it may find life in another module where a complete
emulation of use is desired as a user function or method.
--
31. Not allowed to let sock puppets take responsibility for any of my
actions.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/
Re: Devel::Cover: whence the reference to Class::MethodMaker?
On 2012.3.14 5:49 PM, James E Keenan wrote: > On a machine where I can only install libraries underneath my home directory, > I today tried to install and use Devel::Cover. After considerable effort, I > was able to install Devel-Cover-0.79. But when I went to use it in > conjunction with 'prove' I got error output like this: > > # > $> HARNESS_PERL_SWITCHES=-MDevel::Cover prove t/001_new.t > t/001_new..Devel::Cover: Can't open > blib/lib/Class/MethodMaker/Engine.pm for MD5 digest: No such file or directory > Devel::Cover: Can't open blib/lib/Class/MethodMaker/scalar.pm for MD5 digest: > No such file or directory > Devel::Cover: Can't open blib/lib/Class/MethodMaker/array.pm for MD5 digest: > No such file or directory > Devel::Cover: Can't open blib/lib/Class/MethodMaker/hash.pm for MD5 digest: No > such file or directory > # > > Class::MethodMaker is probably not installed on this machine, and it's > certainly neither installed underneath my home directory nor used in the > testing code or the code being tested. But what's really puzzling to me is > the fact that I cannot locate the string 'MethodMaker' anywhere in the > Devel::Cover source code! Is it possible you had a pre-existing cover_db directory? Try deleting it. -- emacs -- THAT'S NO EDITOR... IT'S AN OPERATING SYSTEM!
Re: What is the "best" code on the CPAN?
On 2012.2.7 6:29 PM, Jeffrey Thalhammer wrote: > I'm working with a group of Perl developers with various backgrounds and > skill levels. > We have instituted a fairly effective code inspection process, but we still > get > bogged down in debates over what "good code" is. Unfortunately, the developers work > in isolated silos and spend the vast majority of the time looking only at their own code. > So they rarely have an opportunity to see what good (or just better) code might actually > look like. If the silos are eliminated and people have to cooperate and work with each other, you'll find that the discussions about what is "good code" become far more practical and people's attitudes become less provincial. > So in your opinion, which distribution on the CPAN best demonstrates these > qualities? > Or do you think there are other more important qualities that I should be looking for? > I realize there is more than one way to do it, so I don't really expect to find the > "best" code. I just want something I can hold up as strong example that > people > (including myself) can learn from and aspire to. At the risk of sounding immodest, with the exception of failing to use other CPAN modules, because it can't, Test-Simple 1.5 matches your criterion pretty well. http://search.cpan.org/~mschwern/Test-Simple-1.005000_002/ -- 91. I am not authorized to initiate Jihad. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Reconsidering Mouse in TB2
On 2011.12.6 10:56 AM, Cosimo Streppone wrote: > I wasn't referring to a dependency problem, > but to the actual code being the same on the > thing-that-tests and thing-that-is-being-tested. It would be a problem if they were the same code, but they're not because TB2::Mouse is a copy of an earlier released version of Mouse. > I would tend to avoid this (who controls the controller etc) > but maybe it's not a problem at all. That wouldn't be possible because both have good test suites. Mouse has to pass its test suite to release a new version. TB2 has to pass its test suite after it embeds a new version of Mouse. There's no more risk than any other dependency. However, TB2 would have to be smart and keep the "oh, Mouse is installed so I'll use that" logic from kicking in. That would be a problem. -- THIS I COMMAND!
Re: Reconsidering Mouse in TB2
On 2011.12.6 1:23 AM, Cosimo Streppone wrote: > Plus, what would happen if at some point Mouse wanted to > use TB2 for its test suite? You might have missed that part of the conversation. TB2 would not depend on Mouse, it has its own embedded copy. It would work fine. -- "Clutter and overload are not an attribute of information, they are failures of design" -- Edward Tufte
Reconsidering Mouse in TB2
As a result of the discussions, I'm now reconsidering using Mouse in TB2. Its interesting, it's not for any of the reasons I thought, but for a distressing encapsulation breech built into Moose. The main concern is that the fact that we're using Mouse leaks out. We must document that we're using Mouse and users must rely on this implementation detail. Anything wishing to consume a role must use Mouse. Anything wishing to take full advantage of subclassing must use Mouse. This is a critical violation of encapsulation. It means TB2 is stuck using Mouse forever. I look ahead a few years and see that becoming a severe limitation. A side issue is the compile time performance. While XS fueled Mouse is the fastest OO implementation, compile time appears to be most critical to Perl test suite performance. So, uhh... I'm going to go do a bad thing. -- If you want the truth to stand clear before you, never be for or against. The struggle between "for" and "against" is the mind's worst disease. -- Sent-ts'an
Re: Need suggestions for terminology
You might want to crib from BackPAN::Index. It has a lot of this terminology. On 2011.12.2 5:21 PM, Jeffrey Thalhammer wrote: > The context is Pinto, which is yet-another suite of libraries and tools > for building a private CPAN-like repository. Does it explode when hit from the rear? No? Joke too old? https://secure.wikimedia.org/wikipedia/en/wiki/Ford_Pinto#Fuel_tank_controversy > Distribution: A Distribution is an abstract concept that defines > relationships between packages. The minimal concrete implementation > of a Distribution would be just a META.json (or equivalent) file. > Distributions also have names and versions like Foo-Bar-1.2 This is what I term a "release". A "distribution" is the whole Foo-Bar series of releases. It has a name, a list of releases, and not much else. All the real data is associated with a distribution's releases. See BackPAN::Index::Dist. A "release" is a specific version of a distribution. It has a CPAN id, date of release, what distribution it's for, a filename, version, what files are in it, etc... basically everything that's in the meta file is associated with a release. See BackPAN::Index::Release. > Distribution Archive: A Distribution Archive is the physical > manifestation of a Distribution, and corresponds to an actual file on the > local disk. For example, /home/jeff/Foo-Bar-1.2.tar.gz or > C:\MyDocuments\Foo-Bar-1.2.tar.gz "Release archive" I guess. > Distribution Path: The Distribution Path is how an Archive is identified > in a CPAN index. It is basically a URL fragment that looks like > A/AU/AUTHOR/Foo-Bar-1.2.tar.gz. This is the term I'm having the most > trouble with. CPAN::DistnameInfo calls this the "prefix" but I don't > really like that either. "Release path". I agree that "prefix" is a poor name. I would store it as just AUTHOR/Foo-Bar-1.2.tar.gz. The A/AU/ is an implementation contrivance and can be calculated. There is value in having just the filename available separate from the rest of the path as well. > Package: A package is just a package, in the usual way. That is, > something declared with the "package" keyword. Agreed. > Module: I actually avoid using the term Module because I think it is > often misused. I feel that a Module is a physical file (i.e. something > that you "use"). But some folks use the terms Module and Package > interchangeably. To be precise, CPAN (and Pinto) index Packages not > Modules. So when you ask cpan[1] to install Foo::Bar, what you actually > install is an Archive that contains some Module that provides Package > Foo::Bar, and that Module may or may not be called Foo/Bar.pm. > At least, that's how I understand it. Agreed that "module" is overloaded. I would associated it either with the .pm file or with the distribution. Probably best to avoid it all together. > Repository: A repository is a general term for any CPAN-like pile of > files. This includes CPAN mirrors, as well as any DarkPAN or mini-cpans. > A repository has a URL that identifies the entry point. For example: > http://cpan.perl.org You might want to record the type of repository, even if there's only "CPAN" for now. It might be worth differentiating "CPAN" from "BackPAN". The structure is the same, but one has some CPAN releases and the later as has all CPAN releases. For example, if you wanted to pull from Debian or Fedora sources. > CPAN: *THE* CPAN is http://cpan.perl.org (or wherever PAUSE publishes > stuff). *A* CPAN is one of the mirrors that replicate *THE* CPAN. > A DarkPAN or mini-cpan or Pinto are *not* CPANs. I'd call it something else, as this can rapidly get confusing. I'm not sure what. -- 44. I am not the atheist chaplain. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: TB2::Mouse will be internal use only... with one hitch.
On 2011.11.30 12:28 PM, Ricardo Signes wrote: >> You're patching Test-Simple. > > Okay. So, I want to write a third-party test output formatter that emits test > results in the binary language of my water vaporators. I subclass some core > formatter with "use parent" and add some methods or something? Sometimes. You can treat it as a generic superclass, with these caveats. 1. You have to override defaults the ad-hoc way, by overriding new() and adding arguments. 2. You can't consume roles. #2 is a problem because some of the public facing things you want to extend are roles. I can change all the public facing roles into classes without too much hassle. I'm not sure which is yickier, degrading the object design or the classes leaking out how they're implemented. Exposing that we're written with Mouse welds us to Mouse. For full control, your Formatter must be a Mouse class. package Format::Moisture::Vaporator; use Mouse; extends 'TB2::Formatter'; # or Format::Binary::LoadLifter The "hitch" described in the original post. All Mo[ou]se subclasses have to use the same OO system if they want to take full advantage of roles and overridable attributes. Mouse subclasses must use Mouse. Moose subclasses must use Moose. TB2::Mouse subclasses must use TB2::Mouse. Here's the tl;dr part: TB2 will use Mouse if Mouse is available. [1] Otherwise it will use TB2::Mouse, its bundled and namespace altered copy of Mouse::Tiny. As far as any extensions are concerned, TB2 uses Mouse since must depend on Mouse. > I want to write a better Test::Foo library. I'm not using "extends 'Tester'" > or something? Depends on what you're doing. If you're just writing a bunch of testing functions you write it just like you do now. No knowledge of Mouse necessary. If you're something like Test::Class or Test::NoWarnings that's changing the behavior of the test you do, because you probably have to write a new event handler. > The only case when the programmer needs to know about the Mouse underpinnings > is when hacking on the dist itself, not not ever reasonable extensions? Yes, unless you consider writing a Formatter and altering the behavior of the whole Test process to be "reasonable". [1] Once I change it to do that. -- 31. Not allowed to let sock puppets take responsibility for any of my actions. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: TB2::Mouse will be internal use only... with one hitch.
On 2011.11.30 7:20 AM, Ricardo Signes wrote: > re: "internal use" > > What does "internal" define here? What are the boundaries of the space to > which TB2::Mouse use is internal? Like any other "internal use only" .pm file in a CPAN module, it can only be used by the Test-Simple distribution (Test::Simple, Test::More, Test::Builder, TB2::Blah...). > In what scenarios do I have to write something that uses it? You're patching Test-Simple. > If I do that, am I likely using Mouse sugar? Yes. More than sugar, you're likely uhh... eating meat and potatoes? I don't know what the food analogy for roles and attribute default overrides would be. -- Alligator sandwich, and make it snappy!
Re: TB2::Mouse will be internal use only... with one hitch.
On 2011.11.30 4:31 AM, David Golden wrote: > On Wed, Nov 30, 2011 at 6:13 AM, Michael G Schwern wrote: >> Unfortunately, that would be a circular dependency between Mouse and TB2 >> unless... > > Not a complete dependency, though, as one can function without the > other. As long as the CPAN clients are smart enough not to follow the > prereq chain endlessly, it might work. *All* CPAN clients. This includes package builders, DarkPAN tools, and so on. Circular dependencies are bad juju. Putting one at the root of the dependency tree would be asking for disaster. It risks breaking the whole automated CPAN bootstrapping and upgrading process. > And possibly TB2's > Makefile could be smart and only add the Mouse prereq if it finds a > version of itself (or, really, of Test::More) sufficient to meet > Mouse's prereq needs. (Bundling might still be easier.) That might work. It puts information about Mouse into the Test-Simple distribution. Information which will change. It means the two distributions must be coupled even more tightly. If Mouse changes its Test::More dep, a new version of Test::More must be cut immediately if not before the Mouse change. Very time sensitive. > Another idea might be for TB2 to use the new "runtime/recomends" > prereq type in v2 CPAN Meta files. Same problem as a regular prereq as they are resolved before building and installation. -- 10. Not allowed to purchase anyone's soul on government time. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: TB2::Mouse will be internal use only... with one hitch.
On 2011.11.30 2:04 AM, David Golden wrote: > On Wed, Nov 30, 2011 at 12:39 AM, Michael G Schwern wrote: >> There is a 3rd choice, which brings us full circle to where this discussion >> started. We write TB2::Object [1] to select load Mouse if its available, >> TB2::Mouse otherwise. That way TB2 classes are compatible with Mouse just in >> time. They also get a performance kick. >> >> The core gets the "for internal use only" marked TB2::Mouse as previously >> discussed. > > Without commenting on the rest of the plan, to which I want to give a > little more thought, it occurs to me TB2 as released to CPAN could > explicitly depend on Mouse, so that any TB2 installed (or upgraded) > via CPAN client gets Mouse installed (or upgraded). Unfortunately, that would be a circular dependency between Mouse and TB2 unless... * Mouse never depends on Test::More newer than the one which shipped with 5.8.1 (0.47). This is a bit much to ask of the Mouse folks. They're already using 0.88 and I don't want to make their life harder to make mine easier. * Mouse bundles Test::More. A distinct possibility. You'd put all the .pm files into t/lib and then C in each test. Modules used to do this all the time, when we weren't sure about this whole new fangled Test::More thing and module authors complained bitterly about having to add a dependency just for testing. -- But there's no sense crying over every mistake. You just keep on trying till you run out of cake. -- Jonathan Coulton, "Still Alive"
Re: TB2::Mouse will be internal use only... with one hitch.
On 2011.11.29 9:49 PM, David E. Wheeler wrote:
> How much slower will this make running the core tests,
> since neither Mouse nor Mouse::XS will every be available
> there (during development, at core installation time).
Doing some back of the envelope calculations, it looks like an extra minute
and a half on my laptop. I don't know how long core tests take right now.
Running...
use Test::More tests => 1;
pass("ok");
100 times via prove...
With 0.98: 2.2s
With 1.5: 10s
An extra 8 seconds per 100 tests. There's about 1100 core tests which use
Test::More. 8 seconds x 11 is about 90 seconds.
That's worst case. I can probably get compile time down at least 20ms which
shaves off about 30 seconds.
--
170. Not allowed to "defect" to OPFOR during training missions.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/
TB2::Mouse will be internal use only... with one hitch.
After the discussion, some pondering and hearing from Goro, I've decided that TB2::Mouse will be marked for internal use only. 1. People are uncomfortable that I'm shipping a copy of Mouse::Tiny, let alone making it public. 2. If it's private now, it can be made public later. Can't do it the other way around. 3. We don't need it for stable. 4. Mouse is a very easy dependency. It has no dependencies and can work without a compiler. There's not much value in providing a bundled copy. They can just depend on Mouse. 5. Goro wants it that way. Except there's a problem. In order to properly subclass a TB2 module your subclass must use TB2::Mouse. You can't use Mouse. For example. package My::TAP::Formatter; use Mouse; extends 'TB2::Formatter::TAP::v13'; has '+show_tap_version' => default => 0; This doesn't work. Mouse doesn't see the superclass's attributes because they were declared by a different namespace, TB2::Mouse. The subclass must use TB2::Mouse if it wants to override attributes. I hadn't realized that. It also must use TB2::Mouse if it wants to consume TB2 roles. This offers two choices: #1 Subclass authors will just have to deal. #2 TB2::Mouse is available IF AND ONLY IF you are subclassing or using a role. #1 isn't much different from the status quo of subclassing a non-Mo[ou]se Perl 5 OO module... except for the problem of roles. #2 is effectively publishing it. This is not academic. TB2::Event and TB2::EventHandler are roles. Writing new Events and EventHandlers are the primary way to extend how TB2 works. They could be turned into classes, but that would make things more difficult to extend. There is a 3rd choice, which brings us full circle to where this discussion started. We write TB2::Object [1] to select load Mouse if its available, TB2::Mouse otherwise. That way TB2 classes are compatible with Mouse just in time. They also get a performance kick. The core gets the "for internal use only" marked TB2::Mouse as previously discussed. The examples in the TB2 documentation can be changed to always show them using Mouse, further distancing TB2::Mouse from user eyeballs who might want to abuse it. The end result is... 1. The core gets a shrouded copy of Mouse::Tiny, with different namespaces, as discussed at the beginning. 2. TB2 gets a performance kick once Mouse is installed. 3. Extension authors can use Mouse to subclass TB2 and consume TB2 roles as normal. 4. TB2 continues to be able to take full advantage of Mouse. Nobody needs to ever know there's a copy of Mouse::Tiny in there. Thoughts? [1] I'd probably reverse it so that TB2::Mouse is the selector and TB2::Object is the Mouse::Tiny copy, but I don't want to make the discussion more confusing than it is. -- s7ank: i want to be one of those guys that types "s/j&jd//.^$ueu*///djsls/sm." and it's a perl script that turns dog crap into gold.
Re: Relying more on Mouse
Once again, I would like to rein this all in. Employing a counter-factual, the WORST CASE is that p5p has to ship a single 110K file which they never have to touch themselves. I am unable to get excited over the consequences. Let's keep our heads. The whole argument against TB2::Mouse is predicated on the idea that shipping TB2::Mouse in the core MIGHT cause future maintenance hassle for p5p. If the alternatives will cause EVEN MORE maintenance hassle for p5p and/or CPAN authors... we should just ship TB2::Mouse. On 2011.11.25 5:07 PM, David Golden wrote: > On Fri, Nov 25, 2011 at 4:02 PM, Michael G Schwern wrote: >> Keeping this all in perspective, TB2::Mouse is one 110K file (compressed down >> to 25K) with no POD. It's part of the Test-Simple distribution. p5p has to >> do NO EXTRA WORK AT ALL to include or maintain it. It just comes with >> Test-Simple like any other .pm file in Test-Simple. >> >> If I didn't bring it up, nobody would know it was there. Think about that. > > That sounds oddly like "security through obscurity" except it's > "compatibility through obscurity". :-) Someone always finds these > things out and then relies on them and then claims the need for > ongoing existence in core. Obscurity, documentation (or lack thereof) and social contract is how Perl enforces encapsulation. Whether we like it or not. Perl doesn't have an infatuation with enforced privacy. It would prefer that you stayed out of its living room because you weren't invited, not because it has a shotgun” ―- Larry Wall People can (and do) rely on internal code and then want us to maintain compatibility. At some point, project willpower to enforce encapsulation has to win out. I quote Tom... You are wicked and wrong to have broken inside and peeked at the implementation and then relied upon it. -- tchrist in <31832.969261130@chthon> Some careful and B documentation in TB2::Mouse can make this point even more clear. At some point, users take responsibility for ignoring the clearly marked warning signs. https://twimg0-a.akamaihd.net/profile_images/63014578/Crocodiles.jpg > As the current maintainer of Parse::CPAN::Meta -- which *is* an exact > copy of YAML::Tiny with a quick paint job (*cough* DZP::Doppelgaenger > *cough*), I'm sensitized to the issues, that's all. Roger. >> The real problem is how does Test::More depend on a non-core CPAN module >> without having a dependency on a non-core CPAN module? One which uses >> Test::More. That problem always remains. Test::More has it. MakeMaker has >> it. And the simple answer is bundling. > > I don't follow. I was speaking to the suggestions to use Role::Tiny or Object::Tiny or Role::Basic instead of a copy of Mouse::Tiny. They all present the same problem as TB2::Mouse. But you bring up some other scenarios while I'll address... > The only things that need to be in core (or need to > be bundled with a CPAN release) are modules sufficient to get a CPAN > client to deal with build_requires (and eventually test_requires). ...and the modules CPAN clients depend on. Very important. ...and the dependencies for everything else we currently ship in core. I would so love if the "only stuff for CPAN" policy was real and retroactive, but it is not. > Test::Hoopy can depend on Test::More 2 and core can have Test::More > 0.99 and CPAN clients will deal. You've already talked about a non > Test-More framework to test TB2, so you've already eliminated that > element of bootstrapping. Here's an example I hope will clarify. 1. CPAN-Meta's tests rely on Test::More. (they do) 2. CPAN-Meta is shipped with the core. (it is) 3. The next version of CPAN-Meta (knowingly or not) relies on a Test::More 1.5.x feature or bug fix in its tests. In order to ship CPAN-Meta, p5p has these options... 1. The core ships with Test::More 1.5.x. 2. CPAN-Meta is forced to downgrade their tests to only use 0.9x features. 3a. CPAN-Meta is forced to downgrade their tests to only use t/test.pl and copy t/test.pl into their CPAN distribution. 3b. Same as 3a, but we fork t/test.pl and make it available is an independent test library... and then ship it with the core. 4. p5p maintains a patched version of CPAN-Meta. 5. p5p stops running CPAN-Meta's tests as part of the build process. I think you'll agree, upgrading Test::More and accepting a single extra file that requires no extra maintenance is the approach with the most flexibility and least hassle for p5p and dual-life modules. This can happen to any module in cpan/ and dist/ http://cpansearch.perl.org/src/SHAY/perl-5.15.5/cpan/ http://cpansearch.perl.org/src/SHAY/perl-5.15.5/dist/ >> BTW What are those downstream maintenance costs? >
Re: Relying more on Mouse
On 2011.11.25 3:04 PM, chromatic wrote: > On Friday, November 25, 2011 at 01:02 PM, Michael G wrote: >> We did it once with Test::Harness::Straps, though not nearly that well >> thought out, and it worked. > > Do you know anyone (besides me) who used it? I know I dealt with some DarkPAN users, but it's been so long I can't enumerate them. There's a handful on CPAN and in Google Code Search. https://metacpan.org/requires/module/Test::Harness::Straps http://www.google.com/codesearch#search/&q=%22use%20Test::Harness::Straps%22%20lang:perl&type=cs I agree it's not a scale test, but the mechanism worked. >> I do see a work around. Document that TB2::Mouse can be used, but you MUST >> add it as a dependency anyway. > > I hate to see the core (even implicitly) blessing Yet Another Object System. The core hasn't blessed any object system, directly or indirectly, unless you count the ad hoc messes previously cobbled together in the old OO tutorials. So it's hard to say "yet another". But I'm willing to say whatever they come up with in the future is going to look an awful lot like Moose. One of the major problems with including a module in the core is its *implicit* blessing. The core never said anything about whether a core module was blessed or better than others, users just assumed it. With TB2::Mouse, we have a chance to *explicitly* state the terms. If it's going to be for internal TB2 use only, we can state that. If it's going to be a stop-gap until a better OO system is core, we can state that. Those old OO tutorials are going away BTW. Moose, Mouse, Class::Accessor, Object::Tiny and Role::Tiny are all talked about in the new OO tutorial. http://cpansearch.perl.org/src/SHAY/perl-5.15.5/pod/perlootut.pod >> It won't work for the simple reason that if any dual-life CPAN module >> decides to use a feature or bug fix of Test::More 1.5.0 then it can't be >> cored. > > I'm also little enamored of pre-emptive coring. > > I'd like to see TB2 *used* outside of bootstrapping for a while first before > considering it a replacement for TB. There's some sort of misunderstanding or terminology mix up. The TB2 modules cannot be separated from Test::More any more than Test::Builder can. Test::Builder is using the TB2 infrastructure. Even if I chucked out the whole event system, Test::Builder would still make use of TB2::Formatter. I expect TB2 to go through an alpha period for a while. The current formal alpha phase for compatibility with existing code, make sure nothing breaks. Then a post-stable 1.5.0 phase where the TB2 interfaces are considered unstable, so we can knock them around. I'm sure there will be plenty of little changes. People can try TB2 out NOW and give feedback. Even if it's "I don't understand this, the docs for X suck". That is useful! I'm so deep in the design its hard for me to see that. There are plenty of modules which could be rewritten right now far easier using TB2. Off the top of my head... Test::SharedFork Test::Aggregate Test::NoWarnings Test::Class Test::Block Fennic Test::DebugOnFail (that's in the examples directory) And I'm still waiting on someone taking a stab at an alternative output format like JUnit or JSON. -- I am somewhat preoccupied telling the laws of physics to shut up and sit down. -- Vaarsuvius, "Order of the Stick" http://www.giantitp.com/comics/oots0107.html
Re: Relying more on Mouse
Keeping this all in perspective, TB2::Mouse is one 110K file (compressed down to 25K) with no POD. It's part of the Test-Simple distribution. p5p has to do NO EXTRA WORK AT ALL to include or maintain it. It just comes with Test-Simple like any other .pm file in Test-Simple. If I didn't bring it up, nobody would know it was there. Think about that. This discussion is EXACTLY THE SAME if TB2 were using Role::Tiny or Role::Basic or any other CPAN module which is not currently core. On 2011.11.25 3:53 AM, David Golden wrote: > To be clear -- I'm not (yet) saying that TB2 should absolutely roll > it's own OO, but I am concerned about hiding yet another copy of a > perfectly decent CPAN module so that we can avoid saying that we're > including another CPAN module in core. It's seductively easy to get > the benefits *now* during the development of TB2, but there are > downstream maintenance costs that we shouldn't discount lightly. That is only a tangential problem. The real problem is how does Test::More depend on a non-core CPAN module without having a dependency on a non-core CPAN module? One which uses Test::More. That problem always remains. Test::More has it. MakeMaker has it. And the simple answer is bundling. BTW What are those downstream maintenance costs? >> Also one of the side benefits of shipping TB2::Mouse is now other Test >> modules >> can rely on it getting the benefit of a fully operational OO system without >> adding further dependencies. > > And potentially *other*, non-test modules could start doing so too. Disallowing non-Test modules is not strictly necessary, it's just something I have in my head to rein this in a bit. I don't think it changes anything. Anyhow, they would be in violation of the documentation (which would be written in bold neon on the first page) and not supported. p5p doesn't always toe that line, but that's their problem. > Is there a guarantee that TB2::Mouse will always be in core (short of > a deprecation cycle)? Would it have to be available "forever" under > "use v5.18"? I'm very leery of adding more core modules when the core > policy around deprecation or lack thereof is in flux. If it's documented, then yes. See above about it being a single file with no extra support costs for p5p. I do see a work around. Document that TB2::Mouse can be used, but you MUST add it as a dependency anyway. That way when we do remove it from the core (and I do agree, it would be nice if we could) then it can be put on CPAN and slurped in like anything else. We did it once with Test::Harness::Straps, though not nearly that well thought out, and it worked. > [ Social hack: include TB2::Mouse but mark it "deprecated" from the > start. :-) ] That would be "for internal use only" and lacking documentation like all the other "for internal use only" .pm files already in core and other CPAN modules. The "everyone else gets an OO framework" part of this plan is just gravy. Right now it's not documented at all, so anyone using it has violated encapsulation and gets no support. > If I had my druthers, I'd like to see Stevan's core MOP work finished, > then I'd like to see a powerful, minimalist object system on top of > *that* built for inclusion in core, then I'd like to see core modules > gradually migrating to that. And I'd like a pony, too. Throw in a time machine so we can all fast forward three years to that version of the future. Wait... eleven years! Gotta wait for all those versions of Perl that don't have a MOP to end-of-life! :-/ > Maybe the related, fundamental question is whether TB2 itself needs to > be in core -- or, if so, *when* it needs to be in core. Maybe core is > just fine with Test::More as is and anyone needing more powerful > testing frameworks can install TB2 from CPAN. (Ditto for any new test > modules that are written that rely on TB2.) Eventually, when TB2 is > well proven on CPAN and the core offers the kind of MOP that TB2 > needs, then it can migrate in. It won't work for the simple reason that if any dual-life CPAN module decides to use a feature or bug fix of Test::More 1.5.0 then it can't be cored. The forward dependency march will drag it in. It also means the perl core gets no bug fixes and no feature enhancements. Unless they want to maintain their own fork, which I sure as hell am not. And that IS an extra maintenance load for p5p. As TB2 is a pretty radical internals restructuring, and I expect new and existing Test modules will want to start using those internals, and people will want to start using those new versions, Test::More 1.5.0 will be required. Holding it back in the core is just YET ANOTHER immediate upgrade that's necessary after a fresh perl install. While we might debate the utility of having certain modules in the core, modules which help install other modules have always been in that set. The test framework used by 80% of CPAN is squarely in that category. Should it be thrown ou
Re: Relying more on Mouse
On 2011.11.23 12:49 AM, Ovid wrote: > As an aside, while I'm sure that Schwern is using far more > than just roles, Role::Basic already handles multiple roles, > is forwards-compatible with Moose::Role and it adheres much > more closely to the traits spec, particularly with regards > to the commutative and associative properties that Role::Tiny > and Moose::Role ignore (https://metacpan.org/module/Role::Basic::Philosophy) Sorry, but Role::Basic is nearly as slow to load as Mouse and only provides one piece of the puzzle. -- s7ank: i want to be one of those guys that types "s/j&jd//.^$ueu*///djsls/sm." and it's a perl script that turns dog crap into gold.
Re: Relying more on Mouse
On 2011.11.23 12:15 AM, Eric Wilhelm wrote: > Was there some work in Moose to generate and load pre-cooked classes for > startup-time basics like the accessors and roles? It seems like being > able to do that work once during `make` would be a big win. I think that was wrt pre-cooking Moose's meta stuff, but if the attributes could be pre-compiled that would be awesome. -- Alligator sandwich, and make it snappy!
Re: Relying more on Mouse
On 2011.11.22 8:06 PM, David Golden wrote: > Attributes -- leaving aside types and coercion -- are you just talking > about the sugar? Mostly, and the sugar is very helpful, but not all of it. There's also the potential runtime performance win. Mouse XS accessors are significantly faster than pure Perl accessors you write by hand, (Mouse pure Perl ones are no slower). It's also handy to be able to list a class' accessors. This is used to allow the dumping of the data contents of an Event. Not strictly necessary, but it will be handy for the JSON formatter/dumper. > And I wonder if Role::Tiny could be extended to do multiple roles. > > Or could you do a poor-man's version of roles using mix-ins? > > I understand the goal of decomposition and why Mo[ou]se rocks, but for > a testing framework, maybe simpler is better. I don't think cobbling together an OO system from bits and bobs, and then relying on it, makes more sense than using a fully baked one. While Role::Tiny might load faster, there's no evidence it will compose roles faster. And there's no evidence loading a bunch of individual OO modules will be any faster than loading Mouse::Tiny. Also one of the side benefits of shipping TB2::Mouse is now other Test modules can rely on it getting the benefit of a fully operational OO system without adding further dependencies. Finally, it's a whole lot of regressive work to tear Mouse out. I'd rather do some investigation into performance optimizations in TB2 and Mouse first before we go into writing YET ANOTHER PERL OO FRAMEWORK, whe! I wonder if I can find an excuse for TB2 to need a template module... ;-) > For example, Moose will detect method clashes with roles, right? (I > assume Mouse does that.) Do you really need that? Or could you > verify the design "by hand" rather than relying on a MOP to do the > work for you? I actually don't know. I sure don't need it. But this presumes that's where the performance problem is. > How much of it is fundamental to what you're trying to do and how much > of it is just making life easier for you in implementing it? Granted, > well tested Mouse might be less buggy than hand-rolled replacement, > but a special-purpose class framework could possibly be simpler (thus > faster and less buggy). None of it is fundamental. All of it is making life easier. -- Reality is that which, when you stop believing in it, doesn't go away. -- Phillip K. Dick
Re: Relying more on Mouse
On 2011.11.22 11:02 AM, Eric Wilhelm wrote: >> By being THE testing framework, it places an upper bound on how fast >> anyone's tests can be. 10 .t files per second, no faster. That >> sucks. > > I agree. But, with XS mouse, you're only cutting the startup time to > 0.07s from 0.09s, correct? And really, 14.3 .t/s also sucks vs the > 50 .t/s with v0.98, no? That's before any further optimization is done. XS Mouse buys us 20ms before we even start. > Is there a way to remove some of the work Mouse is doing at startup? > What is it doing? I don't know about Mouse, but I know Test::Builder2 is eating 13ms building up all the different types of Results at compile time in TB2::Result. Results are over-engineered. I have a not-too-old branch to simplify them. Some of the time is spent applying roles and generating accessors and there isn't much to be done about that in Test::Builder2. Performance enhancements in Mouse itself would be appreciated by all. I've added a branch called "gonzales" which uses Mouse directly. You can experiment with that and NYTProf to examine compile time performance issues yourself. https://github.com/schwern/test-more/commits/gonzales > I did once suggest a pre-loading / forking harness (along the lines of > Test::Aggregate), though it would probably be much better to make the > test code start faster rather than need to work around it. That's an interesting idea. FWIW Test::Aggregate should become much easier with Test::Builder2. Instead of the shenanigans it goes through to merge tests together, it can just intercept set_plan events and turn them into subtests or just scrub them out entirely. -- 184. When operating a military vehicle I may *not* attempt something "I saw in a cartoon". -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Relying more on Mouse
On 2011.11.22 11:22 AM, David Golden wrote: > On Tue, Nov 22, 2011 at 2:02 PM, Eric Wilhelm wrote: >> Is there a way to remove some of the work Mouse is doing at startup? >> What is it doing? > > How much of Mouse is needed? Could Moo be used? (I ask without > having read the details of the OO breakdown of TB2) No. Moo has runtime dependencies, is harder to bundle (multiple files), is not upward compatible with Moose (their type system is incompatible), lacks Moose types, a class can't use more than one role at once, and has no meta system. And it's slower at runtime. I'm also not convinced of Moo's stability. Mouse has strictly adhered to being a subset of Moose. Moo hasn't. The Mouse folks have said they'd test vs Test::Builder2. Finally, Moo has only existed for a year. About the only advantage is Moo loads about 10ms faster than Mouse/XS. The major Mo[ou]se features that Test::Builder2 uses heavily is roles and attributes. It also uses types, coercion, meta classes and method modifiers, but those could be removed if necessary. I've tried to keep it simple to avoid touching any actual or potential Mouse bugs. -- There will be snacks.
Threads working, Test::Builder1.5 is feature complete
Threads are now working in Test::Builder1.5. That's the last missing feature, Test::Builder1.5 can now do everything 0.98 can! Huzzah! If anyone want to see how much working with threads suck, just have a look at this hack. https://github.com/schwern/test-more/blob/Test-Builder1.5/lib/TB2/ThreadSafeFilehandleAccessor.pm Anyhow, there's lots of stuff to do before stable. Some of it is in Test::Builder2, but a lot of it is fixing broken test modules that violated encapsulation. :-/ https://github.com/schwern/test-more/issues?milestone=6 There's a bug in use_ok() that effects 0.98_01 and 1.5. So I'm going to hold off on a new alpha for a day or two and it's either fixed or I'll roll it back. https://rt.cpan.org/Ticket/Display.html?id=67538#txn-1002509 -- 124. Two drink limit does not mean first and last. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Relying more on Mouse
On 2011.11.21 8:30 AM, David Golden wrote: > My bigger concern would be inclusion of Mouse in core as a dependency, > since the direction of Perl seems to be to have fewer core modules, > not more. I'd run that discussion by p5p/Ricardo before getting too > tied to Mouse. Mouse won't be in the core. TB2::Mouse (a copy of Mouse::Tiny, a single 114K file) will, marked only for use by testing modules. This was discussed a long time ago with the Mouse folks and p5p, it's been the plan for quite some time. Neither party wanted Mouse in the core, but they were ok with a copy of Mouse::Tiny for internal use only. > Separately -- once the Perl core gets a MOP, maybe this gets > easier/faster anyway. Great, with my backwards compatibility requirements maybe I'll be able to use it in TB3 10 years from now! :-P -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting annoying in email?
Re: Relying more on Mouse
On 2011.11.21 4:07 AM, David Cantrell wrote: > But then how often does one need to 'use Test::More'? Not enough to > bother optimising it, I'd say. In every single .t file that gets run by just about everybody. By being THE testing framework, it places an upper bound on how fast anyone's tests can be. 10 .t files per second, no faster. That sucks. > To take a real-world example, it occurs 182 times in our test suite at > work, a test suite that takes almost 2 hours to run in full. Those > extra 182 * 0.07 == 13 seconds are of absolutely no importance at all. I run tests a lot while developing. I can see they're slower without even benchmarking it. You wouldn't think you'd notice 70 ms, but you do. I don't want to give anyone an excuse to not run tests often or not upgrade. -- Just call me 'Moron Sugar'. http://www.somethingpositive.net/sp05182002.shtml
Re: Relying more on Mouse
I did some profiling and easy optimizations which sped things up quite a bit, but it didn't translate into real world improvements. Turns out the real problem is startup time. use Test::More; 0.98 real0m0.021s user0m0.016s sys 0m0.004s 1.5 real0m0.092s user0m0.083s sys 0m0.008s That's going to be a tough one to cut down, simply because Test::Builder1.5 contains so many more individual files to load and attributes to set up. There's some things it's doing at compile time that can be hard coded (like setting up all the Result subclasses), but what shaves off a good chunk is using Mouse with XS. 1.5 with XS Mouse real0m0.070s user0m0.061s sys 0m0.007s So that's another point in its favor. -- What we learned was if you get confused, grab someone and swing them around a few times -- Life's lessons from square dancing
Relying more on Mouse
Test::Builder1.5 is slow. How slow? About 3x slower than 0.98. Enough to significantly impact test performance. For example, Spiffy takes 1 second with 0.98 vs 3.3s with 1.5. I deliberately didn't do any profiling or optimization while the design was coming together. This avoided spending time optimizing something that might just be thrown out, and it focused on getting things working. And while my policy has always been that performance is not a priority, a 3x drop is unacceptable. use Test::More; pass() for 1..1; done_testing; 0.98 real0m0.747s user0m0.726s sys 0m0.019s 1.5 real0m2.301s user0m2.271s sys 0m0.027s One thing which I know for sure is slowing things down is its use of a pure Perl Mouse::Tiny. Test::Builder1.5 has a copy of Mouse::Tiny bundled as TB2::Mouse. This both avoids a dependency on Mouse and protects Test::Builder against Mouse bugs. Test::Builder 0.98's performance benefits from being a monolithic object which can quickly grab at it's own guts. Test::Builder1.5 is well factored out into objects with attribute accessors. Pure Perl accessors can't compete with hash lookups. But Mouse boosted with XS can. 1.5-gonzales real0m1.702s user0m1.675s sys 0m0.025s Just switching over to my installed Mouse with XS shaved off a big chunk of the runtime. Enough that I'm confident profiling can knock off the rest, there's a lot of easy fat in there. I've already lopped it down to 1.368s with two small changes. The idea would be to continue to ship a copy of Mouse::Tiny as TB2::Mouse, that takes care of the dependency loop, but to only use it if a good enough version of Mouse is not already installed. To mitigate risk of a new version of Mouse breaking Test::Builder, I'd ask the Mouse folks to test their releases against Test::Builder1.5. I think they're already doing that. Thoughts? -- 60. "The Giant Space Ants" are not at the top of my chain of command. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Dual life t/test.pl?
On 2011.11.17 4:08 AM, Nicholas Clark wrote: >> 1) It's not really my goal to make it distributable as a CPAN module, >>but as just something you copy. > > It continues to be my goal to reduce the amount of effort needed to support > the core. Exposing more of the internals runs counter to this. > >> 2) Test::More would lose the benefit of improvements from p5p. >> >> 3) p5p would lose the benefit of improvements from Test::More. > > The core is using Test::More for pretty much every core-maintained test > outside of t/ > > One of the things I've been working on slowly is trying to turn "pretty > much" to "every", by removing each impediment in turn. > > p5p continues to benefit from Test::More, and certainly I'm grateful for > your continued work on that. (I assume everyone else is) Sorry, I think I wasn't clear. It wasn't meant as a "I'm taking Test::More and going home" screed. I was only commenting on FC's suggestion to fork t/test.pl. That is, when Test::More patches its copy of t/test.pl p5p can benefit. And when p5p patches t/test.pl Test::More can benefit. That only works if I keep them in sync. >> The library upon which 372 core tests rely is undocumented and untested. >> Moving it into its own repo and tracker allows it to be tested and stable >> without having to go through the p5p memorial bike shed. > > That's the entire bloody point. It's not *meant* to be assumed to be stable. > It's an internal implementation detail of testing the core. I think this is the key point on which we fundamentally disagree, and we can't go further without clearing this one up. If I'm hearing you correctly, your argument is that t/test.pl is internal to the core and does not need to be documented or tested. Docs are for publishing stable APIs. Tests are for checking regression against those stable APIs. My argument is that in order to write tests using t/test.pl, core developers need to know how to use it. That's what docs would be for. What does fresh_perl() do, what are it's arguments and limitations? What about run_multiple_progs()? display()? within()? like_yn()? That t/test.pl lacks documentation is a barrier to core developers writing tests. Just because it's internal to the core doesn't mean people magically know how to use it. Tests for t/test.pl have a similar argument. They make sure that the basis for our core testing system works and continues to work. They ensure that changes to how it works are deliberate and not accidental. Docs and tests are something t/test.pl needs, dual-life or not. If we don't agree on that, then this isn't going anywhere. -- 29. The Irish MPs are not after "Me frosted lucky charms". -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Test::Builder 1.5 first alpha
Gentlemen, start your smokers! https://metacpan.org/release/MSCHWERN/Test-Simple-1.005000_001 Thanks very much to the contributors who stepped up: * James E Keenan * Jason Galea * Nóirín Plunkett * Aaron Crane * chromatic * Father Chrysostomos * Larry Leszczynski * Michael Ludwig * Mike Doherty * Pedro Melo * Shawn M Moore * Shawn Sorichetti I'm going to go sleep for a while now. 1.005000_001 Thu Nov 17 01:38:53 PST 2011 This is the first alpha release of what has been termed Test::Builder1.5. It is Test::Builder implemented with the Test::Builder2 internals (formatters, event system, etc...). Aside from thread support, it is feature complete and supports all existing features of Test::Builder, Test::Simple and Test::More. Tests should continue to work as written. Please report any breakages at https://github.com/schwern/test-more/issues/ Incompatible Changes (since 2.00_07) * A lot, too many to list. If you were using an earlier alpha, all bets are off. Incompatible Changes (since 0.98) * The test output format has changed in small ways. A later release will provide the means to format the output just like 0.98 does now. -- THIS I COMMAND!
Subtests work! New alpha imminent.
Subtests are now 100% operational in Test::Builder1.5. It slipped in place very nicely with the new event system, which makes me very happy, and it's much simpler than it used to be. The only vestige of the old system I had to retain was tracking TODO tests since that's still done in the Test::Builder object. Thank you Ovid and others for writing such excellent and comprehensive tests. Before the next alpha can go out, I just have to finish merging from master and do a handful of major class/method name changes so early adoptors don't get screwed again next alpha. https://github.com/schwern/test-more/issues?milestone=5 It looks like Jason Galea stepped up and did the refactorings. There's just one small one left. I'd appreciate a taker. https://github.com/schwern/test-more/issues/210 Then I can collapse with the flu while you all knock around the alpha. -- 170. Not allowed to "defect" to OPFOR during training missions. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Task: Write a TAP v12 formatter for Test::Builder1.5
On 2011.11.16 4:19 PM, James E Keenan wrote: > Should we proceed on the assumption that this specifies v12? > > http://testanything.org/wiki/index.php/TAP_specification > > ... and that this specifies v13? > > http://testanything.org/wiki/index.php/TAP_version_13_specification No, not for our purposes. YAML diagnostics are not implemented. I thought about it some more, and the v12/v13 split isn't what we're really going for here. Rather than calling it the v12 formatter, it should more properly be called the legacy or TB1 formatter. It's not so much about producing TAP v12 as it is replicating what Test::Builder currently outputs. I can break the issue into two parts. Write a v12 TAP formatter (really v12 this time). This will serve as an easy way to split the formatter into extensible pieces, as all the code currently lies in TAP::v13. This is really straightforward. https://github.com/schwern/test-more/issues/219 Write the legacy formatter as a subclass of TAP::v12. This mostly involves putting the strings TAP uses into attributes and making a subclass of TAP::v12 which changes those defaults. https://github.com/schwern/test-more/issues/215 -- 184. When operating a military vehicle I may *not* attempt something "I saw in a cartoon". -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Task: Write a TAP v12 formatter for Test::Builder1.5
I have an important task for getting Test::Builder 1.5 stable. Test::Builder 1.5 outputs TAP version 13. There are minor formatting changes including a TAP version header and changes to the ending commentary. A lot of tests look at this output and so they break. Rather than make everybody change their tests (and have to upgrade their Test::More dependency), I'd rather provide a legacy TAP formatter that outputs like Test::Builder currently does. Then folks can switch it on in their tests, and Test::Builder::Tester will use it by default. That's the task. Write a TAP version 12 formatter. https://github.com/schwern/test-more/issues/215 https://github.com/schwern/test-more/wiki/Preferred-workflow It's a mildly difficult task, mostly because the v13 formatter does things which really should be done in a superclass, but it's way easier than trying to do it in Test::Builder 0.x. If anyone would like to take it on, please say so in the ticket and we'll try to get things off the ground. Thanks. -- 164. There is no such thing as a were-virgin. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Dual life t/test.pl?
First thing to clarify, I don't really envision this being turned into a CPAN module. That would be directly against its purpose. The primary purpose here is to separate and focus development and provide a point for module authors to copy it into their project. It probably will *not* be released to CPAN and it probably will *not* be API stable. The needs of the core outweigh the needs of the few people that use it. On 2011.11.15 6:34 AM, Nicholas Clark wrote: > Other than people writing test modules, who would find it useful? Test::More is about to become a lot more complicated internally, so anyone who does things which Test::More might be using. In addition, .t files which run a bazillion tests would find it useful both for speed and memory usage. I just recommended Buddy Burden use it for some huge Data::Random tests. Abigail could use it for her enormous Regexp::Common test suites. But that's not really what it's about. The library upon which 372 core tests rely is undocumented and untested. Moving it into its own repo and tracker allows it to be tested and stable without having to go through the p5p memorial bike shed. It's a lot easier to get people to work on a single file Perl library than it is to get them to patch the Perl core. In its own project, I can hand a newbie a URL like this and say "pick one of these and do it!" https://github.com/schwern/test-more/issues?labels=Easy >> t/test.pl is very important to the Perl core tests, but it is largely >> undocumented and untested. Going through the p5p process to patch it makes >> working on it difficult and open to bike shedding. If it were dual-lived it >> could be worked on as its own thing with its own commits and tests. > > test.pl is a script, not a package. Our machinery for dealing with dual > life things is really only set up to deal with modules. So doing this is not > going to just drop in to what we have. It will exactly drop in to what we have. Pulling it into its own repository doesn't mean it has to be turned into a package and modularized and all that. It would probably become lib/test.pl and you just copy that into t/test.pl. >> By having a dedicated feed of issues and commits, careful attention could be >> paid to what code and features are going in. Close attention and discussion >> can happen about whether the code is simple enough and the features apt for >> the needs of the core. > > Surely the needs of the core *are* better determined by the core developers, > not some third party group? The groups would obviously overlap and we wouldn't be stingy with the commit bits. > Also, having upstream be anywhere other than the core actively gets in the > way of t/test.pl actually being useful to core developers (or at least to > me, as a core developer who has added things to it). We deal with dual life files all the time, the same procedure applies. If you do it a lot (and you have this year) get a commit bit. Make the changes in a sterile environment with tests. Copy it into blead. Or do it in blead and paste the commit ID into an issue on the project to get someone else to pick it up. To put this in perspective, t/test.pl was edited just 28 times this year. Nearly all of them were by you in March. > Even having it being upstream blead gets in the way of it being useful. > Right now, as all users of the core's t/test.pl are in the core, it's > possible to > > a: add things to it without stopping to think "is this a decent, supportable >API that can last forever", because if the API turns out to be wrong, >it's entirely possible to have a "flag day" commit that changes it and >all its users > b: remove things from it, because nothing is using them any more > c: not worry about what versions of perl it supports, or working round >problems in them >(Although to be fair, if a particular piece of perl syntax had been > buggy before, it's possibly something fragile that should be avoided in > case it breaks again for a different reason) > > Nothing in the above stops anyone taking a copy of test.pl and using it. > Doing this also doesn't add a layer of process into working on the core. I'm ok with all that. The goal is not API stability. Moving it to its own repo does not prevent any of a, b or c. >> If folks are cool with that, I can set up a Github repository for it. We'll >> worry about whether CPAN release is appropriate later. > > I'm distinctly uncool about doing it that way. > I'm unconvinced about doing it with blead upstream. I was thinking blead would be downstream. -- Being faith-based doesn't trump reality. -- Bruce Sterling
Re: Discuss change of namespace Test::Builder2 -> TB2?
On 2011.11.15 6:40 AM, Leon Timmermans wrote: > I'm not seeing the point really. By this logic we can reduce all > frameworks on CPAN to some three letter acronym. To be honest I don't > think Test::Builder is used directly often enough to justify that. Test::Builder was just one monolithic module. It isn't any more. There's lots of pieces now. People who develop Test modules that do interesting things will be typing it more often. People who don't won't even know anything has changed, so they're not really a concern. Elliot Shank uttered: > I'm against abbreviation; it makes things harder to read. Your > text editor's completion mechanism should take care of having to > typing out long names. When you make them up on the spot and use them twice, abbreviations are bad. When you repeat it 1300 times in the project... folks will figure it out. While I agree good editor skills are good, and "I have a crappy editor" isn't an excuse, getting an editor to autocomplete Perl class names is non-trivial. It gets into TAG files and having the editor be aware of the whole project and parsing Perl correctly and guh. I don't know how to do it. I don't expect contributors and users to know or have a copy of Perl Hacks. -- 101. I am not allowed to mount a bayonet on a crew-served weapon. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Problem with running lots of tests (I think)
On 2011.11.15 1:14 AM, Buddy Burden wrote: > Okay, just to follow-up in case anyone cared what the resolution on > this one was, changing the loop full of ok()s to one giant pass() or > fail() per loop fixed _everything_. Plus it runs a lot faster now. I > know I've seen test suites that do thousands and thousands of tests, > but they must be doing it some other way that I'm not privvy to or > somethin'. There is a 3rd option. The perl core maintains a separate implementation of Test::More in t/test.pl. You can drop that into your project and use it for your enormous tests. It does not record test results, that I know, and should not consume memory. I use it to test Test::More and am considering spinning it out into its own project. -- 94. Crucifixes do not ward off officers, and I should not test that. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: TAP - Test::More - fork
On 2011.11.15 1:01 AM, Buddy Burden wrote: > I did not know this ... just to be super-clear, obviously I know that > if I have script.pl and it starts with > > #! /usr/bin/perl -w > > and I make it executable and run it directly, I get perl -w. But > you're saying that even if I type: > > perl script.pl > > I _still_ get perl -w? That, I was not aware of. Yep. Now, want to have your mind blown? $ cat ~/tmp/test #!/usr/bin/python def hello(): print "Hello world!" return hello(); $ python ~/tmp/test Hello world! $ perl ~/tmp/test Hello world! >> Adding -w to the #! line on your tests is a very good idea to avoid the >> gotcha of differences between a test being run with `prove` (does not set >> -w), ... > > Well, yes, prove doesn't _normally_ set -w, but if you run prove -w, > it does, doesn't it? Yes. If you don't set it in #! you, or somebody else, will forget to pass -w to prove and then you'll spent a bunch of time scratching your head wondering why the test is running differently from in "make test". -- Just call me 'Moron Sugar'. http://www.somethingpositive.net/sp05182002.shtml
Smoker setup to compare old vs new versions?
I'd like a smoker setup which can do two things: 1) Run smokes for all the Test:: modules. 2) Compare the results between two different installed versions of Test::More. This will allow me to smoke Test::Builder 1.5 against CPAN, see what it's broken and try to fix it. I've never used the CPAN smoker software before and I'm hoping somebody has a recipe? Thanks. -- package Outer::Space; use Test::More tests => 9;
Dual life t/test.pl?
Having a parallel and featureful testing system is very useful. I use it to test Test::More (in the Test-Builder1.5 branch). Others might find it useful to do the same. t/test.pl is very important to the Perl core tests, but it is largely undocumented and untested. Going through the p5p process to patch it makes working on it difficult and open to bike shedding. If it were dual-lived it could be worked on as its own thing with its own commits and tests. By having a dedicated feed of issues and commits, careful attention could be paid to what code and features are going in. Close attention and discussion can happen about whether the code is simple enough and the features apt for the needs of the core. If folks are cool with that, I can set up a Github repository for it. We'll worry about whether CPAN release is appropriate later. The starting set of collaborators would be picked from the folks who use it and patch it and want a commit bit. -- 24. Must not tell any officer that I am smarter than they are, especially if it's true. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Discuss change of namespace Test::Builder2 -> TB2?
On 2011.11.14 12:41 AM, Philippe Bruhat (BooK) wrote: > I'm more annoyed with the version number being part of the name. > Even if I can understand the reason why (CPAN only knows one way to > upgrade: up). I used to be with you there. I've since found it's a remarkably simple and foolproof way to indicate an API split. The name, code and docs are neatly delineated. It works with all existing CPAN tools. Both API versions can exist in harmony. The name change tells the user this is not their father's Kansas. Besides, "tee bee two" rolls off the mouth nicely and TB:: is a bit too short. -- I have a date with some giant cartoon robots and booze.
Discuss change of namespace Test::Builder2 -> TB2?
I have written "Test::Builder2" more times than I can count. Wait, I can count them. It occurs at least 1280 times in the distribution. That's 18,000 keystrokes. 19,000 if you count shift. It gets a little tiresome. That's just the current count, who knows how many times I've retyped it. In the interest of saving my hands, and the hands of test module authors, I'm considering changing the namespace from Test::Builder2 to just TB2. Like LWP and DBI. Test::Builder2 has a lot of parts, as you can see here: https://metacpan.org/release/MSCHWERN/Test-Simple-2.00_07/ Here's some examples. Before After -- - Test::Builder2 TB2 Test::Builder2::History TB2::History. Test::Builder2::Event::SetPlan TB2::Event::SetPlan Test::Builder2::Formatter::TAP::v13 TB2::Formatter::TAP::v13 Test::Builder, Test::More and Test::Simple remain unchanged. This both saves wear on joints and it establishes a clear "keep outta my namespace" namespace for TB2 to expand into. 3rd party extensions can go into TB2X:: or just Test:: like normal. On the down side, it doesn't contain the word "Test". I'm not too worried as this is the guts of testing, and by the time you need it you should already have picked up enough Perl to find it. It also busts compatibility with the existing TB2 alphas... but that's what alphas are for. I've very well broken compatibility nine other ways this month anyway. Oh, and somebody has to do it. Thoughts? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting annoying in email?
Re: The end of 5.6 is nigh!
On 2011.11.13 8:39 AM, Reini Urban wrote: > I've come around your hammering lately and had this idea: > > Cannot CPAN add logic to avoid downloading your new versions on older > releases? Yes, but it is non-trivial. It would requiring creating a new index which supplies modules and versions for every release (aka tarball and zip) instead of just the latest one. Various folks are working on that, it is a key piece of missing information. And then you need another index of what the minimum version of each release is... if it's declared. And traverse the module's dependency tree to pick the newest version that is compatible. And hope all dependencies are compatible. And then you have to pray that's something like reality. There have also been approaches which use CPAN testers data to select the last compatible version. Contact Dave Cantrell about that. There is my CPANgloss.com project to provide this as a service, but it is not operational. > 5.6.2 still is the fastest perl around. As long as you keep making it > slower and fatter you have to come up with some kind of practical > solution. If you're not completely satisfied with your free software, you get TRIPLE your money back! Ya know, I'll bet DOS would run pretty fast on a modern machine... -- Life is like a sewer - what you get out of it depends on what you put into it. - Tom Lehrer
The end of 5.6 is nigh!
It's that time again! Time when I hammer the last few nails in the coffin of a version of Perl. By which I mean, the next major release of Test::More (aka Test::Builder1.5) will support 5.8.1 and up. ExtUtils::MakeMaker will probably go that way, too. This effectively cuts off most of CPAN from 5.8.0 and down. This will happen in the next few months. Test::More might nudge its requirement a little higher depending on just how difficult it is to work around early threading bugs in the earlier 5.8 releases. At this point I don't imagine this will cause too much disruption. There aren't many serious 5.6 users left and whomever is left has to have already come up with some sort of 5.6PAN solution. Dropping 5.6 will ease maintenance and testing of these modules and make a greater baseline of core modules available. However, if a company or organization would like to see 5.6 compatibility retained, they can contact me about sponsoring the extra work for continued maintenance. Thanks to Sarathy and rgs for some great and long lived releases! It's been a good eight years. Schwern Slayer of Pseudo-Hashes Defender of Lexical Encapsulation Destroyer of Perl Versions
Re: TAP - Test::More - fork
On 2011.11.10 7:44 PM, chromatic wrote:
On Thursday, November 10, 2011 at 05:57 PM, Michael G wrote:
If you don't want global warnings, explicitly turn them off with BEGIN { $^W
= 0 }.
I thought the argument that test modules should set global policy unilaterally
died out when I made Test::MockObject *not* enable UNIVERSAL::isa and
UNIVERSAL::can by default.
That argument happened well after Test::Harness was written to set -w. If
nothing else, it's a major compatibility issue.
It's the 21st century. Why can't Test::Harness find a nice middle ground and
respect the lexical encapsulation of warnings introduced in 2000? (I think it
just might catch on.)
We're discussing how to deal with the reality and smooth over the existing
differences between the various ways to run a test.
If you want to crack open the -w in testing discussion, start another thread
and lay out your argument.
--
Whip me, beat me, make my code compatible with VMS!
Re: TAP - Test::More - fork
On 2011.11.10 4:59 PM, Buddy Burden wrote:
chromatic/Merjin,
Not "use warnings" but the -w command line flag -- the non-lexical, warnings-
on-everywhere one.
no change whatsoever. I've now added -w to all #! lines in the t files
Does that do anything? I didn't think prove respected the shebang
line. Anyway, I thought the -w to prove would be effectively doing
that all along.
Perl respects the *options* on the #! line, prove only adds to them.
Adding -w to the #! line on your tests is a very good idea to avoid the gotcha
of differences between a test being run with `prove` (does not set -w), `make
test` (sets -w), and running it as a normal Perl program (does not set -w).
If you don't want global warnings, explicitly turn them off with BEGIN { $^W =
0 }.
Differences between "make test" and "prove" (was Re: TAP - Test::More - fork)
On 2011.11.10 7:15 AM, H.Merijn Brand wrote: Yes, there indeed is a core (gdb) where #0 0xc0258c70:0 in free+0x1d0 () from /usr/lib/hpux64/libc.so.1 #1 0x4017f7e0:0 in Perl_safesysfree () at util.c:262 #2 0x400d0ab0:0 in perl_destruct () at perl.c:871 #3 0x4009a740:0 in main () at perlmain.c:119 I however do not understand why prove seems to be safe, but make test is not First, try to replicate everything "make test" is doing with prove. For starters, even though test processes are supposed to remain separate, other tests could be interfering and the segfault only happens when you run the whole test suite. So run the whole test suite. prove -wbr t I see you're firing off child processes, if you're doing the same in the previous test it could be that they're not finished from the previous test starts. It could be that your database is in a bad state. The second major difference is "make test" sets the PERL_DL_NONLAZY flag which is helpful for XS testing, so add that into the mix. PERL_DL_NONLAZY=1 prove -wbr t If that doesn't work, go the other way. Strip out everything you can, including prove, and start adding things in. When you want verbose test output, don't use prove -v, just run the program. perl -w -Mblib t/21-uni-regex.t If that doesn't replicate, try adding in subtle things which are different between 'make test', 'prove' and running manually. Try having STDOUT go to a pipe instead of a TTY. perl -w -Mblib t/21-uni-regex.t | cat Try the same for input. echo | perl -w -Mblib t/21-uni-regex.t Try both. echo | perl -w -Mblib t/21-uni-regex.t | cat Or it could be that prove is setting environment variables which Test::Harness is not, check them. They could be affecting your shell command. -- 124. Two drink limit does not mean first and last. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Subtest progress
Subtests work in Test::Builder2. The design we worked out and documented here is implemented. https://github.com/schwern/test-more/wiki/Subtest-Design Test::Builder2->subtest exists and the TAP formatter understands subtest events. However, it has not yet been plugged into Test::Builder's subtests. Before that happens, all the test state must first be removed from Test::Builder (all those things in Test::Builder->reset) and put into either the History object or an event watcher. This is because the Test::Builder object is no longer swapped out when a subtest happens. Instead, the History and other event watchers are. I've set up a milestone to hole issues related to that. https://github.com/schwern/test-more/issues?milestone=4&state=open Fortunately, most of that has already been done previously as part of converting Test::Builder to use Test::Builder2 guts. I'm working on the last of it. The other big piece is moving the logic in Test::Builder->_ending out into event watchers and formatters. Subtest watchers must do that logic when they end, not Test::Builder. While I'm working on that, I'm constantly adding new small tasks to the issue tracker. If other people can do them that's more time I can focus on the hard stuff. Here's easy (at least, straightforward) things: https://github.com/schwern/test-more/issues?labels=Easy%2CTest-Builder2&sort=created&direction=desc&state=open&page=1 Docs: https://github.com/schwern/test-more/issues?labels=Docs%2CTest-Builder2&sort=created&direction=desc&state=open&page=1 Refactorings: https://github.com/schwern/test-more/issues?labels=Gardening%2CTest-Builder2&sort=created&direction=desc&state=open&page=1 James Keenan stepped up to the plate and took care of a few issues, thanks! I've also written up how to work with Git that's smooth for both sides. https://github.com/schwern/test-more/wiki/Recommended-Git-Workflow -- Insulting our readers is part of our business model. http://somethingpositive.net/sp07122005.shtml
Re: What does t/test.pl do?
On 2011.11.6 5:20 AM, James E Keenan wrote: >> You're absolutely correct that it has no docs. Some of this is to avoid >> using >> the inline POD feature, in case it breaks while developing the core. But it >> could be documented either in comments, or POD in an __END__ block, or (the >> safest) in a separate POD file. Documenting it, even a little bit, would be >> very helpful. > > If I or someone else were to take that on, what should I treat as my starting > point? (It's my impression that the t/test.pl in Perl 5.14.2 differs from the > one in the Test-Builder1.5 branch of your github test-more repository.) The Perl core is upstream, work on that in blead. test-more pushes all its improvements upstream to the Perl core. -- 39. Not allowed to ask for the day off due to religious purposes, on the basis that the world is going to end, more than once. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: What does t/test.pl do?
On 2011.11.5 5:23 AM, James E Keenan wrote: > Does anyone actually use t/test.pl? What does it do? What is it for? test.pl has the classic meaning of pl meaning "Perl Library" not the bastardized "Perl Executable". It's a simpler, parallel alternative to Test::More written using simpler Perl features less likely to break. It has most of Test::More's API, plus a few tricks of its own. Its original purpose was back in 2001 when Test::More was a bit more unstable (so was the Perl core) and testing was not entirely accepted. Jarkko started it to use in the most basic perl tests and invoking as few complicated Perl features as possible. I co-opted it to test Test::More for similar purposes. If part of Test::More broke (and it was rather broken during a lot of Test::Builder2 development) test.pl would continue to work. It also resolved the "test yourself" problem without complicated hacks. You're absolutely correct that it has no docs. Some of this is to avoid using the inline POD feature, in case it breaks while developing the core. But it could be documented either in comments, or POD in an __END__ block, or (the safest) in a separate POD file. Documenting it, even a little bit, would be very helpful. Mostly it acts like Test::More, but figuring out what the other functions do will be a matter of looking at the code and also how its used in the Perl core tests. -- 3. Not allowed to threaten anyone with black magic. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Document the delegator or the delegated?
On 2011.11.5 12:37 PM, Aristotle Pagaltzis wrote: > * Michael G Schwern [2011-11-04 04:45]: >> When doing delegation I have a documentation dilemma: do the API docs >> go in the delegator or the delegate? > > This is the classic reference vs tutorial dilemma, isn’t it? Not quite. I'd still like to have the complete API of the delegator available as one document. It's still reference vs reference. A ::Manual would be about working with the event whole system. If a user wants to know what methods they can call on TestState or EventCoordinator a ::Manual would not be the place. -- There will be snacks.
Re: Document the delegator or the delegated?
On 2011.11.3 8:44 PM, Michael G Schwern wrote: > When doing delegation I have a documentation dilemma: do the API docs go in > the delegator or the delegate? To answer my own question, I've decided to do the best of both worlds. I will document each object's native methods in it's own .pm file, which is easiest to maintain. Then a tool will extract the relevant bits of EventCoordinator's (the delegate) docs and insert them into TestState's (the delegator) docs. The result will be written as TestState.pod which `perldoc Test::Builder2::TestState` will display before it goes looking in TestState.pm. I know Pod::Weaver and Pod::Elemental can do this sort of thing, but their docs are pretty sparse. Someone want to tackle it? https://github.com/schwern/test-more/issues/172 -- You are wicked and wrong to have broken inside and peeked at the implementation and then relied upon it. -- tchrist in <31832.969261130@chthon>
Document the delegator or the delegated?
When doing delegation I have a documentation dilemma: do the API docs go in the delegator or the delegate? In this case it's Test::Builder2::TestCase which delegates to a stack of Test::Builder2::EventCoordinator objects. The user will primarily interact with TestCase objects. Only rarely should they ever touch an EventCoordinator directly. Do I put all the documentation in TestCase, because that's what the user will work with, and put a pointer to EventCoordinator? This presents a unified interface to the user... but it's harder to maintain. And if you DO need to use an EventCoordinator it's harder to know which methods are EC's and which are TestCase's. Do I document their native methods in their respective classes and point the TestCase docs at EventCoordinator? It's inconvenient to the user to have to jump between two class's documentation to learn how to use one class. But it is easier to maintain. It also conveys the true nature of things presenting a more accurate model (not necessarily more useful). Meanwhile, I'm going with documenting in place. It's easier while the API is evolving. Hmmm... sectional POD includes anyone? -- Look at me talking when there's science to do. When I look out there it makes me glad I'm not you. I've experiments to be run. There is research to be done On the people who are still alive. -- Jonathan Coulton, "Still Alive"
Re: Small pile of Test::Builder 1.5 refactoring tasks
On 2011.11.2 7:24 PM, James E Keenan wrote: > On 11/2/11 4:35 PM, Michael G Schwern wrote: >> Here's all the current gardening tasks. >> https://github.com/schwern/test-more/issues?labels=Gardening%2CTest-Builder2&sort=created&direction=desc&state=open&page=1 > > I will start to look at these ... but would these be suitable for Google > Code-In students? Yes, except GCI will be too late for most of them. GCI doesn't start until Nov 21st and runs through mid January. Most of that needs to be done ASAP. However, the "Gardening" and "Easy" labels can be used as a dynamic set of tasks for GCI. -- The interface should be as clean as newly fallen snow and its behavior as explicit as Japanese eel porn.
Small pile of Test::Builder 1.5 refactoring tasks
Designing the subtests made me realize a few things about the event system that would be better done a little differently. Mostly renaming things. It would help me immensely if people could do these refactorings so I can focus on the design. They should be straightforward, it's just time consuming. Here's all the current gardening tasks. https://github.com/schwern/test-more/issues?labels=Gardening%2CTest-Builder2&sort=created&direction=desc&state=open&page=1 Thanks! -- Just call me 'Moron Sugar'. http://www.somethingpositive.net/sp05182002.shtml
subtest design in TB1.5
https://github.com/schwern/test-more/wiki/Subtest-Design I've decided on a complete design for subtests and written it up, mostly to get it out of my head before I forget it. :-) Please let me know what you think. Here's a summary... In short, from a Test::Builder point of view, subtests do two things. 1. Set up a new test state isolated from the parent. 2. Communicate the result of the subtest back to the parent. Rather than have each EventHandler have a stack, each nested state has its own EventCoordinator (which contains all the EventHandlers). A stack of these coordinators are maintained by a TestState object (which doesn't exist yet) which delegates to the top of the stack. When a new test state is created, each EventHandler is asked to make a new handler for the subtest. Those are then attached to the new EventCoordinator. This isolates the subtest's state. Communication is accomplished by attaching the subtest's History object to a subtest_end event and passing this around to the parent's EventHandlers. They can do whatever they like with it and have as much information as the subtest did. This design tries to keep as much of the complexity of subtests away from the EventHandler and Formatter authors. Ideally an EventHandler does not need to be aware that it's nested while a Formatter just needs to know how deeply it's nested. -- emacs -- THAT'S NO EDITOR... IT'S AN OPERATING SYSTEM!
Re: Do we need subtests in TAP?
On 2011.10.30 11:15 PM, Eric Wilhelm wrote: > # from Michael G Schwern > # on Sunday 30 October 2011 20:30: > >> The current Test::Builder implementation is a mess and its design >> cannot go forward. They have to be gotten just right to ensure that >> not just nested TAP is supported, but nesting in other formats. Or >> if those formats don't have nesting, then linearizing the subtests in >> those formats. And event watcher (ie. plugins) authors have to be >> shielded from the complexity. > > Maybe have a null or default handler for the subtests -- possibly > allowing authors to use a base class or role which gives them a > subtest_start() returning the null handler and subtest_end() which just > forwards the summary as a single test event. > > Is there a second format being implemented to test this API? There's five. TAP v13, POSIX, Null, PlusMinus and SimpleHTML. [1] [2] Of them all, TAP is the one that's most fleshed out. POSIX has proven most useful as a disparate format. I would love to see a formatter for Jenkins/JUnit, [3] that would probably be the most useful and interesting. I'd also love to see one that just spat out events as a list of JSON objects [4], useful both for debugging and as a lossless format for communicating to a harness. [1] https://github.com/schwern/test-more/tree/Test-Builder1.5/lib/Test/Builder2/Formatter [2] https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/Formatter/SimpleHTML.pm [3] https://github.com/schwern/test-more/issues/158 [4] https://github.com/schwern/test-more/issues/159 -- 24. Must not tell any officer that I am smarter than they are, especially if it's true. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Do we need subtests in TAP?
On 2011.10.30 7:21 PM, David Golden wrote: > I haven't followed the T::B 2 work closely enough, so could I ask you > to please step back and explain the benefits of T::B 1.5 that is worth > stepping backwards in terms of capabilities? What I mean is that we > have TAP::Harness now that processes subtest TAP and we have a T::B > now that produces subtest TAP, so why stop? I'm not advocating one side or the other just yet. I'm investigating what subtests are used for and if they're worth the bother, because I don't use them. Ovid always took care of them. They're a lot of bother, but it looks like it would be even more bother to do them without nested TAP. The current Test::Builder implementation is a mess and its design cannot go forward. They have to be gotten just right to ensure that not just nested TAP is supported, but nesting in other formats. Or if those formats don't have nesting, then linearizing the subtests in those formats. And event watcher (ie. plugins) authors have to be shielded from the complexity. The end result is looking to be fairly simple, but that doesn't mean a lot of work didn't go into it. > This quote from the TB 2 docs scares me a little: "Test::Builder2 is > very generic and doesn't do a lot of the work you've probably come to > expect a test framework to do. This reduction of assumptions increases > flexibility and ensures that TB2 can remain the core of Perl testing > for another decade to come." > > That sounds an awful lot like second system syndrome. And -- hey, > that's great if it works -- I mean no offense is saying that. But > something that is *half* of a second system that loses a nice feature > of the first system seems a suboptimal outcome. D O N ' T P A N I C What you read is about the Test::Builder2 *class*. The class is separate from the Test::Builder2 framework. Sorry if that's not clear. The framework can do what Test::Builder does (with the current exception of subtests) and more. The Test::Builder2 class does a lot less than the Test::Builder class. I intended to provide a provide a pack of roles to add in more functionality, and people could write more. But I'm not worrying about the Test::Builder2 class right now. Test::Builder2 is a user friendly shell to generate test events. Much of the Test::Builder functionality has been (or will be) pushed down into event watchers. Test::Builder 1.5 will use that event system. The benefit is we can continue to have a unified testing framework, but write tests using whatever interface you like and output in whatever format you like. Plugins which effect the whole state of the test (such as Test::NoWarnings) become safer to write and continue to work even as testing styles become different. You can even write your own builder. TB2 is the opposite of the second system syndrome. The second system is traditionally overbuilt with too many features that nobody needs. While the event system might qualify, the Test::Builder2 class itself does not. -- 10. Not allowed to purchase anyone's soul on government time. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Do we need subtests in TAP?
On 2011.10.30 2:58 AM, Adrian Howard wrote: > I prefer the current subtests system for a few reasons: > > * With the new system I would have to re-write TAP streams from other sources > to match the numbering system of the current stream. This makes more work for > folk who are pulling in TAP streams from other boxes/processes. If we do go > this > route you would, ideally, need to have a "standard" system for renumbering/counting > streams. No, TAP::Harness would continue to parse subtest TAP. TAP from other tests could still be embedded by simple indentation. > * I cannot, in a general way, tell the difference between a TAP stream with > subtests and a linear TAP stream. This means I can do things like easily > report > whether certain levels of subtests are passing/failing. I'm using nesting > because the groups make logical sense. Throwing that information away in the > TAP output seems a bad thing to me - it's useful to analyse/view the results > as > well as organising the tests. Agreed. > * With no "standard" way of representing the nesting in the TAP stream output > - everybody who needs to manage any kind of test grouping / hierarchy has to > invent their own way of doing it. True. > * Test::Block kinda uses this system. Never really used by anybody. subtests > are already used by lots of folk. I think part of the reason why may be coz > of the more intention revealing output? That's part of what I'm trying to discover. > * How would a no_plan subtest merge into a planned stream? Just fine, thanks. It would require no work at all. Without the TAP formatting, a no_plan subtest is equivalent to just running some tests. -- ...they shared one last kiss that left a bitter yet sweet taste in her mouth--kind of like throwing up after eating a junior mint. -- Dishonorable Mention, 2005 Bulwer-Lytton Fiction Contest by Tami Farmer
Re: Do we need subtests in TAP?
On 2011.10.29 3:51 AM, Fergal Daly wrote:
> It seems like it's impossible then to declare a global plan in advance
> if you use subtests unless you go counting all the sub tests which is
> no fun,
Yes, that's a very good point.
use Test::More tests => 3;
subtest "first" => sub { ... };
subtest "second" => sub { ... };
subtest "three" => sub { ... };
The main count remains three no matter how the subtests change. I just use
done_testing() but for those who like to count tests, it is more convenient.
--
Don't try the paranormal until you know what's normal.
-- "Lords and Ladies" by Terry Prachett
Re: Do we need subtests in TAP?
On 2011.10.29 1:51 AM, Adrian Howard wrote:
> On 29 Oct 2011, at 09:18, Michael G Schwern wrote:
> [snip]
>> Do you find *blocks with their own name and plan* convenient, or subtests
>> which have their own separate test state (as currently implemented)
>
> This may be me being dim - but I'm not really groking the distinction you're
> making. Could you give an example of the kind of difference that you're
proposing?
It's *mostly* internal, but here's an example to explain the difference.
Here's how subtests currently work:
use Test::More;
pass("First test");
subtest 'An example subtest' => sub {
plan tests => 2;
pass("This is a subtest");
pass("So is this");
};
pass("Third test");
done_testing();
__END__
This would produce.
ok 1 - First test
1..2
ok 1 - This is a subtest
ok 2 - So is this
ok 2 - An example subtest
ok 3 - Third test
1..3
As far as the state of the test is concerned, there's not a lot of difference
between
subtest 'An example subtest' => sub {
plan tests => 2;
pass("This is a subtest");
pass("So is this");
};
and
pass("An example subtest");
The subtest is one test. Its internals have no more effect on the overall
test than any single test, whether it runs one or one hundred.
What I'm asking would it make much difference to how subtests are used if it
was like this:
use Test::More;
pass("First test");
subtest 'An example subtest' => sub {
plan tests => 2;
pass("This is a subtest");
pass("So is this");
};
pass("Last test");
done_testing();
__END__
This would produce.
ok 1 - First test
# An example of a subtest
ok 2 - This is a subtest
ok 3 - So is this
ok 4 - Last test
1..4
Now the subtest is just a block with its own name and plan. The plan is
internally enforced by Test::Builder, TB makes sure that two tests are run
inside the block. Otherwise tests run inside a subtest are no different from
any other.
Another example is:
subtest 'An example subtest' => sub {
plan tests => 2;
plan skip_all => "Because" if $something;
pass("This is a subtest");
pass("So is this");
};
would become exactly equivalent to:
SKIP: {
skip "Because", 2 if $something;
pass("This is a subtest");
pass("So is this");
}
It becomes just a better way of writing a skip block.
David already pointed out that if you're the sort of person who eyeballs TAP
(I do), the indentation is a nice visual distinction. Anything else?
--
Alligator sandwich, and make it snappy!
Re: Do we need subtests in TAP?
On 2011.10.28 6:52 AM, David Golden wrote: > Without looking at the actual code, I would guess that the complexity > is implementing subtests while preserving the legacy procedural > interface that wraps calls to a global singleton. No, that's not really the problem. It was when Ovid originally implemented subtests. In 1.5 the singleton will not contain test state, that will be in event handlers held by the EventCoordinator. So the Test::Builder singleton does not have to change like it currently does. The problem is all the event handlers need nested state. Formatters have to know how to be nested (ex. indentation for TAP or not writing the header/footer for XML). History is no longer a linear set of events and results but a tree. User written event handlers (Test::NoWarnings is one example) also have to cope without pushing complexity onto the test module authors. And if subtest state is implemented as a new object (which it's looking to be) then the result of the subtest has to be communicated back to the parent and assimilated. I finally have solutions for these which don't suck, but I like to solve complex problems by eliminating the problem if possible. So the question is if we need *subtests* meaning a block of tests which is an entirely separate state or if we just need named blocks with a separate plan? This question is orthogonal to the complexity of implementation and it only applies to what Test::Builder produces, not what TAP::Harness reads. With the 1.5 architecture changes in test state are now granular events, rather than it happening all over the place. Subtests are looking easier than I thought. -- Whip me, beat me, make my code compatible with VMS!
Re: Do we need subtests in TAP?
On 2011.10.28 12:23 AM, Ovid wrote: > Echo chamber alert: I've often seen long discussions on this list ignore > the "real world" (though often for good reason). In this case, it sounds > like there's a consideration of removing a feature from TAP. No, not removing from TAP but removing support for producing it from Test::Builder. > I also find subtests so incredibly > convenient and opens up so many possibilities that I would hate to lose them > (and I use them a lot). Do you find *blocks with their own name and plan* convenient, or subtests which have their own separate test state (as currently implemented)? -- "Clutter and overload are not an attribute of information, they are failures of design" -- Edward Tufte
Re: Event handling: One method per event or one method for all?
On 2011.10.27 12:36 AM, chromatic wrote: > I've done this several times. I prefer one method per type of event (hello, > cheap and easy polymorphism!). Yeah, that's my thinking. One method for all makes overriding next to impossible. > This is also one case in which I find an abstract base class acceptable; the > null method pattern means never having to call ->can(). I would agree, but the set of possible events is not finite. New events will be added in the future, and it's actually kosher for users to define their own events. It would suck if older watchers vomited and stopped the whole test for an event it didn't care about. I'd considered adding an AUTOLOAD to make null methods... but yeah. It also makes it difficult to trap all events, as History has to do. It would have to know about and override AUTOLOAD. Bleh. PS The performance spread we're talking about is in the 10-20% range. That's discounting anything the watcher might DO with the event, so in practice it will be negligible. Multiple event handlers with a fallback to accept_event() is the slowest, but arguably the best design. -- 184. When operating a military vehicle I may *not* attempt something "I saw in a cartoon". -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Event handling: One method per event or one method for all?
tl;dr version: Is it better for event handlers to have one method per type of
event? Or one method to accept all events? Or something else?
Currently event handlers (called EventWatchers) in Test::Builder2 implement
two methods: accept_event and accept_result. accept_result() is a special
case to handle results (which are special events) and accept_event() takes
care of everything else.
This means you wind up writing stuff like:
my %event_dispatch = (
"stream start" => "accept_stream_start",
"stream end"=> "accept_stream_end",
"set plan" => "accept_set_plan",
"log" => "accept_log",
);
sub accept_event {
my $self = shift;
my($event, $ec) = @_;
my $type = $event->event_type;
my $method = $event_dispatch{$type};
return unless $method;
$self->$method($event, $ec);
return;
}
Which isn't too bad, but it seems like make-work.
The alternative is for the event handlers to have a method for each event,
(ie. accept_stream_start, etc...) falling back to accept_event if there is no
specific handler.
my $method = type2method($event->event_type);
$handler->can($method) ?
$handler->$method($event) : $handler->accept_event($method);
I chose a single method for all events for two reasons:
1) Speed. The event coordinator doesn't have to call can() on every handler
and every event.
2) It makes it easy to write a handler that slurps up all events, like the
History object.
It turns out #1 can be made moot with some careful programming. #2 can be
handled by falling back to a generic handler.
So now the question is one of good event handler design, which I don't have
experience with. How does one design a good event handler? Is the pattern
one method per type of event? Or one method to do the dispatching? Or
something else?
--
Being faith-based doesn't trump reality.
-- Bruce Sterling
Re: Subtest design in Test::Builder 1.5
On 2011.10.26 12:55 PM, Eric Wilhelm wrote:
> To be specific, I picture the interface as one of:
>
> a) subtest_start() must return the subtest handler object
> (might be a new object, or just $self)
>
> or:
> b) subtest_start() must return the $invocant, $method, %args for
> construction of the subtest handler object
> (might be [$classname,'new', foo => 'bar'] or [$self, sub{shift}])
>
> (Where "return" might be in some way besides return() if warranted.)
>
> I think b is a better way to go if you want to keep more flexibility at
> the framework level. It might seem like a trivial difference, but it's
> really sausage or bacon. By deferring the execution, you will be able
> to put things before the construction/swap, add arguments to the method
> call, and other fun stuff within the framework's scope. If you go with
> a, the constructor already executed and the sausage is already ground.
Given that the event coordinator can't know anything about the interface of
the method they're calling on $classname, the event coordinator can't add
arguments. So I don't see the utility of B.
And given that the logic inside the event coordinator will look something like:
my $subtest_handler = $handler->accept_subtest_start;
The coordinator can do whatever it likes before triggering the handler as well
as after. And the handler can do whatever it likes to generate the subtest
handler.
I don't see the utility of the extra complexity of B in this case. It seems
it adds no extra flexibility, potential or real.
>> That's... almost Orwellian. Strictures are flexibility!
>
> For a long-term framework design, I think what you have seen with
> e.g. 'ok or diag' vs stop-on-failure is very telling. If the execution
> is ceded to the caller (sausage), you limit what features the framework
> can add in the future. But, whenever the "do this" is brought back
> into the scope of the framework in some composable fashion (i.e.
> deferred execution), the framework can add before/after/around (or even
> skip?) features later because you have that extra bit of control.
Sure, I get what you're saying. I couldn't help the 1984 dig. :-)
--
170. Not allowed to "defect" to OPFOR during training missions.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/list/
Re: Do we need subtests in TAP?
Adrian forgot to send this to the list. Original Message Subject: Re: Do we need subtests in TAP? Date: Wed, 26 Oct 2011 14:14:31 +0100 From: Adrian Howard To: Michael G Schwern Hey there, On 26 Oct 2011, at 04:56, Michael G Schwern wrote: > I understand wanting "blocks of tests" and the ability to make plans for just > those blocks, but do we need a discrete test state for that? For example, > Test::Class provides most of what subtests provide without special support. ... and to do that T::C contains a bunch of annoying special case code than is (I think) still wrong in an odd corner case. Everybody who wants to do the things T::C does will also have to do that work. T::C implemented with subtests is _much_ cleaner code. There may be other ways of getting that complexity out of T::C (and similar) and into Test::Builder of course - but I'm not 100% sure what you're suggesting... > It occurred to me because most other testing systems don't have subtests. > They have test methods, but those aren't treated as a sub-state of the test. Some do have different levels of hierarchy though. (e.g. JUnit's Test Case / Suite distinction). > In essence, a "subtest" is nothing more than a block of tests with a name and > it's own plan. The special TAP formatting seems unnecessary. I guess that's > the real question, do we need the special TAP formatting or do we just need > named test blocks with their own plan? One thing subtests TAP formatting gives you is a simple way to nest TAP streams from elsewhere. Any other system would mean you have to rewrite the nested stream (I think?) Cheers, Adrian -- Who invented the eponym?
