Re: rantTesting module madness
On Sat, Sep 10, 2005 at 07:56:11PM +0100, Fergal Daly wrote: What do you mean has become all the modules you encountered are at least 2 years old but apart form anything that's a bit like coplaining about how complex the world of coputers has become, it used to be that I could just switch on, press Shift-RunStop, press play on my tape desk, make a cup of tea and then start playing whatever Commodore 64 game I wanted. It's all so complex now with ADSL, hard drives, wireless, linux distributions and web browsers. I have to agree with Fergal here. While there are modules which have unnecessary dependencies and could use fixing, complaining about dependencies in general is trying to roll back the clock to a day when each application shipped everything. Remember when DOS games came with their own hardware drivers and you had to hope it supported your sound card and configure each game individually and they each had their own audio bugs? Wasn't that FUN?! Wouldn't it be great if CPAN modules were like that? Modularity, its the biggest idea to hit software engineering in the last 40 years. Its not going away. PS You may be interested in http://search.cpan.org/dist/MegaDistro (warning, very alpha). Give it a list of CPAN modules and it spits out an rpm, dpkg or tarball with the modules installed. -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern You know what the chain of command is? It's the chain I go get and beat you with 'til you understand who's in ruttin' command here. -- Jayne Cobb, Firefly
Re: rantTesting module madness
Fergal Daly wrote: There is a genuine problem here though and that's the fact that MakeMaker and possibly Module::Build don't allow you to specify testing requirements separately from building requirements and run time requirements but most people don't ever see it thanks to CPAN.pm. The current CVS version (0.27 to be) of Module::Build does allow 'test_requires', 'test_recommends', etc. Actually, it allows per-action (build, test, install, etc.), as well as run-time requirements recommendations to be specified. Also, they are now evaluated when the action is invoked rather than evaluating all requirements up front so it wont barf on a missing test requirement if you are only performing a build, etc. This information is also (or will be) recorded in META.yml, so CPAN.pm CPANPLUS can make use of it if they choose. The current MakeMaker has support for adding arbitrary data to META.yml, so it can also provide info to installers. However, I doubt MakeMaker itself will every know anything about or use such requirements recommendations. Randy.
Re: rantTesting module madness
You wrote: This is a mini-rant on how complex the tesing world for Perl modules has become. It starts harmless, like you want to install some module. This time it was CPAN-Depency. Since for security reasons your Perl box is not connected to the net, you fetch it and all dependencies from CPAN and transfer them via sneaker net and USB stick. It includes some gems like: 'Test::Deep' = 0, 'Test::Warn' = 0, Huh? Never heard of them, but if it needs them, well, we get 'em. Presumable they are only needed for testing the module, but who knows? However, as you soon find out, Test::Deep needs these two: Test::Tester = '0.04', Test::NoWarnings = '0.02', Put on your high-speed sneakers, grumble shortly and fetch them. Hey! Don't blame me on that! At least I listed the dependencies. There are still many modules that even don't list the test modules they are using. I think I have already changed them from required to recommanded for the next, unreleased version, and skip gracefully if they're not present. -- Sébastien Aperghis-Tramoni Close the world, txEn eht nepO.
Re: rantTesting module madness
Either they're valuable enough that you install their prerequisites or they're not. But how am I supposed to find this out? I dont even know whether the required modules are used for the tests only, without digging through the source... Usually, Test::* modules are only used for the test phase. Maybe there could be some sort of bundle installer that grabs a module and all of its dependencies for people who do offline installations. That might be a great thing. But I'd still need to install all the testing modules just to run the tests prior to install the module itself. Hm. It should be possible to take the offline-prepared bundle and run the testsuite, and only then install the module and the absolute nec. dependencies. What we also miss are the test_requires/test_recommands in Module::Build. This was discussed at some time but I don't remember whether they were added or not. This doesn't solve the problem when you're using Makefile.PL but it can give you hints. -- Sébastien Aperghis-Tramoni Close the world, txEn eht nepO.
Re: rantTesting module madness
Usually, Test::* modules are only used for the test phase. I really don't understand the idea of only used for the test phase, as if the tests don't matter, or if there are levels of failure. Either they install OK on the target system, and you can use them with confidence, and they've done their job, or you're going to ignore the tests completely and then who needs 'em? It's like if I'm installing a washing machine, and I don't have a level. I can say Ah, I only need it for the installation, and it looks pretty level, so I don't need the level, or I can say I'm not using this appliance until I've proven to myself that the machine is level and won't cause me any problems in the future because of an imbalance. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: rantTesting module madness
Andy Lester wrote: Usually, Test::* modules are only used for the test phase. I really don't understand the idea of only used for the test phase, as if the tests don't matter, or if there are levels of failure. Either they install OK on the target system, and you can use them with confidence, and they've done their job, or you're going to ignore the tests completely and then who needs 'em? It's like if I'm installing a washing machine, and I don't have a level. I can say Ah, I only need it for the installation, and it looks pretty level, so I don't need the level, or I can say I'm not using this appliance until I've proven to myself that the machine is level and won't cause me any problems in the future because of an imbalance. I've always thought tests passing should be a requisite of make by default. Make fails if tests fail. There should of course be a -skiptests to opt out of testing for those who insist on not doing it, but for the most part, if tests are so important like we the perl community say they are, then they should be run as part of make. Period. Not a popular opinion, but there it is. -=Chris smime.p7s Description: S/MIME Cryptographic Signature
Re: rantTesting module madness
Andy Lester wrote: Usually, Test::* modules are only used for the test phase. I really don't understand the idea of only used for the test phase, there is clearly a distinction between the code required for a given module to compile and run in a production environment and the code required to make sure a suite of tests runs. as if the tests don't matter, well, tests mean different things to different people. or if there are levels of failure. I wouldn't take that stance, but lots of people do. oh, that test is supposed to fail is still heard all the time, even though those folks hanging out here know how bad something like that really is. Either they install OK on the target system, and you can use them with confidence, and they've done their job, which is really separate from the running the tests, now isn't it. take something as ubiquitous as apache. nobody runs the test suite (mainly because it doesn't come bundled with the software) but people install and run it in droves.do you run tests when installing, say, fedora? no, but you install the software anyway. would you still install it if you ran the tests and some failed? of course (and don't tell me you wouldn't since I'll bet everyone here has had failing perl tests in the past but we all still used it anyway :) now, say that the perl test suite required you do go out and install, say, CuTest. would you say to potential perl users the build process will fail if you don't have the stuff to run the tests or you shouldn't have confidence in perl if you didn't run the tests (even though we made you go through lots of hoops to do so)? I've known many an SA who would say something like LWP has been around for ages, seen many eyes, and if there are any bugs that we see that nobody else has caught yet we'll deal with that later... so I don't bother with 'make test'. you can argue whether that is a sane practice or not, but people do take that position. or you're going to ignore the tests completely and then who needs 'em? well, I'd argue that user-land tests exist (in part) to give me, the developer, and idea of what the issues might be if they report a bug. but even if the users ignore the tests completely that doesn't mean they're meaningless - they still ease the development process, and allow me to say ok, you're having this problem, what does this test say in verbose mode? It's like if I'm installing a washing machine, and I don't have a level. I can say Ah, I only need it for the installation, and it looks pretty level, so I don't need the level, I'd say that's the opinion of the vast majority of SAs I've ever met. or I can say I'm not using this appliance until I've proven to myself that the machine is level and won't cause me any problems in the future because of an imbalance. yeah, well, you could say that. last time I installed my washer I said looks pretty level to me, but I know where my level is if it makes a racket :) --Geoff
Re: rantTesting module madness
yeah, well, you could say that. last time I installed my washer I
said
looks pretty level to me, but I know where my level is if it makes
a racket
That's fine, but I'm still not shipping my washing machines without
explicit instructions to level the damn thing. Similarly, I'm not
making any of my tests optional except in the case of tests where
they don't affect direction operation, as in t/pod{,-coverage}.t.
xoxo,
Andy
--
Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: rantTesting module madness
Andy Lester wrote:
yeah, well, you could say that. last time I installed my washer I said
looks pretty level to me, but I know where my level is if it makes a
racket
That's fine, but I'm still not shipping my washing machines without
explicit instructions to level the damn thing. Similarly, I'm not
making any of my tests optional except in the case of tests where they
don't affect direction operation, as in t/pod{,-coverage}.t.
well, it's nice that you have that luxury with the code you write, but not
every module requires or can live with that kind of behavior :)
consider something real like mod_perl. some of our tests are optional and
are skipped in various circumstances. why? well, not every installation
has LWP, so we skip some tests where we require it when all it does is make
our test writing life easier. not every apache installation has mod_auth
installed, so we skip tests that exercise mod_auth-behaviors. I could go
on, of course, but you get the idea.
the same analogy can hold to various perl modules, where one class interacts
with something that not everyone cares about. say you've got some module
with shared code, client-specific code, and server-specific code. say some
user of that code only cares about the client part. now say that testing
the server code requires 3 additional Test:: modules. you're going to force
the user to install those modules and run those tests even though they will
be exercising code he doesn't care about? I don't think that kind of thing
makes sense at all. furthermore, I think that doing so gives rise to
conversations like the one that started this thread, where people see umteen
test dependencies and get frustrated.
remember, we're in an uphill battle here in the testing world. every time
we frustrate a user we make it harder on ourselves - too many up-front
dependencies and folks will skip the tests altogether and carry that
frustration back to their desk when they decide whether to write tests
themselves...
--Geoff
Re: rantTesting module madness
On Sun, Sep 11, 2005 at 07:01:50AM -0400, Randy W. Sims wrote: The current MakeMaker has support for adding arbitrary data to META.yml, so it can also provide info to installers. However, I doubt MakeMaker itself will every know anything about or use such requirements recommendations. MakeMaker doesn't have to do anything with the requirements or recommendations except communicate them to the CPAN shell. Its the CPAN shell's job to make sure the necessary requirements are available. -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern Reality is that which, when you stop believing in it, doesn't go away. -- Phillip K. Dick
The Right To Blow Off Your Own Foot (was Re: rantTesting module madness)
On Sun, Sep 11, 2005 at 01:52:41PM -0400, Christopher H. Laco wrote: I've always thought tests passing should be a requisite of make by default. Make fails if tests fail. There should of course be a -skiptests to opt out of testing for those who insist on not doing it, but for the most part, if tests are so important like we the perl community say they are, then they should be run as part of make. Period. This is a bad idea. If people don't want to run tests, they won't run the tests. If people want to ignore test failures, they will ignore them. Putting more technology in their way just makes it a bit more annoying (see also the long, expensive War On Piracy that is DRM). This is a social problem that will not be solved with technology, especially not BDSM tech. Tying together the build and test phase and then inserting a new back door doesn't change anything except reducing the flexibility of the build system (now you can't easily tell the difference between a build failure and a test failure) and making people learn the new work around. Doesn't make much difference if I say make; make install or if I say make --skiptests; make install, the tests don't get run. Finally, there's many, many, many instances where a test will fail even though something isn't really wrong. The author made a bad assumption about my environment or maybe the tests are just crap. The former happens all the time with network modules (LWP and mod_perl come to mind) or when installing CPAN modules on a non-Unix system (what do you mean you don't have ctime?!) *I*, the human end-user, get to evaluate the failure and decide to go ahead anyway. Sometimes its better to have a mostly working module than no module at all. This could be called the 2nd ammendment of CPAN: The right to blow off your own foot. It also reminds me of a possibly apocryphal story about the analog targetting computers they used to use on ships in WWII. These things were simple mechanical computers that given such variables as the range, speed, heading and angle off the bow to your target plus your own speed and heading, plus wind conditions, plus information about the gun you're firing would tell you at what angle and elevation to lay the gun and how much powder to use to transport a lump of metal onto the enemy's deck. During heavy use they had a tendency to overheat. The computer had safeguards to shut itself down before it damaged itself, it doesn't to do break your expensive targetting computer during a training exercise. It had a switch called Battle Mode. When in this mode rather than shutting down when it started to overheat or detected a fault it would just light warning lights but continue to supply firing solutions. Right answers, wrong answers... any answer is better than no answer when you're beign shot at... and it would keep right on pumping out answers until it melted down. Because the last thing you want is your guns to stop firing because some programmer put in an overzealous assert()! -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern I do have a cause though. It's obscenity. I'm for it. - Tom Lehrer
Re: rantTesting module madness
On 9/11/05, Adam Kennedy [EMAIL PROTECTED] wrote: And for something simple as the tests don't generate warnings, I would think module has excessive dependencies is a bug in Test::Deep, rather than a more general problem. I'd say obvious, necessary and a simple idea but if you think it's simple, off you go and golf it (don't forget the stack traces). I'd actually say that the need to _install_ something to test that you don't generate warnings is the bug. When it comes to unit tests, the warning detector is something that you should have to switch _off_. This should not be taken as a criticism of Test::Simple by the way, the fact that I can so easily create T::NW is pretty cool, F
Re: rantTesting module madness
On Sun, Sep 11, 2005 at 23:51:56 +1000, Adam Kennedy wrote: I've found that by using Module::Install, I can often just bundle those small miscellaneous testing modules, and occasionally a few of their dependencies. And for something simple as the tests don't generate warnings, I would think module has excessive dependencies is a bug in Test::Deep, rather than a more general problem. Don't do that! If you get hit by a truck and someone updates Test::Moose to take care of a bug, who will update your bundled version? If you don't get hit by a truck but simply don't realize that Test::Moose was updated? If you do realize, but it takes you 3 days to update once mainline was fixed, and it takes mainline 3 days to update once Joe Random's patch was submitted, why should new users from these 3 days get a buggy, outdated version when a better one is updated? Aren't the 3 days till the patch is applied enough? CPAN.pm handles module dependencies. All Tels really has to do is set 'follow' instead of 'ask', or use one of the tools available to make a copy that is network indepenedant. -- () Yuval Kogman [EMAIL PROTECTED] 0xEBD27418 perl hacker /\ kung foo master: /me kicks %s on the nose: neeyah! pgpgAb4Tnu9jC.pgp Description: PGP signature
Bottom-up testing and the Space Shuttle
The following excerpt is from Richard Feynman's essay Personal observations on the reliability of the Shuttle. http://www.fotuva.org/feynman/challenger-appendix.html I find it covers a testing topic I have a little trouble explaining to testing newbies, that being the advantages of bottom up testing. The usual way that such [complex, liquid fueled] engines are designed (for military or civilian aircraft) may be called the component system, or bottom-up design. First it is necessary to thoroughly understand the properties and limitations of the materials to be used (for turbine blades, for example), and tests are begun in experimental rigs to determine those. With this knowledge larger component parts (such as bearings) are designed and tested individually. As deficiencies and design errors are noted they are corrected and verified with further testing. Since one tests only parts at a time these tests and modifications are not overly expensive. Finally one works up to the final design of the entire engine, to the necessary specifications. There is a good chance, by this time that the engine will generally succeed, or that any failures are easily isolated and analyzed because the failure modes, limitations of materials, etc., are so well understood. There is a very good chance that the modifications to the engine to get around the final difficulties are not very hard to make, for most of the serious problems have already been discovered and dealt with in the earlier, less expensive, stages of the process. The Space Shuttle Main Engine was handled in a different manner, top down, we might say. The engine was designed and put together all at once with relatively little detailed preliminary study of the material and components. Then when troubles are found in the bearings, turbine blades, coolant pipes, etc., it is more expensive and difficult to discover the causes and make changes. For example, cracks have been found in the turbine blades of the high pressure oxygen turbopump. Are they caused by flaws in the material, the effect of the oxygen atmosphere on the properties of the material, the thermal stresses of startup or shutdown, the vibration and stresses of steady running, or mainly at some resonance at certain speeds, etc.? How long can we run from crack initiation to crack failure, and how does this depend on power level? Using the completed engine as a test bed to resolve such questions is extremely expensive. One does not wish to lose an entire engine in order to find out where and how failure occurs. Yet, an accurate knowledge of this information is essential to acquire a confidence in the engine reliability in use. Without detailed understanding, confidence can not be attained. A further disadvantage of the top-down method is that, if an understanding of a fault is obtained, a simple fix, such as a new shape for the turbine housing, may be impossible to implement without a redesign of the entire engine. -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern 'All anyone gets in a mirror is themselves,' she said. 'But what you gets in a good gumbo is everything.' -- Witches Abroad by Terry Prachett
Re: rantTesting module madness
On Mon, Sep 12, 2005 at 09:36:28 +1000, Adam Kennedy wrote: If you get hit by a truck and someone updates Test::Moose to take care of a bug, who will update your bundled version? Simple. Because I don't bundle it by hand, Module::Install does it. Bundling things by hand would be WAY too much work. The next time someone else rolls the build, it automagically grabs the newest latest version they have without any need for effort at all. No, that's not the point... The issue is that the old version of Test::Moose is frozen into your tarball. Someone still needs to roll the build and there is really no reason to do that. If you don't get hit by a truck but simply don't realize that Test::Moose was updated? If the tests pass still, does it matter? Yes, because the bug could be triggered by an environment change or something like that. Modules get updated for a reason. No let me retort - if the tests how is a newbie going to deal with this? Because it was obviously good enough _already_ to allow the testing to proceed successfully. And if it was a problem, well I would have updated my dependency on that testing version to the current one, and when I reroll the dist if fails, wanting the newer version. /me stops maintaining all his modules because they are good enough already. Periodical bug fixes will hence forth be published with a one year delay. This isn't some permanent module. The user is ONLY going to need it for the half a second my module is being tested. They can go without some minor fix for some bizarre edge case or your latest feature addition just fine. It still promotes: * needless duplication of library code across distribution * subverting the very functional dependency mechanism for the purpose of convenience * abusing tools made for installing packed applications and software deployment where standard library installations work and are reccomended * inconsistency in the notion of the latest version of a module being installed by a user who is not expecting a bundled support module -- () Yuval Kogman [EMAIL PROTECTED] 0xEBD27418 perl hacker /\ kung foo master: /me wields bonsai kittens: neeyah pgpzLk2HHAn3Y.pgp Description: PGP signature
Re: rantTesting module madness
On Sep 11, 2005, at 7:25 PM, chromatic wrote: I don't feel as confident as you do that if the tests all passed on your machine that they'll automatically pass everywhere. THIS is my biggest point. It's not about the quality of the code so much as making sure the code fits with the target machine. xoxo, Andy -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
