On 12 June 2014 06:37, Jens Rehsack <rehs...@gmail.com> wrote: > > 1. begin test > > 2. load test networking module module > > 3. is NO_NETWORK_TESTING? SKIP! > > 4. can access specified resources? yes -> run tests > > no -> SKIP! > > In that case I suggest to FAIL instead of skip. Maybe is that part of the > error.
Really whether or not that fails would be an installer choice. For instance, me, I would basically say "If you cant access the internet, don't bother doing tests that require the internet". After all, the running of the test suite is not to give me a "pass/fail: Your internet works", but to test the code in question functions as intended. The only time I'd want the tests to fail is if I was running the tests _AND_ want assurredness that the network tests in fact work as intended, and that the non-network tests are not sufficient for my personal installer satisfaction. Then those tests would fail every time they couldn't use the internet, and I'd know I have to fix my internet and run them again. But that case is very much not the usecase of a bulk smoke tester. If a tester box doesn't have internet visible during 'make test', I'd expect tests that require internet access to be skipped ( or exit N/A somehow ), not give the developer of the module spurious FAIL results that revolve around something the developer can do nothing about, FAIL reports that don't indicate the code is broken, merely that the person who ran the tests has broken internet. Based on some of the comments on #perl-qa , it seems there may also be a use for declaring 2 or 3 grades of internet access. 1. Loopback only access: Basic network functions work, you can spawn a HTTP test server for something like PSGI, and connect to it from another Perl process and IPC between the two. 2. LAN only access: There are other machines on a LAN, some semblance of DNS will probably work and resolve HOSTNAME to a network routable IP, but internet traffic is still expected to not work. ( Or there's a (broken?) firewall standing between LAN and WAN than can resolve global DNS addresses but not relay packets ) 3. Full internet Now we probably don't need to represent all of these, just they're scopes to take into consideration. You'd think case #1 would never need blocking off, however, the person who raised that case mentioned in the context of "Localhost is firewalled". And that would mess things up. ( And I doubt anyone is interested in going through a firewall just to allow open ports for some random Perl test ) Most of those levels we could indeed check for with code, but having a "Don't even check, its not worth the cycles, you'll probably get it wrong, and trying may even cause an explosion because somebody attached C4 to a network device somewhere in the house" control would be still useful. ( And I guess the "weird networks are weird all over the place" problem is one reason why the default is to have network tests opt-in instead of opt-out. ) -- Kent
