Hello folks,
In the past I already sent this for discussion, but didn't get much
attention. I hope I can get more feedback this time.
We all create distroprefs to block distributions to run on our smokers.
Although we can always publish those on something like Github, it is not
simple to keep sync with them.
That's bad for several reasons:
1. We are not taking advantage of each other work of smoking test
distributions. It took me several months of work to go over the ~37K
distributions available on CPAN. This is a huge amount of resources
wasted considering that we don't have a "watchdog" or something like
to kill tests that were running for more than a hour, for instance.
It is almost funny that for many days I left the Smoker from running
unattended after hours just to find out the tests stopped after 15
minutes I left.
2. We cannot classify easily the distroprefs (for instance, I want to
use only distroprefs created for Linux or FreeBSD, or even for a
specific perl version) to download and use in our smokers.
3. We don't have any simple way to contact the distribution author to
let him/her know their respective distribution is blocked for some
reason.
4. Kwalitee reports don't take into consideration that a distribution
is blocked.
5. We can't easily implement policies like blocking distributions that
don't include any test files at all (we shouldn't have to download,
unpack and check that there aren't any test files).
6. Not every CPAN contributor writes tests that allow execution of
tests in parallel, or even to run in parallel with another perl
executing the same tests at the same time.
7. Not every CPAN contributor writes tests that skips installation when
there aren't requirements in place like a specific OS. Why would we
want to smoke test a distribution that will only run on Win32 from a
UNIX-like based smoker?
Due my experiments with OpenBSD running a CPAN::Reporter::Smoker, I was
able to fetch a set of 183 distroprefs and the vast majority of them
have the following comment:
comment: Tests hang smoker
This "default" comment is due a limitation of the program dblock (see
CPAN-Reporter-Smoker-OpenBSD) that I made available on CPAN. For some
cases, I tried to add something more meaningful:
GRAVATTJ.Backup-EZ.yml:
comment: Requires root access to test/install
But at the end I did it manually. That could be easily fixed adding a
command line option, but that's just an example. Even if I implement
that, the author will not get anything automatically.
There are some other cases that the distribution used a huge amount of
/tmp space and other that created directories on CPAN build_dir
parameter with permissions that didn't allowed removal after the tests.
I'm considering starting a project to share this kind of information
from a website with the following features:
1. Front end to allow a developer to search for anything blocked based
on this CPAN ID.
2. REST interface to allow the same search (anonymous access) and
submission of distroprefs (authenticated) in JSON format.
3. Send and e-mail to the author when something is received for his/her
distributions.
For that, I reviewed the distroprefs specification (from the CPAN module
POD) and there aren't fields that I think would be required to include
into the YAML:
* The OS name and version
* The reporter ID
* Reason (we might define some standard values)
* perl details
* creation date of the distroprefs
* the version of the distribution that was being tested when the
distroprefs was created
I guess I could declare multiple "comment" entries with that, but that
would be a hack at best.
Since I'm not sure if anybody is working on something like that, I'm
hoping to get some feedback like suggestions, complains and maybe, who
knows, some code. :-)
Thanks!
Alceu
