On 2019/08/15 12:07, Karen Etheridge wrote:
> On Thu, Aug 15, 2019 at 11:21 AM L A Walsh <cpan-tes...@tlinx.org> wrote:
>
>> This is very worrisome as it indicates old modules that used to
>> work may no longer work even though nothing changed. AFAIK, this
>> shouldn't be possible.
>>
>
> Of course it's possible. Your code might not have changed, but other
> things change all the time -- other modules that your code is using,
> system libraries used by everything, the operating system itself, or
> the hardware that it's running on.
>
---
Partly true. However, I didn't rebuild my perl, nor did I download
upgraded modules. From what I can tell, more than one tester did the same.
What could have changed are the dynamically linked libraries from
the system perl is running on and the kernel).
Now there are a couple of problems that I can see.
1) Something changed in the past 4 years that causes this to fail.
Build requires ExtUtils::MakeMaker 6.36, mem 0.3.2, Test::More
config requires ExtUtils::MakeMaker 6.36
test requires mem 0.3.2
None of those were reinstalled. That means perl or a library it
uses changed, right (if not, what else is there?)
2) This _could_ affect every module on CPAN (unlikely, but like the problem
in Types::Core, we wouldn't know because apparently it's not been
retested
in 4 years (or has it?). I'm thinking that along with perl's policy of
only supporting the last 2 perls(cur+prev), no testing is regularly or
officially done on older perl distros to see if they are still
buildable,
if they still pass their test suit, and if they still pass the test
suites
of all the modules that were tested and passed against it.
> analysis.cpantesters.org performs statistical analysis on all the
> reports in the system in an attempt to find patterns that can isolate
> what the relevant factors are to changing test results -- but it can
> only analyze data in the reports.
---
right. What's missing is current information. There may be other
things
missing, but the most important lack right now is knowing which modules that
did work against perl-5.X still work.
I think there is a broken assumption in that if perl doesn't change
and the
module-chain doesn't chain, then old cpan test results should still be
valid.
Is that a tacit assumption in the perl+cpan development process?
> It's likely that the relevant
> information (OS/platform, dependent modules, compile-time parameters
> etc) are included in the reports, but it's also possible that it's
> not, in which case you'd need to make a new release that included
> additional information in the test output.
>
----
Since the breakage was unknown before today (4 years after
everything tested
clean), we are looking at a very long time to discover anything is
wrong. Since
the information used to build modules is included in the cpan-test
infrastructure, it doesn't seem useful to enhance output in 1 cpan
module nor even in ALL cpan modules, since we don't know what failed or
when or what would be needed for inclusion since whatever caused this
failure might be only 1 of a class of failures that manifest with
different symptoms with other modules.
One way this is likely to have been prevented would be if all of the
binary
modules perl linked with 4 years ago were kept in a separate perl-lib
path so
we can reproduce test results from "some time ago", *or* statically link
those in (!heresy!). I'm not saying either approach doesn't have
problems, and I would hope there are better solutions.
At the very least, it seems testing for modules need to be redone
"periodically" to see if the last known test results are still valid. A
_crude_ way would be for module authors to touch the source file and
resubmit the module. Hopefully some automated method would provide a
superior outcome.
If perl.org doesn't support old versions, maybe all testing against
unsupported versions should be halted (sound of lead balloon hitting
ground).
I mean why are we testing against old versions if we don't know if those
tests
are still valid a year (or 4) later?
