Bug#842794: autopkgtests fail (since perl 5.24?): Failed test 'Testing vcf-fix-ploidy .. cat ...

[Andreas Tille]

On Thu, Dec 15, 2016 at 09:10:43AM +0200, Niko Tyni wrote:
> > What did happen around the time was the removal of '.' from @INC.
> 
> That's indeed what caused the regression.
> ...
> Patch attached.

Works.  Thanks a lot

Andreas.

-- 
http://fam-tille.de

Bug#842794: autopkgtests fail (since perl 5.24?): Failed test 'Testing vcf-fix-ploidy .. cat ...

[gregor herrmann]

On Thu, 15 Dec 2016 09:10:43 +0200, Niko Tyni wrote:

> > What did happen around the time was the removal of '.' from @INC.
> That's indeed what caused the regression.
[..]
> > The test call can be shortened to:
> > # vcf-fix-ploidy -s fix-ploidy.samples < fix-ploidy.vcf | grep 61098
> You missed the '-p' parameter above, which makes all the difference.
> It's loading Perl code from the specified file with 'do', which no longer
> looks in cwd. The load fails silently and the program continues on with
> wrong results.

*sigh*

I did see the "do $file", and I tried with and without -p ... and
then must have had a "knot in my brain". Sorry & thanks!


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at/ - Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
   `-   BOFH excuse #173:  Recursive traversal of loopback mount points 

Bug#842794: autopkgtests fail (since perl 5.24?): Failed test 'Testing vcf-fix-ploidy .. cat ...

[Niko Tyni]

On Wed, Dec 14, 2016 at 08:00:00PM +0100, gregor herrmann wrote:
> On Wed, 14 Dec 2016 08:58:30 +0100, Andreas Tille wrote:
> 
> > On Tue, Nov 01, 2016 at 11:05:02AM +, Iain Lane wrote:
> > > I noticed in Ubuntu, where we run autopkgtests as part of britney
> > > migration, that vcftools fails now. You can see on ci.debian.net[0]
> > > 
> > > > not ok 26 - Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl -I../. 
> > > > -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p fix-ploidy.txt 
> > > > 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n'
> > > > #   Failed test 'Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl 
> > > > -I../. -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p 
> > > > fix-ploidy.txt 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n''
> > > > #   at ./test.t line 452.
> > > > # Structures begin differing at:
> > > > #  $got->[0] = '61098   M1 0/1 0,9,72,5,6,7 M2 0/0 
> > > > 0,15,140,5,6,7   F3 1 147,0,5F4 0 0,131,5M5 0/0 0,9,83,5,6,7
> > > >  M6 0/0 0,6,56,5,6,7
> > > > # '
> > > > # $expected->[0] = '61098   M1 0 0,9,72,5,6,7   M2 0 
> > > > 0,15,140,5,6,7 F3 1/1 147,0,5  F4 0/0 0,131,5  M5 0 0,9,83,5,6,7
> > > >M6 0 0,6,56,5,6,7
> > > > # '

> The tests started failing on 2016-09-01.
> perl 5.24 entered unstable on 23 Sep 2016.
> 
> What did happen around the time was the removal of '.' from @INC.

That's indeed what caused the regression.

> The test call can be shortened to:
> 
> # vcf-fix-ploidy -s fix-ploidy.samples < fix-ploidy.vcf | grep 61098
> 20  61098   .   C   A,T 999 PASS
> AC1=41;AF1=0.2104;DP4=209,284,67,76;DP=658;FQ=999;MQ=45;PV4=0.39,4.4e-10,0.0034,0.2
>  GT:PL:DP:SP:GQ  0/1:0,9,72,5,6,7:3:212:12   0/0:0,15,140,5,6,7:5:458752:18  
> 1:147,0,5:7:384:24  0:0,131,5:5:208:18  0/0:0,9,83,5,6,7:3:392:12   
> 0/0:0,6,56,5,6,7:2:204:9
> 
> Which already has the 0/1:0,9,72,5,6,7 sequence which is not the
> expected "0 0,9,72,5,6,7". And I don't know what either of them mean

You missed the '-p' parameter above, which makes all the difference.
It's loading Perl code from the specified file with 'do', which no longer
looks in cwd. The load fails silently and the program continues on with
wrong results.

Patch attached.
-- 
Niko Tyni   nt...@debian.org
>From 8435d883d7d514618c8dcd8b60a722e163a261f6 Mon Sep 17 00:00:00 2001
From: Niko Tyni 
Date: Thu, 15 Dec 2016 01:39:16 +0200
Subject: [PATCH] vcf-fix-ploidy: make -p work without . on @INC

'do $file' only looks in @INC for unqualified file names, so
prefix those with './' in case '.' is not on @INC.

While at it, add diagnostics if sourcing the file fails altogether.

Bug-Debian: https://bugs.debian.org/842794
---
 src/perl/vcf-fix-ploidy | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/src/perl/vcf-fix-ploidy b/src/perl/vcf-fix-ploidy
index 4fd380d..0140a21 100644
--- a/src/perl/vcf-fix-ploidy
+++ b/src/perl/vcf-fix-ploidy
@@ -75,7 +75,17 @@ sub parse_params
 };
 while (defined(my $arg=shift(@ARGV)))
 {
-if ( $arg eq '-p' || $arg eq '--ploidy' ) { my $file=shift(@ARGV); my $x=do $file; $$opts{ploidy}=$x; next }
+if ( $arg eq '-p' || $arg eq '--ploidy' ) {
+my $file=shift(@ARGV);
+$file = "./$file" if $file !~ m{^/};
+my $x=do $file;
+if (!defined $x) {
+error("problem parsing \"$file\": $@") if $@;
+error("problem doing \"$file\": $!");
+}
+$$opts{ploidy}=$x;
+next
+}
 if ( $arg eq '-s' || $arg eq '--samples' ) { $$opts{samples}=shift(@ARGV); next }
 if ( $arg eq '-a' || $arg eq '--assumed-sex' ) { $$opts{assumed_sex}=shift(@ARGV); next }
 if ( $arg eq '-l' || $arg eq '--fix-likelihoods' ) { $$opts{fix_likelihoods}=1; next }
-- 
2.10.2

Bug#842794: autopkgtests fail (since perl 5.24?): Failed test 'Testing vcf-fix-ploidy .. cat ...

[Andreas Tille]

Hi Gregor,

thanks for your analysis.

On Wed, Dec 14, 2016 at 08:00:00PM +0100, gregor herrmann wrote:
> > > I noticed in Ubuntu, where we run autopkgtests as part of britney
> > > migration, that vcftools fails now. You can see on ci.debian.net[0]
> > > 
> > > > not ok 26 - Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl -I../. 
> > > > -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p fix-ploidy.txt 
> > > > 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n'
> > > > #   Failed test 'Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl 
> > > > -I../. -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p 
> > > > fix-ploidy.txt 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n''
> > > > #   at ./test.t line 452.
> > > > # Structures begin differing at:
> > > > #  $got->[0] = '61098   M1 0/1 0,9,72,5,6,7 M2 0/0 
> > > > 0,15,140,5,6,7   F3 1 147,0,5F4 0 0,131,5M5 0/0 0,9,83,5,6,7
> > > >  M6 0/0 0,6,56,5,6,7
> > > > # '
> > > > # $expected->[0] = '61098   M1 0 0,9,72,5,6,7   M2 0 
> > > > 0,15,140,5,6,7 F3 1/1 147,0,5  F4 0/0 0,131,5  M5 0 0,9,83,5,6,7
> > > >M6 0 0,6,56,5,6,7
> > > > # '
> > 
> > Since the bug is menitoning the Perl version bump I wonder whether the
> > perl team would have an explanation / fix for the issue.
> 
> The tests started failing on 2016-09-01.
> perl 5.24 entered unstable on 23 Sep 2016.

Ahhh, that's the changelog date from 1.3.1-3 with the following patch
included:

   
https://anonscm.debian.org/cgit/debian-med/htslib.git/tree/debian/patches/define_PATH_MAX.patch

While there is some time coincicence I can not imagine that this
specific patch would be able to cause this result.  To be really sure I
tried with a tabix where I reverted the patch and there is the same test
failure.
 
> What did happen around the time was the removal of '.' from @INC.
> 
> But still, I think the failure is unrelated;
> https://ci.debian.net/data/packages/unstable/amd64/v/vcftools/20160901_192825.log
> (the first failure) doesn't show any changes in dependencies. But
> has perl 5.22.2-4. But re-adding . to @INC in
> /etc/perl/sitecustomize.pl doesn't change anything.
> 
> The log from the failed test before is:
> https://ci.debian.net/data/packages/unstable/amd64/v/vcftools/20160901_164433.log
> 
> -tabix 1.3.1-2
> +tabix 1.3.1-3
> 
> Not sure if and how this is related.

I think its a nice hint anyway even if the actual change in this package
does not have caused the failed test.  I even fetched tabix 1.3.1-2 +
libhts1 1.3.1-2 from snapshot.d.o and the test failure remains.
 
> The test call can be shortened to:
> 
> # vcf-fix-ploidy -s fix-ploidy.samples < fix-ploidy.vcf | grep 61098
> 20  61098   .   C   A,T 999 PASS
> AC1=41;AF1=0.2104;DP4=209,284,67,76;DP=658;FQ=999;MQ=45;PV4=0.39,4.4e-10,0.0034,0.2
>  GT:PL:DP:SP:GQ  0/1:0,9,72,5,6,7:3:212:12   0/0:0,15,140,5,6,7:5:458752:18  
> 1:147,0,5:7:384:24  0:0,131,5:5:208:18  0/0:0,9,83,5,6,7:3:392:12   
> 0/0:0,6,56,5,6,7:2:204:9
> 
> Which already has the 0/1:0,9,72,5,6,7 sequence which is not the
> expected "0 0,9,72,5,6,7". And I don't know what either of them mean
> ...

I confirm that I have no idea what this means but it somehow looks
"quite similar".  I'll ask upstream about this.
 
> And /usr/share/perl5/Vcf.pm has 3523 lines which I'm not going to
> read and try to understand now :)

How comes? ;-)
 
Thanks for your very helpful analysis

 Andreas.

-- 
http://fam-tille.de

Bug#842794: autopkgtests fail (since perl 5.24?): Failed test 'Testing vcf-fix-ploidy .. cat ...

[gregor herrmann]

On Wed, 14 Dec 2016 08:58:30 +0100, Andreas Tille wrote:

> On Tue, Nov 01, 2016 at 11:05:02AM +, Iain Lane wrote:
> > I noticed in Ubuntu, where we run autopkgtests as part of britney
> > migration, that vcftools fails now. You can see on ci.debian.net[0]
> > 
> > > not ok 26 - Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl -I../. 
> > > -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p fix-ploidy.txt 
> > > 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n'
> > > #   Failed test 'Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl 
> > > -I../. -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p 
> > > fix-ploidy.txt 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n''
> > > #   at ./test.t line 452.
> > > # Structures begin differing at:
> > > #  $got->[0] = '61098   M1 0/1 0,9,72,5,6,7 M2 0/0 
> > > 0,15,140,5,6,7   F3 1 147,0,5F4 0 0,131,5M5 0/0 0,9,83,5,6,7 
> > > M6 0/0 0,6,56,5,6,7
> > > # '
> > > # $expected->[0] = '61098   M1 0 0,9,72,5,6,7   M2 0 
> > > 0,15,140,5,6,7 F3 1/1 147,0,5  F4 0/0 0,131,5  M5 0 0,9,83,5,6,7  
> > >  M6 0 0,6,56,5,6,7
> > > # '
> 
> Since the bug is menitoning the Perl version bump I wonder whether the
> perl team would have an explanation / fix for the issue.

The tests started failing on 2016-09-01.
perl 5.24 entered unstable on 23 Sep 2016.

What did happen around the time was the removal of '.' from @INC.

But still, I think the failure is unrelated;
https://ci.debian.net/data/packages/unstable/amd64/v/vcftools/20160901_192825.log
(the first failure) doesn't show any changes in dependencies. But
has perl 5.22.2-4. But re-adding . to @INC in
/etc/perl/sitecustomize.pl doesn't change anything.

The log from the failed test before is:
https://ci.debian.net/data/packages/unstable/amd64/v/vcftools/20160901_164433.log

-tabix 1.3.1-2
+tabix 1.3.1-3

Not sure if and how this is related.


The test call can be shortened to:

# vcf-fix-ploidy -s fix-ploidy.samples < fix-ploidy.vcf | grep 61098
20  61098   .   C   A,T 999 PASS
AC1=41;AF1=0.2104;DP4=209,284,67,76;DP=658;FQ=999;MQ=45;PV4=0.39,4.4e-10,0.0034,0.2
 GT:PL:DP:SP:GQ  0/1:0,9,72,5,6,7:3:212:12   0/0:0,15,140,5,6,7:5:458752:18  
1:147,0,5:7:384:24  0:0,131,5:5:208:18  0/0:0,9,83,5,6,7:3:392:12   
0/0:0,6,56,5,6,7:2:204:9

Which already has the 0/1:0,9,72,5,6,7 sequence which is not the
expected "0 0,9,72,5,6,7". And I don't know what either of them mean
...

And /usr/share/perl5/Vcf.pm has 3523 lines which I'm not going to
read and try to understand now :)


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at/ - Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member of VIBE!AT & SPI, fellow of the Free Software Foundation Europe
   `-   NP: Rolling Stones


signature.asc
Description: Digital Signature

Bug#842794: autopkgtests fail (since perl 5.24?): Failed test 'Testing vcf-fix-ploidy .. cat ...

[Andreas Tille]

Hi,

On Tue, Nov 01, 2016 at 11:05:02AM +, Iain Lane wrote:
> I noticed in Ubuntu, where we run autopkgtests as part of britney
> migration, that vcftools fails now. You can see on ci.debian.net[0]
> 
> > not ok 26 - Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl -I../. 
> > -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p fix-ploidy.txt 
> > 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n'
> > #   Failed test 'Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl -I../. 
> > -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p fix-ploidy.txt 
> > 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n''
> > #   at ./test.t line 452.
> > # Structures begin differing at:
> > #  $got->[0] = '61098   M1 0/1 0,9,72,5,6,7 M2 0/0 
> > 0,15,140,5,6,7   F3 1 147,0,5F4 0 0,131,5M5 0/0 0,9,83,5,6,7 M6 
> > 0/0 0,6,56,5,6,7
> > # '
> > # $expected->[0] = '61098   M1 0 0,9,72,5,6,7   M2 0 0,15,140,5,6,7 
> > F3 1/1 147,0,5  F4 0/0 0,131,5  M5 0 0,9,83,5,6,7   M6 0 
> > 0,6,56,5,6,7
> > # '

Since the bug is menitoning the Perl version bump I wonder whether the
perl team would have an explanation / fix for the issue.

Kind regards

Andreas.
 
> [0] 
> https://ci.debian.net/data/packages/unstable/amd64/v/vcftools/20161028_112724.autopkgtest.log.gz

-- 
http://fam-tille.de

Bug#842794: autopkgtests fail (since perl 5.24?): Failed test 'Testing vcf-fix-ploidy .. cat ...

[Iain Lane]

Package: vcftools
Version: 0.1.14+dfsg-3
Severity: normal

Hi,

I noticed in Ubuntu, where we run autopkgtests as part of britney
migration, that vcftools fails now. You can see on ci.debian.net[0]

> not ok 26 - Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl -I../. -MVcf 
> /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p fix-ploidy.txt 2>/dev/null | 
> vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n'
> #   Failed test 'Testing vcf-fix-ploidy .. cat fix-ploidy.vcf | perl -I../. 
> -MVcf /usr/bin/vcf-fix-ploidy -s fix-ploidy.samples -p fix-ploidy.txt 
> 2>/dev/null | vcf-query -f '%POS[\t%SAMPLE %GTR %PL]\n''
> #   at ./test.t line 452.
> # Structures begin differing at:
> #  $got->[0] = '61098   M1 0/1 0,9,72,5,6,7 M2 0/0 0,15,140,5,6,7 
>   F3 1 147,0,5F4 0 0,131,5M5 0/0 0,9,83,5,6,7 M6 0/0 0,6,56,5,6,7
> # '
> # $expected->[0] = '61098   M1 0 0,9,72,5,6,7   M2 0 0,15,140,5,6,7   
>   F3 1/1 147,0,5  F4 0/0 0,131,5  M5 0 0,9,83,5,6,7   M6 0 0,6,56,5,6,7
> # '

Any ideas?

Cheers,

-- 
Iain Lane  [ i...@orangesquash.org.uk ]
Debian Developer   [ la...@debian.org ]
Ubuntu Developer   [ la...@ubuntu.com ]

[0] 
https://ci.debian.net/data/packages/unstable/amd64/v/vcftools/20161028_112724.autopkgtest.log.gz