Bug#815980: tracker.debian.org: All packages have uscan issues

[Osamu Aoki]

Hi,

On Sat, Mar 05, 2016 at 07:15:46AM +0100, Lucas Nussbaum wrote:
...
> Looking at the errors, uscan doesn't seem to use the  tag, but instead
> output its errors in the  tag. (but that was already the case).

Correct.

When uscan faces "die" situation, it calls "uscan_die" and create
 tag data.  When uscan faces "warn" situation, it calls
"uscan_warn" and create  data.  Most watch file parsing errors
are "warn" situation since uscan may be looping ober multiple watch file
which is better to be continued.

After my recent code review, I found a few unwrapped "die" and "warn" in
the code.  They are fixed now so all  and  are
properly generated for all cases with the next upload.

Regards,

Osamu

Bug#815980: tracker.debian.org: All packages have uscan issues

[Osamu Aoki]

Hi again,

I have a second thought.  UDD used to work around --dehs 
problem.  Now you do not need to do so.  Let's make thing simpler.

Since I never coded in ruby, you must give some break for me making
buggy code but attached patch (after bug fixes if needed) should address
issues on UDD web page.

This simplifies UDD code and keeps uscan away from another bloat.

On Thu, Mar 03, 2016 at 09:17:37PM +0900, Osamu Aoki wrote:
> Hi,
> 
> I think I can propose peaceful solution.
> 
> On Sun, Feb 28, 2016 at 03:18:21PM +0100, Lucas Nussbaum wrote:
> > On 26/02/16 at 19:10 -0500, James McCoy wrote:
> ...
> > > > My guess is that this was triggered in UDD when devscripts 2.16.1 got
> > > > uploaded to jessie-backports. In devscripts 2.15.10, the interface to
> > > > uscan changed with everything being printed to stderr (even if it's not
> > > > an error) if --dehs is used.
> > > 
> > > Then this is what needs to be fixed.  The behavior, especially for
> > > something that's intended to be used by automation, shouldn't have
> > > changed such that it broke the major user of such functionality.
> > 
> > The code that runs uscan in UDD is
> > http://anonscm.debian.org/cgit/collab-qa/udd.git/tree/rimporters/upstream.rb
> > (starting around line 50).
> 
> I have never learned ruby but the exit code "status" obtained by:
>stdout, stderr, status = Open3.capture3(cmd)
> is not used to determin if the program suceeded or not.
> 
> Instead stderr != '' check is used.  Unexpected :-)

Pathced code still do not check exit status :-)

> > As you can see, the code is already quite complicated:
> > uscan warnings are outputted in the  tag, but not all uscan
> > errors are: sometimes they are in an  tag, sometimes they are
> > just displayed on stderr.

The same goes with uscan. Adding another option seems to be bad idea.

> The old code turned off verbose outputs and warnings if --dehs is used.
> This was bad for tweaking watch file etc.  So I activated them while
> making sure stdout os used only by --dehs and stderr has them.
>  
> > It would be better if all uscan errors were displayed in . Then
> > UDD could just ignore the stderr output.
> 
> Now the dehs output marked with  has all uscan errors, if uscan
> starts to process watch file(s) :-)  (I think it was not the case
> previously.)

So attached patch should work. (I hope)

> But reading what James said, I can satisfy my thought by introducing new
> option --dehs-verbose while making --dehs to be quiet for warning.
> 
> This way, no pain for everyone.

This is pain for uscan.  Much less code change for UDD :-)

> Let me think a bit more but the above seems to be most rationale thing
> to do.

That is my thought tonight.

Regards,

Osamu

>From 97b4959be37bf2c3964bd962030d63a48450f3ca Mon Sep 17 00:00:00 2001
From: Osamu Aoki 
Date: Thu, 3 Mar 2016 21:44:06 +0900
Subject: [PATCH] uscan dehs supports  properly now

---
 rimporters/upstream.rb | 21 +++--
 1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/rimporters/upstream.rb b/rimporters/upstream.rb
index 87ab2e6..961b87b 100755
--- a/rimporters/upstream.rb
+++ b/rimporters/upstream.rb
@@ -85,11 +85,6 @@ module Upstream
 :warnings => nil
   }
 end
-if stderr != ''
-  stderr = "uscan output on stderr: #{stderr}\n"
-else
-  stderr = nil
-end
 if d.root.elements.count == 0
   return {
 :debian_uversion => nil,
@@ -97,19 +92,9 @@ module Upstream
 :upstream_version => nil,
 :upstream_url => nil,
 :status => 'error',
-:errors => stderr,
+:errors => nil,
 :warnings => 'uscan returned an empty output'
   }
-elsif d.root.elements['errors']
-  return {
-:debian_uversion => nil,
-:debian_mangled_uversion => nil,
-:upstream_version => nil,
-:upstream_url => nil,
-:status => 'error',
-:errors => (stderr || '') + d.root.elements['errors'].text,
-:warnings => nil
-  }
 elsif not (['debian-uversion', 'debian-mangled-uversion', 'upstream-version', 'upstream-url', 'status'] - d.root.elements.to_a.map { |e| e.name }).empty?
   # At least one of the required fields is missing. Count as error.
   return {
@@ -118,7 +103,7 @@ module Upstream
 :upstream_version => (e = d.root.elements['upstream-version']) ? e.text : nil,
 :upstream_url => (e = d.root.elements['upstream-url']) ? e.text : nil,
 :status => 'error',
-:errors => stderr,
+:errors => (e = d.root.elements['errors']) ? e.text : nil,
 :warnings => (e = d.root.elements['warnings']) ? e.text : nil
   }
 else
@@ -128,7 +113,7 @@ module Upstream
 :upstream_version => d.root.elements['upstream-version'].text,
 :upstream_url => d.root.elements['upstream-url'].text,
 :status => d.root.elements['status'].text,
-:errors => stderr,
+:errors => (e = 

Bug#815980: tracker.debian.org: All packages have uscan issues

[Lucas Nussbaum]

On 03/03/16 at 21:17 +0900, Osamu Aoki wrote:
> Hi,
> 
> I think I can propose peaceful solution.
> 
> On Sun, Feb 28, 2016 at 03:18:21PM +0100, Lucas Nussbaum wrote:
> > On 26/02/16 at 19:10 -0500, James McCoy wrote:
> ...
> > > > My guess is that this was triggered in UDD when devscripts 2.16.1 got
> > > > uploaded to jessie-backports. In devscripts 2.15.10, the interface to
> > > > uscan changed with everything being printed to stderr (even if it's not
> > > > an error) if --dehs is used.
> > > 
> > > Then this is what needs to be fixed.  The behavior, especially for
> > > something that's intended to be used by automation, shouldn't have
> > > changed such that it broke the major user of such functionality.
> > 
> > The code that runs uscan in UDD is
> > http://anonscm.debian.org/cgit/collab-qa/udd.git/tree/rimporters/upstream.rb
> > (starting around line 50).
> 
> I have never learned ruby but the exit code "status" obtained by:
>stdout, stderr, status = Open3.capture3(cmd)
> is not used to determin if the program suceeded or not.
> 
> Instead stderr != '' check is used.  Unexpected :-)
> 
> > As you can see, the code is already quite complicated:
> > uscan warnings are outputted in the  tag, but not all uscan
> > errors are: sometimes they are in an  tag, sometimes they are
> > just displayed on stderr.
> 
> The old code turned off verbose outputs and warnings if --dehs is used.
> This was bad for tweaking watch file etc.  So I activated them while
> making sure stdout os used only by --dehs and stderr has them.
>  
> > It would be better if all uscan errors were displayed in . Then
> > UDD could just ignore the stderr output.
> 
> Now the dehs output marked with  has all uscan errors, if uscan
> starts to process watch file(s) :-)  (I think it was not the case
> previously.)
> 
> But reading what James said, I can satisfy my thought by introducing new
> option --dehs-verbose while making --dehs to be quiet for warning.
> 
> This way, no pain for everyone.
> 
> Let me think a bit more but the above seems to be most rationale thing
> to do.

Well, since you are changing options anyway, what about introducing --xml
that would only output XML (nothing on stderr, never something else than
XML on stdout), displaying warnings and errors using  and
 tags?

If you prefer to keep the changes minimal, I think that --dehs should
continue to display warnings in  so that UDD can import them.

But anyway, I'm fine with changing UDD to match whatever you choose to
do, as long as it doesn't increase the complexity too much.

Lucas

Bug#815980: tracker.debian.org: All packages have uscan issues

[Osamu Aoki]

Hi,

I think I can propose peaceful solution.

On Sun, Feb 28, 2016 at 03:18:21PM +0100, Lucas Nussbaum wrote:
> On 26/02/16 at 19:10 -0500, James McCoy wrote:
...
> > > My guess is that this was triggered in UDD when devscripts 2.16.1 got
> > > uploaded to jessie-backports. In devscripts 2.15.10, the interface to
> > > uscan changed with everything being printed to stderr (even if it's not
> > > an error) if --dehs is used.
> > 
> > Then this is what needs to be fixed.  The behavior, especially for
> > something that's intended to be used by automation, shouldn't have
> > changed such that it broke the major user of such functionality.
> 
> The code that runs uscan in UDD is
> http://anonscm.debian.org/cgit/collab-qa/udd.git/tree/rimporters/upstream.rb
> (starting around line 50).

I have never learned ruby but the exit code "status" obtained by:
   stdout, stderr, status = Open3.capture3(cmd)
is not used to determin if the program suceeded or not.

Instead stderr != '' check is used.  Unexpected :-)

> As you can see, the code is already quite complicated:
> uscan warnings are outputted in the  tag, but not all uscan
> errors are: sometimes they are in an  tag, sometimes they are
> just displayed on stderr.

The old code turned off verbose outputs and warnings if --dehs is used.
This was bad for tweaking watch file etc.  So I activated them while
making sure stdout os used only by --dehs and stderr has them.
 
> It would be better if all uscan errors were displayed in . Then
> UDD could just ignore the stderr output.

Now the dehs output marked with  has all uscan errors, if uscan
starts to process watch file(s) :-)  (I think it was not the case
previously.)

But reading what James said, I can satisfy my thought by introducing new
option --dehs-verbose while making --dehs to be quiet for warning.

This way, no pain for everyone.

Let me think a bit more but the above seems to be most rationale thing
to do.

Osamu

Bug#815980: tracker.debian.org: All packages have uscan issues

[Lucas Nussbaum]

On 26/02/16 at 19:10 -0500, James McCoy wrote:
> Control: reassign -1 devscripts 2.15.10
> Control: retitle -1 "uscan --dehs" shouldn't send all output to stderr
> Control: affects -1 qa.debian.org
> 
> On Fri, Feb 26, 2016 at 12:29:10PM +, James Cowgill wrote:
> > > jcowgill@quantz:~$ psql service=udd -x -c "SELECT * FROM upstream WHERE 
> > > source='easytag'"
> > > -[ RECORD 1 
> > > ]---+-
> > > source  | easytag
> > > version | 2.4.2-1
> > > distribution| debian
> > > release | sid
> > > component   | main
> > > watch_file  | version=3
> > > | 
> > > http://ftp.gnome.org/pub/GNOME/sources/easytag/(\d.)+/ \
> > > | easytag-([\d.]+)\.tar\.xz
> > > | 
> > > signing_key_pgp | 
> > > signing_key_asc | 
> > > debian_uversion | 2.4.2
> > > debian_mangled_uversion | 2.4.2
> > > upstream_version| 2.4.2
> > > upstream_url| 
> > > http://ftp.gnome.org/pub/GNOME/sources/easytag/2.4/easytag-2.4.2.tar.xz
> > > errors  | uscan output on stderr: uscan: Newest version 
> > > of easytag on remote site is 2.4.2, local version is 2.4.2
> > > | uscan warn: No upstream tarball downloaded.  No 
> > > further processing with mk_origtargz ...
> > > | 
> > > | 
> > > warnings| No upstream tarball downloaded.  No further 
> > > processing with mk_origtargz ...
> > > | 
> > > status  | up to date
> > > last_check  | 2016-02-26 00:04:48.688531
> > 
> > My guess is that this was triggered in UDD when devscripts 2.16.1 got
> > uploaded to jessie-backports. In devscripts 2.15.10, the interface to
> > uscan changed with everything being printed to stderr (even if it's not
> > an error) if --dehs is used.
> 
> Then this is what needs to be fixed.  The behavior, especially for
> something that's intended to be used by automation, shouldn't have
> changed such that it broke the major user of such functionality.

The code that runs uscan in UDD is
http://anonscm.debian.org/cgit/collab-qa/udd.git/tree/rimporters/upstream.rb
(starting around line 50).

As you can see, the code is already quite complicated:
uscan warnings are outputted in the  tag, but not all uscan
errors are: sometimes they are in an  tag, sometimes they are
just displayed on stderr.

It would be better if all uscan errors were displayed in . Then
UDD could just ignore the stderr output.

- Lucas

Bug#815980: tracker.debian.org: All packages have uscan issues

[James McCoy]

Control: reassign -1 devscripts 2.15.10
Control: retitle -1 "uscan --dehs" shouldn't send all output to stderr
Control: affects -1 qa.debian.org

On Fri, Feb 26, 2016 at 12:29:10PM +, James Cowgill wrote:
> > jcowgill@quantz:~$ psql service=udd -x -c "SELECT * FROM upstream WHERE 
> > source='easytag'"
> > -[ RECORD 1 
> > ]---+-
> > source  | easytag
> > version | 2.4.2-1
> > distribution| debian
> > release | sid
> > component   | main
> > watch_file  | version=3
> > | 
> > http://ftp.gnome.org/pub/GNOME/sources/easytag/(\d.)+/ \
> > | easytag-([\d.]+)\.tar\.xz
> > | 
> > signing_key_pgp | 
> > signing_key_asc | 
> > debian_uversion | 2.4.2
> > debian_mangled_uversion | 2.4.2
> > upstream_version| 2.4.2
> > upstream_url| 
> > http://ftp.gnome.org/pub/GNOME/sources/easytag/2.4/easytag-2.4.2.tar.xz
> > errors  | uscan output on stderr: uscan: Newest version of 
> > easytag on remote site is 2.4.2, local version is 2.4.2
> > | uscan warn: No upstream tarball downloaded.  No 
> > further processing with mk_origtargz ...
> > | 
> > | 
> > warnings| No upstream tarball downloaded.  No further 
> > processing with mk_origtargz ...
> > | 
> > status  | up to date
> > last_check  | 2016-02-26 00:04:48.688531
> 
> My guess is that this was triggered in UDD when devscripts 2.16.1 got
> uploaded to jessie-backports. In devscripts 2.15.10, the interface to
> uscan changed with everything being printed to stderr (even if it's not
> an error) if --dehs is used.

Then this is what needs to be fixed.  The behavior, especially for
something that's intended to be used by automation, shouldn't have
changed such that it broke the major user of such functionality.

Reassigning to devscripts.

Cheers,
-- 
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy 

Bug#815980: tracker.debian.org: All packages have uscan issues

[James Cowgill]

Control: reassign -1 qa.debian.org
Control: retitle -1 UDD: upstream watch file checker is broken with devscripts 
2.16.1

On Fri, 2016-02-26 at 11:45 +0100, Vincent Danjean wrote:
> Package: tracker.debian.org
> Severity: normal
> 
>   Hi,
> 
>   In the PTS, all packages I currently look at display the following
> "action needed" item:
> * Problems while searching for a new upstream version
> 
>   When looking at the details, it seems to be a connectivity problem
> (and I checked for a few packages that the watch file is correct).

I think this is rooted in UDD which tracker.debian.org gets its data
from. For example, If I run this against UDD it returns "errors" none
of which are actually errors:

> jcowgill@quantz:~$ psql service=udd -x -c "SELECT * FROM upstream WHERE 
> source='easytag'"
> -[ RECORD 1 
> ]---+-
> source  | easytag
> version | 2.4.2-1
> distribution| debian
> release | sid
> component   | main
> watch_file  | version=3
> | 
> http://ftp.gnome.org/pub/GNOME/sources/easytag/(\d.)+/ \
> | easytag-([\d.]+)\.tar\.xz
> | 
> signing_key_pgp | 
> signing_key_asc | 
> debian_uversion | 2.4.2
> debian_mangled_uversion | 2.4.2
> upstream_version| 2.4.2
> upstream_url| 
> http://ftp.gnome.org/pub/GNOME/sources/easytag/2.4/easytag-2.4.2.tar.xz
> errors  | uscan output on stderr: uscan: Newest version of 
> easytag on remote site is 2.4.2, local version is 2.4.2
> | uscan warn: No upstream tarball downloaded.  No 
> further processing with mk_origtargz ...
> | 
> | 
> warnings| No upstream tarball downloaded.  No further 
> processing with mk_origtargz ...
> | 
> status  | up to date
> last_check  | 2016-02-26 00:04:48.688531

My guess is that this was triggered in UDD when devscripts 2.16.1 got
uploaded to jessie-backports. In devscripts 2.15.10, the interface to
uscan changed with everything being printed to stderr (even if it's not
an error) if --dehs is used. If all errors uscan gives are always given
in the relevant dehs element, can the output from stderr simply be
ignored?

James

signature.asc
Description: This is a digitally signed message part

Bug#815980: tracker.debian.org: All packages have uscan issues

[Vincent Danjean]

Package: tracker.debian.org
Severity: normal

  Hi,

  In the PTS, all packages I currently look at display the following
"action needed" item:
* Problems while searching for a new upstream version

  When looking at the details, it seems to be a connectivity problem
(and I checked for a few packages that the watch file is correct).


  I report the bug against the tracker.debian.org pseudo-package because
the problem is shown here. But I CC debian-devel as I'm convinced that the root
issue comes from elsewhere (the PTS is just displaying one information source).
I hope someone will be able to find the root issue and reassing this bug
accordingly.

  Regards,
Vincent

-- System Information:
Debian Release: stretch/sid
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'squeeze-lts'), (500, 
'oldstable-updates'), (500, 'oldoldstable'), (500, 'unstable'), (500, 
'testing'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386, armel, mipsel

Kernel: Linux 4.4.0-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=fr_FR.utf8, LC_CTYPE=fr_FR.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)