Re: [Soc-coordination] project idea: Debian metric portal

[Stefano Zacchiroli]

On Thu, Mar 14, 2013 at 04:54:57PM +0800, Paul Wise wrote:
 On Thu, Mar 14, 2013 at 4:46 PM, Stefano Zacchiroli wrote:
 
  '''Debian metrics portal'''
 
 I guess this isn't about per-package metrics (Mole/PTS is good for
 that), but about things like the ones listed on the stats wiki page?

Correct, there's a link to the Statistics page in the project
description. Given you asked, I'll try to clarify that even further
before posting to the wiki (e.g. by adding project-wide [metrics]
where appropriate).

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: project idea: Debian metric portal

[Stefano Zacchiroli]

On Thu, Mar 14, 2013 at 03:03:57PM +0100, Andreas Tille wrote:
 seems you are desperately seeking new tasks for you past-leader time. ;-)

Indeed: freedom, at last! :)

  It's essentially a refactoring/reengineering initiative to consolidate
  all the scattered metrics graphing initiatives we have throughout the
  project. Those of you familiar with Tor might recognize some
  similarities with http://metrics.torproject.org
 
 Do you also consider including the results of the teammetrics projects
 of GSoC 2011 and 2012[1].
 [1] https://lists.debian.org/debian-project/2013/01/msg00011.html

I don't consider including *specific* data sources as an objective for
the project. But surely including any relevant project-wide metric out
there should be easy enough, otherwise I wouldn't consider the result
successful. On the other hand, team metrics are kind of weird, as
they're neither project-wide (meaning: one metric per project), nor as
abundant as package (meaning: one metric per package). So, I still don't
know whether it would be a good fit. We'll see, I guess. Let's first get
the infrastructure in place, then you'll be free to (ab)use it as you
please :-)

Thanks for your feedback,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: [GSoC] Using a messaging system to interface the different parts of Debian infrastructure

[Stefano Zacchiroli]

On Tue, Apr 09, 2013 at 10:47:40PM +0200, Nicolas Dandrimont wrote:
  I'm a prospective GSoC student as well as a Debian user and contributor,
  and was looking for project ideas to submit for this year's GSoC when
  Nicolas Dandrimont (olasd) mentionned to me fedmsg[1] and the current
  state of interfaces throughout the Debian infrastructure.
 
 Well, you beat me to writing that mail. Great!

Hear hear. I've never heard of fedmsg before, but it does sound a great
architecture we should look into.

  So, anyone interested in the project, esp. for mentoring ? :-)
 
 I'd be glad to. It'd be interesting to get a willing service maintainer or two
 onboard, so that real services can get wired into the PoC. Also, maybe someone
 from Fedora could be interested in joining (I haven't talked about that yet, 
 so
 it's just an idea).

It's great to read that you're up to mentoring this, Nicolas, thanks.

I've already committed all my mentoring bandwidth for GSoC 2013 to other
projects, but if you two could keep -qa in the loop of this project
discussions I'll be happy to chime-in whenever feasible, especially in
connection with the PTS.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Supporting semantic formats for meta-data would be interesting - Was: Re: [GSoC] Using a messaging system to interface the different parts of Debian infrastructure

[Stefano Zacchiroli]

On Fri, Apr 12, 2013 at 10:20:43AM +0200, Simon Chopin wrote:
 While I like the idea of semantics in the data, if I receive a message
 from dak saying that apache2 has been uploaded, I don't expect it to
 contain anything else than the name of the package for the simple reason
 that it's pretty much all dak knows about it at the time of the upload.

+1

The reference universe of Debian source packages (at any given point
in time) is entirely well defined within Debian.  If external sources
need to clarify that the package name is specific to Debian, then it
should be up to them to add the appropriate (URI) prefix.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: PTS rewrite in Django

[Stefano Zacchiroli]

On Sat, Apr 20, 2013 at 10:20:13AM +0800, Paul Wise wrote:
 On Sat, Apr 20, 2013 at 3:01 AM, Pankaj Kumar Sharma wrote:
  Till now I have gone through the codebase of PTS and have written a
  small patch as suggested in [1] and it was really fun (and got to
  learn many things). Offcourse team QA needs to be thanked for that.
 
 Could you share the patch so we can include it in the current PTS?

I think he referred to the trial patch meant to get familiar with the
PTS code base, nothing particularly useful for the PTS. That will come
later ;-)

-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Guidance required for GSoC project PTS rewrite in Django

[Stefano Zacchiroli]

On Wed, Apr 24, 2013 at 10:28:07PM +0530, Pankaj Kumar Sharma wrote:
 In the present system the content is loaded explicitly via cron.  The
 confusion that bounds me is that what should be methodology that we should
 use in the upcoming Django project ? Should the data be loaded at the time
 when some one asks for that or it should be present in the databases ?

The short answer is: we'll have to experiment with that :)

Some of the information that the PTS exposes (e.g. those related to the
status of the archive) are fairly static, meaning that they change at
most 4 times a day. Others are very dynamic (e.g. bug information) and
ideally should really be live, as it could be really confusing for a
user to see that, say, a package has 1 RC bug, click on the bugs link,
and discover that that's not true. It's not true *anymore*, but the
random user would have no way of understanding that and think it's a
bug. This kind of incoherences has been an endless source of (bogus) bug
reports along the PTS life.

A separate question is how to make all this efficient, in term of
caching. Obviously, the current solution with static HTML pages is very
fast and is also easy to mirror in case of need.  A purely dynamic
solution would be on the opposite end of the spectrum in terms of
performances. We will probably need to stay somehow in the middle, and
benchmark the scalability of the new solution (as mentioned in the
project description).

Ideally, we should cache heavily, either by using Django caching, or by
producing actual HTML pages via Django templates (as mentioned by Paul
in this thread). And add on top of it heavy cache invalidation
mechanisms for live information, like bugs.  Alternatively, we might
want to cache only the information that are seldomly updated and be
entirely dynamic on the live information.

Regarding where the data come from, my dream would be to develop a
Python abstraction layer over all the data that the PTS uses. And then
have various implementation (backends) of it. One can for instance
access directly UDD, another can access a local cache updated by cron
(as in the current PTS deployment), another be entirely live, and yet
another use mixed solutions. That would allow to more easily experiment
with the different solutions.

Hope this explains that we don't have yet written-in-stone-answers to
your question, and that finding out, via experiments, the right
trade-offs will be part of the actual project.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Guidance required for GSoC project PTS rewrite in Django

[Stefano Zacchiroli]

On Fri, Apr 26, 2013 at 10:13:44AM +0800, Paul Wise wrote:
  I'm not expecting the students to contribute code to UDD.
 
 Stefano is an admin in collab-qa and a mentor for this project, so
 that should be enough?

To clarify: I'm still a UDD admin for historical reasons, having
contributed to the initial design/implementation. But I haven't really
done any significant hacks on UDD since then. If anything, I should
probably be removed as an UDD admin to bring the perception of who's
active hacking on UDD closer to reality.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: PTS: RC bugs in dependencies

[Stefano Zacchiroli]

On Fri, May 10, 2013 at 02:24:56PM +0800, Paul Wise wrote:
  I started from RC bugs (grave, critical, serious) tagged help.  We 
  currently
  have 18 RC bugs tagged help.  I matches these 18 bugs with all reverse
  dependencies, recursively, both build- and plain reverse dependencies.

Thanks for doing this work, Bart!

  Comments ?
 I wonder if doing this recursively is a good idea or not. If we decide
 to do that, your approach to limit the recursion is a good one.

AOL on the fact that doing it recursively is _not_ a good idea. My main
argument for it is that direct dependencies are those that the
maintainers took the time to explicitly add (either manually, or by
delegating that choice to some helper tool that fills in substvars). As
such, those dependencies are those that are most likely to be meaningful
to the package maintainer.  And of course there is the bloat risk, that
could diminish the usefulness of this check.

Other than that, it sounds like a very good idea to me too.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: PTS: revamp style

[Stefano Zacchiroli]

On Mon, May 27, 2013 at 06:12:04PM +0200, David Suárez wrote:
 On Lunes, 27 de mayo de 2013 06:57:59 Paul Wise escribió:
  Unfortunately both of these URLs just give me some PHP errors. Could
  you attach a patch or new CSS file instead?
 
 Done.

Hey David, thanks for your proposed changes.

 -a:link{ color: #0755d7; text-decoration: none; }
 -a:visited { color: #0755d7; text-decoration: none; }
 -a:hover   { color: #0755d7; text-decoration: underline; }
 -a:active  { color: #f00;text-decoration: none; }
 +a:link{ color: #005090; text-decoration: none; font-weight: bold; }
 +a:visited { color: #005090; text-decoration: none; font-weight: bold; }
 +a:hover   { color: #D70767; text-decoration: none; font-weight: bold; }
 +a:active  { color: #f00;text-decoration: none; font-weight: bold; }

I've some comments about this. Personally, I think your example uses
bold typefaces way too much. Bold typefaces should be used sparingly, to
highlight important stuff, not as a de facto default to typeset all the
informative content of a page. That is, in fact, the first thing that
caught my eye when looking at the page you posted here for comments.

Can you please reconsider the use of bold typefaces?

(I haven't yet looked much at the other changes you propose, so I can't
comment on them ATM.)

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

[RFC] introducing sources.d.n

[Stefano Zacchiroli]

Heya, as anticipated yesterday on IRC, I've a new toy to present and I'm
looking for comments, suggestions, ... and patches! ;) for it. Ladies
and gentlemen:

  http://sources.debian.net/

My initial need was to have a stable link base to cross-reference
results of static analysis tools to actual source code in the Debian
archive, without having to reinvent the code browsing/highlighting wheel
each time. E.g.:

  
http://sources.debian.net/src/cowsay/3.03%2Bdfsg1-4/cowsay?hl=22:28msg=22:Cowsay:See?%20Cowsay%20variables%20are%20declared%20here.#L22

in a single place. But it quickly turned into something that sounded
useful as a more general tool, so I've grabbed the sources.d.n DNS
entry. People thus far seem to like it: sources.d.n has already been
integrated into codesearch.d.n, which will link back to sources.d.n
(with graceful degradation if needed), as well as into the PTS, which
has grown a new browse source code link.

The code base is updated 4 times a day and currently follows stable,
testing, unstable, and experimental. Also, disappeared versions are kept
around for 14 days before being removed, to reduce the likelihood of
dangling links from services that are updated less often
(e.g. codesearch itself, which is updated weekly).

If you've other ideas about where sources.d.n should be integrated,
please shout. The code base is still young, and there are plenty of
pending items to be worked on (see TODO file in the Git repo [1]). Feel
free to mention additional issues here.

[1]: http://anonscm.debian.org/gitweb/?p=qa/debsources.git;a=blob;f=TODO;hb=HEAD


Acknowledgements


Matthieu Caneill, who is subscribed to this list, is the main developer
of the web part of sources.d.n; the service wouldn't exist without him.
Thanks a lot, Matthieu! Other people have contributed patches to
integrate sources.d.n with various parts of the Debian infrastructure
already: many thanks to Michael Stapelberg (for codesearch.d.n
integration) and Paul Wise (for PTS integration). See [2] for a full
list of contributors.

[2]: 
http://anonscm.debian.org/gitweb/?p=qa/debsources.git;a=blob;f=AUTHORS;hb=HEAD


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#699936: clang integrated in PTS

[Stefano Zacchiroli]

On Sat, Aug 17, 2013 at 02:34:03PM +0200, Giovanni Mascellani wrote:
 Mostly I don't like PTS links to be too volatile; for me, the presence
 of a clang link just says that the clang service is meaningful with
 respect to this (source) package. It is reasonable to add another small
 piece of information to avoid wasting the users' time, but just loosing
 the link to clang is (in my opinion) not very friendly.
 
 Anyway, I think it is not the first time I have a different opinion from
 the design principles of PTS. I can live with it, my opinion is not
 really strong.

So, the design principle at stake here is slightly different. The point
of it has always been to avoid having visitor click on a link just to
discover whether there is something for them there or not.

Such a principle has often been implemented by making links volatile,
as you say, but that's not the only way to do it.  In this specific case
one can for instance have a link and decorate it with a trailing (X)
where X is the number of things to be seen there.

This way, the visitor can learn that they do not need to click if the
annotation is not there, and everyone else who want the link to be
always present can be happy too.

OTOH, if it really there is *nothing to see* in the (0) case, it is
indeed probably pointless to have the link in the first place. YMMV.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#752375: QA page shows Git ERROR because it uses only debian master branches

[Stefano Zacchiroli]

Hi Thomas,

On Mon, Jun 23, 2014 at 02:18:26PM +0800, Thomas Goirand wrote:
 For some of the packages I maintain, I get a Git ERROR showing.
 Clicking on the Git link shows a working git repository. Clicking on
 the ERROR link shows:

Can you please be specific? In particular, can you name a specific
package you maintain that currently exhibits the problem you're
reporting?

That way it'd be easier to verify this bug (and possibly a fix for it).

TIA,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Help me with tracker.debian.org

[Stefano Zacchiroli]

On Fri, Jul 04, 2014 at 02:34:58PM +0200, Raphael Hertzog wrote:
 What about just updating those links everywhere to point to the new
 package tracker?
 
 Do people feel that it's too early for this?

IMHO, the right way to go about this is having a grace period during
which both services are available and mutually link to each other.

For now we should:

- patch the old PTS to prominently show a link like hey, you know
  that there is tracker.d.o now? take a look!. The link should exist on
  every package page and should point to the corresponding package page
  on the new PTS

- patch the new PTS to have a link like wanna go back to the old PTS?
  click here, and maybe add some language pointing people that miss
  some of the old features in the new implementation to how they can
  contribute. Bonus point: make the link appear only for people that
  come from the old PTS (e.g. by inspecting referrer HTTP headers)

When we feel ready to do the big switch we should:

- not bother going all around the Debian infrastructure to fix links to
  the old PTS in a big batch, but rather redirect old URLs to the new
  PTS at the HTTP level

- bonus point: create new URLs that point to the old PTS
  (e.g. old-pts.debian.net) and keep them around for a while, depending
  on the acceptance that the new PTS will have gained in the community


My 2 cents,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#756834: use https://tracker.d.o links to point to source packages

[Stefano Zacchiroli]

Package: how-can-i-help
Version: 6
Severity: minor
Tags: patch

Given tracker.d.o is on its way to replace the old PTS interface, it would be
nice for how-can-i-help to use the new adress scheme.

The attached patch should implement this,
Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages how-can-i-help depends on:
ii  ruby 1:2.1.0.1
ii  ruby-debian  0.3.8+b4
ii  ruby-json1.8.1-1+b1

how-can-i-help recommends no packages.

how-can-i-help suggests no packages.

-- no debconf information
From 850bbb372c8cb2be194f7ddf571441bef81a721f Mon Sep 17 00:00:00 2001
From: Stefano Zacchiroli z...@debian.org
Date: Sat, 2 Aug 2014 12:26:55 +0200
Subject: [PATCH] use https://tracker.d.o links to point to source packages

---
 bin/how-can-i-help | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/bin/how-can-i-help b/bin/how-can-i-help
index a3ad7b9..28b0497 100755
--- a/bin/how-can-i-help
+++ b/bin/how-can-i-help
@@ -229,7 +229,7 @@ end
 if notesting.length  0
   puts $old ? 'Packages removed from Debian \'testing\' (the maintainer might need help):' : 'New packages removed from Debian \'testing\' (the maintainer might need help):'
   notesting.sort { |a, b| a['source'] = b['source'] }.each do |r|
-puts  - #{r['package']} - https://packages.qa.debian.org/#{r['source']}
+puts  - #{r['package']} - https://tracker.debian.org/pkg/#{r['source']}
   end
   puts
 end
@@ -245,7 +245,7 @@ if autoremoval.length  0
 else
   bugs =  (bug: #{bugs[0]})
 end
-puts  - #{r['source']} - https://packages.qa.debian.org/#{r['source']} - removal on #{Time.at(r['removal_time']).to_date.to_s}#{bugs}
+puts  - #{r['source']} - https://tracker.debian.org/pkg/#{r['source']} - removal on #{Time.at(r['removal_time']).to_date.to_s}#{bugs}
   end
   puts
 end
-- 
2.0.1

Re: Bug#756117: Resurrecting patch-tracker.d.o

[Stefano Zacchiroli]

On Tue, Sep 02, 2014 at 09:03:59PM +0100, Colin Watson wrote:
 Wouldn't a better approach be to resurrect the existing service as-is
 for the sake of the people who want to use it today, and *then* worry
 about getting somebody to reimplement it on top of a different
 framework?

Yes, definitely.

(Unless we have multiple volunteers for doing the work, including some
that are ready to do both the resurrection and the refactoring --- but
that doesn't seem to be the case yet.)

-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761077: debsources: support tarball-in-tarball

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
Control: usertags -1 + debsources

Debsources should support tarball-in-tarballs, offering the ability to browse
tarball content.

Rationale #1: support (bad) packaging habits

Rationale #2: support tarballs shipped by upstreams

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140910152922.11699.17104.reportbug@timira.takhisis.invalid

Bug#761079: debsources: support multiple archives

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
Control: user qa.debian@packages.debian.org
Control: usertags -1 + debsources

Debsources currently only supports one (live) archive, i.e., the main
Debian archive as available from the mirror network.  We want to support
multiple archives. The main use case is the ability to inject the
security mirror into Debian Sources, but there are others (e.g.,
derivatives).

To do so, we will need to do separate debmirror runs, one for each
archive, and decide what's the policy for running the Debsource updater
(at each archive update? periodically? etc.)

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761084: debsources: support for the security archive

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
Control: block -1 by 761079

Packages shipped by the Debian security archive should be imported by,
and visible via Debian Sources.


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761083: debsources: inject binary packages metadata into the DB

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal

The Debsources DB schema has tables for collecting binary package
metadata (i.e. binaries, binary_names), but the updater does not
fill them in at present. It should.

Having this information would allow to implement binary package -
source package redirection and more.


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140910163238.ga16...@upsilon.cc

Bug#761085: debsources: automatic binary-source package name redirection

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
Control: block -1 by 761083

As per subject (and as implemented by several other source package
related services in the Debian ecosystem, e.g., tracker.d.o).


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761100: debsources: show suites/releases when listing package versions

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist

When visiting URLs like http://sources.debian.net/src/linux, the user
should be able to which package version belongs to which suites /
releases. (Note that one version might be in multiple suites.)

Sample desired output (proposed by Stuart Prescott):

  1.15.11 (main) [squeeze]
  1.16.15 (main) [wheezy]
  1.17.10 (main)
  1.17.12 (main)
  1.17.13 (main) [jessie, sid]


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761103: debsources: highlight #Lxxx lines by default

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist

[ bug originally reported by Joachim Breitner ]

URLs like
http://sources.debian.net/src/cairo/1.12.14-4/boilerplate/cairo-boilerplate-xcb.c#L395
should highlight the line mentioned in the anchor by default, without
requiring the user to pass ?hl.

Ideally, the line should be put in the middle of the screen, instead of
the top.

(Implementing this likely requires some JavaScript fiddling.)

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761105: debsources: on the fly package diff / debdiff

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist

Add the ability to diff arbitrary version of packages available in
Debsources, producing a debdiff as a result.

This should probably come as a new /diff URL, taking package
names/versions as parameters and returning the debdiff.

Bonus point: offer the ability to alternatively download or typeset the
debdiff in the browser (with diff-like syntax highlight).


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761106: debsources: cannot list dir content for packages with unicode issues

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal

[ bug originally reported by Luciano Bello ]

As an example of this issue, http://sources.debian.net/src/cvsnt/latest/
currently returns 505, due to a contained file named testáéíóú.txt
(and non utf8 clean).

The failure is at debsources/web/models.py, line 571, at os.path.join()


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761108: debsources: file name / path search

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist

We already have all paths in the DB, it would trivial and very useful to
offer file/path-based search functionalities.

The only caveat is query execution team.
Some preliminary micro-benchmarks:

- it is too slow without decent index:

   debsources= select count(*) from files where path like '%stdio%';
   count
  ---
9643
  (1 row)

  Time: 7736,861 ms

- but it works pretty decently with trigram indexing:

  debsources= \d ix_files_path_trgm
Index public.ix_files_path_trgm
   Column |  Type   |  Definition
  +-+--
   encode | integer | encode(path, 'escape'::text)
  gin, for table public.files

  debsources= select count(*) from files where encode(path, 'escape') like 
'%stdio%';
   count
  ---
9643
  (1 row)

  Time: 77,558 ms

  To work, the above requires CREATE EXTENSION 'trgm';


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761117: debsources: file-level deduplication

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist

We already have all the file checksums in the database. Removing
(file-level) duplication in the file storage, using hard-links, can be
safely implemented offline, i.e., as long as no debsources update is
ongoing.

Micro-benchmark (from my DebConf14 Debsources talk) of the expected disk
space saving:

select count(*) from checksums;- 35'370'653
select count(distinct sha256) from checksums;  - 15'822'745
  --
  = deduplicated core: ~45%

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761118: debsources: more statistics

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist

Several interesting statistics and graphs have been computed for the
ESEM 2014 paper [1] by Debsources' authors.  Unfortunately, only some of
them have been turned into live data that is recomputed at each update
and published at http://sources.debian.net/stats/

We should do so.

[1]: https://upsilon.cc/~zack/research/publications/debsources-esem-2014.pdf


On the same vein, we should refine the disk usage graph to also take
into account the postgres and mirror sizes.  Finally, once binary
package information will be injected in the DB, we should graph the
number of *binary* packages over time.


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761119: debsources: suite-based navigation

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist

Debsources should offer per-suite/release navigation, e.g., with URLs
like http://sources.debian.net/suite/jessie/prefix/d/


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761121: debsources: allow symlink within same source package/version

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist

Currently, all symlinks shipped as part of source packages are forbidden
and cannot be accessed for security reasons. Debsources should be more
clever about that, and allow access to symlinks that point with the same
package/version.


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761124: debsources: proper pythonic package layout

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal

The layout of Debsources as upstream software should be changed to
match Python best practices (provide a top-level debsources module, have
use setuptools, etc.).

This will make it easier to package Debsources, deploy it as a regular
Debian package, and upload it to the Debian archive to the benefit of
other distros.


Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761124: debsources: proper pythonic package layout

[Stefano Zacchiroli]

Note (to self and others): work on this issue, and more generally on the
Debian packaging of Debsources, is in progress in the debian-packaging
branch of the Git repo:

  http://anonscm.debian.org/cgit/qa/debsources.git/log/?h=debian-packaging

Feedback and patches from expert Python packagers are welcome!

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761079: debsources: support multiple archives

[Stefano Zacchiroli]

On Thu, Sep 11, 2014 at 01:42:20PM +0800, Paul Wise wrote:
 On Wed, Sep 10, 2014 at 11:56 PM, Stefano Zacchiroli wrote:
 
  (e.g., derivatives).
 
 For derivatives, the best thing will be replacing debmirror with
 rsyncing sources.list files (and or apt directories) from the derivs
 census plus apt-get update and apt-get source.

Do you maybe already have code that does this (at list the first part,
up to some sort of containerized apt-get update) as part of the census
script? If so, mind posting to this bug report pointers to it?

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761105: debsources: on the fly package diff / debdiff

[Stefano Zacchiroli]

On Thu, Sep 11, 2014 at 01:47:33PM +0800, Paul Wise wrote:
 On Thu, Sep 11, 2014 at 3:01 AM, Stefano Zacchiroli wrote:
 
  Add the ability to diff arbitrary version of packages available in
  Debsources, producing a debdiff as a result.
 
 FYI, we were thinking about adding debdiff capabilities to
 snapshot.debian.org, that might make more sense since it has more
 package versions?

We're gonna need it on debsources anyhow, in particular to implement the
edit feature suggested by Raphael Geissert. What we could do is to
factorize as much as possible the common parts in a common place, e.g.,
python-debian. But I'm not sure there is actually that *much* code to
write, considering that 1) we will probably invoke the real debdiff as
an external program anyhow and 2) I plan to delegate diff highlighting
to the javascript toolkit we already use. What else is there to be done?

-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761117: debsources: file-level deduplication

[Stefano Zacchiroli]

On Thu, Sep 11, 2014 at 02:09:35PM +0800, Paul Wise wrote:
 A hash based filesystem layout like we use on snapshot.d.o.
 
 Use a filesystem with deduplication support like btrfs.

I thought about btrfs back in the days, and ruled out the idea because
it imposes a fairly important deployment requirement.

Regarding a hash-based filesystem layout, that will get in the way of
dpkg-source -x, meaning you will need to massage the files into the
has layout after package extraction. Plus, you lose the ability to use
the natural file organization as the url structure that you present to
the user.

All in all, offline deduplication seems much more appealing and, up to
now, it seems to have no drawbacks whatsoever (except a negligible lag
between the extraction time and the deduplication run).

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761105: debsources: on the fly package diff / debdiff

[Stefano Zacchiroli]

On Thu, Sep 11, 2014 at 03:48:38PM +0800, Paul Wise wrote:
 On Thu, Sep 11, 2014 at 2:58 PM, Stefano Zacchiroli wrote:
 
  We're gonna need it on debsources anyhow, in particular to implement the
  edit feature suggested by Raphael Geissert. What we could do is to
  factorize as much as possible the common parts in a common place, e.g.,
  python-debian. But I'm not sure there is actually that *much* code to
  write, considering that 1) we will probably invoke the real debdiff as
  an external program anyhow and 2) I plan to delegate diff highlighting
  to the javascript toolkit we already use.
 
 I see, some factors I can think of:

So, from your examples I've realized that I've mentioned debdiff, but I
actually don't need it. Debsources currently have unpacked packages on
the filesystem, organized in per-version directories. So Debsources can
simply recursively diff the two directories, instead of using debdiff
(which AFAICT, doesn't even work on directories). We'll just need to
look into whether debdiff uses specific diff options that we want to use
as well, just to ensure that the output format is more or less the same.

(FWIW, this seems to be yet another good argument against some
hash-based file-layout, that we have been discussing in #761117.)

Most of the legitimate problems you've mentioned seems to be related to
source package format and, arguably, we won't have those problems with
Debsources, or at least not in the diffing part. (We will have them at
source package unpack time, of course.)

  What else is there to be done?
 
 Mainly having a full set of packages to debdiff between, from our
 discussions at DebConf14 it seems like you plan to get a copy of the
 snapshot archive anyway so maybe sources.d.n is indeed the right place
 to do this?

That's the plan yes, even though importing snapshot.d.o is a very
different feature/goal.

However, it is important to observe that sources.d.n still aims at being
source-package-only, whereas 1) debdiff is capable of diffing .deb and
2) .debs are indeed available on snapshot.d.o. So if you are interested
in diffing .debs, as of now the only place where to implement that
specific feature is snapshot.d.o.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761227: debsources: add totals to all /stats tables

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

Some tables under http://sources.debian.net/stats/ have totals (i.e., not only
per-suite entries), some don't. We should uniform the rendering, and provide
totals for all tables.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140911214113.20752.19278.reportbug@timira.takhisis.invalid

Bug#761228: debsources: make pie charts more readable

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

Most pie charts --- both under /stats and on the per-release pages,
e.g. http://sources.debian.net/stats/squeeze/ --- are almost unreadable, due to
the superposition of language labels. We should hide minor languages, i.e. pie
slices that don't reach a given thresholds, and add them up under a common
other label. The precise numbers are visible in the table anyhow for those
who are interested.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140911214301.20814.90443.reportbug@timira.takhisis.invalid

Bug#761231: debsources: add ctags count to the package infobox

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

The package infobox (i.e. the box shown on the right when navigating a specific
version of a package, see e.g. http://sources.debian.net/src/ocaml/latest/)
currently shows only some statistics about the package. It should show all
available ones. In particular, it should show the ctags count.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140911214707.20917.50545.reportbug@timira.takhisis.invalid

Bug#761232: debsources: release pages should mention release number and date

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

Release pages (e.g. http://sources.debian.net/src/ocaml/4.01.0-4/) should
expose the release meta-data currently stored in the Debsources DB, in
particular release date and release version.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140911214828.20983.5919.reportbug@timira.takhisis.invalid

Bug#761861: allow to override detected language type

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
User: qa.debian@packages.debian.org
Usertags: debsources

[ bug originally reported by Stuart Prescott ]

The heuristics used to detect language type could be wrong (of course), it
would be nice to allow overriding detected language type with a ?lang=... URL
parameter.

A related problem is that we might wrongly detect that some file is not a
text-like file, and hence only offer to download it, rather than render it on
the web. E.g. http://sources.debian.net/src/make/latest/doc/make.info-4/
currently can only be downloaded, in spite of info being a textual file format.

In terms of code the semantic of a new lang URL parameter should therefore be
twofold:

- on one hand its presence should override the detection done by libmagic to
  decide whether to render or download a file (see
  http://anonscm.debian.org/cgit/qa/debsources.git/tree/python/models.py#n652 )

- on the other hand the language specified should override the heuristic based
  language detection. To simply force web display without requiring any
  highlighting, we could use something like ?lang=none


-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140916121836.15349.10059.reportbug@timira.takhisis.invalid

Bug#761863: debsources: use relative paths in cache/sources.txt

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
User: qa.debian@packages.debian.org
Usertags: debsources

A typical cache/sources.txt line currently looks like this:

  susv3   6.1   contrib 
/srv/debsources/testdata/mirror/pool/contrib/s/susv3/susv3_6.1.dsc 
/srv/debsources/sources/contrib/s/susv3/6.1 jessie,squeeze,wheezy,sid

the use of absolute paths is undesirable and has no good reason to exist.
We should use relative paths:

- 4th field should be relative to conf['mirror_dir']
- 5th field should be relative to conf['sources_dir']

(see test_updater.py for reference)

bin/foreach should then be adapted to work properly with relative paths
sources.txt.


-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/2014091614.15661.70245.reportbug@timira.takhisis.invalid

Bug#761864: debsources: /latest redirection for /data URLs

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

[ bug originally reported by Simon Paillard ]

/latest redirection currently works only for the webapp, but not for the static
data served directly by the web server, and most notably stuff under /data . We
should be uniform and implement /latest redirection for /data

To implement that, we could either generate static redirection maps for Apache
at each update run. Or decide that /data stuff can be served by the webapp
(assessing the corresponding performance penalty) and reuse the code we already
have in place for the redirection.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140916122532.15868.50687.reportbug@timira.takhisis.invalid

Bug#761867: debsources: conjunctive (AND-ed) ctags search

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

We should have a new kind of search under /search that allows to search for
files that contain all of a given number of ctags symbols provided by the user.

Note that, differently from the current ctags search that returns indivudual
locations within files, this new search should probably return entire files,
possibly highlighting the lines containing the requested ctags.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140916122813.15935.4791.reportbug@timira.takhisis.invalid

Bug#761869: debsources: update statistics stage is too slow

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
User: qa.debian@packages.debian.org
Usertags: debsources

The update statistics stage of Debsources updated is currently too slow,
taking ~12 minutes on the current sources.d.n machine.

It could be easily optimized by avoiding redoing queries for each live suites
(currently: 9), where each query will do a sequential scan (due to count(*),
despite index mostly scan) over the same data.  Instead, we can use GROUP BY
queries, taking at once stats for all suites.

(See proof of concept and benchmarks available in
doc/update-stats-query.bench.sql)


-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140916123840.17340.76075.reportbug@timira.takhisis.invalid

Bug#762109: tracker.debian.org: could use changelog parser to pretty print debian changelogs

[Stefano Zacchiroli]

On Fri, Sep 19, 2014 at 10:46:31AM +0200, Raphael Hertzog wrote:
 In any case, Distro Tracker is 100% Python and I don't see us relying on
 libparse-debianchangelog-perl to generate pretty changelogs. So this
 wishlist is likely to sit here for a long time.

python-debian ships a changelog module, that can be used to traverse
changelogs. AFAIR there is no (HTML) renderer, though. If someone is
interested in implementing one to fix this feature request against
tracker.d.o, it will be nice to push it upstream as a proper
python-debian feature; I suspect it might be useful to others.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#754658: please display the package's description

[Stefano Zacchiroli]

[ quoted text edit to add references ]

On Tue, Sep 16, 2014 at 11:39:13PM +0800, Paul Wise wrote:
 1) When only one binary package, use the description from it.
 
 2) When more than one binary package but one has the same name as the
 source package, use the description from that.
 
 3) Otherwise, use source package.

Uhm, interesting, I always thought the heuristic was slightly
difference.

Is there an argument for not replacing point (3) above with:

  (3*) Otherwise, use the description of the first binary in
  debian/control order

My gut feeling is that (2) is enough to counter most of the misleading
results that (3*) alone would produce.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#762933: debsources: webapp should not require python-matplotlib

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
User: qa.debian@packages.debian.org
Usertags: debsources

Since the refactoring with the top-level debsources/ module, the webapp
requires python-matplotlib to be installed to run properly. That is
unfortunate, as matplotlib is needed only by the updater, and induces weird
requirements on the webserver user (e.g., a writable /var/www/.matplotlib ?!!).

We should track down and remove this dependency.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140926123053.12792.80598.reportbug@timira.takhisis.invalid

Bug#762931: debsources: flake8 compliance

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
User: qa.debian@packages.debian.org
Usertags: debsources

At present:

  /srv/debsources$ flake8 debsources/ | wc -l
  228

We would like to make all Python source files flake8 (i.e. PEP8 + pyflakes)
compliant, modulo justified (and documented) exceptions.

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140926122901.12742.76984.reportbug@timira.takhisis.invalid

Bug#762934: debsources: refactoring - move queries out of models.py

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
User: qa.debian@packages.debian.org
Usertags: debsources

models.py contains several classes which are used only by the web app and that
have nothing to do with the ORM layer, e.g. Location, Directory, SourceFile.
Similarly, models.py also contains static methods
(e.g. PackageName.list_versions_from_name) that should better be moved to a
query-specific module.

We should refactor the code so that models.py contains only the ORM
abstraction, and move query code to separate modules.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140926123517.12853.42598.reportbug@timira.takhisis.invalid

Bug#762935: debsources: namespace qualified ctags

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
User: qa.debian@packages.debian.org
Usertags: debsources

[ originally reported by Olly Betts on #debian-devel ]

Ctags indexing is currently non-qualified, so indexing on OO langauges will not
contain class names, namespaces, etc.

To fix this we should add --extra=+q to ctags invocation. About this, the ctags
documentation says: However, that this could potentially more than double the
size of the tag file.

Micro benchmarks:
- computation time is ~10-15% slower with +q
- size
  - libreoffice (C++)
zack@tytso:~/ctags-bench/libreoffice-4.1.5$ wc ../libreoffice*ctags
561396   3872260  61123731 ../libreoffice-4.1.5.ctags
972942   6799455 117133590 ../libreoffice-4.1.5.extra+q.ctags
  - chromium-browser (C/C++)
zack@tytso:~/ctags-bench/chromium-browser-32.0.1700.123$ wc 
../chromium-browser*ctags
1618199  10931585 205005488 ../chromium-browser-32.0.1700.123.ctags
2472627  17064351 342772532 ../chromium-browser-32.0.1700.123.extra+q.ctags
  - linux (C)
zack@tytso:~/ctags-bench/linux-3.12.9$ wc ../linux*ctags
2146431  14340875 208057034 ../linux-3.12.9.ctags
2610525  17780208 269391932 ../linux-3.12.9.extra+q.ctags

Bottom line: it's doable.  (But will need to rerun all ctags indexing after
having changed the ctags plugin code.)

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140926124101.13044.73655.reportbug@timira.takhisis.invalid

Bug#762941: debsources: popup messages between lines instead of on the right margin

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
User: qa.debian@packages.debian.org
Usertags: debsources

We should add the ability of showing popup messages between source code lines,
rather than showing them on the right margin.

- rationale #1: popups might be overlooked for very long source code lines
  e.g.:
  
http://sources.debian.net/src/vim/2:7.3.547-7/runtime/syntax/sicad.vim?hl=133msg=133%3Atest%3Atest#L133

- rationale #2: between-the-lines is the only sane way to show multiple
  pop-ups, and at the same time aviud that they clash with one another

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140926125719.13645.98477.reportbug@timira.takhisis.invalid

Bug#762944: debsources: make .pc/ exclusion a configuration parameter

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

Currently, the exclusion of the .pc/ dir from source package navigation is
hard-coded, see:

  
http://anonscm.debian.org/cgit/qa/debsources.git/tree/debsources/models.py#n578

It should become a configuration entry, likely using file-globbing.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140926130426.16655.41448.reportbug@timira.takhisis.invalid

Bug#762946: debsources: multiple popup messages

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

Currently, the URL scheme to associate popup messages to specific lines of code
(see http://sources.debian.net/doc/url/) supports only one message at a time.
We should support an arbitrary number of them (with some length limit to avoid
abuses). That would be needed to support single URLs that pinpoint several
issues found by static analysis tools in a single file.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140926130826.16846.74918.reportbug@timira.takhisis.invalid

Bug#762951: debsources: increase/maximize test coverage

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

Debsources' test coverage is decent:

  /srv/debsources$ nosetests --with-coverage debsources/ -v 
--cover-package=debsources
  [...]
  /srv/debsources$ python-coverage report 
  NameStmts   Miss  Cover
  ---
  debsources/__init__ 2  0   100%
  debsources/app/__init__ 2  0   100%
  debsources/app/app_factory 40  393%
  debsources/app/extract_stats9  0   100%
  debsources/app/forms5  0   100%
  debsources/app/infobox 54 1180%
  debsources/app/pagination  20  195%
  debsources/app/sourcecode  71 3255%
  debsources/app/views  342 6780%
  debsources/archiver   104 2279%
  debsources/charts  42 3029%
  debsources/consts   9  0   100%
  debsources/dbutils 51  198%
  debsources/debmirror  128 3473%
  debsources/excepts 16  0   100%
  debsources/filetype58 2360%
  debsources/fs_storage  68 2071%
  debsources/hashutil20  860%
  debsources/local_info  17  382%
  debsources/mainlib145 7747%
  debsources/models 328 2592%
  debsources/plugins/__init__ 0  0   100%
  debsources/plugins/hook_checksums  73  790%
  debsources/plugins/hook_ctags  99 1486%
  debsources/plugins/hook_hello  10 10 0%
  debsources/plugins/hook_metrics51  0   100%
  debsources/plugins/hook_sloccount  70  790%
  debsources/sqla_session 8  0   100%
  debsources/statistics 157 3578%
  debsources/subprocess_workaround3  167%
  debsources/tests/__init__   0  0   100%
  debsources/tests/db_testing43  491%
  debsources/tests/test_archiver118  0   100%
  debsources/tests/test_stats71  0   100%
  debsources/tests/test_updater 164  895%
  debsources/tests/test_webapp  153  199%
  debsources/tests/testdata   4  0   100%
  debsources/tests/updater_testing6  0   100%
  debsources/updater318 6978%
  ---
  TOTAL287951382%

But still short of 100%, which is required to give some basic guarantees
of code well-formedness.

Please increase (and/or maximize to 100%) Debsources' test coverage.

FWIW, the first command shown above will give hints of the lines of code
which are currently not exercised by the test suite. Patches that add
new tests touching those lines are more than welcome.

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20140926133818.9713.88293.reportbug@timira.takhisis.invalid

Bug#763921: debsources: detailed directory listing with file types and permissions

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

Currently,
  file permissions are shown by Debsources only when visiting individual files,
adding something like permissions: rw-rw-r-- to the title bar of the file
content window.

It would be nice to change the directory listing from the current ls-like
style, to a more ls -l-like style, which associates extra details to each
entry (no matter if it is a file or a directory), and in particular associate
permissions to each entry when showing a directory content.  Other information
that can be shown are the file type, and in particular whether a file is a
symlink or not.

Doing the above would also have the nice side effect of fixing the current
inability to show the permissions of directories.

At the implementation level, it would be nice to implement this as a template
option, so that it is easy to switch on/off this detailed listing feature.

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20141003190746.15812.98042.reportbug@timira.takhisis.invalid

Bug#764077: debsources: case-insensitive browsing by package prefix

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
User: qa.debian@packages.debian.org
Usertags: debsources

The Browse by prefix box on sources.debian.net homepage has entries like
... 7 8 9 W a b c d ... w x y z, where (capital) W is an artifact of very
old Debian releases that didn't inforce the rule that package names should only
contain lower case letters.

We should enforce the fact that package prefixes (stored in cache/pkg-prefixes
at update time) contain only lowercase letters, and make prefix browsing
(e.g. http://sources.debian.net/prefix/W/) case-insensitive, so that both
/prefix/w and /prefix/W give the same result.

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20141005100927.22702.74266.reportbug@timira.takhisis.invalid

Bug#766012: debsources: checksums_package_id_file_id_key unique constraint violation when adding some packages

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
User: qa.debian@packages.debian.org
Usertags: debsources

Some packages cannot be added to the Debsources DB due to violation of
the unique constraint in subject.

Here are the current 2 offender, from debsources.log on the machine
hosting sources.debian.net:

2014-10-18 22:29:57 updater:ERROR plugin hooks for add-package on 
shinken/1.4.2-3 failed
2014-10-18 22:29:57 updater:ERROR failed to add shinken/1.4.2-3
Traceback (most recent call last):
  File /srv/debsources/debsources/updater.py, line 211, in _add_package
notify(conf, 'add-package', session, pkg, pkgdir, file_table)
  File /srv/debsources/debsources/updater.py, line 122, in notify
file_table=file_table)
  File /srv/debsources/debsources/updater.py, line 135, in notify_plugins
action(session, pkg, pkgdir, file_table)
  File /srv/debsources/debsources/plugins/hook_checksums.py, line 114, in 
add_package
session.execute(insert_q, insert_params)
  File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py, line 802, 
in execute
clause, params or {})
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 1449, 
in execute
params)
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 1584, 
in _execute_clauseelement
compiled_sql, distilled_params
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 1698, 
in _execute_context
context)
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 1680, 
in _execute_context
context)
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py, line 
328, in do_executemany
cursor.executemany(statement, parameters)
IntegrityError: (IntegrityError) duplicate key value violates unique constraint 
checksums_package_id_file_id_key
DETAIL:  Key (package_id, file_id)=(126874, 90046831) already exists.
 'INSERT INTO checksums (package_id, file_id, sha256) VALUES (%(package_id)s, 
%(file_id)s, %(sha256)s)' ({'package_id': 126874, 'sha256': 
'c5a9faf4689be0028bb8d5daa60e9be37cdd24a53bea8ce224fd1d5f078b7695', 'file_id': 
90044182}, {'package_id': 126874, 'sha256': 
'2198e3cce9c1559864d101f500143c3bab05672c6cb1ef3f6b2cd64b2e11e194', 'file_id': 
90044542}, {'package_id': 126874, 'sha256': 
'24e371cf8ba52ce17c329f4100263f8e459e8048cf8ca73f35df5dfa2d36715a', 'file_id': 
90044533}, {'package_id': 126874, 'sha256': 
'be51cb3d5bb127cd2919d17e8585abe1585a0f4403ff83a55507579f6b16b42a', 'file_id': 
90047153}, {'package_id': 126874, 'sha256': 
'aab284a9765e95d92e52165f8d18cfbe206651c726a4843b9820b0cb952bda98', 'file_id': 
90045039}, {'package_id': 126874, 'sha256': 
'e05b1de9220f1b04490c31be71562cd63e97682285ebe56d01cf672503ab4f58', 'file_id': 
90043347}, {'package_id': 126874, 'sha256': 
'2f4d70bac5bc84f4d50b459303e949c16f4f57d5497b64d9ea417a531e9eb15a', 'file_id': 
90045569}, {'package_id': 126874, 'sha256':
 '89716f728b8df8431cc077ff1b56dd2276faa75d7594a74ffade1fdbbc3ce7e2', 'file_id': 
90046203}  ... displaying 10 of 4226 total bound parameter sets ...  
{'package_id': 126874, 'sha256': 
'5e19bb5502e896b86d4132af9fce430c94ec8b66dcd3ff71e986c7d8ffcb15a7', 'file_id': 
90045590}, {'package_id': 126874, 'sha256': 
'950691d36b0228b800cf21b451bca2a9e1b6cd3993c37b4e7d1c1939047ef3bd', 'file_id': 
90044703})

2014-10-19 03:53:44 updater:ERROR plugin hooks for add-package on weboob/0.j-1 
failed
2014-10-19 03:53:44 updater:ERROR failed to add weboob/0.j-1
Traceback (most recent call last):
  File /srv/debsources/debsources/updater.py, line 211, in _add_package
notify(conf, 'add-package', session, pkg, pkgdir, file_table)
  File /srv/debsources/debsources/updater.py, line 122, in notify
file_table=file_table)
  File /srv/debsources/debsources/updater.py, line 135, in notify_plugins
action(session, pkg, pkgdir, file_table)
  File /srv/debsources/debsources/plugins/hook_checksums.py, line 114, in 
add_package
session.execute(insert_q, insert_params)
  File /usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py, line 802, 
in execute
clause, params or {})
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 1449, 
in execute
params)
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 1584, 
in _execute_clauseelement
compiled_sql, distilled_params
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 1698, 
in _execute_context
context)
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py, line 1680, 
in _execute_context
context)
  File /usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py, line 
328, in do_executemany
cursor.executemany(statement, parameters)
IntegrityError: (IntegrityError) duplicate key value violates unique constraint 
checksums_package_id_file_id_key
DETAIL:  Key (package_id, file_id)=(126917, 90065484) already exists.
 'INSERT INTO checksums (package_id, file_id, sha256) VALUES (%(package_id)s, 
%(file_id)s, %(sha256)s)' ({'package_id': 

Bug#766162: qa.debian.org: Use 'env' instead of /usr/bin/python to make source more portable.

[Stefano Zacchiroli]

severity 766162 normal
tags 766162 + confirmed
user debian-qa@lists.debian.org
usertag 766162 + gift
thanks

On Tue, Oct 21, 2014 at 05:00:49PM +0800, sophiejjj wrote:
 Currently somefiles are hard-coded with /usr/bin/python. e.g.,
 debiansources-run-app and the alike.
 
 This makes the code less portable, and inhibiting the use of 
 virtualenv.

I've nothing against switching shebang lines to /usr/bin/env python
for all [Python] Debsources binary under bin/.  If doing so makes it
easier to deploy/test debsources using virtualenv, even better.

Thanks for your bug report,
patches welcome :-)

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#766857: debsources: import mirror suites as archive.d.o (AKA sticky) suites, without reimporting

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
User: qa.debian@packages.debian.org
Usertags: debsources

Currently, archive.debian.org suites are imported manually into Debsources
(using bin/debsources-suite-archive), and not watched periodically by the cron
job using bin/debsources-update.

This poses a problem at release time: when oldstable is removed from the mirror
network (more precisely: when stable becomes the new oldstable),
debsources-update will notice and remove all (old) oldstable packages from its
database. Those packages will have to be reimported manually using
debsources-suite-archive, when the archive suite shows up on archive.d.o.

A proper solution for the underlying problem requires automating the watching
of archive.d.o suites, which might need a separate bug report.

In the meantime, we should add an import command to debsources-suite-archive,
that allows to import manually, before the next stable-oldstable-archive
rotation, a currently live suite as an archive.d.o suite (AKA sticky suite),
inhibiting its deletion. To implement that, it might be enough to simply set
sticky=True on all packages belonging to that suite. But some reflection is
needed about what might happen during debsources-update runs, in the interim
between the import and the disappearing of the imported suite from the mirror
network.

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20141026110915.15459.66522.reportbug@timira.takhisis.invalid

Bug#766965: debsources: RSS/Atom feed of data changes

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

Debsources should offer a parseable feed of changes to its dataset (package
addition, removal, suite changes, etc). RSS/Atom is the most natural format to
use here. Implementing this will need a bit of design work on the type and
granularity of events to support.

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20141027093011.23110.73758.reportbug@timira.takhisis.invalid

Bug#766967: debsources: refactoring - make web UI parts conditional on plugin enablement

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: normal
User: qa.debian@packages.debian.org
Usertags: debsources

Various parts of Debsources UI should really be exposed only if specific
plugins are enabled, e.g., ctags-based search, sloccount statistics, etc. The
way to go about this with Flask are pluggable views
http://flask.pocoo.org/docs/latest/views/

To proper implement this, we will probably need to improve the plugin API,
which is at present rather thin.

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20141027093322.23252.42589.reportbug@timira.takhisis.invalid

Bug#766968: debsources: test suite - differentiate sid and jessie

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
User: qa.debian@packages.debian.org
Usertags: debsources

Debsources test suite spans several Debian releases/suites. However, at present
sid and jessie in the test suite are made of exactly the same packages. This is
suboptimal and might hide errors.

We should add/remove packages to one or the other, to ensure the two sets of
packages are different. This will probably break some tests, that will need to
be fixed accordingly.

Cheers.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20141027093545.23372.32384.reportbug@timira.takhisis.invalid

Bug#764178: debsources: infobox CSS alignment problem with short files

[Stefano Zacchiroli]

First of all: thanks a lot for your patch, Jason!

On Fri, Oct 31, 2014 at 07:42:23AM -0400, Jason Pleau wrote:
 From 6cc9f15d51dd35a5afb82a2c3680e3e5dfc0f93b Mon Sep 17 00:00:00 2001
 From: Jason Pleau ja...@jpleau.ca
 Date: Fri, 31 Oct 2014 00:05:26 -0400
 Subject: [PATCH] source_file: fix text overlapping the infobox

I'm no CSS expert, so I'm unable to comment on your patch at the
moment. Matthieu: can you have a look and comment on Jason's approach at
fixing #764178.

I've a separate comment though:

 When browsing a file's source on sources.debian.net, if the file
 didn't contain enough text its content would overlap onto the infobox
 to the right.

Your commit message essentially restates the bug report, rather than
explaining how the corresponding change fixes it. The commit message
should really do the latter, rather than the former.

Particularly in this case, I see no obvious reason why changing the
right padding of codetable (an horizontal spacing matter) would fix the
bug (which seems to be a vertical spacing matter). I'm sure it *does*
fix the bug, but the commit should explain why it does so, so that even
CSS illiterates as myself could understand the rationale :-)

Jason: do you think you can update your patch to do so?

Many thanks in advance,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#764178: debsources: infobox CSS alignment problem with short files

[Stefano Zacchiroli]

On Tue, Nov 04, 2014 at 07:03:38PM -0500, Jason Pleau wrote:
 My commit adds a padding-right to make sure that even if the file has
 one short line, it's content will be left-aligned.

Thanks Jason, your explanation makes a lot of sense to me now. And the
description in the commit is now good enough to explain what's going on
also to a CSS-illiterate like myself :)

 The 450px that I've written is a bit arbitrary.

This is the only remaining aspect that concerns me. It is indeed
arbitrary, and I'm pretty sure it will lead to nasty side effects on
small screens.

I've reviewed the rest of the current CSS files, and it seems to me that
all other use cases for large px values are more safe (e.g. input field
widths) than the one your patch is introducing. How about using a
relative width instead?

If CSS gurus are reading this, please advise on what you think is the
best solution here.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20141105091445.ga16...@upsilon.cc

Bug#761103: debsources: highlight #Lxxx lines by default

[Stefano Zacchiroli]

Hello again, Jason.

To JavaScript hackers lurking here: your comments on this patch are most
welcome!

On Sat, Nov 01, 2014 at 11:37:56AM -0400, Jason Pleau wrote:
 I have attached a patch that solves this bug.

Thanks a lot for this patch. It works like a charm on my local
deployment. As I'm no JavaScript guru, I could use extra eyeballs to
review it, and I do have some questions for you (see below), but I see
no reason why this couldn't be integrated.

 -a id=L{{ i }} href=#L{{ i }}{{ i }}/abr /
 +a id=L{{ i }} href=#L{{ i }} class=linenumber data-line={{ i 
 }}{{ i }}/abr /

Is it really needed to add class=linenumber to all a elements?

It seems that the JavaScript only uses that class as a selector. So
can't one instead rely on the id=sourcelinenumbers that is already on
the parent pre element, and consider all its a children? That would
avoid bloat in the generated HTML, reduce load time, etc.

 +script type=text/javascript
[...]
 +/script

I'd like to move this JavaScript snippet to a separate .js file, rather
than re-shipping it every time a file is rendered. Can you change your
patch to do so?

I suggest to name it something like
DEBSOURCES_ROOT/debsources/app/static/javascript/debsources.js (so that
we can use it in the future for other JavaScript-related needs). It will
then be accessible from the HTML as /static/javascript/debsources.js
(note: *not* as /javascript/debsources.js).

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761103: debsources: highlight #Lxxx lines by default

[Stefano Zacchiroli]

On Thu, Nov 06, 2014 at 08:09:15PM -0500, Jason Pleau wrote:
 The attached patch implements your previous suggestions :

Awesome, thanks! Last comments:

1) I've added to the new .js file a copyright header pointing to
   you. Can you confirm you're OK with contributing your code under
   AGPLv3, as the rest of Debsources?

   You can find attached a version of your patch that includes the
   copyright header (and fixes some whitespace issues that make Git
   cry :-))

 +change_hash_without_scroll(callerElement, L + 
 callerElement.getAttribute('data-line'));
 +} else {
 +var first_line = 
 parseInt(last_clicked.getAttribute('data-line'));
 +var second_line = 
 parseInt(callerElement.getAttribute('data-line'));
[...]
 +a id=L{{ i }} href=#L{{ i }} data-line={{ i }}{{ i }}/abr 
 /

2) In the same anti-bloat vein of my previous comments, do we really
   need to another attribute here, given that its content is precisely
   the same of the text child of a? Can't you just use the data
   content of that text node?

   (Yes, I understand that in the future the a node might contain more
   complex markup, but there will always be a text leaf in the DOM tree
   that we can use; if not directly a DOM method that return the textual
   value of the whole sub-tree.)

What do you think?
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761103: debsources: highlight #Lxxx lines by default

[Stefano Zacchiroli]

On Fri, Nov 07, 2014 at 06:24:58PM +0100, Stefano Zacchiroli wrote:
You can find attached a version of your patch that includes the
copyright header (and fixes some whitespace issues that make Git
cry :-))

Now with an attachment.

-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »
From eaa36cc8aa2ba936932ec4b2f99ba1b9fef41982 Mon Sep 17 00:00:00 2001
From: Jason Pleau ja...@jpleau.ca
Date: Sat, 1 Nov 2014 10:44:24 -0400
Subject: [PATCH] web app: automatically highlight #Lxx lines

Add automatic line numbers from #Lxx location.hash

It supports #L50 and #L50-L150 (The latter will highlight from line 50 to line
150).

It also changes the hash if we click on a line, and if we click on a second
line holding the SHIFT key, it will highlight the whole range, and update the
hash as well.

Closes: #761103

Signed-off-by: Stefano Zacchiroli z...@upsilon.cc
---
 debsources/app/static/javascript/debsources.js | 137 +
 debsources/app/templates/source_file.html  |   1 +
 debsources/app/templates/source_file_code.inc.html |   7 +-
 3 files changed, 140 insertions(+), 5 deletions(-)
 create mode 100644 debsources/app/static/javascript/debsources.js

diff --git a/debsources/app/static/javascript/debsources.js b/debsources/app/static/javascript/debsources.js
new file mode 100644
index 000..2a382bb
--- /dev/null
+++ b/debsources/app/static/javascript/debsources.js
@@ -0,0 +1,137 @@
+/* Copyright (C) 2014  Jason Pleau ja...@jpleau.ca
+ *
+ * This file is part of Debsources.
+ *
+ * Debsources is free software: you can redistribute it and/or modify it under
+ * the terms of the GNU Affero General Public License as published by the Free
+ * Software Foundation, either version 3 of the License, or (at your option)
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/.
+ */
+
+/*
+ * Highlight line numbers according to data received in the anchor
+ *
+ * Example: file.cpp#L50-L275 will highlight lines 50 to 275.
+ *
+ * There's also support to select one line or a range of lines (by clicking on
+ * a line or shift-clicking on a range of lines). The URL will be updated with
+ * the selection.
+ *
+ */
+
+(function() {
+function highlight_lines(start, end) {
+// First, remove the highlight class from elements that might already have it
+var elements = document.querySelectorAll(span.highlight);
+for (i = 0; i  elements.length; ++i) {
+var element = elements[i];
+element.className = element.className.replace(/\bhighlight\b/, '');
+}
+
+// Then, add the highlight class to elements that contain the lines we want to highlight
+for (i = start; i = end; ++i) {
+var element = document.getElementById(line + i);
+element.className = element.className +  highlight ;
+}
+}
+
+var hash_changed = function(event, scroll) {
+
+event = typeof event !== 'undefined' ? event: null;
+scroll = typeof scroll !== 'undefined' ? scroll: false;
+
+// Will match strings like #L15 and #L15-L20
+var regex = /#L(\d+)(-L(\d+))*$/;
+
+var match = regex.exec(window.location.hash);
+if (match != null) {
+var first_line = second_line = null;
+first_line = parseInt(match[1]);
+
+if (typeof match[3] !== 'undefined'  match[3].length  0) {
+second_line = parseInt(match[3]);
+} else {
+second_line = first_line;
+}
+
+// If we get something like #L20-L15, just swap the two line numbers so the loop will work
+if (second_line  first_line) {
+var tmp = first_line;
+first_line = second_line;
+second_line = tmp;
+}
+
+highlight_lines(first_line, second_line);
+
+if (scroll) {
+window.scroll(0, document.getElementById(L+first_line).offsetTop);
+}
+}
+}
+
+
+function change_hash_without_scroll(element, hash) {
+// This is necessary because when changing window.location.hash, the window will
+// scroll to the element's id if it matches the hash
+var id = element.id;
+element.id = id+'-tmpNoScroll';
+window.location.hash = hash;
+element.id = id;
+}
+
+var

Re: Facilitating contributions by newcomers

[Stefano Zacchiroli]

On Tue, Nov 11, 2014 at 02:13:09PM -0800, Don Armstrong wrote:
 On Tue, 11 Nov 2014, Lucas Nussbaum wrote:
  Have you considered adding the 'gift' tag[0] to some BTS bugs?
 
 I probably should do that; I actually wasn't that familiar with the
 gift tag before this e-mail.

Just to share a recent experience: thanks to proper triaging of gift
bugs for debsources (see [1]), and to the fact that how-can-i-help now
lists infrastructure bugs, we've gained a new prolific contributor to
debsources (hi Jason!).

[1]: http://deb.li/debsrceasy

I highly recommend to anyone taking the time to triage bugs in their own
packages and tag them +gift as needed.

FWIW, similar experiences in other projects (e.g., easy hacks in
LibreOffice) confirm that the whole approach of triaging bugs for new
contributors does work.

  [0] https://wiki.debian.org/qa.debian.org/GiftTag
 
 Does anyone have any thoughts about elevating the gift tag to a
 fully-fledged BTS tag?

As the person responsible for adding +gift support to devscripts back in
the days, I've always regretted not having pushed for a full-fledged BTS
tag at the time. We should definitely do that, but also think at
backward compatibility. For instance, how to guarantee that existing
queries like [1] above do not break?

Also, a while ago (cannot find the thread right now) we discussed with
Lucas how the name gift could be misunderstood as a patronizing
characterization, if you think that's a gift from Debian Project to the
newbie contributor. Whereas in fact it was meant to be the other way
around.  So, in picking an actual tag name, we should probably change it
to something like easyhack (LibreOffice name), or newbie, or
bitesize (Ubuntu name).

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Making entry-point nee gift a real BTS tag [Re: Facilitating contributions by newcomers]

[Stefano Zacchiroli]

On Wed, Nov 12, 2014 at 03:54:05PM -0800, Don Armstrong wrote:
 Excellent; thanks. I'm going to make these gender-neutral, and then I'll
 commit them.
 
 This is the last chance for someone to object to entry-point as the tag
 name. If I hear no objections, I'll put this in place on Friday, around
 18:00 UTC.

I'm fine with entry-point. However, about its description:

On Wed, Nov 12, 2014 at 07:42:33PM +0100, Stéphane Aulery wrote:
 entry-point
 The maintainer can easily solve this bug by himself, but he
 wants to take it to new contributors who wish to get involved
 in Debian. Bugs of any difficulty can be offered in order to
 attract and increase skill.

I don't think we should include the initial part about maintainer can
easily solve..., as it does seems a bit patronizing (that's easy for
me, but I won't do it because...). My take: just include the part about
being a good entry point (hence the name) for new contributors and scrap
the rest.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761149: debsources: allow redirects to package versions based on suite/codename

[Stefano Zacchiroli]

On Tue, Nov 11, 2014 at 07:03:17AM -0500, Jason Pleau wrote:
 Updated patch

Hey Jason, once again thanks a lot for your patch! It looks good in
general. Some comments/requests:

  debsources/app/views.py | 16 +++-
  debsources/migrate/007-to-008.sql   | 14 ++
  debsources/models.py| 10 ++
  debsources/tests/test_webapp.py | 10 ++
  doc/db-schema/debsources.dia| 23 +++
  doc/db-schema/debsources.dot|  2 +-
  doc/db-schema/debsources.html   | 18 +-
  doc/db-schema/debsources.neato  |  2 +-
  doc/db-schema/debsources.xml| 16 
  doc/db-schema/debsources.zigzag.dia | 23 +++

Can you please separate your patch into two separate commits: one that
implements your real changes, and a subsequent one that only
regenerates the doc/db-schema/* files?  That makes both the history more
readable/clear and reviewing easier.

 diff --git a/debsources/migrate/007-to-008.sql 
 b/debsources/migrate/007-to-008.sql
 new file mode 100644
 index 000..838346e
 --- /dev/null
 +++ b/debsources/migrate/007-to-008.sql
 @@ -0,0 +1,14 @@
 +ALTER TABLE suites_info
 +  ADD COLUMN alias VARCHAR;

Regarding this, which is an important design decision here, I'm not
particularly happy about having a maximum of 1 alias per suite. Why
couldn't a suite have more?

Now, implement this properly in SQL would require a separate table to
allow 1:N suite-aliases mappings, which is probably a tad overkill for
what we need here. So how about instead: 1/ renaming the new column to
aliases, and 2/ document that it is a comma-separated list of aliases,
that all map to the current row in the suites_info table?

Of course your changes to views.py and models.py will need to be adapted
accordingly.

 +UPDATE suites_info
 +  SET alias='unstable' WHERE name='sid';
 +
 +UPDATE suites_info
 +  SET alias='testing' WHERE name='jessie';
 +
 +UPDATE suites_info
 +  SET alias='stable' WHERE name='wheezy';
 +
 +UPDATE suites_info
 +  SET alias='oldstable' WHERE name='squeeze';

This is another part of the design that bothers me a little. The above
is OK for deploying quickly the change, but I certainly do not want to
introduce another place in Debsources which will have to be manually
maintained/updated at each Debian release.

So what I think we should do here is modifying the Debsources updated to
retrieve aliases from Release files, and re-fill the suites_info table
(including aliases information) at each update run. What do you think?

 @@ -194,15 +194,17 @@ class SuiteInfo(Base):
  version = Column(String, nullable=True)
  release_date = Column(Date, nullable=True)
  sticky = Column(Boolean, nullable=False)
 +alias = Column(String, nullable=True)

If you're OK with my proposed change, it'd be nice to have automatically
split of the aliases list every time a SuiteInfo object is created, but
I'm not sure what's the most appropriate way to do that in SQLAlchemy.
If it's annoying to do, we can certainly live with keeping that as a
comma-separated string and split it when needed.

 +def __init__(self, name, sticky=False, version=None, release_date=None,
 + alias=None):
  self.name = name
  if version:
  self.version = version
  if release_date:
  self.release_date = release_date
  self.sticky = sticky
 -
 +self.alias = alias

Shouldn't this be instead:

if alias:  # aliases, in the future
self.alias = alias

 diff --git a/doc/db-schema/debsources.dia b/doc/db-schema/debsources.dia
 index 5487cb0..960f76e 100644
 --- a/doc/db-schema/debsources.dia
 +++ b/doc/db-schema/debsources.dia
 @@ -3344,6 +3344,29 @@

From now on there is the auto generated stuff which I think should go in
a separate commit.

HTH,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761149: debsources: allow redirects to package versions based on suite/codename

[Stefano Zacchiroli]

On Thu, Nov 13, 2014 at 06:30:45PM -0500, Jason Pleau wrote:
 Thanks for your feedback on this. I know how time consuming it can be
 to review someone else's code when they're new to a project, so please
 know I do appreciate it !

Likewise! :)

 The logical thing would be to create a separate table to allow the
 association between a suite and its aliases. I think it would ease the
 rest of the code since we'd let SQLAlchemy handle creating and saving
 the 'aliases' array.

I agree that would be the best design. I just didn't want to impose it
on you, but if you're motivated for changing the SQLAlchemy accordingly,
by all means go for it!

 Agreed about having the suites and their aliases auto-updated.
 
 I remember we talked about it a bit on #debian-qa with pabs, there was
 the possibility to use an API once it's finished:
 https://api.ftp-master.debian.org/suites

I think there are two separate concerns here.

One is automating the maintenance of Debsources configuration
information, even in presence of bugs that forces to hardcode release
names (e.g., #714726). This concern could be addressed using the API you
mention.

A different concern is interpreting information that we already have in
the mirrors anyhow (like suite aliases) and injecting it in Debsources
database. This, I believe, is the concern that is at stake in this bug
report. There is no need of the ftp-master.d.o API to address it.

 If the API is not close to release, I can look into parsing Release
 files, are you aware of a parsing tool in python? I was looking at
 python-apt, but it seems to rely on a sources.list file, I don't think
 that's not what we need.

Debsources already depends on python-debian, which has support for
parsing all sort of Deb822-like files, including Release files. The
class you want for that is debian.deb822.Release.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

migration gift - newcomer [Was: newcomer will be this tag.]

[Stefano Zacchiroli]

On Fri, Nov 14, 2014 at 06:29:45PM -0800, Don Armstrong wrote:
 The winner is newcomer. I'll add this and update the documentation
 later tonight.

Thanks Don!

To help with the migration I've just:

- updated https://wiki.debian.org/qa.debian.org/GiftTag adding a
  deprecation warning at the top of the page

- patched devscripts adding support for newcomer, retaining backward
  compatibility for gift (which will do both +gift and +newcomer for a
  while). See
  
http://anonscm.debian.org/cgit/collab-maint/devscripts.git/commit/?id=5c7978d4508fcb895463b2e9ce828e4dd8f79be4

- requested support for newcomer in how-can-i-help:
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=769640

Don, what have you decided in the end for migrating old URLs pointing to
the gift tag? Do you have a way to add redirects or should we just cope
with the breakages?

Personally, I care particularly about http://deb.li/debscreasy and I'd
be fine with making the underlying URL a redirect to the corresponding
newcomer query. But if that's not possible, I'll switch to a new short
URL.

TIA,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: migration gift - newcomer [Was: newcomer will be this tag.]

[Stefano Zacchiroli]

On Sat, Nov 15, 2014 at 02:56:38PM +0100, Tomasz Nitecki wrote:
 how-can-i-help will treat 'newcomer' tags exactly as it treats 'gift'
 tags now. I'm only going to change retrieval logic from 'get all gift'
 to 'get all (gift or newcomer)'. This change requires only a minor
 addition to UDD generate-how-can-i-help script (and is already done -
 just needs uploading). It doesn't require any changes in how-can-i-help
 itself so we are freeze safe.

Awesome, thanks!

 However, how-can-i-help contains some references to 'gift' bugs (both in
 documentation and in configuration options) and those can't be changed
 without modifying hcih itself. That means that the version available in
 Jessie will refer to both 'gift' and 'newcomer' bugs as 'gift' bugs. In
 the next hcih version, I'll change all those references from 'gift' to
 'newcomer' and I'll add a note about 'gift'-'newcomer' tag change.

If the change can be isolated from other changes and backported to the
how-can-i-help version currently I jessie, I wouldn't exclude the
possibility that the release team can accept it. After all, jessie's
how-can-i-help will be with us for quite some time, and it is an
important tool for attracting new contributors.

Would you mind preparing an unblock request and run it through release
team review?

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: tracker pts

[Stefano Zacchiroli]

On Mon, Dec 01, 2014 at 10:31:56AM +0100, Holger Levsen wrote:
 is there any timeline to make tracker.d.o the official pts replacement
 in place?
 
 I'm wondering whether I should update
 piuparts/jenkins/security-tracker links pointing to the pts or whether
 this should wait some more...

You should do that no matter the timeline of the previous point.

The old PTS is already deprecated, hence the message pointing to the
tracker on each page. Any new part of the Debian infrastructure should
point to the tracker, and migrating old links from the old PTS to the
tracker is encouraged (we discussed this in the past, IIRC).

HTH,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: tracker pts

[Stefano Zacchiroli]

On Mon, Dec 01, 2014 at 11:00:09AM +0100, Holger Levsen wrote:
 #757443 layout broken with konqueror from wheezy
 #756721 Please use same font size as in the PTS.
 
 These might seem minor but are actually my reasons why I havent
 updated the links yet... (and why I still use the old pts.)

OK, those are legitimate concerns (and blockers for your use cases,
IIUC), but they are unrelated to the actual timeline for shutting down
the old PTS. That is to say: you are not migrating your links for
reasons other than the tracker hasn't been officially blessed as PTS
replacement yet, right?

I have no idea if someone is maintaining a list of legitimate blockers
that are inhibiting a complete migration to the new tracker (including a
shutdown of the old one). My tentative answer about this would be no
one is doing that, but I might be wrong.  Personally, I certainly
consider the new tracker good enough and I've stopped using the old PTS
long ago. The main reason for keeping around the old PTS, at least for
me, is currently in the realm of let's give people enough time to
migrate their links, habits, etc.

In the meantime, if people have migration blockers, the best way forward
is providing patches :) ; and the second best let this list know (as you
just did).

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#770365: debsources: 403 on /src/beignet/1.0.0-1/README.md/

[Stefano Zacchiroli]

retitle 770365 debsources: should not return 403 on internal symlinks
severity 761121 normal
forcemerge 761121 770365
thanks

On Sun, Nov 30, 2014 at 09:55:12PM +0100, Matthieu Caneill wrote:
 On 20 November 2014 at 20:05, Jérémy Bobbio lu...@debian.org wrote:
  When visiting https://sources.debian.net/src/beignet/1.0.0-1/README.md/
  I'm told “403 Permission Denied”. This is a bit annoying as the file is
  listed on https://sources.debian.net/src/beignet/1.0.0-1/
 
 Due to security reasons, we deactivated all symbolic links on
 Debsources (even the ones internal to a package, but this isn't
 implemented yet).
 Zack what do you think?

Oooh, I see, I didn't realize, before your message, that these 404 where
due to symlinks. So this bug is effectively the same of #761121, which
was the wishlist bug requesting the support for symlinks that point
within the same source package.

Arguably, we could provide a better user experience than 403 (especially
because the user has no way of knowing a priori that a given file is a
symlink, until #763921 is implemented, that is). But the best fix here
will indeed be to do support internal symlinks.

I'm therefore retitling/merging as needed.

Thanks Lunar for your bug report!
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Announcing the new Debian Code Search Instant

[Stefano Zacchiroli]

Context:
https://people.debian.org/~stapelberg/2014/12/03/debian-code-search-instant.html

 Instant indexing
[...]
 The time between uploading a package and being able to find it in Debian
 Code Search therefore now ranges from a couple of minutes to
 about an hour, instead of about a week!

Hey Michael, great exciting news on the Code Search front, thanks for
working on them!

I was wondering, does the above mean that we can lower the 2-week grace
period before expiring packages on Debsources?  The main reason for such
a long grace period was avoiding breaking codesearch.d.n links, but
seems moot now(?).

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Announcing the new Debian Code Search Instant

[Stefano Zacchiroli]

[ still mailing you explicitly, as I'm unsure you're subscribed to -qa ]

On Thu, Dec 04, 2014 at 10:11:17AM +0100, Michael Stapelberg wrote:
 Yeah, we can lower that period now. I suggest that starting from next
 week (2014-12-08), we set it to 1 week. That should give us plenty of
 time to investigate issues when the import should start to fail for
 some reason.
 
 Does that work for you?

Sure, thanks for your prompt feedback.
I've just reduced the grace period on sources.d.n to 1 week.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#772560: debsources: please add a CA-signed SSL setup

[Stefano Zacchiroli]

On Tue, Dec 23, 2014 at 11:50:28AM -0500, James McCoy wrote:
 Would that just be a matter of using scheme-relative links (i.e., a
 href=//foo)?

Actually, yes! I was assuming that something more complex was needed,
but apparently scheme-relative URLs are enough for our case. So now the
browser is fully happy when browsing sourrces.d.n.

Thanks for the heads up.

I've converted to scheme-relative all URLs that, upon manual checking of
their https:// variant, led to SSL-enabled web sites with valid
certificates. If anyone checks, please let me know if I've missed some.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#761106: web-app: Add support for non-ascii filenames

[Stefano Zacchiroli]

tags 761106 + patch
thanks

On Mon, Mar 16, 2015 at 11:41:59AM +0100, Orestis Ioannou wrote:
 Subject: [PATCH] web app: Add support for non-ascii filenames and directories

Hey Orestis, thanks for your patch. I'm Cc:-ing the bug log to record
that a patch is available.

However, while it seems to fix the problem on my development machine, it
doesn't seem to be enough to fix it on sources.debian.net.

With your patch applied, sources.d.n still fails with the following
backtrace:

2015-03-16 14:21:00,712 ERROR: 'ascii' codec can't decode byte 0xc3 in position 
4: ordinal not in range(128) [in /srv/debsources/debsources/app/views.py:137]
  Traceback (most recent call last):
File /srv/debsources/debsources/app/views.py, line 193, in 
dispatch_request
  context = self.get_objects(**kwargs)
File /srv/debsources/debsources/app/sources/views.py, line 330, in 
get_objects
  return self._render_location(package, version, path)
File /srv/debsources/debsources/app/sources/views.py, line 129, in 
_render_location
  return self._render_directory(location)
File /srv/debsources/debsources/app/sources/views.py, line 147, in 
_render_directory
  content = directory.get_listing()
File /srv/debsources/debsources/models.py, line 691, in get_listing
  for f in os.listdir(self.sources_path))
File /srv/debsources/debsources/models.py, line 691, in genexpr
  for f in os.listdir(self.sources_path))
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: 
ordinal not in range(128)

Whereas without your patch it failed with this (note that the backtrace
is indeed different, even though the origin line is the same):

2015-03-16 14:23:48,108 ERROR: 'ascii' codec can't decode byte 0xc3 in position 
4: ordinal not in range(128) [in /srv/debsources/debsources/app/views.py:137]
  Traceback (most recent call last):
File /srv/debsources/debsources/app/views.py, line 193, in 
dispatch_request
  context = self.get_objects(**kwargs)
File /srv/debsources/debsources/app/sources/views.py, line 330, in 
get_objects
  return self._render_location(package, version, path)
File /srv/debsources/debsources/app/sources/views.py, line 129, in 
_render_location
  return self._render_directory(location)
File /srv/debsources/debsources/app/sources/views.py, line 147, in 
_render_directory
  content = directory.get_listing()
File /srv/debsources/debsources/models.py, line 687, in get_listing
  for f in os.listdir(self.sources_path))
File /srv/debsources/debsources/models.py, line 687, in genexpr
  for f in os.listdir(self.sources_path))
File /srv/debsources/debsources/models.py, line 680, in get_type
  if os.path.isdir(os.path.join(self.sources_path, f)):
File /usr/lib/python2.7/posixpath.py, line 78, in join
  path +=  b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal 
not in range(128)

Any idea what might be going on here?

TIA,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »
From 994c768d3d1efaa2cddac2e16696cd6ca45f4a90 Mon Sep 17 00:00:00 2001
From: Orestis Ioannou ores...@oioannou.com
Date: Mon, 16 Mar 2015 11:38:53 +0100
Subject: [PATCH] web app: support non-ASCII filenames and directories

Closes: #761106
---
 .../app/sources/templates/sources/source_folder.html  |  4 ++--
 debsources/models.py  | 11 ---
 debsources/tests/test_webapp.py   | 19 +++
 3 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/debsources/app/sources/templates/sources/source_folder.html b/debsources/app/sources/templates/sources/source_folder.html
index 54d73a1..1c78cc6 100644
--- a/debsources/app/sources/templates/sources/source_folder.html
+++ b/debsources/app/sources/templates/sources/source_folder.html
@@ -42,7 +42,7 @@
 {% if config[DIR_LS_LONG] %}td class=stat-typespan{{ dir.stat.type }}/span/td{% endif %}
 td class=stat-permsspan{{ dir.stat.perms }}/span/td
 {% if config[DIR_LS_LONG] %}td class=stat-sizespan{{ {:,d}.format(dir.stat.size) }}/span/td{% endif %}
-td class=item-namea href={{ url_for('.source', path_to=path+'/'+dir.name) }}{{ dir.name }}/a
+td class=item-namea href={{ url_for('.source', path_to=path+'/'+dir.name.decode('utf-8')) }}{{ dir.name.decode('utf-8') }}/a
   {% if config[DIR_LS_LONG] %}
 {% if dir.stat.symlink_dest is not none %}{{  →  + dir.stat.symlink_dest }}{% endif %}
   {% endif %}
@@ -56,7 +56,7 @@
 {% if config[DIR_LS_LONG] %}td class=stat-typespan{{ file_.stat.type }}/span/td{% endif %}
 td class=stat-permsspan{{ file_.stat.perms }}/span/td
 {% if config[DIR_LS_LONG] %}td class=stat-sizespan{{ {:,d

Bug#761106: debsources: cannot list dir content for packages with unicode issues

[Stefano Zacchiroli]

On Wed, Sep 10, 2014 at 09:04:39PM +0200, Stefano Zacchiroli wrote:
 As an example of this issue, http://sources.debian.net/src/cvsnt/latest/
 currently returns 505, due to a contained file named testáéíóú.txt
 (and non utf8 clean).

FYI, the Debsources test data now contains a copy of cvsnt, with the
incriminated file above. (Yes, the bug is still open, but it should now
be easier to reproduce for people interested in debugging it.)

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150315214732.ga11...@upsilon.cc

Bug#761867: Webapp: Add conjunctive search for ctags

[Stefano Zacchiroli]

[ just documenting here a quick summary of IRC discussions ]

On Thu, Mar 12, 2015 at 02:33:03PM +0100, Orestis Ioannou wrote:
 I have attached the patch for the conjunctive search.
 Raw sql commands are the following:

The patch looks good, but there are concerns that this feature will
allow to DoS the DB, given the ctags table currently contains ~380
million tuples.

We've pinged friendly neighborhood Postgres experts (Myon, mnencia) who
are looking into this to give feedback about: whether the feature is a
good idea at all and/or whether there are better ways to implement this
query.

I duly notice that we already have non-conjunctive ctags-based search.
And it's not given us problems thus far. If we can make the conjunctive
query, performance-wise, as bad as the non-conjunctive version, then
it's probably good to do.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150313104753.ga27...@upsilon.cc

Bug#761149: debsources: allow redirects to package versions based on suite/codename

[Stefano Zacchiroli]

On Thu, Mar 12, 2015 at 06:45:37PM -0400, Jason Pleau wrote:
 As discussed on IRC, I rebase my patches for this bug.

Thanks Jason, I've made minor changes to your patches, and pushed them
to the feature/suite-aliasing branch here:

  http://anonscm.debian.org/cgit/qa/debsources.git/log/?h=feature/suite-aliasing

I've also updated the testdata submodule, and pushed it to the
feature/suite-aliasing branch here:

  
http://git.upsilon.cc/?p=hacks/debsources-testdata.git;a=shortlog;h=refs/heads/feature/suite-aliasing

I haven't yet merged the branch because while make test works well,
make test-all (which runs all the testsuite, including updater tests),
fails in 3 tests due to constraint violation errors, e.g.:

excludeFiles (debsources.tests.test_updater.Updater) ... ERROR
garbageCollects (debsources.tests.test_updater.Updater) ... ERROR
producesReferenceSourcesTxt (debsources.tests.test_updater.Updater) ... ERROR
[...]
IntegrityError: (IntegrityError) update or delete on table suites_info 
violates foreign key constraint suites_aliases_suite_fkey on table 
suites_aliases
DETAIL:  Key (name)=(jessie) is still referenced from table suites_aliases.
 'DELETE FROM suites_info WHERE suites_info.name = %(name_1)s' {'name_1': 
'jessie'}

Which looks like an actual regression.

Can you look into this?

As fixes, I'd appreciate an *additional* patch wrt the above branches,
which I'll then squash onto your previous commits in that branch.

TIA,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150313151016.ga7...@upsilon.cc

Bug#783461: debsources: add test case for news listing on the index page

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
Tags: newcomer
User: qa.debian@packages.debian.org
Usertags: debsources

Debsources can show news items on the index page (see News section at the
bottom of http://sources.debian.net/). The feature broke during the recent
blueprint refactoring, when the expected file name was renamed from news.html
to sources_news.html. We didn't notice because no test case actually checks
that feature. One such test case should be added.

To do so, we should populate a temporary local/ directory during test, and
check that the news item shipped in there shows up on the index page.

Cheers.

-- System Information:
Debian Release: 8.0
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.utf8, LC_CTYPE=it_IT.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20150427082817.13662.89152.reportbug@timira.takhisis.invalid

Bug#779400: tracker.debian.org: add a patches panel

[Stefano Zacchiroli]

On Sat, Feb 28, 2015 at 01:38:28PM +0800, Paul Wise wrote:
 I would like to have a patches panel on the distro-tracker that might
 look something like the attached mockup for Debian. The possible fields
 of the Debian version of the panel are as follows, many of them can't be
 implemented right now but are still interesting to mention for future
 work once the services backing them exist.
 
 Debian: the count of patches Debian applies to this package, as reported
 by the Debian patch tracking service (currently down).

So, implementing something like the old patch-tracker.d.o system on top
of Debsources/sources.debian.net is currently on the roadmap of
Debsources, and it is possible (though not certain yet) that we will get
to it during the upcoming GSoC projects that I'm mentoring.

I'd like to avoid duplication of work, so I've subscribed to this bug
report to stay on top of the tracker.d.o plans on this front.

I think it makes sense to _generate_ Debian patch information in
Debsources, because we already have all the source packages, in both
packed and unpacked form.

For what concerns tracker.d.o, am I right in assuming that tracker will
be happy with being able to simply _consume_ Debian patch information
generated by some other service, possibly Debsources? AFAICT that would
be enough to implement what Paul is asking here (for Debian; other
distros pose different problems, and are currently not indexed by
Debsources).

If so, we should standardize a patch information exchange format.
Ideally, something that is derivative agnostic.

Thanks for this bug report, Paul.
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#783911: debsources: old news page

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
Tags: newcomer
User: qa.debian@packages.debian.org
Usertags: debsources

Similarly to the current local/sources_news.html page, we should support
sources_old_news.html, and (manually) move there old news that are no longer
relevant enough to be shown on the front page.

When sources_old_news.html is present on disk, the front page should also have
a old news... link pointing to it.

Cheers.

-- System Information:
Debian Release: 8.0
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.utf8, LC_CTYPE=it_IT.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20150501082745.5425.89821.reportbug@timira.takhisis.invalid

Bug#783832: debsources: reenable the file count assertion that fails on travis builds

[Stefano Zacchiroli]

retitle 783832 re-enable all tests in travis build environment
thanks

On Thu, Apr 30, 2015 at 04:52:09PM +0200, Stefano Zacchiroli wrote:
 The assertion disabled in that commit should be reenabled. Before doing that,
 though, we should figure out why it fails in the travis CI environment.

The above is now fixed --- in the sense that the assertion has been
reenabled (for general use), and selectively disabled only in travis
builds, via the @notravis nose attribute.

However, we do want to run all tests also in travis builds. This bug
will track the need of doing that.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150510191403.ga19...@upsilon.cc

Re: Python 2, Python 3, Stretch Buster

[Stefano Zacchiroli]

Hey Paul,

On Thu, Apr 16, 2015 at 09:50:02AM -0400, Paul Tagliamonte wrote:
 - If *you* maintain or work on a Python 2 project that's used in Debian
   Development (buildd, release tools, QA tools, ftpteam tools), please
   email me a link to the project. An accurate census will help hugely. If
   it works on Python 2, Python 2 and Python 3 or just Python 3, you should
   include those details as well.

I co-maintain Debsources, which is already in the Debian service census
(https://wiki.debian.org/Services), with an entry at
https://wiki.debian.org/Services/DebianSources

Debsources is currently deployed on Python 2 (migrating to Python 3 is
tracked at #764961).

I've recently migrated it to a code base that _should_ work with both
Python 2 and Python 3, using a mixture of python-modernize, refactoring,
and peer review (thanks Niels for spotting many glitches!). But that's
just the theory, as nobody has ever tried to actually _deploy_ it with
Python 3.

And the reason for that are that at least the following dependencies of
Debsources are missing Python 3 package in Debian:

- flask-wtf: #764959
- wtforms: #780600

I _think_ in both cases there is Python 3 support upstream, but I
haven't thoroughly investigated this. Other people on debian-qa (Cc:-ed)
might have.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Bug#788226: debsources: add package/version blacklist

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: wishlist
User: qa.debian@packages.debian.org
Usertags: debsources

It would be nice to have a configurable package/version (and possibly /suite)
blacklist, to exclude specific packages from processing.

Use case: due to #753219 and #787271, we now have katoob/0.5.9.1-1.2+squeeze1
and moodle/1.9.9.dfsg2-2.1+squeeze4 which are not extractable. But nonetheless
the updater try to extract them at every single update run, and fails.  It's
not a big deal for such small packages, but if it were to happen for lager
packages it would be a problem.

A blacklist would allow to ignore specific packages during update runs.

Cheers.

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.utf8, LC_CTYPE=it_IT.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20150609150306.8061.51193.reportbug@timira.takhisis.invalid

Re: Bug#808216: debmirror: Debmirror seeems to have problems with new SHA256 in Packages.diff/Index files

[Stefano Zacchiroli]

On Thu, Dec 17, 2015 at 06:45:13PM +0800, Michael Bergbauer wrote:
> For a fews days (Dec 09, to be exact, in GMT+8), my debmirror cronjob pulling
> from ftp.us is running in an infinite loop - using up 100% of a core. After
> having a closer look, it seems that the culprit code is in lines 2460ff with a
> $_ value of e.g.  'SHA256-Current:
> d4228ed8d1591732f9a3af33f4064c4e0d173d16218d12b930d1c5de3673d7ce39582357\n',
> when parsing my .temp/dists/sid/main/binary-i386/Packages.diff/Index file.

Confirmed here too: Debsources/sources.debian.net hasn't been updated
since December 8th due to this bug.

It'd be great to have a backported debmirror with this fix.

I'll try Colin's oneliner patch as a work-around to get Debsources back
up running.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader . . . . . @zacchiro . . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: PGP signature

Bug#818324: debsources: migrate from Sources.gz to Sources.xz

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: important
User: qa.debian@packages.debian.org
Usertags: debsources

Due to the mail below (and also at
https://lists.debian.org/debian-devel-announce/2016/03/msg6.html ) we need
to migrate the mirror lister module of Debsources from using Sources.gz to
using Sources.xz.

See:
https://anonscm.debian.org/cgit/qa/debsources.git/tree/debsources/debmirror.py#n195
for a *starting* point.

Cheers.

- Forwarded message from Joerg Jaspert  -

Date: Tue, 15 Mar 2016 23:15:16 +0100
From: Joerg Jaspert 
To: debian-devel-annou...@lists.debian.org
Subject: Archive changes
Message-ID: <87k2l3xua3@gkar.ganneff.de>
Resent-From: debian-devel-annou...@lists.debian.org

Hi,

I've just activated a few changes to the archive we talk(ed) about for a
long time. And while it is not exactly the start of this release cycle,
it should still work out nicely (so one hopes).

As of now, InRelease/Release files, Packages and Sources no longer
provide MD5Sum and SHA1sums, only SHA256.

Additionally I turned off generating gzip compressed versions of those
files, xz is there.

To test it, this is limited to experimental. We hope nothing breaks on it,
but lets try for a few days. If that works out, we should adjust
unstable, and another short time later coordinate with the release team
to adjust testing, so it ends up in the next release.

Have fun.

- End forwarded message -


-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.3.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.utf8, LC_CTYPE=it_IT.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Bug#818324: updates of packages in experimental are currently ignored

[Stefano Zacchiroli]

severity 818324 serious
thanks

I'm raising the severity of 818324, because Sources.gz are already gone
for experimental, meaning that we are effectively ignoring them on
sources.debian.net until this is fixed.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader . . . . . @zacchiro . . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: PGP signature

Bug#818498: migrate from debmirror to archvsync

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: important
User: qa.debian@packages.debian.org
Usertags: debsources

debmirror shortcomings have been bugging debsources for quite a while lately.
Let's migrate away from it to archvsync [1], as that will put us in a more
"normal" situation w.r.t. how actual Debian mirrors are maintained around the
world.

[1]: https://ftp-master.debian.org/git/archvsync.git

I'm marking this bug as important because, after the recent archive changes,
debmirror isn't properly mirroring experimental (due to its having been
switched to Sources.xz).

Cheers.

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.3.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.utf8, LC_CTYPE=it_IT.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Bug#835033: debsources: [PATCH] Retain line context on 404s by appending the hash for specific links

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: minor
User: qa.debian@packages.debian.org
Usertags: debsources

Forwarding to the BTS for record keeping.

- Forwarded message from Chris Lamb <la...@debian.org> -

Date: Sun, 21 Aug 2016 11:41:27 +0100
From: Chris Lamb <la...@debian.org>
To: debian-qa@lists.debian.org
Cc: i...@sources.debian.net
Subject: debsources: [PATCH] Retain line context on 404s by appending the hash 
for specific links
Message-Id: <1471776087.3092845.701483073.410a5...@webmail.messagingengine.com>

Hi,

Attached is the following:

  commit a0a47c965d364fb41d1b0fde2480252a9c328479
  Author: Chris Lamb <la...@debian.org>
  Date:   Sun Aug 21 11:39:25 2016 +0100
  
  Retain line context on 404s by appending the hash for specific links
  
  Signed-off-by: Chris Lamb <la...@debian.org>
  
   debsources/app/sources/templates/sources/404_suggestions.html | 2 +-
   debsources/app/templates/footer.inc.html  | 8 
   2 files changed, 9 insertions(+), 1 deletion(-)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

>From a0a47c965d364fb41d1b0fde2480252a9c328479 Mon Sep 17 00:00:00 2001
From: Chris Lamb <la...@debian.org>
Date: Sun, 21 Aug 2016 11:39:25 +0100
Subject: [PATCH] Retain line context on 404s by appending the hash for
 specific links

Signed-off-by: Chris Lamb <la...@debian.org>
---
 debsources/app/sources/templates/sources/404_suggestions.html | 2 +-
 debsources/app/templates/footer.inc.html  | 8 
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/debsources/app/sources/templates/sources/404_suggestions.html 
b/debsources/app/sources/templates/sources/404_suggestions.html
index bf4609e..f1127f7 100644
--- a/debsources/app/sources/templates/sources/404_suggestions.html
+++ b/debsources/app/sources/templates/sources/404_suggestions.html
@@ -11,7 +11,7 @@ The file you are looking for might exist in one of the 
following
 versions:
 
 {% for s in suggestions %}
-  {{ s }}
+  {{ s }}
 {% endfor %}
 
 
diff --git a/debsources/app/templates/footer.inc.html 
b/debsources/app/templates/footer.inc.html
index 29d1d14..8636672 100644
--- a/debsources/app/templates/footer.inc.html
+++ b/debsources/app/templates/footer.inc.html
@@ -32,3 +32,11 @@
   Last update: {{ last_update }}.
 
 
+
+
+var elems = document.querySelectorAll('.js-append-window-location-hash');
+
+for (var i = 0; i < elems.length; ++i) {
+  elems[i].setAttribute('href', elems[i].getAttribute('href') + 
window.location.hash);
+}
+
-- 
2.9.3



- End forwarded message -

-- 
Stefano Zacchiroli . z...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
Computer Science Professor . CTO Software Heritage . . . . . o . . . o o
Former Debian Project Leader . OSI Board Director  . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

Bug#844201: debsources: updater broken by recent (Sources.gz removal, maybe sha1 removal) dak changes

[Stefano Zacchiroli]

Package: qa.debian.org
Severity: serious
User: qa.debian@packages.debian.org
Usertags: debsources

Context is:

  https://lists.debian.org/debian-devel-announce/2016/11/msg5.html

Here are some relevant entries from debsources.log:

  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Use of uninitialized value $sha1 in string eq at /usr/bin/debmirror line 1639.
  Errors:
   Download of dists/squeeze-lts/Release failed
   Ignoring failed Release files
   Ignoring missing Release file for dists/sid/main/source/Sources.gz
   Ignoring missing Release file for dists/sid/contrib/source/Sources.gz
   Ignoring missing Release file for dists/sid/non-free/source/Sources.gz
   Ignoring missing Release file for dists/squeeze-lts/main/source/Sources.gz
   Ignoring missing Release file for dists/squeeze-lts/contrib/source/Sources.gz
   Ignoring missing Release file for 
dists/squeeze-lts/non-free/source/Sources.gz
   Ignoring missing Release file for dists/experimental/main/source/Sources.gz
   Ignoring missing Release file for 
dists/experimental/contrib/source/Sources.gz
   Ignoring missing Release file for 
dists/experimental/non-free/source/Sources.gz
  /srv/debsources/bin/debsources-main: I: debsources-update...

Cheers.

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.7.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=it_IT.utf8, LC_CTYPE=it_IT.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Re: Debian infra services and tools looking for programming contributions

[Stefano Zacchiroli]

Heya, thanks for this initiative!

On Thu, May 21, 2020 at 03:36:15PM -0300, Antonio Terceiro wrote:
> I'm planning a talk titled "I'm a programmer, how can I help Debian?" in
> which I intend to present contribution opportunities for people who are
> programmers, but are not necessarily interested in packaging. My plan is
> to present several Debian infrastructure services and tools that could
> receive contributions, highlighting a few where contributions could have
> a larger impact in the community (IMO).
> 
> For services, my starting point is https://wiki.debian.org/Services For
> tools, I currently have a list of the ones I usually contribute to, but
> can add more.
> 
> Not the part where I need your help. I'm looking for people who maintain
> or contribute to a Debian infrastructure service or tool that could use
> some help with programming, have the availability to provide some
> mentoring for someone who is already a programmer but not necessarily
> already involved with Debian, and would like your project to be
> highlighted in such a talk.
> 
> If that's you, please reply to this message and provide some information
> about your service or tool. Package names are enough for tools in the
> archive, otherwise links/wiki pages/etc are appreciated. Please also
> mention a contact point (IRC channel, mailing list etc).

sources.debian.org, AKA Debsources, could use some help. I'm definitely
MIA on it, and the bulk of code maintenance is being assured by Mathieu
alone, including migration to Python 3 (thanks!). Having someone else
would be good, and I think it might be a piece of infra that might be
interesting to work on even for people that don't have a lot of Debian
insider knowledge.

Links:

- service: https://sources.debian.org/
- code: https://salsa.debian.org/qa/debsources
- bugs:
  
https://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;include=subject%3Adebsources;package=qa.debian.org

Hope this helps and thanks again !
Cheers
-- 
Stefano Zacchiroli . z...@upsilon.cc . upsilon.cc/zack . . o . . . o . o
Computer Science Professor . CTO Software Heritage . . . . . o . . . o o
Former Debian Project Leader & OSI Board Director  . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: PGP signature