Re: Using release-monitoring.org [was: uscan roadmap]

[Stephan Lachnit]

On Sat, Dec 4, 2021 at 3:34 AM Paul Wise  wrote:
>
> Repology gets you mappings for all the source packages in Debian in one
> download (assuming it has an export of the mappings, that may need to
> be added), while the Anitya mapping requires a human to manually add a
> mapping for each of the thousands of source packages in Debian. Not all
> maintainers are going to bother and repetitive clicking is going to get
> boring for the folks trying to make up for that.

I'm sure there would be a way to automate this with repology data.

> > Also, mapping on Repology sometimes needs to be adjusted manually. And
> > sometimes they disagree and instead tell you to rename the source
> > package in the distro (happened to me once), which is not really
> > viable in Debian.
>
> I wasn't aware of the renaming part, seems kind of weird.

See e.g. [1]:
"Will not merge: Modules (e.g. python) without consistent prefix (e.g.
python- or python3-) (common problem for Slackbuilds and Debian source
packages). [...]"

> > Yes it can't, but also I don't think this is something *release
> > monitoring* should do. It is definitely a good use case and that is
> > why there is a link to repology on the tracker (called "other
> > distros"), but it has IMHO nothing to do with *automatic* release
> > monitoring. Don't get me wrong, I actually like repology exactly for
> > this particular reason.
>
> I was taking the thread topic to be the slightly more general area of
> "monitoring when a package needs updating to a new upstream release,
> snapshot or fork". New VCS snapshots in other distros fits that IMO.

Ah right I see, I guess we then should separate more between fetching
the tarball and scanning for versions to inform the maintainer - I
don't think that these necessarily need to use the same technique. For
informing the maintainer, repology might as well be an additional very
useful tool.

> The other issue with using Anitya is that Debian and Fedora have
> different policies and culture for choosing which upstream versions to
> update to. Debian strongly prefers LTS versions while Fedora are all
> about the latest and greatest, which is a bit of a culture clash and is
> likely to mean for some packages we couldn't use Anitya.

I don't think this is an issue here - see [2]. The response
differentiates between stable versions and other versions. I'm not
sure how RCs are handled, but it would not be that difficult to
implement.

> In addition to independence there is the issue Jonas mentioned
> elsewhere in the initial uscan thread that some Debian people prefer
> the info to be maintained in the source package instead of elsewhere.

Of course this would be optional. Regarding bootstrapping it might not
be that good of an idea to use it for essential packages anyway.

> > This sounds more reasonable to me than writing a tool that converts a
> > new standard to the old one just as backup.
>
> Given the above, perhaps a way to sync a locally stored file and the
> Anitya one, and then have uscan understand the Anitya format?

I've been looking at the api (see [2]) for a bit and while not trying
it out myself, afaik there is no functionality to download a tarball.
While I like the idea of release-monitoring, I now feel like it
doesn't fulfill the needs of Debian and probably newer will. So
putting all watch files in a single salsa repo probably makes more
sense, or something similar to offload it.

On Sat, Dec 4, 2021 at 3:44 AM Scott Kitterman  wrote:
>
> I think that there's a security consideration associated with all these 
> proposals for externalizing finding upstream updates.  Currently watch files 
> and at least the redirectors I know of all run on Debian infrastructure or on 
> the systems of the Debian person doing the update.
>
> If one of these services were ever compromised it would provide a vector for 
> offering substitute upstream code (at least for the cases where upstream 
> releases aren't both signed by upstream and verified in Debian).  I find that 
> prospect concerning.

Good point, but I think this can be mitigated relatively easily - just
always print the url of the tarball that is downloaded (which is a
good idea anyway). The maintainer should know the url where the
sources are hosted, and if the printed url somehow looks weird, it can
be easily checked.


Stephan


[1] https://repology.org/project/symfit/report
[2] https://release-monitoring.org/static/docs/api.html#post--api-v2-versions-

Re: Using release-monitoring.org [was: uscan roadmap]

[Wouter Verhelst]

On Sat, Dec 04, 2021 at 02:43:56AM +, Scott Kitterman wrote:
> I think that there's a security consideration associated with all these
> proposals for externalizing finding upstream updates.  Currently watch files
> and at least the redirectors I know of all run on Debian infrastructure or on
> the systems of the Debian person doing the update.

I don't see how? At least repology just tells you "there is a new
upstream release", it doesn't tell you where to get it. It's up to the
maintainer to know where to download a new release.

Obviously if upstream is compromised and a new "release" is produced
that contains malicious code then there is a problem, but that is a
problem that is neither exacerbated nor mitigated by using repology.

> If one of these services were ever compromised it would provide a
> vector for offering substitute upstream code (at least for the cases
> where upstream releases aren't both signed by upstream and verified in
> Debian).  I find that prospect concerning.

Validating that upstream releases are valid is part of the job of being
a maintainer in Debian. Having some helper service that tells you there
is a new release doesn't change that.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: Using release-monitoring.org [was: uscan roadmap]

[Andrius Merkys]

On 2021-12-03 00:51, Paul Wise wrote:
> The one issue I can think of with using release-monitoring.org is that
> Debian becomes more reliant on an external service, while currently we
> are completely independent of other distros for version checking. 
> 
> Converting the release-monitoring.org check to a watch file might be an
> alternative to using it directly that maintains our independence.

+1

Best,
Andrius

Re: uscan roadmap

[Yadd]

On 05/12/2021 10:26, Timo Röhling wrote:

Hi Yadd,

thank you very much for your work on uscan. That new version 5
format looks really promising.

* Yadd  [2021-12-01 09:11]:

* Version 5:
  * Main (first) paragraph contains "Version: 5" and optional options
    that change default values for source-paragraph
  * URL and regex are separated
  * Some default values change. For example, `dversionmangle` default
    value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

[...]

Of course, comments are welcome!
I have a feature request regarding signature verification. As luck would 
have it, I maintain three packages with upstream

signatures; one of them is me being my own upstream, and the other
two do not use the "standard" approach with one GnuPG signature per
source tarball:

- cmake releases its sources in multiple archive formats and signs
   them indirectly (a text file with SHA256 hashes) [1].
- liblzf uses the BSD signify tool [2] and only GnuPG-signs the
   signify key.

I don't know if any of these schemes are used elsewhere (more likely
for the CMake approach, less likely for liblzf, I'd guess), but it
would be nice if uscan offered some support for this; maybe a hook
to run the signature verification by an external script with
autopkgtest semantics (fail if output occurs on stderr the script
returns with a non-zero exit code).


Hi,

this was ~impossible using previous format. With new format, things like 
pgpmode=previous/next will still be accepted (to avoid regressions) but 
new things will be possible. Example (not yet implemented):


  Version: 5

  Source: https://url...
  Version-Regex: @PACKAGE@@ANY_VERSION@@ARCHIVE_EXT@
  Pgp-Version-Regex:  @PACKAGE@@SAME_VERSION@@ARCHIVE_EXT@.asc

Then it will be possible to implement new signature workflows, maybe 
something like:


  Version: 5

  Source: https://url...
  Version-Regex: @PACKAGE@@ANY_VERSION@@ARCHIVE_EXT@
  Pgp-Mode: sha256-hashes
  Sha256-Version-Regex: .*@SAME_VERSION@.sig

Cheers,
Yadd

Re: uscan roadmap

[Timo Röhling]

Hi Yadd,

thank you very much for your work on uscan. That new version 5
format looks really promising.

* Yadd  [2021-12-01 09:11]:

* Version 5:
  * Main (first) paragraph contains "Version: 5" and optional options
that change default values for source-paragraph
  * URL and regex are separated
  * Some default values change. For example, `dversionmangle` default
value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

[...]

Of course, comments are welcome!
I have a feature request regarding signature verification. 
As luck would have it, I maintain three packages with upstream

signatures; one of them is me being my own upstream, and the other
two do not use the "standard" approach with one GnuPG signature per
source tarball:

- cmake releases its sources in multiple archive formats and signs
  them indirectly (a text file with SHA256 hashes) [1].
- liblzf uses the BSD signify tool [2] and only GnuPG-signs the
  signify key.

I don't know if any of these schemes are used elsewhere (more likely
for the CMake approach, less likely for liblzf, I'd guess), but it
would be nice if uscan offered some support for this; maybe a hook
to run the signature verification by an external script with
autopkgtest semantics (fail if output occurs on stderr the script
returns with a non-zero exit code).


Cheers
Timo

[1] https://cmake.org/install/#download-verification
[2] http://dist.schmorp.de/signing-key.txt

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Using release-monitoring.org [was: uscan roadmap]

[Scott Kitterman]

On December 5, 2021 1:51:48 AM UTC, Paul Wise  wrote:
>On Sat, 2021-12-04 at 02:43 +, Scott Kitterman wrote:
>
>> I think that there's a security consideration associated with all these
>> proposals for externalizing finding upstream updates. 
>
>Good point.
>
>> If one of these services were ever compromised it would provide a
>> vector for offering substitute upstream code (at least for the cases
>> where upstream releases aren't both signed by upstream and verified in
>> Debian).  I find that prospect concerning.
>
>I think the same concern should also apply to centralised upstream
>development infrastructure like GitHub and also individual upstream
>developers themselves. There isn't really any mitigation for malicious
>code being pushed out beyond commit/release signing (both unpopular)
>and (distributed) downstream code review.
>
>To mitigate the concern for upstream version monitoring we could prefer
>debian/watch when it exists but fall back to release-monitoring.org
>when one doesn't exist, have a tool to convert the Ayanita format into
>debian/watch and have dh_make and similar try to create an initial
>debian/watch by default.
>
>We need a culture of doing change review before updating to new
>upstream releases, but often that isn't necessarily feasible,
>especially for large projects with rapid change or when switching to
>new forks of existing tools.
>
>> Currently watch files and at least the redirectors I know of all run
>> on Debian infrastructure or on the systems of the Debian person doing
>> the update.
>
>Some run on debian.org servers, and many run on debian.net domains.
>However I don't think that that makes them immune to compromise.
>
Never claimed it did, but the knowledge that our security posture isn't 
invulnerable isn't an argument for making it worse.

Scott K

Re: Using release-monitoring.org [was: uscan roadmap]

[Paul Wise]

On Sat, 2021-12-04 at 02:43 +, Scott Kitterman wrote:

> I think that there's a security consideration associated with all these
> proposals for externalizing finding upstream updates. 

Good point.

> If one of these services were ever compromised it would provide a
> vector for offering substitute upstream code (at least for the cases
> where upstream releases aren't both signed by upstream and verified in
> Debian).  I find that prospect concerning.

I think the same concern should also apply to centralised upstream
development infrastructure like GitHub and also individual upstream
developers themselves. There isn't really any mitigation for malicious
code being pushed out beyond commit/release signing (both unpopular)
and (distributed) downstream code review.

To mitigate the concern for upstream version monitoring we could prefer
debian/watch when it exists but fall back to release-monitoring.org
when one doesn't exist, have a tool to convert the Ayanita format into
debian/watch and have dh_make and similar try to create an initial
debian/watch by default.

We need a culture of doing change review before updating to new
upstream releases, but often that isn't necessarily feasible,
especially for large projects with rapid change or when switching to
new forks of existing tools.

> Currently watch files and at least the redirectors I know of all run
> on Debian infrastructure or on the systems of the Debian person doing
> the update.

Some run on debian.org servers, and many run on debian.net domains.
However I don't think that that makes them immune to compromise.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: uscan roadmap

[Yadd]

On 01/12/2021 22:16, Yadd wrote:

On 01/12/2021 21:07, Patrice wrote:


Really great!
And could the new uscan read a watch file from version 3/4/5 and output a
version 5 of it by its own (in-place or stdout)?
uscan --standardize
:-)


Yes but without optimization neither scheme (except some few fields). 
Example:


   version=4

   opts=uversionmangle=s/-/~/g,pgpmode=none \
    https://...  .*(\d[\d\.]*)@ARCHIVE_EXT@

will be translated into:

   Version: 5

   Source: https://...
   Regex: .*(\d[\d\.]*)@ARCHIVE_EXT@
   Uversionmangle: s/-/~/g

You'll have to manually modify it into

   Version: 5
   Scheme: stable

   Source: https://...


Done, uscan from 
https://salsa.debian.org/debian/devscripts/-/merge_requests/251 is already:


 * natively working with watchFiles version 5
 * able to read old versions transparently (using a wrapper)
 * able to convert files from version 3/4 to 5 (using the same
   wrapper), and probably versions 1/2 (not fully tested):

 uscan --update-watchfile

   Don't us it for now except for tests, key names will probably change

Looking at test results: no regression, all passed

Re: Using release-monitoring.org [was: uscan roadmap]

[Simon McVittie]

On Sat, 04 Dec 2021 at 10:33:55 +0800, Paul Wise wrote:
> The other issue with using Anitya is that Debian and Fedora have
> different policies and culture for choosing which upstream versions to
> update to. Debian strongly prefers LTS versions while Fedora are all
> about the latest and greatest, which is a bit of a culture clash and is
> likely to mean for some packages we couldn't use Anitya.

For quite a few packages that have this branching structure, Debian
would ideally have both - latest development versions in experimental,
and LTS versions in unstable/testing/stable.

In some packages I use a different d/watch for unstable and experimental
(so that at least I can do a "uscan --download" on each branch and
get the right thing), but that's one more thing to adjust when merging
between branches, and the PTS can only remind me about one of them. It
would be great if uscan (or equivalent) could show us both from one
configuration file.

smcv

Re: Using release-monitoring.org [was: uscan roadmap]

[Scott Kitterman]

On December 3, 2021 12:12:47 PM UTC, Stephan Lachnit 
 wrote:
>On Thu, Dec 2, 2021 at 11:52 PM Paul Wise  wrote:
>>
>> On Thu, 2021-12-02 at 23:36 +0100, Stephan Lachnit wrote:
>>
>> > If I understand correctly, release-monitoring already offers such a
>> > mapping [1].
>>
>> It seems like the Ayanita distro mapping needs to be done manually once
>> per package, while using the Repology data would automatically get us
>> the mapping for each existing package and all future packages.
>
>I mean it looks rather easy to do, just a couple of mouse clicks.
>Compare that to writing a watch file at the moment (assuming one has
>to do more than copy and paste the github example).
>
>> > Hm, I can't really think of an example where such a thing couldn't
>> > also be implemented in release-monitoring.org.
>>
>> None of the three use-cases I listed can be done by it AFAICT.
>>
>> It can't check things that it doesn't have a check for, while
>> individual package maintainers in various distros will update their
>> packages and Repology will notice the new versions.
>
>Then the maintainer would just have to write a check, just like they
>have to do now.
>
>Also, mapping on Repology sometimes needs to be adjusted manually. And
>sometimes they disagree and instead tell you to rename the source
>package in the distro (happened to me once), which is not really
>viable in Debian.
>
>> It presumably doesn't look at the versions for all distros, so it can't
>> do the cross-distro VCS snapshot choice check, while individual package
>> maintainers in various distros know their packages well and might
>> upgrade to a VCS snapshot in their distro, which Repology notices.
>
>Yes it can't, but also I don't think this is something *release
>monitoring* should do. It is definitely a good use case and that is
>why there is a link to repology on the tracker (called "other
>distros"), but it has IMHO nothing to do with *automatic* release
>monitoring. Don't get me wrong, I actually like repology exactly for
>this particular reason.
>
>> It also isn't going to check locations it doesn't check yet, while
>> individual package maintainers in other distros might do that after
>> noticing their package hasn't been updated recently and then going
>> searching for a new upstream and updating, which Repology notices.
>
>Fair point, but if we would work together on release-monitoring.org
>with Fedora, there are more eyes on it as well as in the current
>situation.
>Repology still has more eyes of course, but then again the link to
>Repology is right there on the tracker already if one is curious.
>
>> > Just one quick idea I had: what about a "fake" uscan backend? I.e.
>> > something like `Version: release-monitoring.org` in d/watch. In that
>> > case uscan will launch an external program that fetches the data from
>> > there and gives it back to uscan, so that other tools stay unaffected
>> > until a full transition is done.
>>
>> Excellent idea, that would be great to have.
>
>One more thought on this. If we go with version 5, maybe something like:
>
>Version: 5
>Source: release-monitoring.org
>
>Would also work for multiple sources then and in general would fit
>nicely to the current idea for v5. It also solves the problem with the
>tooling, watch files and uscan would still exist, but the "searching"
>portion is offloaded.
>
>> The one issue I can think of with using release-monitoring.org is that
>> Debian becomes more reliant on an external service, while currently we
>> are completely independent of other distros for version checking.
>>
>> Converting the release-monitoring.org check to a watch file might be an
>> alternative to using it directly that maintains our independence.
>
>Hm right, independence is a valid concern. Anitya itself is open
>source [1] so we could host it easily, but of course the real problem
>would be the stored data of the projects. I don't know if they are
>hosted somewhere, but I'm sure the Fedora guys would be open to share
>them with us, so that we could easily spin up a mirror in case there
>are any problems (it's probably a good idea to host a read-only mirror
>just in case).
>
>This sounds more reasonable to me than writing a tool that converts a
>new standard to the old one just as backup.

I think that there's a security consideration associated with all these 
proposals for externalizing finding upstream updates.  Currently watch files 
and at least the redirectors I know of all run on Debian infrastructure or on 
the systems of the Debian person doing the update.

If one of these services were ever compromised it would provide a vector for 
offering substitute upstream code (at least for the cases where upstream 
releases aren't both signed by upstream and verified in Debian).  I find that 
prospect concerning.

Scott K

Re: Using release-monitoring.org [was: uscan roadmap]

[Paul Wise]

On Fri, 2021-12-03 at 13:12 +0100, Stephan Lachnit wrote:

> I mean it looks rather easy to do, just a couple of mouse clicks.
> Compare that to writing a watch file at the moment (assuming one has
> to do more than copy and paste the github example).

Repology gets you mappings for all the source packages in Debian in one
download (assuming it has an export of the mappings, that may need to
be added), while the Anitya mapping requires a human to manually add a
mapping for each of the thousands of source packages in Debian. Not all
maintainers are going to bother and repetitive clicking is going to get
boring for the folks trying to make up for that.

> Then the maintainer would just have to write a check, just like they
> have to do now.

Or you could get the most recent distro version for free without manual
work by using the Repology data. While it isn't always the true latest
release from upstream, often the latest distro version being newer than
the Debian version is good enough to notify the Debian maintainer to
update the package to the true latest release from upstream.

> Also, mapping on Repology sometimes needs to be adjusted manually. And
> sometimes they disagree and instead tell you to rename the source
> package in the distro (happened to me once), which is not really
> viable in Debian.

I wasn't aware of the renaming part, seems kind of weird.

> Yes it can't, but also I don't think this is something *release
> monitoring* should do. It is definitely a good use case and that is
> why there is a link to repology on the tracker (called "other
> distros"), but it has IMHO nothing to do with *automatic* release
> monitoring. Don't get me wrong, I actually like repology exactly for
> this particular reason.

I was taking the thread topic to be the slightly more general area of
"monitoring when a package needs updating to a new upstream release,
snapshot or fork". New VCS snapshots in other distros fits that IMO.

> Fair point, but if we would work together on release-monitoring.org
> with Fedora, there are more eyes on it as well as in the current
> situation.
> Repology still has more eyes of course, but then again the link to
> Repology is right there on the tracker already if one is curious.

Sure, I think we need all three solutions as they all fit different
use-cases within the "update to latest release/snapshot/fork" arena;
(see below for more about why) debian/watch, Anitya and Repology. There
is already a bug about more Repology integration within the package
tracker, and I was the one who did the existing tracker integration.

> One more thought on this. If we go with version 5, maybe something
> like:
> 
> Version: 5
> Source: release-monitoring.org

Looks good.

> Would also work for multiple sources then and in general would fit
> nicely to the current idea for v5. It also solves the problem with the
> tooling, watch files and uscan would still exist, but the "searching"
> portion is offloaded.

Agreed.

> Hm right, independence is a valid concern. Anitya itself is open
> source [1] so we could host it easily, but of course the real problem
> would be the stored data of the projects. I don't know if they are
> hosted somewhere, but I'm sure the Fedora guys would be open to share
> them with us, so that we could easily spin up a mirror in case there
> are any problems (it's probably a good idea to host a read-only mirror
> just in case).

The other issue with using Anitya is that Debian and Fedora have
different policies and culture for choosing which upstream versions to
update to. Debian strongly prefers LTS versions while Fedora are all
about the latest and greatest, which is a bit of a culture clash and is
likely to mean for some packages we couldn't use Anitya.

In addition to independence there is the issue Jonas mentioned
elsewhere in the initial uscan thread that some Debian people prefer
the info to be maintained in the source package instead of elsewhere.

> This sounds more reasonable to me than writing a tool that converts a
> new standard to the old one just as backup.

Given the above, perhaps a way to sync a locally stored file and the
Anitya one, and then have uscan understand the Anitya format?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Using release-monitoring.org [was: uscan roadmap]

[Stephan Lachnit]

On Thu, Dec 2, 2021 at 11:52 PM Paul Wise  wrote:
>
> On Thu, 2021-12-02 at 23:36 +0100, Stephan Lachnit wrote:
>
> > If I understand correctly, release-monitoring already offers such a
> > mapping [1].
>
> It seems like the Ayanita distro mapping needs to be done manually once
> per package, while using the Repology data would automatically get us
> the mapping for each existing package and all future packages.

I mean it looks rather easy to do, just a couple of mouse clicks.
Compare that to writing a watch file at the moment (assuming one has
to do more than copy and paste the github example).

> > Hm, I can't really think of an example where such a thing couldn't
> > also be implemented in release-monitoring.org.
>
> None of the three use-cases I listed can be done by it AFAICT.
>
> It can't check things that it doesn't have a check for, while
> individual package maintainers in various distros will update their
> packages and Repology will notice the new versions.

Then the maintainer would just have to write a check, just like they
have to do now.

Also, mapping on Repology sometimes needs to be adjusted manually. And
sometimes they disagree and instead tell you to rename the source
package in the distro (happened to me once), which is not really
viable in Debian.

> It presumably doesn't look at the versions for all distros, so it can't
> do the cross-distro VCS snapshot choice check, while individual package
> maintainers in various distros know their packages well and might
> upgrade to a VCS snapshot in their distro, which Repology notices.

Yes it can't, but also I don't think this is something *release
monitoring* should do. It is definitely a good use case and that is
why there is a link to repology on the tracker (called "other
distros"), but it has IMHO nothing to do with *automatic* release
monitoring. Don't get me wrong, I actually like repology exactly for
this particular reason.

> It also isn't going to check locations it doesn't check yet, while
> individual package maintainers in other distros might do that after
> noticing their package hasn't been updated recently and then going
> searching for a new upstream and updating, which Repology notices.

Fair point, but if we would work together on release-monitoring.org
with Fedora, there are more eyes on it as well as in the current
situation.
Repology still has more eyes of course, but then again the link to
Repology is right there on the tracker already if one is curious.

> > Just one quick idea I had: what about a "fake" uscan backend? I.e.
> > something like `Version: release-monitoring.org` in d/watch. In that
> > case uscan will launch an external program that fetches the data from
> > there and gives it back to uscan, so that other tools stay unaffected
> > until a full transition is done.
>
> Excellent idea, that would be great to have.

One more thought on this. If we go with version 5, maybe something like:

Version: 5
Source: release-monitoring.org

Would also work for multiple sources then and in general would fit
nicely to the current idea for v5. It also solves the problem with the
tooling, watch files and uscan would still exist, but the "searching"
portion is offloaded.

> The one issue I can think of with using release-monitoring.org is that
> Debian becomes more reliant on an external service, while currently we
> are completely independent of other distros for version checking.
>
> Converting the release-monitoring.org check to a watch file might be an
> alternative to using it directly that maintains our independence.

Hm right, independence is a valid concern. Anitya itself is open
source [1] so we could host it easily, but of course the real problem
would be the stored data of the projects. I don't know if they are
hosted somewhere, but I'm sure the Fedora guys would be open to share
them with us, so that we could easily spin up a mirror in case there
are any problems (it's probably a good idea to host a read-only mirror
just in case).

This sounds more reasonable to me than writing a tool that converts a
new standard to the old one just as backup.


Regards,
Stephan

[1] https://github.com/fedora-infra/anitya

Re: Using release-monitoring.org [was: uscan roadmap]

[Paul Wise]

On Thu, 2021-12-02 at 23:36 +0100, Stephan Lachnit wrote:

> If I understand correctly, release-monitoring already offers such a
> mapping [1].

It seems like the Ayanita distro mapping needs to be done manually once
per package, while using the Repology data would automatically get us
the mapping for each existing package and all future packages.

> Hm, I can't really think of an example where such a thing couldn't
> also be implemented in release-monitoring.org.

None of the three use-cases I listed can be done by it AFAICT.

It can't check things that it doesn't have a check for, while 
individual package maintainers in various distros will update their
packages and Repology will notice the new versions.

It presumably doesn't look at the versions for all distros, so it can't
do the cross-distro VCS snapshot choice check, while individual package
maintainers in various distros know their packages well and might
upgrade to a VCS snapshot in their distro, which Repology notices.

It also isn't going to check locations it doesn't check yet, while
individual package maintainers in other distros might do that after
noticing their package hasn't been updated recently and then going
searching for a new upstream and updating, which Repology notices.

> Just one quick idea I had: what about a "fake" uscan backend? I.e.
> something like `Version: release-monitoring.org` in d/watch. In that
> case uscan will launch an external program that fetches the data from
> there and gives it back to uscan, so that other tools stay unaffected
> until a full transition is done.

Excellent idea, that would be great to have.

The one issue I can think of with using release-monitoring.org is that
Debian becomes more reliant on an external service, while currently we
are completely independent of other distros for version checking. 

Converting the release-monitoring.org check to a watch file might be an
alternative to using it directly that maintains our independence.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Using release-monitoring.org [was: uscan roadmap]

[Stephan Lachnit]

On Thu, 2 Dec 2021, 23:17 Paul Wise,  wrote:

> At minimum we would need a way to map from release-monitoring.org
> package names to Debian source package names. Assuming they use Fedora
> source package names, then the Repology service provides such a mapping
> and we could presumably could get a periodic export of that.
>

If I understand correctly, release-monitoring already offers such a mapping
[1].


> I see using Repology as a complement to release-monitoring.org and
> uscan, not as an alternative to them. It enables use-cases that aren't
> possible with the other two. We automatically get version monitoring
> for packages that don't have other version monitoring mechanisms. We
> get monitoring of whether or not a particular package needs updating to
> a VCS snapshot instead of waiting for an official release. We get
> monitoring of versions even when upstream has moved to a different
> location not monitored by other mechanisms. There are probably other
> use-cases I can't think of right now.
>

Hm, I can't really think of an example where such a thing couldn't also be
implemented in release-monitoring.org.


> > Yes that makes sense, what I wonder is how much change is needed for
> > putting watch files in a separate repo compared to going with
> > release-monitoring.org (I don't know enough about the inner workings
> > of our tools to answer this question).
>
> For the VCS idea it would be minimal, just vcswatch needs to also pull
> debian/watch files out of VCS repos with commits not yet pushed to
> Debian and then the version checking infra (zero idea where that went)
> needs to pay attention to that data.
>

Oh right, that actually sounds pretty smart.


> For fully moving debian/watch (and Homepage) out of Debian source
> packages there would need to be a lot more work, probably migrating to
> release-monitoring.org would be the way to go.
>

Just one quick idea I had: what about a "fake" uscan backend? I.e.
something like `Version: release-monitoring.org` in d/watch. In that case
uscan will launch an external program that fetches the data from there and
gives it back to uscan, so that other tools stay unaffected until a full
transition is done.

In case there are others interested in this approach, I would be down to do
some coding (when I find time).


Regards,
Stephan

[1]
https://release-monitoring.org/static/docs/user-guide.html#creating-new-distribution-mapping

Re: Using release-monitoring.org [was: uscan roadmap]

[Scott Talbert]

On Fri, 3 Dec 2021, Paul Wise wrote:


I think this would be the best path forward - it would probably be not
easy given that it changes entirely how the current system works, but
it might be well worth the effort. Working together with another
distribution would share the work for the distro. I'm sure if we are
willing to join them they would accommodate us if there are any
changes we would require (e.g. login via salsa instead of a fedora
account).


At minimum we would need a way to map from release-monitoring.org
package names to Debian source package names. Assuming they use Fedora
source package names, then the Repology service provides such a mapping
and we could presumably could get a periodic export of that.


release-monitoring.org has the ability to configure distribution-specific 
names for each package.  Take for example pycurl, which has mappings for 
Fedora, Alpine, and Timesys:


https://release-monitoring.org/project/7973/

It appears there are 314 packages that already have Debian mappings.

Scott

Re: uscan roadmap

[Paul Wise]

On Thu, 2021-12-02 at 10:16 +0100, Yadd wrote:

> Yes but the redirector often responded with 500 codes

500 codes probably just mean bugs in the redirector, which should be
easy to fix for anyone with access to the redirector source code.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Using release-monitoring.org [was: uscan roadmap]

[Paul Wise]

On Thu, 2021-12-02 at 15:57 +0100, Stephan Lachnit wrote:

> I think this would be the best path forward - it would probably be not
> easy given that it changes entirely how the current system works, but
> it might be well worth the effort. Working together with another
> distribution would share the work for the distro. I'm sure if we are
> willing to join them they would accommodate us if there are any
> changes we would require (e.g. login via salsa instead of a fedora
> account).

At minimum we would need a way to map from release-monitoring.org
package names to Debian source package names. Assuming they use Fedora
source package names, then the Repology service provides such a mapping
and we could presumably could get a periodic export of that.

> This however I think is not a good idea. Repology is very nice to
> check what versions other distros have, but for projects that don't
> have any external language-specific package repository like e.g.
> python, it would mean that we could easily miss a new release (think
> small projects written in C that are not in any other distro) and
> wrongly formatted version from other distros would impact us
> (unlikely, but still bad in theory).

I see using Repology as a complement to release-monitoring.org and
uscan, not as an alternative to them. It enables use-cases that aren't
possible with the other two. We automatically get version monitoring
for packages that don't have other version monitoring mechanisms. We
get monitoring of whether or not a particular package needs updating to
a VCS snapshot instead of waiting for an official release. We get
monitoring of versions even when upstream has moved to a different
location not monitored by other mechanisms. There are probably other
use-cases I can't think of right now.

> And since it requires the same infrastructure changes as going with
> release-monitoring.org, it would be better to just use that.

I think it would need different changes, especially since Debian
doesn't have the same realtime notifications stuff as Fedora does.

> Yes that makes sense, what I wonder is how much change is needed for
> putting watch files in a separate repo compared to going with
> release-monitoring.org (I don't know enough about the inner workings
> of our tools to answer this question).

For the VCS idea it would be minimal, just vcswatch needs to also pull
debian/watch files out of VCS repos with commits not yet pushed to
Debian and then the version checking infra (zero idea where that went)
needs to pay attention to that data.

For fully moving debian/watch (and Homepage) out of Debian source
packages there would need to be a lot more work, probably migrating to
release-monitoring.org would be the way to go.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Using release-monitoring.org [was: uscan roadmap]

[Stephan Lachnit]

On Thu, Dec 2, 2021 at 12:51 AM Paul Wise  wrote:
>
> It might be a idea to look at how other distributions do checking for
> new upstream releases and adopt some of their improvements.
>
> I note Fedora uses a service (that isn't Fedora specific) for this:
>
> https://release-monitoring.org
> https://docs.fedoraproject.org/en-US/package-maintainers/Upstream_Release_Monitoring/

I think this would be the best path forward - it would probably be not
easy given that it changes entirely how the current system works, but
it might be well worth the effort. Working together with another
distribution would share the work for the distro. I'm sure if we are
willing to join them they would accommodate us if there are any
changes we would require (e.g. login via salsa instead of a fedora
account).

> Another idea would be to use the Repology service to notice when other
> distros include a newer version of a package than Debian does.
>
> https://repology.org/

This however I think is not a good idea. Repology is very nice to
check what versions other distros have, but for projects that don't
have any external language-specific package repository like e.g.
python, it would mean that we could easily miss a new release (think
small projects written in C that are not in any other distro) and
wrongly formatted version from other distros would impact us
(unlikely, but still bad in theory).

And since it requires the same infrastructure changes as going with
release-monitoring.org, it would be better to just use that.

> I also wonder if it is time to split debian/watch out of Debian source
> packages, since upstream download locations generally change
> independently of the Debian package and so information about upstream
> download locations probably should be maintained independently.

Yes that makes sense, what I wonder is how much change is needed for
putting watch files in a separate repo compared to going with
release-monitoring.org (I don't know enough about the inner workings
of our tools to answer this question).


Anyway, if using release-monitoring.org is too much work or nobody is
willing to do it, I like the proposals for version 5 so far.

Regards,
Stephan

Re: uscan roadmap

[Gard Spreemann]

Jonas Smedegaard  writes:

> Quoting Gard Spreemann (2021-12-02 13:09:17)
>> 
>> Jonas Smedegaard  writes:
>> 
>> > Quoting Gard Spreemann (2021-12-02 12:31:30)
>> >> 
>> >> Paul Wise  writes:
>> >> 
>> >> > I also wonder if it is time to split debian/watch out of Debian 
>> >> > source packages, since upstream download locations generally 
>> >> > change independently of the Debian package and so information 
>> >> > about upstream download locations probably should be maintained 
>> >> > independently.
>> >> 
>> >> I very much agree. I don't understand the logic of tying upstream 
>> >> checking to a particular version of a source package. The fact that 
>> >> some version of a package once upon a time could locate and 
>> >> download new upstream versions using a particular recipe is of no 
>> >> use if said recipe becomes outdated at any later time.
>> >> 
>> >> It makes a lot more sense to provide this service in a way that 
>> >> allows it to be modified/updated/improved/fixed with no regards to 
>> >> the actual packages that may use it. That could be as simple as a 
>> >> uscan service with watch files that can be individually and 
>> >> independently modified.
>> >
>> > I find it helpful for our packages to include information about 
>> > where and how (at the time of its release) that package was 
>> > monitoring for upstream releases.  It helps working decentralized - 
>> > both for preparing packages for Debian and for working on 
>> > Debian-derived packages, both without needing access to somewhere 
>> > central for this "watch" information.
>> 
>> Would it make sense for a package to contain a snapshot of the 
>> relevant metadata in the hypothetical "centralized-and-often-updated 
>> watch service" at the time in enters into the archives?
>
> Not _instead_ of current location: What I find helpful is to have the 
> watch file available with the source package.  I am unaware if there 
> would be some benefit of _additionally_ embedding the watch file in 
> binary packages (if that's what you meant).
>
>
>> > Therefore I like the proposal to have Debian project scanners 
>> > aggressively look for _newest_ watch file for a packaging project, 
>> > including looking up declared Vcs-* hints for not-yet-released work.
>> 
>> Indeed, that sounds like a better idea than what I suggest above!
>
> Not sure if you noticed, but (since I won't steal credit) I basically 
> emphasized Pabs' suggestion in last paragraph of what you previously 
> quoted:
>
> Quoting Paul Wise (2021-12-02 00:47:44)
>> Alternatively, perhaps we could workaround outdated debian/watch files
>> by having vcswatch extract debian/watch files from the repo specified
>> in the Vcs-* URLs.

Apologies; I somehow thought that he meant auto-generating watch files
from *upstream* VCSs. My bad, and thanks for clarifying!

 -- Gard
 


signature.asc
Description: PGP signature

Re: uscan roadmap

[Julien Puydt]

Hi

Le jeu. 2 déc. 2021 à 11:36, Yadd  a écrit :

>
> Another idea to have a compromise:
>   * uscan is released with versioned schemes (GitHub.json, sf.json,...)
>   * when launched, it tries to download new version from a new Debian API
> (static json files)
> * if no response or no new version, uscan uses its own scheme or a
>   previously downloaded update (verifying signature)
> * if a new version is available from new redirector:
>   * it verifies GPG signature of new scheme
> * if not OK, it warns and uses cached scheme
> * if OK, it stores it with signature in ~/.cache/uscan/schemes
>

What I don't like is that it will need time to check new profiles on a
central site, which looks like an invitation for DoS situations.

I propose a variation of this: an explicit
"uscan --update" will update the profiles, and all other calls will use the
known profiles.

Cheers,

J. Puydt

Re: uscan roadmap

[Jonas Smedegaard]

Quoting Gard Spreemann (2021-12-02 13:09:17)
> 
> Jonas Smedegaard  writes:
> 
> > Quoting Gard Spreemann (2021-12-02 12:31:30)
> >> 
> >> Paul Wise  writes:
> >> 
> >> > I also wonder if it is time to split debian/watch out of Debian 
> >> > source packages, since upstream download locations generally 
> >> > change independently of the Debian package and so information 
> >> > about upstream download locations probably should be maintained 
> >> > independently.
> >> 
> >> I very much agree. I don't understand the logic of tying upstream 
> >> checking to a particular version of a source package. The fact that 
> >> some version of a package once upon a time could locate and 
> >> download new upstream versions using a particular recipe is of no 
> >> use if said recipe becomes outdated at any later time.
> >> 
> >> It makes a lot more sense to provide this service in a way that 
> >> allows it to be modified/updated/improved/fixed with no regards to 
> >> the actual packages that may use it. That could be as simple as a 
> >> uscan service with watch files that can be individually and 
> >> independently modified.
> >
> > I find it helpful for our packages to include information about 
> > where and how (at the time of its release) that package was 
> > monitoring for upstream releases.  It helps working decentralized - 
> > both for preparing packages for Debian and for working on 
> > Debian-derived packages, both without needing access to somewhere 
> > central for this "watch" information.
> 
> Would it make sense for a package to contain a snapshot of the 
> relevant metadata in the hypothetical "centralized-and-often-updated 
> watch service" at the time in enters into the archives?

Not _instead_ of current location: What I find helpful is to have the 
watch file available with the source package.  I am unaware if there 
would be some benefit of _additionally_ embedding the watch file in 
binary packages (if that's what you meant).


> > Therefore I like the proposal to have Debian project scanners 
> > aggressively look for _newest_ watch file for a packaging project, 
> > including looking up declared Vcs-* hints for not-yet-released work.
> 
> Indeed, that sounds like a better idea than what I suggest above!

Not sure if you noticed, but (since I won't steal credit) I basically 
emphasized Pabs' suggestion in last paragraph of what you previously 
quoted:

Quoting Paul Wise (2021-12-02 00:47:44)
> Alternatively, perhaps we could workaround outdated debian/watch files
> by having vcswatch extract debian/watch files from the repo specified
> in the Vcs-* URLs.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: uscan roadmap

[Gard Spreemann]

Jonas Smedegaard  writes:

> Quoting Gard Spreemann (2021-12-02 12:31:30)
>> 
>> Paul Wise  writes:
>> 
>> > I also wonder if it is time to split debian/watch out of Debian 
>> > source packages, since upstream download locations generally change 
>> > independently of the Debian package and so information about 
>> > upstream download locations probably should be maintained 
>> > independently.
>> 
>> I very much agree. I don't understand the logic of tying upstream 
>> checking to a particular version of a source package. The fact that 
>> some version of a package once upon a time could locate and download 
>> new upstream versions using a particular recipe is of no use if said 
>> recipe becomes outdated at any later time.
>> 
>> It makes a lot more sense to provide this service in a way that allows 
>> it to be modified/updated/improved/fixed with no regards to the actual 
>> packages that may use it. That could be as simple as a uscan service 
>> with watch files that can be individually and independently modified.
>
> I find it helpful for our packages to include information about where 
> and how (at the time of its release) that package was monitoring for 
> upstream releases.  It helps working decentralized - both for preparing 
> packages for Debian and for working on Debian-derived packages, both 
> without needing access to somewhere central for this "watch" 
> information.

Would it make sense for a package to contain a snapshot of the relevant
metadata in the hypothetical "centralized-and-often-updated watch
service" at the time in enters into the archives?

> Therefore I like the proposal to have Debian project scanners 
> aggressively look for _newest_ watch file for a packaging project, 
> including looking up declared Vcs-* hints for not-yet-released work.

Indeed, that sounds like a better idea than what I suggest above!


 -- Gard
 


signature.asc
Description: PGP signature

Re: uscan roadmap

[Jonas Smedegaard]

Quoting Gard Spreemann (2021-12-02 12:31:30)
> 
> Paul Wise  writes:
> 
> > I also wonder if it is time to split debian/watch out of Debian 
> > source packages, since upstream download locations generally change 
> > independently of the Debian package and so information about 
> > upstream download locations probably should be maintained 
> > independently.
> 
> I very much agree. I don't understand the logic of tying upstream 
> checking to a particular version of a source package. The fact that 
> some version of a package once upon a time could locate and download 
> new upstream versions using a particular recipe is of no use if said 
> recipe becomes outdated at any later time.
> 
> It makes a lot more sense to provide this service in a way that allows 
> it to be modified/updated/improved/fixed with no regards to the actual 
> packages that may use it. That could be as simple as a uscan service 
> with watch files that can be individually and independently modified.

I find it helpful for our packages to include information about where 
and how (at the time of its release) that package was monitoring for 
upstream releases.  It helps working decentralized - both for preparing 
packages for Debian and for working on Debian-derived packages, both 
without needing access to somewhere central for this "watch" 
information.

Therefore I like the proposal to have Debian project scanners 
aggressively look for _newest_ watch file for a packaging project, 
including looking up declared Vcs-* hints for not-yet-released work.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: uscan roadmap

[Gard Spreemann]

Paul Wise  writes:

> I also wonder if it is time to split debian/watch out of Debian source
> packages, since upstream download locations generally change
> independently of the Debian package and so information about upstream
> download locations probably should be maintained independently.

I very much agree. I don't understand the logic of tying upstream
checking to a particular version of a source package. The fact that some
version of a package once upon a time could locate and download new
upstream versions using a particular recipe is of no use if said recipe
becomes outdated at any later time.

It makes a lot more sense to provide this service in a way that allows
it to be modified/updated/improved/fixed with no regards to the actual
packages that may use it. That could be as simple as a uscan service
with watch files that can be individually and independently modified.


 Best,
 Gard
 


signature.asc
Description: PGP signature

Re: uscan roadmap

[Geert Stappers]

On Thu, Dec 02, 2021 at 11:36:08AM +0100, Yadd wrote:
> On 02/12/2021 10:16, Yadd wrote:
> > On 02/12/2021 00:34, Paul Wise  wrote:
> > > On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:
> > > 
> > > > Personally I dislike redirectors.
> > > 
> > > A redirector service is superior to including the redirector code
> > > within uscan itself or within a debian/watch file, since when the
> > > upstream website breaks the existing code, a service can be updated in
> > > one place immediately, while uscan in Debian stable will be broken
> > > until the next point release if it gets fixed at all and one in
> > > debian/watch requires every package using the site to get updated.

So true


> > 
> > Yes but the redirector often responded with 500 codes
> 
> Another idea to have a compromise:
>  * uscan is released with versioned schemes (GitHub.json, sf.json,...)
>  * when launched, it tries to download new version from a new Debian API
>(static json files)
>* if no response or no new version, uscan uses its own scheme or a
>  previously downloaded update (verifying signature)
>* if a new version is available from new redirector:
>  * it verifies GPG signature of new scheme
>* if not OK, it warns and uses cached scheme
>* if OK, it stores it with signature in ~/.cache/uscan/schemes
> 
> Then:
>  * no more redirector with an heavy load, but just some JSON schemes
>statically stored
>  * uscan still works if Debian website doesn't respond
> 
> What do you think about this idea?


Way too optimistic   :-)

The original problem was (and is) dealing with various upstream websites.

Putting a translator, a redirector, between uscan and a single upstream
website solves the problem for that particular website.

IMNSHO is building (hard to upgrade and distribute) "solutions"
for redirectors with 500s or whatever error effort at the wrong place.

Explaining to the user (us, debian maintainers) what is happing is a
better approach.   Especial when the redirector can explain the 500 is
due problems with the actual upstream website.



Groeten
Geert Stappers
-- 
Silence is hard to parse

Re: uscan roadmap

[Yadd]

On 02/12/2021 10:16, Yadd wrote:

On 02/12/2021 00:34, Paul Wise  wrote:

On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:


Personally I dislike redirectors.


A redirector service is superior to including the redirector code
within uscan itself or within a debian/watch file, since when the
upstream website breaks the existing code, a service can be updated in
one place immediately, while uscan in Debian stable will be broken
until the next point release if it gets fixed at all and one in
debian/watch requires every package using the site to get updated.



Yes but the redirector often responded with 500 codes


Another idea to have a compromise:
 * uscan is released with versioned schemes (GitHub.json, sf.json,...)
 * when launched, it tries to download new version from a new Debian API
   (static json files)
   * if no response or no new version, uscan uses its own scheme or a
 previously downloaded update (verifying signature)
   * if a new version is available from new redirector:
 * it verifies GPG signature of new scheme
   * if not OK, it warns and uses cached scheme
   * if OK, it stores it with signature in ~/.cache/uscan/schemes

Then:
 * no more redirector with an heavy load, but just some JSON schemes
   statically stored
 * uscan still works if Debian website doesn't respond
 * GPG permits to be sure that scheme isn't corrupted (released files
   are as protected as uscan itself because owned by root)
 * easy update if upstream store changes its behavior: just to update
   one JSON file

What do you think about this idea? Which GPG keys will be accepted?

More than one scheme could be used. Example:

  Schemes: GitHub, stable

  Schemes: GitHub, semver

GitHub scheme:

  {
"Version": "1.0",
"filenamemangle": "auto",
"regex": "archive/.*/v?(@DEFAULT_VERSION_REGEX@)@ARCHIVE_EXT@$",
...
  }

"stable" scheme:

  {
"Version": "1.0",
"Default-Version-Regex": "(?:0|[1-9]\d*)(?:0|[1-9]\d*)*"
  }

Re: uscan roadmap

[Yadd]

Le 2 décembre 2021 00:34:27 GMT+01:00, Paul Wise  a écrit :
>On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:
>
>> Personally I dislike redirectors.
>
>A redirector service is superior to including the redirector code
>within uscan itself or within a debian/watch file, since when the
>upstream website breaks the existing code, a service can be updated in
>one place immediately, while uscan in Debian stable will be broken
>until the next point release if it gets fixed at all and one in
>debian/watch requires every package using the site to get updated.
>

Yes but the redirector often responded with 500 codes
-- 
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma 
brièveté.

Re: uscan roadmap

[Paul Wise]

On Wed, 2021-12-01 at 09:11 +0100, Yadd wrote:

> after few discussions with some devscripts maintainers, we decided to
> build a new "version=5" format for debian/watch.

It might be a idea to look at how other distributions do checking for
new upstream releases and adopt some of their improvements.

I note Fedora uses a service (that isn't Fedora specific) for this:

https://release-monitoring.org
https://docs.fedoraproject.org/en-US/package-maintainers/Upstream_Release_Monitoring/

Another idea would be to use the Repology service to notice when other
distros include a newer version of a package than Debian does. 

https://repology.org/

I also wonder if it is time to split debian/watch out of Debian source
packages, since upstream download locations generally change
independently of the Debian package and so information about upstream
download locations probably should be maintained independently.

Alternatively, perhaps we could workaround outdated debian/watch files
by having vcswatch extract debian/watch files from the repo specified
in the Vcs-* URLs.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: uscan roadmap

[Paul Wise]

On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:

> sf.net because it needs JS interpretation

The sf.net redirector uses the RSS feed of the files.
This is documented at the top of the redirector HTML:

$ curl -s https://qa.debian.org/watch/sf.php/NSIS/ | grep -i rss
(https://sourceforge.net/projects/NSIS/rss?limit=1000;>RSS)
pages. File listing converted from the RSS.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: uscan roadmap

[Paul Wise]

On Wed, 2021-12-01 at 12:53 +0100, Yadd wrote:

> Personally I dislike redirectors.

A redirector service is superior to including the redirector code
within uscan itself or within a debian/watch file, since when the
upstream website breaks the existing code, a service can be updated in
one place immediately, while uscan in Debian stable will be broken
until the next point release if it gets fixed at all and one in
debian/watch requires every package using the site to get updated.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: uscan roadmap

[Yadd]

On 01/12/2021 21:07, Patrice wrote:


Really great!
And could the new uscan read a watch file from version 3/4/5 and output a
version 5 of it by its own (in-place or stdout)?
uscan --standardize
:-)


Yes but without optimization neither scheme (except some few fields). 
Example:


  version=4

  opts=uversionmangle=s/-/~/g,pgpmode=none \
   https://...  .*(\d[\d\.]*)@ARCHIVE_EXT@

will be translated into:

  Version: 5

  Source: https://...
  Regex: .*(\d[\d\.]*)@ARCHIVE_EXT@
  Uversionmangle: s/-/~/g

You'll have to manually modify it into

  Version: 5
  Scheme: stable

  Source: https://...

Re: uscan roadmap

[Patrice]

Really great!
And could the new uscan read a watch file from version 3/4/5 and output a
version 5 of it by its own (in-place or stdout)?
uscan --standardize
:-)

Wishes,
Patrice

Re: uscan roadmap

[Tomas Pospisek]

On 01.12.21 12:50, Mattia Rizzolo wrote:


Likewise, I would love if uscan could just learn how github, gitlab,
launchpad, etc are made so prople won't have to bother with sticking
urls into watchfiles, such as:

   Source: GitHub
   Source-Options:
namespace: trendmicro
project: tlsh
match-on: tags|releases


Excellent idea: +1

However at the very moment that abstractions get introduced (which is 
+1), please, please, please do keep the poor users in mind when stuff 
*does not* work. I.e. please make uscan trivially debuggable.


Something like:

$ uscan --verbose
... found GitHub source definition:
Source: GitHub
Source-Options:
  namespace: trendmicro
  project: tlsh
  match-on: tags|releases
... using that GitHub source definition to find new release under 
https://github.com/foo/bar/releases/bar-1.2.3.tar (uscan.py:line 3498)


One of the regularly recurring frustrations I am encountering is that 
I'm using some SW that has abstracted something and I know in principle 
how the thing it has abstracted works, but am completely unable to find 
out where the abstraction gets all wired up and falls on its face. (Hi 
k8s!). The famous "computer says no" moment.


OK, this is just me whining and not contributing any code at all, so 
please take it as just that: a wishlist item.

*t

Re: uscan roadmap

[Yadd]

On 01/12/2021 18:39, Thomas Goirand wrote:

Hi Yadd,

Thanks a lot for working on this. What you are proposing (ie: using a
mime thing, which is easy to parse instead of the dirty command-line
oriented thingy of version 3 and 4) feels much nicer than what we
currently have.

On 12/1/21 12:53 PM, Yadd wrote:

Fix: will be

   Version: 5

   Source: https://qa.debian.org/watch/sourceforge/
   Regex: -(.+)\.tar\.gz


That's much nicer than previous proposal!


And I don't think "uupdate" is still useful.


IMO, it is needed. That's what is nice with calling scripts: it can take
care with programming of things you didn't even think of. If you remove
it, the risk is that maintainers will continue to use version 3 or 4,
because they still need an update script.

How about:

Update-Script: uupdate

?


Looks good to me

Re: uscan roadmap

[Thomas Goirand]

Hi Yadd,

Thanks a lot for working on this. What you are proposing (ie: using a
mime thing, which is easy to parse instead of the dirty command-line
oriented thingy of version 3 and 4) feels much nicer than what we
currently have.

On 12/1/21 12:53 PM, Yadd wrote:
> Fix: will be
> 
>   Version: 5
> 
>   Source: https://qa.debian.org/watch/sourceforge/
>   Regex: -(.+)\.tar\.gz

That's much nicer than previous proposal!

> And I don't think "uupdate" is still useful.

IMO, it is needed. That's what is nice with calling scripts: it can take
care with programming of things you didn't even think of. If you remove
it, the risk is that maintainers will continue to use version 3 or 4,
because they still need an update script.

How about:

Update-Script: uupdate

?

Cheers,

Thomas Goirand (zigo)

Re: uscan roadmap

[Russ Allbery]

Yadd  writes:

> after few discussions with some devscripts maintainers, we decided to
> build a new "version=5" format for debian/watch.

> Principles:
>  * keep compatibility with versions 3 and 4, no need to change all
>debian/watch files
>  * new version 5 format using the same syntax than other debian/* files
>(rfc822 + "# comments")

I have no detailed feedback on your proposal, but just wanted to say thank
you so much for doing this.  Being able to write a uscan file using a
clear key/value syntax will make me very happy.

-- 
Russ Allbery (r...@debian.org)  

Re: uscan roadmap

[Yadd]

On 01/12/2021 13:14, Jonas Smedegaard wrote:

Quoting Yadd (2021-12-01 13:04:09)

On 01/12/2021 12:50, Mattia Rizzolo wrote:

Possibly, I'm indeed kind of unimpressed that we grew a parse for
nodejs' package.json and perl's META.json.  Though I accepted it
because I saw some value, I'm totally in awe of universes where that
is actually needed..

This is very useful for package with components. This is the only way
to be able to `uscan --download-current-version`.


Speakin of components, it would be quite helpful if possible to handle
versions of components - e.g. upgrade all components except
SomeComponent like this:

   `uscan --download-current-version-SomeComponent`


Yes, maybe with a --download-compat-version-SomeComponent which will 
accept upgrades only if it is semver-compatible (no major updates)


Note that this will be in a separated MR (not related to version change)

Re: uscan roadmap

[Jonas Smedegaard]

Quoting Yadd (2021-12-01 13:04:09)
> On 01/12/2021 12:50, Mattia Rizzolo wrote:
> > Possibly, I'm indeed kind of unimpressed that we grew a parse for 
> > nodejs' package.json and perl's META.json.  Though I accepted it 
> > because I saw some value, I'm totally in awe of universes where that 
> > is actually needed..
> This is very useful for package with components. This is the only way 
> to be able to `uscan --download-current-version`.

Speakin of components, it would be quite helpful if possible to handle 
versions of components - e.g. upgrade all components except 
SomeComponent like this:

  `uscan --download-current-version-SomeComponent`


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: uscan roadmap

[Yadd]

On 01/12/2021 12:50, Mattia Rizzolo wrote:

On Wed, Dec 01, 2021 at 12:39:41PM +0100, Geert Stappers wrote:

Summary: unhide redirectors


And not only.


On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:

after few discussions with some devscripts maintainers, we decided to build
a new "version=5" format for debian/watch.


To be clear, this is a *very* non-ready proposal that we are getting
through the wider community.  Nothing of this is implemented anywhere.


* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
  value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...


I honestly would like to add website-aware functionalities to uscan,
such as exactly this.


I think the move from v4 to v5 is an excellent opportunity
to express in the watch file that there is a dependency on a redirector.

[..]

  Version: 5
  Source: https://qa.debian.org/watch/sourceforge/ 
-(.+)\.tar\.gz debian uupdate


I would like something like:
   Source: qa-redirector
   Source-Options:
name: sourceforge
project: 


Likewise, I would love if uscan could just learn how github, gitlab,
launchpad, etc are made so prople won't have to bother with sticking
urls into watchfiles, such as:

   Source: GitHub
   Source-Options:
namespace: trendmicro
project: tlsh
match-on: tags|releases


OK, "Source" will be:
 * a keyword known by uscan DB
 * a URL for other websites

And what about default version value, "stable" (only digits), "semver" 
or "any" (== @ANY_VERSION@)



To go either matching on https://github.com/trendmicro/tlsh/tags or
https://github.com/trendmicro/tlsh/releases. using then Scheme (a name
that, tbh, I don't particularly like right now) for the tags or releases
regex.


And I think such change will allow removal of

bare
Disable all site specific special case code such as URL
redirector uses and page content alterations.

from the uscan code and uscan manual page  (they are in /usr/bin/uscan )


The goal is to have documented that there are extra components being used.
Avoiding nasty surprises.


this feels like the opposite direction I'm proposing above :D


Seconded


Awareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter.


Possibly, I'm indeed kind of unimpressed that we grew a parse for
nodejs' package.json and perl's META.json.  Though I accepted it because
I saw some value, I'm totally in awe of universes where that is actually
needed..
This is very useful for package with components. This is the only way to 
be able to `uscan --download-current-version`.

Re: uscan roadmap

[Yadd]

On 01/12/2021 12:39, Geert Stappers wrote:

Summary: unhide redirectors

On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:

Hi,

after few discussions with some devscripts maintainers, we decided to build
a new "version=5" format for debian/watch.

Principles:
  * keep compatibility with versions 3 and 4, no need to change all
debian/watch files
  * new version 5 format using the same syntax than other debian/* files
(rfc822 + "# comments")
  * no option renaming (becomes case-insensitive to be compliant with
all formats)
  * Version 5:
* Main (first) paragraph contains "Version: 5" and optional options
  that change default values for source-paragraph
* URL and regex are separated
* Some default values change. For example, `dversionmangle` default
  value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

Example:

   Version: 5


   


Of course, comments are welcome!



I think the move from v4 to v5 is an excellent opportunity
to express in the watch file that there is a dependency on a redirector.


Example

  version=4
  https://sf.net// -(.+)\.tar\.gz debian uupdate


becomes something like

  Version: 5
  Source: https://qa.debian.org/watch/sourceforge/ 
-(.+)\.tar\.gz debian uupdate


Fix: will be

  Version: 5

  Source: https://qa.debian.org/watch/sourceforge/
  Regex: -(.+)\.tar\.gz

And I don't think "uupdate" is still useful.

Personally I dislike redirectors. For npmjs, I wrote a 
"serachMode=plain" to be able to parse npm JSON result, also because 
there are a lot of npmjs packages and redirector was often unable to 
respond. This is different of course for sf.net because it needs JS 
interpretation



And I think such change will allow removal of

bare
Disable all site specific special case code such as URL
redirector uses and page content alterations.

from the uscan code and uscan manual page  (they are in /usr/bin/uscan )


Yes, doc will be updated


The goal is to have documented that there are extra components being used.
Avoiding nasty surprises.

Groeten
Geert Stappers


Thanks!


P.S.
Awareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter

Re: uscan roadmap

[Mattia Rizzolo]

On Wed, Dec 01, 2021 at 12:39:41PM +0100, Geert Stappers wrote:
> Summary: unhide redirectors

And not only.

> On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:
> > after few discussions with some devscripts maintainers, we decided to build
> > a new "version=5" format for debian/watch.

To be clear, this is a *very* non-ready proposal that we are getting
through the wider community.  Nothing of this is implemented anywhere.

> >* URL and regex are separated
> >* Some default values change. For example, `dversionmangle` default
> >  value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
> >  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

I honestly would like to add website-aware functionalities to uscan,
such as exactly this.

> I think the move from v4 to v5 is an excellent opportunity
> to express in the watch file that there is a dependency on a redirector.
[..]
>  Version: 5
>  Source: https://qa.debian.org/watch/sourceforge/ 
> -(.+)\.tar\.gz debian uupdate

I would like something like:
  Source: qa-redirector
  Source-Options:
   name: sourceforge
   project: 


Likewise, I would love if uscan could just learn how github, gitlab,
launchpad, etc are made so prople won't have to bother with sticking
urls into watchfiles, such as:

  Source: GitHub
  Source-Options:
   namespace: trendmicro
   project: tlsh
   match-on: tags|releases

To go either matching on https://github.com/trendmicro/tlsh/tags or
https://github.com/trendmicro/tlsh/releases. using then Scheme (a name
that, tbh, I don't particularly like right now) for the tags or releases
regex.

> And I think such change will allow removal of
> 
>bare
>Disable all site specific special case code such as URL
>redirector uses and page content alterations.
> 
> from the uscan code and uscan manual page  (they are in /usr/bin/uscan )
> 
> 
> The goal is to have documented that there are extra components being used.
> Avoiding nasty surprises.

this feels like the opposite direction I'm proposing above :D

> Awareness of redirectors will get us more redirectors.
> Those redirectors will help us to prevent that `uscan`
> must get a javascript interpreter.

Possibly, I'm indeed kind of unimpressed that we grew a parse for
nodejs' package.json and perl's META.json.  Though I accepted it because
I saw some value, I'm totally in awe of universes where that is actually
needed..

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: uscan roadmap

[Geert Stappers]

Summary: unhide redirectors

On Wed, Dec 01, 2021 at 09:11:17AM +0100, Yadd wrote:
> Hi,
> 
> after few discussions with some devscripts maintainers, we decided to build
> a new "version=5" format for debian/watch.
> 
> Principles:
>  * keep compatibility with versions 3 and 4, no need to change all
>debian/watch files
>  * new version 5 format using the same syntax than other debian/* files
>(rfc822 + "# comments")
>  * no option renaming (becomes case-insensitive to be compliant with
>all formats)
>  * Version 5:
>* Main (first) paragraph contains "Version: 5" and optional options
>  that change default values for source-paragraph
>* URL and regex are separated
>* Some default values change. For example, `dversionmangle` default
>  value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
>  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...
> 
> Example:
> 
>   Version: 5
> 
  
> 
> Of course, comments are welcome!


I think the move from v4 to v5 is an excellent opportunity
to express in the watch file that there is a dependency on a redirector.


Example

 version=4
 https://sf.net// -(.+)\.tar\.gz debian uupdate


becomes something like

 Version: 5
 Source: https://qa.debian.org/watch/sourceforge/ 
-(.+)\.tar\.gz debian uupdate



And I think such change will allow removal of

   bare
   Disable all site specific special case code such as URL
   redirector uses and page content alterations.

from the uscan code and uscan manual page  (they are in /usr/bin/uscan )


The goal is to have documented that there are extra components being used.
Avoiding nasty surprises.




Groeten
Geert Stappers


P.S.
Awareness of redirectors will get us more redirectors.
Those redirectors will help us to prevent that `uscan`
must get a javascript interpreter.


-- 
Silence is hard to parse

uscan roadmap

[Yadd]

Hi,

after few discussions with some devscripts maintainers, we decided to 
build a new "version=5" format for debian/watch.


Principles:
 * keep compatibility with versions 3 and 4, no need to change all
   debian/watch files
 * new version 5 format using the same syntax than other debian/* files
   (rfc822 + "# comments")
 * no option renaming (becomes case-insensitive to be compliant with
   all formats)
 * Version 5:
   * Main (first) paragraph contains "Version: 5" and optional options
 that change default values for source-paragraph
   * URL and regex are separated
   * Some default values change. For example, `dversionmangle` default
 value will be "auto" (drop +dfsg, ~ds,...), uversionmangle=s/-/~/g,
 filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@)/@PACKAGE@-$1/...

Example:

  Version: 5

  Source: https://...


Some new features to simplify debian/watch:
 * a scheme option with the following values:
   * "stable": digit-only versions (maybe default value)
   * "semver": semver versions only (see semver.org)
   * "any": version matching current @ANY_VERSION@ macro

Example:

  Version: 5
  Scheme: stable

  Source: https://...

  Component: foo
  Source: https://...

  Component: bar
  Source: https://...
  Scheme: semver

Then a current GitHub changes from:

  version=4
  opts=\
  dversionmangle=auto,\
  filenamemangle=s/.*?(\d[\d\.]*@ARCHIVE_EXT@/@PACKAGE@-$1/ \
https://github.com/foo/bar/tags .*/archive/.*/v?([\d\.]+).tar.gz

to:

  Version: 5
  Scheme: stable

  Source: https://github.com/foo/bar/tags

Then to include rc/beta releases:

  Version: 5
  Scheme: semver

  Source: https://github.com/foo/bar/tags

Version 5 adds also new macros (used by Scheme of course):
 * @STABLE_VERSION@
 * @SEMVER@

Uscan API will also provide a "write" function usable by a provided 
script to convert old formats to new.


Of course, comments are welcome!

Cheers,
Yadd