Re: Debian Policy 4.1.4.0 released

[Andreas Tille]

On Wed, Jul 04, 2018 at 10:26:20PM +0100, Sean Whitton wrote:
> >Provide get-orig-source target if (and only if) uscan would fail.
> >
> > The previous discussion seem to show a tendency that this bug will be
> > at best tagged wontfix which for the moment prevents me from calling
> > reportbug right now.
> 
> It wouldn't be an immediate wontfix because it is not just a proposal to
> revert the change.

May be not immediate but I see no tendency that it has a promising
chance.
 
> An issue with this proposal is that it might suddenly make a lot of
> packages buggy that don't provide this facility.

I have no record of precious changes in policy but from a pure gut
feeling / wild guessing this might happen with policy changes.

> My suggestion to allow
> README.source might be a good first step towards what you are
> suggesting.

My main point is that README.source is just text and no code that I can
run and test.  That's different from my proposal.

Kind regards

Andreas.

-- 
http://fam-tille.de

Re: Debian Policy 4.1.4.0 released

[Bill Allombert]

On Tue, Jul 03, 2018 at 01:35:49PM +0200, Andreas Tille wrote:
> Not really, I do not want to use README.source or something like this.
> I have a *personal* policy:  I will not sponsor any package if there is
> no code I could run that recreates the source tarball.  May be I'm to
> strict and the sponsee might find some other sponsor.  The Debian policy
> change simply weakens my point where I think there are very good reasons
> for.

There are two different situations:

-- Given one upstream tarball and a Debian tarball, check you can
regenerate the Debian tarball (so it has not been doctored)

-- Given a new upstream tarball, generate a corresponding Debian tarball.

Unfortunately, it is not always possible to write a script that will
correctly anticipate the changes needed for a new upstream version.
Doubly so for Files-excluded stanzas. In that case README.source can
be useful.

Cheers,
-- 
Bill. 

Imagine a large red swirl here. 

Re: Debian Policy 4.1.4.0 released

[Sean Whitton]

Hello Andreas,

On Tue, Jul 03 2018, Andreas Tille wrote:

> I would love to create a new bug report but this would rather be:
>
>Provide get-orig-source target if (and only if) uscan would fail.
>
> The previous discussion seem to show a tendency that this bug will be
> at best tagged wontfix which for the moment prevents me from calling
> reportbug right now.

It wouldn't be an immediate wontfix because it is not just a proposal to
revert the change.

An issue with this proposal is that it might suddenly make a lot of
packages buggy that don't provide this facility.  My suggestion to allow
README.source might be a good first step towards what you are
suggesting.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Debian Policy 4.1.4.0 released

[Andreas Tille]

Hi Sean,

On Tue, Jul 03, 2018 at 09:59:55AM +0100, Sean Whitton wrote:
> [trimming the CC a bit; Russ and Ian read -devel]
> 
> Hello Jonathan, Andreas,
> 
> I don't think that what either of you have said is a response to the
> reasons that there were for removing this optional target from Policy.
> 
> The thought driving this is that not every trick in a Debian package
> maintainer's toolbox should be detailed in Policy.  What we want to
> write down here are techniques that apply to most packages, where the
> cases to which the techniques do not apply can be written down too.
> 
> The problem with get-orig-source is that it is always an edge case.
> It's only needed when d/watch and Files-Excluded aren't sufficient,
> which already puts you out in the weeds, and it is going to need to work
> differently in every package that uses it.  So it doesn't make sense to
> standardise it.

We had discussed this some weeks ago.  My answer was:  Policy should
enforce that any random person gets a tool to reproduce the upstream
tarball.  Usually it is uscan but if there is no chance to accomplish
that task with uscan some alternative should be provided.  I have seen
many such "edge cases" and IMHO policy should cover all packages
including edge cases.  We are extremely picky to document the copyright
and license of every single file in a source tarball.  I fail to see a
good reason why we should not encourage maintainers to also provide code
to fetch those files.
 
> However, the use cases that you raise are valid, and I agree with you
> that package maintainers should make it possible for users to extract
> the information that the two of you are trying to extract.  They might
> do that by providing a rules target called get-orig-source, which is
> perfectly allowed.  But that's now package-specific machinery.

I understand that the sense of the policy change is that it is package
specific and that I'm allowed to use it.  However, I'm now missing a
highly ranked documentation to point a newcomer to which answers the
question how to fetch the source code if uscan fails.
 
> It seems that what you actually want is not a very loose and unhelpful
> get-orig-source convention, but a recommendation or requirement that
> package maintainers make it possible to obtain a list of the files that
> were excluded, or similar.  Maintainers could fulfill that using
> Files-Excluded, a rules target, text in README.source, or whatever.

Not really, I do not want to use README.source or something like this.
I have a *personal* policy:  I will not sponsor any package if there is
no code I could run that recreates the source tarball.  May be I'm to
strict and the sponsee might find some other sponsor.  The Debian policy
change simply weakens my point where I think there are very good reasons
for.

> If you think you know exactly what that recommendation or requirement
> should be, please file a new bug proposing its addition to Policy.
> That's something that it makes sense to standardise, unlike
> get-orig-source.

I would love to create a new bug report but this would rather be:

   Provide get-orig-source target if (and only if) uscan would fail.

The previous discussion seem to show a tendency that this bug will
be at best tagged wontfix which for the moment prevents me from
calling reportbug right now.

Kind regards

 Andreas.
 
> On Tue, Jul 03 2018, Andreas Tille wrote:
> 
> > Since this does not exist any more I'm afraid we will end up with more
> > upstream tarballs a third person will not have any clue how to fetch
> > the source.  IMHO that's an unfortunate change in policy.
> 
> I don't see why removing an /optional/ target, which you can still use
> if you want to, makes it likely we'll end up with more such source
> packages.  The fact that the target isn't there won't make it much less
> likely someone will think, "I should ensure the tarball is fetchable."
> 
> -- 
> Sean Whitton



-- 
http://fam-tille.de

Re: Debian Policy 4.1.4.0 released

[Emmanuel Bourg]

> Git mode for uscan helps as well in many cases.

Is the git mode currently broken? I keep getting the error message
"fatal: Not a valid object name" when fetching a new release:

  $ uscan --download-version 9.2.25
  uscan: Newest version of jetty9 on remote site is 9.2.25, specified download 
version is 9.2.25
  Cloning into bare repository '../jetty9-temporary.15124.git'...
  remote: Counting objects: 6674, done.
  remote: Compressing objects: 100% (4546/4546), done.
  remote: Total 6674 (delta 2037), reused 3517 (delta 785), pack-reused 0
  Receiving objects: 100% (6674/6674), 18.04 MiB | 7.95 MiB/s, done.
  Resolving deltas: 100% (2037/2037), done.
  fatal: Not a valid object name
  uscan die: git archive failed

Emmanuel Bourg

Re: Debian Policy 4.1.4.0 released

[Sean Whitton]

[trimming the CC a bit; Russ and Ian read -devel]

Hello Jonathan, Andreas,

I don't think that what either of you have said is a response to the
reasons that there were for removing this optional target from Policy.

The thought driving this is that not every trick in a Debian package
maintainer's toolbox should be detailed in Policy.  What we want to
write down here are techniques that apply to most packages, where the
cases to which the techniques do not apply can be written down too.

The problem with get-orig-source is that it is always an edge case.
It's only needed when d/watch and Files-Excluded aren't sufficient,
which already puts you out in the weeds, and it is going to need to work
differently in every package that uses it.  So it doesn't make sense to
standardise it.

However, the use cases that you raise are valid, and I agree with you
that package maintainers should make it possible for users to extract
the information that the two of you are trying to extract.  They might
do that by providing a rules target called get-orig-source, which is
perfectly allowed.  But that's now package-specific machinery.

It seems that what you actually want is not a very loose and unhelpful
get-orig-source convention, but a recommendation or requirement that
package maintainers make it possible to obtain a list of the files that
were excluded, or similar.  Maintainers could fulfill that using
Files-Excluded, a rules target, text in README.source, or whatever.

If you think you know exactly what that recommendation or requirement
should be, please file a new bug proposing its addition to Policy.
That's something that it makes sense to standardise, unlike
get-orig-source.

On Tue, Jul 03 2018, Andreas Tille wrote:

> Since this does not exist any more I'm afraid we will end up with more
> upstream tarballs a third person will not have any clue how to fetch
> the source.  IMHO that's an unfortunate change in policy.

I don't see why removing an /optional/ target, which you can still use
if you want to, makes it likely we'll end up with more such source
packages.  The fact that the target isn't there won't make it much less
likely someone will think, "I should ensure the tarball is fetchable."

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Debian Policy 4.1.4.0 released

[Andreas Tille]

Hi Russ,

On Mon, Jul 02, 2018 at 04:40:43PM -0700, Russ Allbery wrote:
> Jonathan Nieder  writes:
> 
> > Context: I have run into a few packages that used the +dfsg convention
> > without documenting what they removed from the tarball and I was not
> > able to locally update them. :(
> 
> This is one of the cases that now has a better solution and more standard
> tools than the get-orig-source target, specifically Files-Excluded in
> debian/copyright.  See the documentation of Files-Excluded in uscan(1).

Files-Excluded helps a lot if there exists an upstream tarball for
download (and I actually invented it to get rid of many get-orig-source
scripts in the Debian Med team - nice that it is now widely used).  Git
mode for uscan helps as well in many cases.

However, we have remaining software in not supported VCS and sometimes
upstream throws files on a ftpserver without creating an archive.  I'm
not happy about the policy change that we are now lacking a clear
guideline how to fetch those sources.  I agree with all the deficits of
get-orig-source described in this thread - but I insist that it was an
advantage to say: If you can not write a proper watch file, provide a
target in d/rules which creates the upstream tarball.  Since this does
not exist any more I'm afraid we will end up with more upstream tarballs
a third person will not have any clue how to fetch the source.  IMHO
that's an unfortunate change in policy.

Kind regards

 Andreas.

-- 
http://fam-tille.de

Re: Debian Policy 4.1.4.0 released

[Russ Allbery]

Jonathan Nieder  writes:

> Context: I have run into a few packages that used the +dfsg convention
> without documenting what they removed from the tarball and I was not
> able to locally update them. :(

This is one of the cases that now has a better solution and more standard
tools than the get-orig-source target, specifically Files-Excluded in
debian/copyright.  See the documentation of Files-Excluded in uscan(1).

-- 
Russ Allbery (r...@debian.org)   

Re: Debian Policy 4.1.4.0 released

[Jonathan Nieder]

Hi,

Sean Whitton wrote:
> On Mon, Jul 02 2018, Jonathan Nieder wrote:

>> I'm a bit confused: wasn't it already specified pretty precisely?
>
> Please take a look through the bug's discussion.  It's explained why the
> wording was not thought to be good enough.

Thanks.  This looks like a classic case of letting the perfect be the
enemy of the good (or perhaps, of not understanding the use case for
which the existing practice was good enough).  Some quotes from the
bug:

| According to codesearch.d.n, get-orig-source is implemented by less than
| 3000 source packages. This is not very low, but neither a high adoption
| rate. It certainly makes using get-orig-source somewhat useless on a
| distribution-scale.

Certainly it's even more useful to have a debian/watch file, but this
doesn't make it clear to me what I'm supposed to do with those 3,000
source packages now.

|  * The requirement that get-orig-source may be invoked from any
|directory is difficult to fulfil and often times not implemented. A

This hasn't been an obstacle to my use.  I can even try multiple
directories.  What's helpful for me is that it works from *somewhere*.

|  * It is not clear whether the most recent upstream version or the
|currently packaged version should be downloaded.

Likewise, either works fine for my use.

|  * It is not clear where required tools should be declared.

As long as the command prints an error about the required tool, I can
install it.

| We're just reducing
| the documented interfaces of packages a bit based on current trends,
| which is useful for newcomers to Debian.

What is a newcomer supposed to do now when they encounter a package
that does not provide an explanation of how to generate the upstream
tarball?

Jonathan

Re: Debian Policy 4.1.4.0 released

[Sean Whitton]

Hello Jonathan,

On Mon, Jul 02 2018, Jonathan Nieder wrote:

> I'm a bit confused: wasn't it already specified pretty precisely?

Please take a look through the bug's discussion.  It's explained why the
wording was not thought to be good enough.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Debian Policy 4.1.4.0 released

[Jonathan Nieder]

Hi,

Sean Whitton wrote:
> On Wed, Apr 11 2018, Russ Allbery wrote:

>> I'm pretty reluctant to specify this sort of optional target that
>> works differently in every package that uses it back in Policy because
>> it's really not standardized, nor do I think it's possible to
>> standardize.  If we really want to write something down about the
>> target, maybe the Developer's Reference would be a better spot?  There
>> were a whole host of issues with having this in Policy that were
>> resolved by moving it outside the scope of Policy, such as how to
>> document dependencies required for running the get-orig-source target.
>
> The Developer's Reference seems like a more appropriate place for a
> convention that it is not possible to specify precisely.

I'm a bit confused: wasn't it already specified pretty precisely?

I would be in favor of adding the target back, with a description
along the lines of "If you provide a get-orig-source target, it should
satisfy .  If you provide neither a get-orig-source
target nor a debian/watch file and you do not use an archive from
upstream as-is, please include clear instructions in README.source to
allow a human to produce an upstream tarball."

Context: I have run into a few packages that used the +dfsg convention
without documenting what they removed from the tarball and I was not
able to locally update them. :(

Thanks,
Jonathan

Re: Debian Policy 4.1.4.0 released

[Ian Jackson]

Paul Wise writes ("Re: Debian Policy 4.1.4.0 released"):
> uscan is used in situations where one does not want arbitrary code
> >from source packages automatically run by uscan. As long as `uscan
> --safe` ignores that fallback, that should be fine I guess though.

I think safety should be the default, so I have filed
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895546

Thanks,
Ian.

Re: Debian Policy 4.1.4.0 released

[Paul Wise]

On Thu, Apr 12, 2018 at 10:27 AM, Russ Allbery wrote:

> Personally, I'd probably add an interactive prompt warning about the
> dangers and stressing that the source package needs to be trusted if stdin
> and stdout are connected to a tty, and otherwise fail and require some
> flag to use the fallback from the source package.  But happy to let
> whoever implements this pick their strategy.  :)

That would break tools that supervise the output of uscan and other
tools, pass fake ttys to the tools and run multiple tools at the same
time. For e.g. `check-all-the-things -j2` would have uscan waiting
forever on that prompt. The existing uscan --safe option should be
enough here though, no need for interactive input.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Debian Policy 4.1.4.0 released

[Russ Allbery]

Paul Wise  writes:
> On Thu, Apr 12, 2018 at 5:02 AM, Russ Allbery wrote:

>> Rather than documenting this fallback in Policy, why not add that
>> fallback directly to uscan?

> uscan is used in situations where one does not want arbitrary code from
> source packages automatically run by uscan. As long as `uscan --safe`
> ignores that fallback, that should be fine I guess though.

Personally, I'd probably add an interactive prompt warning about the
dangers and stressing that the source package needs to be trusted if stdin
and stdout are connected to a tty, and otherwise fail and require some
flag to use the fallback from the source package.  But happy to let
whoever implements this pick their strategy.  :)

-- 
Russ Allbery (r...@debian.org)   

Re: Debian Policy 4.1.4.0 released

[Paul Wise]

On Thu, Apr 12, 2018 at 5:02 AM, Russ Allbery wrote:

> Rather than documenting this fallback in Policy, why not add that fallback
> directly to uscan?

uscan is used in situations where one does not want arbitrary code
from source packages automatically run by uscan. As long as `uscan
--safe` ignores that fallback, that should be fine I guess though.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Debian Policy 4.1.4.0 released

[Russ Allbery]

Andreas Tille  writes:

> That is exactly what I wanted to express.  I do not mind the actual
> implementation but writing down in policy that there should be some
> common interface to obtain the upstream source as a fallback to uscan
> (and only as fallback if there is really no chance to use uscan) seems
> important to me,

Rather than documenting this fallback in Policy, why not add that fallback
directly to uscan?

-- 
Russ Allbery (r...@debian.org)   

Re: Debian Policy 4.1.4.0 released

[Andreas Tille]

On Wed, Apr 11, 2018 at 02:29:25PM +0100, Ian Jackson wrote:
> Andreas Tille writes ("Re: Debian Policy 4.1.4.0 released"):
> > In other words: I'm fine with removing the target in rules and replace
> > it by:
> > 
> > If there are reasons why uscan can not fetch the upstream source it
> > is recommended to provide a script debian/get-orig-source .
> > 
> > If we do not provide *code* to get the upstream source and just settle
> > to some (however vague) description in d/README.source random package
> > developers who might work on a package will end up with different
> > results.  This should not be the outcome of a policy change.
> 
> I'm not sure why you would change from a rules target to a custom
> script.

I personally did it since you can save a lot of syntax things like "; \"
at end of lines etc.  I would not make it a common rule but I became
simply used to it.
 
> There is an argument that rules-as-makefile is not the ideal interface
> and implementation strategy, but I doubt that here and now is the
> right time to be making an accidental transition away from that.

The argument of the bug report was that dh does not know the target.
Since d/rules is usually interpretet by dh I understand the arguing
to not describe something that can not be interpreted by dh.
 
> If the only reason is that the docs for the target are being removed
> from policy, then:
> 
> (i) The fact that a target is no longer documented in policy does not
> mean it should not be provided; the fact that a target is being
> removed from policy does not mean it should be removed from packages.
> It means merely that the target's purpose and inclusion should be
> reconsidered.

Yes.
 
> (ii) You make a very good argument that policy should continue to give
> guidance for this kind of situation.  The target should probably be
> put back in policy, but with an explicit note saying it's not normally
> desirable, or something.

That is exactly what I wanted to express.  I do not mind the actual
implementation but writing down in policy that there should be some
common interface to obtain the upstream source as a fallback to uscan
(and only as fallback if there is really no chance to use uscan) seems
important to me,

A wording that makes bug reports of high severity possible saying:

   Package is using get-orig-source instead of uscan

is fine for me. 

Kind regards

   Andreas.

-- 
http://fam-tille.de

Re: Debian Policy 4.1.4.0 released

[Andrey Rahmatullin]

On Wed, Apr 11, 2018 at 02:29:25PM +0100, Ian Jackson wrote:
> (ii) You make a very good argument that policy should continue to give
> guidance for this kind of situation.  The target should probably be
> put back in policy, but with an explicit note saying it's not normally
> desirable, or something.
That would require deciding what should the target do first.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Debian Policy 4.1.4.0 released

[Ian Jackson]

Andreas Tille writes ("Re: Debian Policy 4.1.4.0 released"):
> In other words: I'm fine with removing the target in rules and replace
> it by:
> 
> If there are reasons why uscan can not fetch the upstream source it
> is recommended to provide a script debian/get-orig-source .
> 
> If we do not provide *code* to get the upstream source and just settle
> to some (however vague) description in d/README.source random package
> developers who might work on a package will end up with different
> results.  This should not be the outcome of a policy change.

I'm not sure why you would change from a rules target to a custom
script.

There is an argument that rules-as-makefile is not the ideal interface
and implementation strategy, but I doubt that here and now is the
right time to be making an accidental transition away from that.

If the only reason is that the docs for the target are being removed
from policy, then:

(i) The fact that a target is no longer documented in policy does not
mean it should not be provided; the fact that a target is being
removed from policy does not mean it should be removed from packages.
It means merely that the target's purpose and inclusion should be
reconsidered.

(ii) You make a very good argument that policy should continue to give
guidance for this kind of situation.  The target should probably be
put back in policy, but with an explicit note saying it's not normally
desirable, or something.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Debian Policy 4.1.4.0 released

[Andreas Tille]

On Sun, Apr 08, 2018 at 10:58:53AM +0200, Ole Streicher wrote:
> >
> > Imho Sean's last mail sums it up pretty well
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=515856#94
> 
> I have read this, but it does not convince me. My rule to get the
> upstream packagage was always: use uscan, if d/watch exists, otherwise
> use get-orig-source. Sounds pretty simple and straigt-forward. If it
> fails, I had a starting point where to debug (usually just a missing
> dep). I see no reaso why this should be given up.

I agree with Ole.  While I took over some ideas from this thread how to
get rid of some get-orig-source targets by using mode=git I think it
makes perfectly sense to define a target name that should be used in
case uscan can not be used exclusively to fetch upstream source.  I can
not see in how far definition of the target name will harm - but from
the point of team maintenance it helps to know where to look first to
download a new upstream version.

BTW, I have standardized all my packages to

get-orig-source:
. debian/get-orig-source

In other words: I'm fine with removing the target in rules and replace
it by:

If there are reasons why uscan can not fetch the upstream source it
is recommended to provide a script debian/get-orig-source .

If we do not provide *code* to get the upstream source and just settle
to some (however vague) description in d/README.source random package
developers who might work on a package will end up with different
results.  This should not be the outcome of a policy change.

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: Debian Policy 4.1.4.0 released

[Russ Allbery]

kact...@gnu.org writes:

> It breaks my workflow ;) I use pristine-tar(1) to store orig tarballs
> with their upstream signature in git.

> dpkg-buildpackage(1) do not extract orig.tar.gz from `pristine-tar'
> automatically, so I add `get-orig-source' rule that invokes
> `pristine-tar(1)' with proper arguments.

Have you considered using origtargz instead?  This is pretty close to the
use case it was designed for, and it's a standard tool that works with a
lot of packages in various situations.

get-orig-source in its initial conception was more intended to get the
tarball for a new release that you would then commit with pristine-tar,
not as a way of getting the source for the current release (although this
turned out to be very ambiguous and people interpreted it different ways,
which was one of the problems with the target).

-- 
Russ Allbery (r...@debian.org)   

Re: Debian Policy 4.1.4.0 released

[Mike Gabriel]

Hi Sean,

On  So 08 Apr 2018 00:36:45 CEST, Sean Whitton wrote:


Hello,

On Sun, Apr 08 2018, kact...@gnu.org wrote:


It breaks my workflow ;) I use pristine-tar(1) to store orig tarballs
with their upstream signature in git.


You can continue to use your target.


this is good news. Thanks for stating this so explicitly.

For dozens of project I use debian/rules get-orig-source as _the_  
universal command to obtain new tarball release, git snapshot orig  
tarballs, etc.


Greets from a relieved...
sunweaver (aka Mike)
--

DAS-NETZWERKTEAM
mike gabriel, herweg 7, 24357 fleckeby
mobile: +49 (1520) 1976 148
landline: +49 (4354) 8390 139

GnuPG Fingerprint: 9BFB AEE8 6C0A A5FF BF22  0782 9AF4 6B30 2577 1B31
mail: mike.gabr...@das-netzwerkteam.de, http://das-netzwerkteam.de



pgpa3HLpu_9f3.pgp
Description: Digitale PGP-Signatur

Re: Debian Policy 4.1.4.0 released

[Ole Streicher]

Andreas Metzler  writes:
> Ole Streicher  wrote:
>> Sean Whitton  writes:
>>> On Sat, Apr 07 2018, Ole Streicher wrote:
> [...]
 Sure, but why do we give up a common rule? I think the cases where
 d/watch does not work are not so rare (at least I have quite a number
 of them), and keeping them unified is not the worst thing we can do.
>
>>> See discussion in #515856.
>
>> Maybe I didn't read it too carefully, but I didn't find the argument why
>> get-orig-source is not kept for the cases where uscan doesn't do the
>> job.
>
>> And when I extrapolate from my packages, this is not an exceptionally
>> rare case.
>
> Imho Sean's last mail sums it up pretty well
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=515856#94

I have read this, but it does not convince me. My rule to get the
upstream packagage was always: use uscan, if d/watch exists, otherwise
use get-orig-source. Sounds pretty simple and straigt-forward. If it
fails, I had a starting point where to debug (usually just a missing
dep). I see no reaso why this should be given up.

Cheers

Ole

Re: Debian Policy 4.1.4.0 released

[Ole Streicher]

Paul Wise  writes:
> On Sat, Apr 7, 2018 at 8:49 PM, Ole Streicher wrote:
>
>> I have a number of "uncommon" upstreams:
>
> It would be really nice if these folks could switch to something more
> standard. Have they considered using a version control system for a
> start?

I asked, but this did not result in a change.

>> * aladin, download http://aladin.unistra.fr/java/download/AladinSrc.jar
>>   look for the VERSION string in cds/aladin/Aladin.java and remove the
>>   leading "v" (and for Pre-releases, download AladinBetaSrc.jar, or a
>>   ad-hoc named jar, like AladinSrcV10Premiere.jar).
>
> The version number can be obtained here:
>
> http://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=downloading
>
> With a pagemangle, uscan could detect the version and associated
> tarball.

Only when the version number on the web page does match to the version
number in the file. Which is often enough not the case.

>> * coyote, http://www.idlcoyote.com/programs/zip_files/coyoteprograms.zip
>>   look for the latest file date in the zip file (no upstream versioning)
>
> There is apparently a github org for this:
>
> https://github.com/idl-coyote
>
> With a pagemangle, you can get the latest date from that:
>
> https://sources.debian.org/src/purple-discord/latest/debian/watch

But it is a difference between a github commit and a released zip
file. Even if it does not contain a version number, it is usually
somehow better curated.

>> * idlastro, https://idlastro.gsfc.nasa.gov/ftp/astron.tar.gz
>>   similar (latest file date; no upstream versioning)
>
> There is apparently a github repo for this, use pagemangle:
>
> https://github.com/wlandsman/IDLAstro

Dito.

>> * mpfit, https://cow.physics.wisc.edu/~craigm/idl/down/mpfit.tar.gz
>>   Take version number from "Revision" in mpfit.pro and add the latest
>>   file date
>
> This contains the mpfit.pro Revision and all dates, use pagemangle:
>
> https://cow.physics.wisc.edu/~craigm/idl/down/cmtotal.txt

This file is three times of the size of than mpfit.tar.gz, containing
a lot more (versionized) files.

> PS: the $Id things are from CVS, could you ask upstream to publicly
> expose their CVS repository?

In that specific case, I didn't. However, also here I would resist in
using the plain RCS version, but rely on upstreams version curation.

>> * skyview, https://skyview.gsfc.nasa.gov/current/jar/skyview.jar
>>   look for "Version=" in skyview.settings
>
> The version number is listed on two pages, use pagemangle:
>
> https://skyview.gsfc.nasa.gov/current/cgi/titlepage.pl
> https://skyview.gsfc.nasa.gov/blog/
>
> Unfortunately both are outdated compared to the jar...

That is the problem here. When the version number is unreliable, it
cannot be used.

>> * starjava-*, download via svn (subdir of
>> https://github.com/Starlink/starjava)
>>   add the main LICENSE.txt file,
>>   get the version from build.xml property, and add latest file date
>>   (but download only tagged versions of starjava-topcat, starjava-ttools,
>>   and starjava-table).
>
> uscan can deal with git repos just fine.

Also with subtrees?

>> The rules may change over time (since I try to convince them to be more
>> friendly here), so unless there is a flexible way for me to change them
>> myself, I doubt it would be a good idea to put it there.
>
> We could add you to the salsa qa group so you can commit them, but
> given the above I think most are best dealt with by uscan.

I would think that I am just a random example, and you probably wouldn't
want to give write access to all that are affected. But could you point
me to the documentation how to write a redirector? I could also do a
Merge Request.

Best regards

Ole

Re: Debian Policy 4.1.4.0 released

[Andreas Metzler]

Ole Streicher  wrote:
> Sean Whitton  writes:
>> On Sat, Apr 07 2018, Ole Streicher wrote:
[...]
>>> Sure, but why do we give up a common rule? I think the cases where
>>> d/watch does not work are not so rare (at least I have quite a number
>>> of them), and keeping them unified is not the worst thing we can do.

>> See discussion in #515856.

> Maybe I didn't read it too carefully, but I didn't find the argument why
> get-orig-source is not kept for the cases where uscan doesn't do the
> job.

> And when I extrapolate from my packages, this is not an exceptionally
> rare case.

Imho Sean's last mail sums it up pretty well
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=515856#94

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Debian Policy 4.1.4.0 released

[Paul Wise]

On Sat, Apr 7, 2018 at 8:49 PM, Ole Streicher wrote:

> I have a number of "uncommon" upstreams:

It would be really nice if these folks could switch to something more
standard. Have they considered using a version control system for a
start?

> * aladin, download http://aladin.unistra.fr/java/download/AladinSrc.jar
>   look for the VERSION string in cds/aladin/Aladin.java and remove the
>   leading "v" (and for Pre-releases, download AladinBetaSrc.jar, or a
>   ad-hoc named jar, like AladinSrcV10Premiere.jar).

The version number can be obtained here:

http://aladin.u-strasbg.fr/java/nph-aladin.pl?frame=downloading

With a pagemangle, uscan could detect the version and associated tarball.

> * coyote, http://www.idlcoyote.com/programs/zip_files/coyoteprograms.zip
>   look for the latest file date in the zip file (no upstream versioning)

There is apparently a github org for this:

https://github.com/idl-coyote

With a pagemangle, you can get the latest date from that:

https://sources.debian.org/src/purple-discord/latest/debian/watch

> * idlastro, https://idlastro.gsfc.nasa.gov/ftp/astron.tar.gz
>   similar (latest file date; no upstream versioning)

There is apparently a github repo for this, use pagemangle:

https://github.com/wlandsman/IDLAstro

> * mpfit, https://cow.physics.wisc.edu/~craigm/idl/down/mpfit.tar.gz
>   Take version number from "Revision" in mpfit.pro and add the latest
>   file date

This contains the mpfit.pro Revision and all dates, use pagemangle:

https://cow.physics.wisc.edu/~craigm/idl/down/cmtotal.txt

PS: the $Id things are from CVS, could you ask upstream to publicly
expose their CVS repository?

> * skyview, https://skyview.gsfc.nasa.gov/current/jar/skyview.jar
>   look for "Version=" in skyview.settings

The version number is listed on two pages, use pagemangle:

https://skyview.gsfc.nasa.gov/current/cgi/titlepage.pl
https://skyview.gsfc.nasa.gov/blog/

Unfortunately both are outdated compared to the jar...

> * starjava-*, download via svn (subdir of 
> https://github.com/Starlink/starjava)
>   add the main LICENSE.txt file,
>   get the version from build.xml property, and add latest file date
>   (but download only tagged versions of starjava-topcat, starjava-ttools,
>   and starjava-table).

uscan can deal with git repos just fine.

> The rules may change over time (since I try to convince them to be more
> friendly here), so unless there is a flexible way for me to change them
> myself, I doubt it would be a good idea to put it there.

We could add you to the salsa qa group so you can commit them, but
given the above I think most are best dealt with by uscan.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Debian Policy 4.1.4.0 released

[Sean Whitton]

Hello,

On Sun, Apr 08 2018, kact...@gnu.org wrote:

> It breaks my workflow ;) I use pristine-tar(1) to store orig tarballs
> with their upstream signature in git.

You can continue to use your target.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Debian Policy 4.1.4.0 released

[KAction]

[2018-04-07 10:35] Ben Finney 
> Sean Whitton  writes:
> 
> > I just pushed Debian Policy 4.1.4.0 to sid. Thank you to the ~20
> > people who contributed to this release, which includes several first
> > time contributors of patches.
> > […]
> >
> > 4.9
> > The ``get-orig-source`` rules target has been removed.  Packages
> > should use ``debian/watch`` and uscan instead.

It breaks my workflow ;) I use pristine-tar(1) to store orig tarballs
with their upstream signature in git.

dpkg-buildpackage(1) do not extract orig.tar.gz from `pristine-tar'
automatically, so I add `get-orig-source' rule that invokes
`pristine-tar(1)' with proper arguments.

I have debian/watch too, but it `uscan(1)` would require network access.

How can I do better with new Policy?

Re: Debian Policy 4.1.4.0 released

[Ole Streicher]

Sean Whitton  writes:
> On Sat, Apr 07 2018, Ole Streicher wrote:
>
>> Adam Borowski  writes:
>>> get-orig-source merely isn't described by the Policy any more, it is
>>> no different from an arbitrary private target you have in
>>> debian/rules.
>>
>> Sure, but why do we give up a common rule? I think the cases where
>> d/watch does not work are not so rare (at least I have quite a number
>> of them), and keeping them unified is not the worst thing we can do.
>
> See discussion in #515856.

Maybe I didn't read it too carefully, but I didn't find the argument why
get-orig-source is not kept for the cases where uscan doesn't do the
job.

And when I extrapolate from my packages, this is not an exceptionally
rare case.

Best regards

Ole

Re: Debian Policy 4.1.4.0 released

[Sean Whitton]

Hello,

On Sat, Apr 07 2018, Ole Streicher wrote:

> Adam Borowski  writes:
>> get-orig-source merely isn't described by the Policy any more, it is
>> no different from an arbitrary private target you have in
>> debian/rules.
>
> Sure, but why do we give up a common rule? I think the cases where
> d/watch does not work are not so rare (at least I have quite a number
> of them), and keeping them unified is not the worst thing we can do.

See discussion in #515856.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Debian Policy 4.1.4.0 released

[Ole Streicher]

Paul Wise  writes:
> On Sat, Apr 7, 2018 at 4:40 PM, Ole Streicher wrote:
>
>> I have some packages where the version is not encoded in the file name,
>> but must be extracted from the file content. Shall one keep
>> get-orig-source here to be consistent, or what would be the right
>> solution here?
>
> If uscan can find the right tarball, it will rename it correctly.
>
> If uscan cannot find a tarball, it will need assistance from a redirector.
>
> The fakeupstream CGI is a good place to add new redirectors.
>
> Please include some details and maybe we can add something.

I have a number of "uncommon" upstreams:

* aladin, download http://aladin.unistra.fr/java/download/AladinSrc.jar
  look for the VERSION string in cds/aladin/Aladin.java and remove the
  leading "v" (and for Pre-releases, download AladinBetaSrc.jar, or a
  ad-hoc named jar, like AladinSrcV10Premiere.jar).

* coyote, http://www.idlcoyote.com/programs/zip_files/coyoteprograms.zip
  look for the latest file date in the zip file (no upstream versioning)

* idlastro, https://idlastro.gsfc.nasa.gov/ftp/astron.tar.gz
  similar (latest file date; no upstream versioning)

* mpfit, https://cow.physics.wisc.edu/~craigm/idl/down/mpfit.tar.gz
  Take version number from "Revision" in mpfit.pro and add the latest
  file date

* skyview, https://skyview.gsfc.nasa.gov/current/jar/skyview.jar
  look for "Version=" in skyview.settings

* starjava-*, download via svn (subdir of https://github.com/Starlink/starjava)
  add the main LICENSE.txt file,
  get the version from build.xml property, and add latest file date
  (but download only tagged versions of starjava-topcat, starjava-ttools,
  and starjava-table).

The rules may change over time (since I try to convince them to be more
friendly here), so unless there is a flexible way for me to change them
myself, I doubt it would be a good idea to put it there.

But feel free to do so :-)

Best

Ole

Re: Debian Policy 4.1.4.0 released

[Ole Streicher]

Adam Borowski  writes:
> On Sat, Apr 07, 2018 at 10:40:42AM +0200, Ole Streicher wrote:
>> Ben Finney  writes:
>> > Sean Whitton  writes:
>> >> 4.9
>> >> The ``get-orig-source`` rules target has been removed.  Packages
>> >> should use ``debian/watch`` and uscan instead.
>> >
>> > Especially for this, my ‘debian/rules’ files thank you.
>> 
>> I have some packages where the version is not encoded in the file name,
>> but must be extracted from the file content. Shall one keep
>> get-orig-source here to be consistent, or what would be the right
>> solution here?
>
> get-orig-source merely isn't described by the Policy any more, it is no
> different from an arbitrary private target you have in debian/rules.

Sure, but why do we give up a common rule? I think the cases where
d/watch does not work are not so rare (at least I have quite a number of
them), and keeping them unified is not the worst thing we can do.

Best

Ole

Re: Debian Policy 4.1.4.0 released

[Paul Wise]

On Sat, Apr 7, 2018 at 4:40 PM, Ole Streicher wrote:

> I have some packages where the version is not encoded in the file name,
> but must be extracted from the file content. Shall one keep
> get-orig-source here to be consistent, or what would be the right
> solution here?

If uscan can find the right tarball, it will rename it correctly.

If uscan cannot find a tarball, it will need assistance from a redirector.

The fakeupstream CGI is a good place to add new redirectors.

Please include some details and maybe we can add something.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Debian Policy 4.1.4.0 released

[Adam Borowski]

On Sat, Apr 07, 2018 at 10:40:42AM +0200, Ole Streicher wrote:
> Ben Finney  writes:
> > Sean Whitton  writes:
> >> 4.9
> >> The ``get-orig-source`` rules target has been removed.  Packages
> >> should use ``debian/watch`` and uscan instead.
> >
> > Especially for this, my ‘debian/rules’ files thank you.
> 
> I have some packages where the version is not encoded in the file name,
> but must be extracted from the file content. Shall one keep
> get-orig-source here to be consistent, or what would be the right
> solution here?

get-orig-source merely isn't described by the Policy any more, it is no
different from an arbitrary private target you have in debian/rules.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢰⠒⠀⣿⡁ 
⢿⡄⠘⠷⠚⠋⠀ ... what's the frequency of that 5V DC?
⠈⠳⣄

Re: Debian Policy 4.1.4.0 released

[Ole Streicher]

Ben Finney  writes:
> Sean Whitton  writes:
>> I just pushed Debian Policy 4.1.4.0 to sid. Thank you to the ~20
>> people who contributed to this release, which includes several first
>> time contributors of patches.
>> […]
>>
>> 4.9
>> The ``get-orig-source`` rules target has been removed.  Packages
>> should use ``debian/watch`` and uscan instead.
>
> Especially for this, my ‘debian/rules’ files thank you.

I have some packages where the version is not encoded in the file name,
but must be extracted from the file content. Shall one keep
get-orig-source here to be consistent, or what would be the right
solution here?

Best regards

Ole

Re: Debian Policy 4.1.4.0 released

[Sébastien Villemot]

On Sat, Apr 07, 2018 at 08:02:11AM +0200, Andreas Tille wrote:

> On Sat, Apr 07, 2018 at 10:35:02AM +1000, Ben Finney wrote:
> > Sean Whitton  writes:
> > >
> > > 4.9
> > > The ``get-orig-source`` rules target has been removed.  Packages
> > > should use ``debian/watch`` and uscan instead.
> > 
> > Especially for this, my ‘debian/rules’ files thank you.
> 
> While I really like to have this consistent approach but it seems I've
> missed how uscan can spot new versions in for instance untagged VCS or
> download files with changing content but no version number.  Is there
> some way to do this with something else than a manually craftet script?

Since devscripts 2.18.1, uscan is able to track the HEAD of a git a repository.

See the section titled "direct access to the git repository (HEAD)" in
uscan(1) for an example. It is possible to choose how to customize how the
upstream version string is created using the "pretty" option.

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  http://sebastien.villemot.name
⠈⠳⣄  http://www.debian.org


signature.asc
Description: PGP signature

Re: Debian Policy 4.1.4.0 released

[Stuart Prescott]

Hi Andreas

>> > 4.9
>> > The ``get-orig-source`` rules target has been removed.  Packages
>> > should use ``debian/watch`` and uscan instead.
>> 
>> Especially for this, my ‘debian/rules’ files thank you.
> 
> While I really like to have this consistent approach but it seems I've
> missed how uscan can spot new versions in for instance untagged VCS or
> download files with changing content but no version number.  Is there
> some way to do this with something else than a manually craftet script?

yes, d/watch can use the qa.debian.org fakeupstream service to create a fake 
new release for every 
commit. I use this on projects that have very occasional (bugfix-only) commits 
and don't seem to be 
interested in actually making releases any more:

https://sources.debian.org/src/svn-all-fast-export/1.0.10+git20160822-3/debian/watch/


opts="uversionmangle=s/.*date=(\d{4})-(\d\d)-(\d\d)T.*/1.0.10+git$1$2$3/, \
filenamemangle=s/.*date=(\d{4})-(\d\d)-(\d\d)T.*/1.0.10+git$1$2$3.tar.gz/" \

https://qa.debian.org/cgi-bin/fakeupstream.cgi?upstream=github_commits_package_json/svn-all-fast-export/svn2git
 \
.*/archive/(.*\.tar\.gz?.*)

A version 1.0.10+git20180406 would therefore appear from a commit made 
yesterday and if I were to package 
and upload that version, that would also be the upstream part of the version 
string I'd use. With uscan 
integration, tools like the UDD Maintainer Dashboard also show when new commits 
are made.

(Thanks to Paul Wise for creating this a couple of years ago when I was musing 
on how to track this sort 
of upstream)

cheers
Stuart

-- 
Stuart Prescotthttp://www.nanonanonano.net/   stu...@nanonanonano.net
Debian Developer   http://www.debian.org/ stu...@debian.org
GPG fingerprint90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

Re: Debian Policy 4.1.4.0 released

[Andreas Tille]

Hi,

On Sat, Apr 07, 2018 at 10:35:02AM +1000, Ben Finney wrote:
> Sean Whitton  writes:
> >
> > 4.9
> > The ``get-orig-source`` rules target has been removed.  Packages
> > should use ``debian/watch`` and uscan instead.
> 
> Especially for this, my ‘debian/rules’ files thank you.

While I really like to have this consistent approach but it seems I've
missed how uscan can spot new versions in for instance untagged VCS or
download files with changing content but no version number.  Is there
some way to do this with something else than a manually craftet script?

Kind regards

 Andreas.

-- 
http://fam-tille.de

Re: Debian Policy 4.1.4.0 released

[Ben Finney]

Sean Whitton  writes:

> I just pushed Debian Policy 4.1.4.0 to sid. Thank you to the ~20
> people who contributed to this release, which includes several first
> time contributors of patches.
> […]
>
> 4.9
> The ``get-orig-source`` rules target has been removed.  Packages
> should use ``debian/watch`` and uscan instead.

Especially for this, my ‘debian/rules’ files thank you.

-- 
 \“Institutions will try to preserve the problem to which they |
  `\ are the solution.” —Clay Shirky, 2012 |
_o__)  |
Ben Finney