Re: Enabling uscan to simply remove files from upstream source

[Jon Dowland]

On Sat, Aug 25, 2012 at 05:18:57PM +0900, Charles Plessy wrote:
> I also considered, but the information of what files are excluded when 
> creating
> the Debian source package is not relevant to upstream.
> 
> More simply, I think that if debian/watch would radically change its format to
> YAML or the Debian control data file syntax, it would be a good place for the
> Files-Excluded field.

The copyright file is, as the name suggests, for copyright information.

The watch file is, as the name suggests, for watching for upstream releases.

Can we please just have a new file for a new purpose, rather than overloading
ones that are succinct, single-purpose and consistent?

We can then pick a syntax that fits the problem, rathen than trying to bend and
contort one that doesn't.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120907125431.GC4705@debian

Re: Enabling uscan to simply remove files from upstream source

[Lars Wirzenius]

On Sat, Aug 25, 2012 at 12:34:28PM +0200, Jonas Smedegaard wrote:
> Hi Charles,
> 
> On 12-08-25 at 01:03pm, Charles Plessy wrote:
> > a paragraph must not contain multiple instances of the same field.
> 
> Interesting.  Where is that defined?

Policy, 5.1, "Syntax of control files"
(http://www.debian.org/doc/debian-policy/ch-controlfields#s-controlsyntax),
fourth paragraph, in its entirety:

A paragraph must not contain more than one instance of a particular
field name.

copyright-format/1.0 refers to Policy section 5.1
in the "File syntax" section (second paragraph,
http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/#file-syntax:

The syntax of the file is the same as for other Debian control
files, as specified in the Debian Policy Manual. See its section
5.1 for details.

Happy hacking.

-- 
I wrote a book on personal productivity: http://gtdfh.branchable.com/


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Jonas Smedegaard]

Hi Charles,

On 12-08-25 at 01:03pm, Charles Plessy wrote:
> a paragraph must not contain multiple instances of the same field.

Interesting.  Where is that defined?


 - Jonas

P.S.

I realize now that in this conversations I've wrongly used "section" 
when referring to "paragraph", and "paragraph" when referring to 
"field".  Sorry for the confusion :-(

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Jonas Smedegaard]

On 12-08-24 at 01:06pm, Stefano Zacchiroli wrote:
> On Fri, Aug 24, 2012 at 12:33:09PM +0200, Jonas Smedegaard wrote:
> > > Files-Excluded:
> > > # ignore copy of lua
> > > lua_5.1.4/
> […]
> > Above means inventing a *new* syntax for files, instead of reusing 
> > the existing one from Files: paragraphs.
> 
> debian/control, which is 822-like, already supports #-comments. So, 
> strictly speaking, we will just reusing a (meta-)syntax that already 
> exists elsewhere among debian/* files.
> 
> Note, however, that #-comments in debian/control must *start* the 
> line, so the above example wouldn't work.

Support for #-comments is 822-like files is specific to debian/control, 
and Jakub Wilk pointed out in <20120823192353.ga5...@jwilk.net>.

Debian Policy, §5.1:
> Lines starting with # without any preceding whitespace are comments 
> lines that are only permitted in source package control files 
> (`debian/control').


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Charles Plessy]

Le Sat, Aug 25, 2012 at 07:24:06AM +0200, Andreas Tille a écrit :
> On Sat, Aug 25, 2012 at 01:03:03PM +0900, Charles Plessy wrote:
> 
> > I also think that the current proposal would be a good opportunity to 
> > transfer
> > the information about source location and excluded files in a separate file
> > that would be parsed by uscan and others, and which format would be easier 
> > than
> > debian/watch.
> 
> I guess you mean your DEP12 proposal, right?[1]

Hi Andreas,

I also considered, but the information of what files are excluded when creating
the Debian source package is not relevant to upstream.

More simply, I think that if debian/watch would radically change its format to
YAML or the Debian control data file syntax, it would be a good place for the
Files-Excluded field.  And it already contains the information about where the
upstream source is distributed, so for most packages, it would only take one
patch to Debhelper, one to the PTS and one to packages.debian.org to get the
information distribued to our users.

Cheers,

-- 
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120825081857.gb3...@falafel.plessy.net

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

On Sat, Aug 25, 2012 at 01:03:03PM +0900, Charles Plessy wrote:
> > So would be nice to check that the implementation properly includes all 
> > of the following items:
> > 
> > Format: 
> >  http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
> > Source: http://susy.oddbird.net/
> >   Repackaged, excluding non-DFSG licensed fonts and source-less
> >   JavaScript
> > Files-Excluded:
> >   docs/source/javascripts/jquery-1.7.1.min.js
> >   docs/source/javascripts/modernizr-2.5.3.min.js
> > Files-Excluded-comment: Exlude source-less JavaScript
> > Files-Excluded: foo/bar
> >   docs/source/fonts/*
> >   baz
> >   boom/boom/
> > Files-Excluded-comment: Exlude non-DFSG licensed fonts and more
> >  Just for demonstration purpose, this paragraph has multiple
> >  lines.
> 
> Hi all,
> 
> a paragraph must not contain multiple instances of the same field.

I can confirm that I tried this yesterday and it really fails (just
forgot to report this here).

> Perhaps
> the example above suggests that the Files-Excluded field is not well suited
> for the header paragraph ?

Hmmm, even if the suggestion for a commenting feature for single entries
is a bit spoiled technically - logically I would expect this information
somehow in the header.

> I also think that the current proposal would be a good opportunity to transfer
> the information about source location and excluded files in a separate file
> that would be parsed by uscan and others, and which format would be easier 
> than
> debian/watch.

I guess you mean your DEP12 proposal, right?[1]

I admit this alternative came to my mind yesterday once Ian had trouble
with debian/copyright but later he has withdrawn his objection and I had
not seen any need to bring in this idea any more.

>From my perspective the debian/copyright file is the ideal file because:

  1. Content wise the information is license / DFSG related and it
 is a close to perfect way to document the removal here inside
 the copyright file.
  2. Technically it is perfectly possible (see #685787).
  3. The unability to comment every single exclusion in the current
 form is not as important enough (to me) to seek actively for
 a different solution (even if I made some similar comments in
 some get-orig-source scripts)

Kind regards

Andreas.

[1] http://lists.debian.org/debian-project/2012/06/msg00164.html 

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120825052406.gc21...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Charles Plessy]

> On 12-08-22 at 09:39am, Paul Wise wrote:
> > 
> > In comparison to the current method for repacking (debian/rules
> > get-orig-source), this doesn't allow per-file-set comments about why
> > the file-set is being removed. I often use this to document in more
> > detail why I am removing files. So I view this spec as a downgrade
> > from what we have now.
> 

Le Wed, Aug 22, 2012 at 10:09:38AM +0200, Jonas Smedegaard a écrit :
>
> So would be nice to check that the implementation properly includes all 
> of the following items:
> 
> Format: 
>  http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
> Source: http://susy.oddbird.net/
>   Repackaged, excluding non-DFSG licensed fonts and source-less
>   JavaScript
> Files-Excluded:
>   docs/source/javascripts/jquery-1.7.1.min.js
>   docs/source/javascripts/modernizr-2.5.3.min.js
> Files-Excluded-comment: Exlude source-less JavaScript
> Files-Excluded: foo/bar
>   docs/source/fonts/*
>   baz
>   boom/boom/
> Files-Excluded-comment: Exlude non-DFSG licensed fonts and more
>  Just for demonstration purpose, this paragraph has multiple
>  lines.

Hi all,

a paragraph must not contain multiple instances of the same field.  Perhaps
the example above suggests that the Files-Excluded field is not well suited
for the header paragraph ?  In that case, even if the current specification
does not allow it, maybe you can explore the possibility to make 
Files-Excluded paragraphs, like in the following:

Files-Excluded:
  docs/source/javascripts/jquery-1.7.1.min.js
  docs/source/javascripts/modernizr-2.5.3.min.js
Comment: Exlude source-less JavaScript

Files-Excluded: foo/bar
  docs/source/fonts/*
  baz
  boom/boom/
Comment: Exlude non-DFSG licensed fonts and more
 Just for demonstration purpose, this paragraph has multiple
 lines.

This will only cause problems if existing parsers are detecting paragraph type
by exclusion, like for instance inferring that a paragraph is a stand-alone
License paragraph because 1) it is not a Header paragraph and 2) it does not
contain a Files field.

I also think that the current proposal would be a good opportunity to transfer
the information about source location and excluded files in a separate file
that would be parsed by uscan and others, and which format would be easier than
debian/watch.

However, for the moment packages.debian.org or the PTS do not display arbitrary
information from files in the source packages, so the easiest way to present
the information to our users is through the Debian copyright file.  Altogether,
no solution is entirely satisfactory.  Things may become easier once we have a
solid distribution of all our package metadata, both on-line and as an off-line
copy for the roaming or disconnected systems.

Have a nice day,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120825040303.ge2...@falafel.plessy.net

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

On Fri, Aug 24, 2012 at 01:06:14PM +0200, Stefano Zacchiroli wrote:
> > Above means inventing a *new* syntax for files, instead of reusing the 
> > existing one from Files: paragraphs.
> 
> debian/control, which is 822-like, already supports #-comments. So,
> strictly speaking, we will just reusing a (meta-)syntax that already
> exists elsewhere among debian/* files.
> 
> Note, however, that #-comments in debian/control must *start* the line,
> so the above example wouldn't work.

I'm perfectly fine with just not inventing the new syntax because this
would cause myself more work for something I personally would not use
anyway.  The '#' inside the paragraph just came to my mind - so if
somebody really want this he is free to discuss + implement it but I'll
leave it untouched for now.

Kind regards

   Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120824114324.gi10...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Stefano Zacchiroli]

On Fri, Aug 24, 2012 at 12:33:09PM +0200, Jonas Smedegaard wrote:
> > Files-Excluded:
> > # ignore copy of lua
> > lua_5.1.4/
[…]
> Above means inventing a *new* syntax for files, instead of reusing the 
> existing one from Files: paragraphs.

debian/control, which is 822-like, already supports #-comments. So,
strictly speaking, we will just reusing a (meta-)syntax that already
exists elsewhere among debian/* files.

Note, however, that #-comments in debian/control must *start* the line,
so the above example wouldn't work.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Jonas Smedegaard]

On 12-08-24 at 11:16am, Andreas Tille wrote:
> Hi Paul,
> 
> On Wed, Aug 22, 2012 at 09:39:00AM +0800, Paul Wise wrote:
> > On Tue, Aug 21, 2012 at 6:21 PM, Andreas Tille wrote:
> > 
> > > Any further hints / remarks?
> > 
> > In comparison to the current method for repacking (debian/rules
> > get-orig-source), this doesn't allow per-file-set comments about why
> > the file-set is being removed. I often use this to document in more
> > detail why I am removing files. So I view this spec as a downgrade
> > from what we have now.
> 
> Besides the suggestion of Jonas which I like in principle and would be
> the easiest way to implemet it came to my mind that we could be free to
> ignore everything past '#' to make these strings valid:
> 
> Files-Excluded:
> # ignore copy of lua
> lua_5.1.4/
> sqlite3.* # we use Debian's own sqlite
> #  The dada files are that long that there is no point in leaving
> #  them so the complete directory is removed
> data/samples/
> *.pdf # We do not have the source of the pdfs
> 
> I have some preference for Jonas' suggestion using
> Files-Excluded-Comment because it looks cleaner - but both should
> solve your problem.

Above means inventing a *new* syntax for files, instead of reusing the 
existing one from Files: paragraphs.

If really sensible to use hash for comments, then let's relax the 
general format, instead of inventing new formats for each use case!

Personally I am fine using 822-style paragraphs for comments in an 
822-style file.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Jonas Smedegaard]

On 12-08-24 at 11:28am, Andreas Tille wrote:
> On Fri, Aug 24, 2012 at 10:59:10AM +0200, Jonas Smedegaard wrote:
> > > Anyway, I thought I wanted a separate file, but then I remembered that 
> > > uscan already uses 'debian/watch' for configuration.  The syntax of a 
> > > watch file is pretty awkward, being based on (logical) lines rather 
> > > than stanzas, and using "opts=foo=1,bar=2" instead of something like 
> > > "foo=1 bar=2", but it does seem like the right place to put additional 
> > > uscan configuration.  And the watch file format can presumably be 
> > > fixed, as it is explicitly versioned.
> > 
> > Sounds sensible.
> 
> Uhmm, that's the first case in my memory that you drifted away from one
> of your reasonable proposals. ;-)

Sorry if I was too brief: sensible, even if I do not agree.

I find the argument sensible that not all are comfortable with using 
machine-readable copyright file for expressing information to be 
readable by machines.

I am still a big fan of DEP-5.  I won't use an eventual alternative 
uscan syntax.

...or, since it was later pointed out that "uscan" is narrowly for 
scanning for upstream source, not repackaging it, such alternative newly 
invented syntax could instead be stuffed into a brand new 
debian/exclude-from-uscan or whatever.



> > How about, in addition to looking for excludes in DEP-5 copyright files, 
> > have uscan also support an exclude option, so that e.g. 
> > compass-susy-plugin could alternatively have this debian/watch file:
> > 
> > version=4
> > opts=dversionmangle=s/\~dfsg.*// \
> > exclude="docs/source/fonts/* \ 
> > docs/source/javascripts/jquery-1.7.1.min.js \ 
> > docs/source/javascripts/modernizr-2.5.3.min.js" \
> > https://github.com/ericam/susy/tags .*/tarball/v?(\d[\d\.]+)
> 
> While I suggested in my initial proposal a separate file (not actually 
> debian/watch but this solution seems to be comparable to a separate 
> file) I think we are lacking the feature of documentation what was 
> removed and what not and from my (and Jonas' previous point of view 
> debian/copyright is a very good place to do this).

Yes, I am perfectly aware of that.  I was simply trying to play along 
with the "DEP-5 is wrong, uscan is adequate" mindset.

I'll leave it to fans of that mindset to drive it further...


> > > Hence if you're going to automate repacking, I just wanted to 
> > > suggest generating a README file to put into the repacked tarball.  
> > > And as I said, I haven't heard of anyone else doing this, so 
> > > perhaps I'm the only one who thinks it makes sense.
> > 
> > Sorry, I missed the detail of it being in the tarball.  I find 
> > adequate the common practice of renaming the tarball (and then 
> > documenting the why it was done externally from the tarball), so 
> > let's just agree to disagree on this one :-)
> 
> In my practice (several years ago) I used the *.orig.tar.gz files from 
> a Debian mirror to build my own system inside $HOME on a HP-UX 
> machine. The advantage of using the Debian mirror was that you find 
> everything you need in one place.  For this purpose it would have been 
> helpful to have a short notice what was changed right inside the 
> changed tarball. I admit that is a rare use case these days - but it 
> might be nice anyway.

I find that it is an argument of putting the notice inside the box 
instead of aside it (in the debian tarball/diff).  Good luck driving 
that further.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

On Fri, Aug 24, 2012 at 10:59:10AM +0200, Jonas Smedegaard wrote:
> > Anyway, I thought I wanted a separate file, but then I remembered that 
> > uscan already uses 'debian/watch' for configuration.  The syntax of a 
> > watch file is pretty awkward, being based on (logical) lines rather 
> > than stanzas, and using "opts=foo=1,bar=2" instead of something like 
> > "foo=1 bar=2", but it does seem like the right place to put additional 
> > uscan configuration.  And the watch file format can presumably be 
> > fixed, as it is explicitly versioned.
> 
> Sounds sensible.

Uhmm, that's the first case in my memory that you drifted away from one
of your reasonable proposals. ;-)
 
> How about, in addition to looking for excludes in DEP-5 copyright files, 
> have uscan also support an exclude option, so that e.g. 
> compass-susy-plugin could alternatively have this debian/watch file:
> 
> version=4
> opts=dversionmangle=s/\~dfsg.*// \
> exclude="docs/source/fonts/* \ 
> docs/source/javascripts/jquery-1.7.1.min.js \ 
> docs/source/javascripts/modernizr-2.5.3.min.js" \
> https://github.com/ericam/susy/tags .*/tarball/v?(\d[\d\.]+)

While I suggested in my initial proposal a separate file (not actually
debian/watch but this solution seems to be comparable to a separate
file) I think we are lacking the feature of documentation what was
removed and what not and from my (and Jonas' previous point of view
debian/copyright is a very good place to do this).
 
> > Hence if you're going to automate repacking, I just wanted to suggest 
> > generating a README file to put into the repacked tarball.  And as I 
> > said, I haven't heard of anyone else doing this, so perhaps I'm the 
> > only one who thinks it makes sense.
> 
> Sorry, I missed the detail of it being in the tarball.  I find adequate 
> the common practice of renaming the tarball (and then documenting the 
> why it was done externally from the tarball), so let's just agree to 
> disagree on this one :-)

In my practice (several years ago) I used the *.orig.tar.gz files from a
Debian mirror to build my own system inside $HOME on a HP-UX machine.
The advantage of using the Debian mirror was that you find everything
you need in one place.  For this purpose it would have been helpful to
have a short notice what was changed right inside the changed tarball.
I admit that is a rare use case these days - but it might be nice anyway.

Kind regards

  Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120824092810.gg10...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

On Thu, Aug 23, 2012 at 05:15:35PM -0500, Peter Samuelson wrote:
> Well, two reasons not to bundle it into DEP-5 format files.  First,
> there may be a lot of people like me who would find value in a
> config-file-driven 'get-orig-source' but who do not find any value in
> maintaining debian/copyright in DEP-5 format.

Cool reason to make people switch to DEP5. ;-)

> > > Unrelated: when I've repacked tarballs, I add a file 
> > > "README.Debian-tarball" to the top level source directory, explaining 
> > > what I did.  Nobody ever suggested this to me, it just seems like 
> > > common sense that information about the new tarball should be, well, 
> > > in the new tarball.  Not just in the .diff.gz.
> > 
> > Nowadays such info is commonly put into README.source
> 
> I know, but that's typically not in the orig.tar.gz.  If someone grabs
> foo_1.0+dfsg.orig.tar.gz by itself, I think it is useful to have a
> README in there that explains how it is different from an upstream
> foo-1.0.tar.gz.

I admit that I see some point in documenting the removal of files
*inside* the orig.tar.gz.
 
> generating a README file to put into the repacked tarball.  And as I
> said, I haven't heard of anyone else doing this, so perhaps I'm the
> only one who thinks it makes sense.

+1

Kind regards

   Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120824092021.gf10...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

Hi Paul,

On Wed, Aug 22, 2012 at 09:39:00AM +0800, Paul Wise wrote:
> On Tue, Aug 21, 2012 at 6:21 PM, Andreas Tille wrote:
> 
> > Any further hints / remarks?
> 
> In comparison to the current method for repacking (debian/rules
> get-orig-source), this doesn't allow per-file-set comments about why
> the file-set is being removed. I often use this to document in more
> detail why I am removing files. So I view this spec as a downgrade
> from what we have now.

Besides the suggestion of Jonas which I like in principle and would be
the easiest way to implemet it came to my mind that we could be free to
ignore everything past '#' to make these strings valid:

Files-Excluded:
# ignore copy of lua
lua_5.1.4/
sqlite3.* # we use Debian's own sqlite
#  The dada files are that long that there is no point in leaving
#  them so the complete directory is removed
data/samples/
*.pdf # We do not have the source of the pdfs

I have some preference for Jonas' suggestion using
Files-Excluded-Comment because it looks cleaner - but both should
solve your problem.

Kind regards

  Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120824091622.ge10...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Jonas Smedegaard]

On 12-08-23 at 05:15pm, Peter Samuelson wrote:
> 
> > > Automating get-orig-source is a fine idea, but tying it to DEP-5 
> > > would be unfortunate.
> 
> [Jonas Smedegaard]
> > You mean that you prefer a separate file for this info?
> > 
> > What should be its name? What should be its syntax?
> > 
> > ...and why start from scratch with this - or does something else 
> > already exist, comparable to the work of DEP-5?
> 
> Well, two reasons not to bundle it into DEP-5 format files.  First, 
> there may be a lot of people like me who would find value in a 
> config-file-driven 'get-orig-source' but who do not find any value in 
> maintaining debian/copyright in DEP-5 format.  Tying the two together 
> basically means I probably won't use it, as managing my own .orig 
> generation seems easier than having to maintain a DEP-5 file.  By 
> making me choose to migrate to DEP-5 in order to get the uscan 
> feature, you're making the feature less useful.
> 
> Second, debian/copyright isn't a config file.  I'd rather see 
> configuration in a config file.  Perhaps the DEP-5 mindset is such 
> that you do see debian/copyright as a config file now, but I think its 
> purpose has always been documentation, not configuration.  I guess you 
> can tell I never really cared for "literate programming"
> 
> Anyway, I thought I wanted a separate file, but then I remembered that 
> uscan already uses 'debian/watch' for configuration.  The syntax of a 
> watch file is pretty awkward, being based on (logical) lines rather 
> than stanzas, and using "opts=foo=1,bar=2" instead of something like 
> "foo=1 bar=2", but it does seem like the right place to put additional 
> uscan configuration.  And the watch file format can presumably be 
> fixed, as it is explicitly versioned.

Sounds sensible.

How about, in addition to looking for excludes in DEP-5 copyright files, 
have uscan also support an exclude option, so that e.g. 
compass-susy-plugin could alternatively have this debian/watch file:

version=4
opts=dversionmangle=s/\~dfsg.*// \
exclude="docs/source/fonts/* \ 
docs/source/javascripts/jquery-1.7.1.min.js \ 
docs/source/javascripts/modernizr-2.5.3.min.js" \
https://github.com/ericam/susy/tags .*/tarball/v?(\d[\d\.]+)



> > > Unrelated: when I've repacked tarballs, I add a file 
> > > "README.Debian-tarball" to the top level source directory, 
> > > explaining what I did.  Nobody ever suggested this to me, it just 
> > > seems like common sense that information about the new tarball 
> > > should be, well, in the new tarball.  Not just in the .diff.gz.
> > 
> > Nowadays such info is commonly put into README.source
> 
> I know, but that's typically not in the orig.tar.gz.  If someone grabs 
> foo_1.0+dfsg.orig.tar.gz by itself, I think it is useful to have a 
> README in there that explains how it is different from an upstream 
> foo-1.0.tar.gz.
> 
> Hence if you're going to automate repacking, I just wanted to suggest 
> generating a README file to put into the repacked tarball.  And as I 
> said, I haven't heard of anyone else doing this, so perhaps I'm the 
> only one who thinks it makes sense.

Sorry, I missed the detail of it being in the tarball.  I find adequate 
the common practice of renaming the tarball (and then documenting the 
why it was done externally from the tarball), so let's just agree to 
disagree on this one :-)


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Jon Dowland]

On Thu, Aug 23, 2012 at 12:38:14PM -0500, Peter Samuelson wrote:
> ...Or add the exclude list to a file called 'debian/watch'.

I struggle to see why. I suppose because uscan reads debian/watch, but
the point of that file is to document where to find upstream sources,
the name implies that, and putting lists of files to strip into it would
seem counter-intuitive to me.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120824083627.GA19780@debian

Re: Enabling uscan to simply remove files from upstream source

[Peter Samuelson]

> > Automating get-orig-source is a fine idea, but tying it to DEP-5
> > would be unfortunate.

[Jonas Smedegaard]
> You mean that you prefer a separate file for this info?
> 
> What should be its name? What should be its syntax?
> 
> ...and why start from scratch with this - or does something else already 
> exist, comparable to the work of DEP-5?

Well, two reasons not to bundle it into DEP-5 format files.  First,
there may be a lot of people like me who would find value in a
config-file-driven 'get-orig-source' but who do not find any value in
maintaining debian/copyright in DEP-5 format.  Tying the two together
basically means I probably won't use it, as managing my own .orig
generation seems easier than having to maintain a DEP-5 file.  By
making me choose to migrate to DEP-5 in order to get the uscan feature,
you're making the feature less useful.

Second, debian/copyright isn't a config file.  I'd rather see
configuration in a config file.  Perhaps the DEP-5 mindset is such that
you do see debian/copyright as a config file now, but I think its
purpose has always been documentation, not configuration.  I guess you
can tell I never really cared for "literate programming"

Anyway, I thought I wanted a separate file, but then I remembered that
uscan already uses 'debian/watch' for configuration.  The syntax of a
watch file is pretty awkward, being based on (logical) lines rather
than stanzas, and using "opts=foo=1,bar=2" instead of something like
"foo=1 bar=2", but it does seem like the right place to put additional
uscan configuration.  And the watch file format can presumably be
fixed, as it is explicitly versioned.


> > Unrelated: when I've repacked tarballs, I add a file 
> > "README.Debian-tarball" to the top level source directory, explaining 
> > what I did.  Nobody ever suggested this to me, it just seems like 
> > common sense that information about the new tarball should be, well, 
> > in the new tarball.  Not just in the .diff.gz.
> 
> Nowadays such info is commonly put into README.source

I know, but that's typically not in the orig.tar.gz.  If someone grabs
foo_1.0+dfsg.orig.tar.gz by itself, I think it is useful to have a
README in there that explains how it is different from an upstream
foo-1.0.tar.gz.

Hence if you're going to automate repacking, I just wanted to suggest
generating a README file to put into the repacked tarball.  And as I
said, I haven't heard of anyone else doing this, so perhaps I'm the
only one who thinks it makes sense.

Peter


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120823221535.gc5...@p12n.org

Re: Enabling uscan to simply remove files from upstream source

[Jakub Wilk]

* Jonas Smedegaard , 2012-08-23, 19:47:

This is valid DEP-5 syntax as well:

Format:
http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Source: http://susy.oddbird.net/
 Repackaged, excluding non-DFSG licensed fonts and source-less
 JavaScript
# Exlude source-less JavaScript
Files-Excluded:
 docs/source/javascripts/jquery-1.7.1.min.js
 docs/source/javascripts/modernizr-2.5.3.min.js


No, it's not valid. # comments are not allowed in control files other 
than debian/control. See last but one paragraph of Policy §5.1.


Please note that adding a machine-readable header section does *not* 
force you to rewrite the whole copyright file: You are free to add 
non-machine-readable chunks below the machine-readable parts.


If you don't mind lintian warnings…

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120823192353.ga5...@jwilk.net

Re: Enabling uscan to simply remove files from upstream source

[Jonas Smedegaard]

On 12-08-23 at 12:27pm, Peter Samuelson wrote:
> 
> [Jonas Smedegaard]
> > Format: 
> >  http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
> > Source: http://susy.oddbird.net/
> >   Repackaged, excluding non-DFSG licensed fonts and source-less
> >   JavaScript
> > Files-Excluded:
> >   docs/source/javascripts/jquery-1.7.1.min.js
> >   docs/source/javascripts/modernizr-2.5.3.min.js
> > Files-Excluded-comment: Exlude source-less JavaScript
> 
> A file format in which a comment starts with "Files-Excluded-comment:"
> instead of, say, "#", is a file format I just can't get excited about.

This is valid DEP-5 syntax as well:

Format: 
 http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Source: http://susy.oddbird.net/
  Repackaged, excluding non-DFSG licensed fonts and source-less
  JavaScript
# Exlude source-less JavaScript
Files-Excluded:
  docs/source/javascripts/jquery-1.7.1.min.js
  docs/source/javascripts/modernizr-2.5.3.min.js

Does that please you?


> Automating get-orig-source is a fine idea, but tying it to DEP-5 would 
> be unfortunate.

Why?  Because it is better to have multiple independent machine-readable 
formats?

Please note that adding a machine-readable header section does *not* 
force you to rewrite the whole copyright file: You are free to add 
non-machine-readable chunks below the machine-readable parts.


> And there is something to be said for the dpkg-source / debhelper 
> style, in which each configuration parameter lives in its own tiny 
> file (e.g., 'debian/source/format', 'debian/compat', 
> 'debian/pyversions') rather than as fields of a larger file that is 
> only tangentially related to the task at hand.

You mean that you prefer a separate file for this info?

What should be its name? What should be its syntax?

...and why start from scratch with this - or does something else already 
exist, comparable to the work of DEP-5?


> Unrelated: when I've repacked tarballs, I add a file 
> "README.Debian-tarball" to the top level source directory, explaining 
> what I did.  Nobody ever suggested this to me, it just seems like 
> common sense that information about the new tarball should be, well, 
> in the new tarball.  Not just in the .diff.gz.  If you're going to 
> generate the tarball with uscan, could you either generate a 
> README.Debian-tarball in the new orig.tar.gz, or actually use that 
> location for configuration in the first place?  (I'm not wedded to 
> that filename, of course, but I do think it should be in the 
> orig.tar.gz, and thus outside debian/.)

I also used to add a custom README.* file for some of "my" packages.

Nowadays such info is commonly put into README.source, which has grown 
to be the common place to document unusual things in the package for 
fellow developers e.g. doing NMUs.


Regards,

 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Peter Samuelson]

[Peter Samuelson]
> And there is something to be said for the dpkg-source / debhelper
> style, in which each configuration parameter lives in its own tiny
> file (e.g., 'debian/source/format', 'debian/compat',
> 'debian/pyversions') rather than as fields of a larger file that is
> only tangentially related to the task at hand.

...Or add the exclude list to a file called 'debian/watch'.  That makes
even more sense to me.  I mean, that file format isn't perfect (having
options start with a prefix of "opts=" is a bit clunky, as is the
design of one self-contained logical line per source, where there is
almost always just a single source) but the format is explicitly
versioned, so these things can be fixed.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120823173814.gb5...@p12n.org

Re: Enabling uscan to simply remove files from upstream source

[Peter Samuelson]

[Jonas Smedegaard]
> Format: 
>  http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
> Source: http://susy.oddbird.net/
>   Repackaged, excluding non-DFSG licensed fonts and source-less
>   JavaScript
> Files-Excluded:
>   docs/source/javascripts/jquery-1.7.1.min.js
>   docs/source/javascripts/modernizr-2.5.3.min.js
> Files-Excluded-comment: Exlude source-less JavaScript

A file format in which a comment starts with "Files-Excluded-comment:"
instead of, say, "#", is a file format I just can't get excited about.
Automating get-orig-source is a fine idea, but tying it to DEP-5 would
be unfortunate.

And there is something to be said for the dpkg-source / debhelper
style, in which each configuration parameter lives in its own tiny file
(e.g., 'debian/source/format', 'debian/compat', 'debian/pyversions')
rather than as fields of a larger file that is only tangentially
related to the task at hand.

Unrelated: when I've repacked tarballs, I add a file
"README.Debian-tarball" to the top level source directory, explaining
what I did.  Nobody ever suggested this to me, it just seems like
common sense that information about the new tarball should be, well, in
the new tarball.  Not just in the .diff.gz.  If you're going to
generate the tarball with uscan, could you either generate a
README.Debian-tarball in the new orig.tar.gz, or actually use that
location for configuration in the first place?  (I'm not wedded to that
filename, of course, but I do think it should be in the orig.tar.gz,
and thus outside debian/.)

Peter


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120823172752.ga5...@p12n.org

Re: Enabling uscan to simply remove files from upstream source

[Jonas Smedegaard]

On 12-08-21 at 12:21pm, Andreas Tille wrote:
> Regarding the implementation there was some uncertainity about the 
> actual Perl module to use.  In the attached example script I decided 
> to stick to Dpkg::Control and left the code for Parse::DebControl as a 
> comment which could pretty easily could replace the other parser.

Do Dpkg::Control::Hash provide the equivalent of flags discardCase and 
singleBlock for Parse::DebControl, or are those considered unimportant?

As I recall, Copyright Format 1.0 is enabled by its identifying Format: 
line being the first line of the file, and my proposal is for 
Files-Excluded lines to be in that same section (and only there).  Also, 
I seem to recall during the development of the format that it should be 
ok to add non-machine-readable chunks _after_ the machine-readable 
parts.  That's why I found it quite compelling to me that those two 
flags existed in Parse::DebControl: You don't accidentally (although 
arguably quite rare) pickup unintended hints further down the file.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Jonas Smedegaard]

On 12-08-22 at 09:39am, Paul Wise wrote:
> On Tue, Aug 21, 2012 at 6:21 PM, Andreas Tille wrote:
> 
> > Any further hints / remarks?
> 
> In comparison to the current method for repacking (debian/rules
> get-orig-source), this doesn't allow per-file-set comments about why
> the file-set is being removed. I often use this to document in more
> detail why I am removing files. So I view this spec as a downgrade
> from what we have now.

Good point.

Copyright file format 1.0 allows multiple paragraphs of same name.

So would be nice to check that the implementation properly includes all 
of the following items:

Format: 
 http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Source: http://susy.oddbird.net/
  Repackaged, excluding non-DFSG licensed fonts and source-less
  JavaScript
Files-Excluded:
  docs/source/javascripts/jquery-1.7.1.min.js
  docs/source/javascripts/modernizr-2.5.3.min.js
Files-Excluded-comment: Exlude source-less JavaScript
Files-Excluded: foo/bar
  docs/source/fonts/*
  baz
  boom/boom/
Files-Excluded-comment: Exlude non-DFSG licensed fonts and more
 Just for demonstration purpose, this paragraph has multiple
 lines.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Paul Wise]

On Tue, Aug 21, 2012 at 6:21 PM, Andreas Tille wrote:

> Any further hints / remarks?

In comparison to the current method for repacking (debian/rules
get-orig-source), this doesn't allow per-file-set comments about why
the file-set is being removed. I often use this to document in more
detail why I am removing files. So I view this spec as a downgrade
from what we have now.

-- 
bye,
pabs

http://wiki.debian.org/PaulWise


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/caktje6hhd392gepadjmxkqe3vfg5j++qg1fkdrqn4sjf9bz...@mail.gmail.com

Re: Enabling uscan to simply remove files from upstream source

[Russ Allbery]

Charles Plessy  writes:
> Le Tue, Aug 21, 2012 at 03:48:33PM +0200, Andrew Shadura a écrit :

>> By the way, how about adding something under debian/source/... to allow
>> automatic repacking regardless of Files-Excluded? (Or please tell me
>> how to repack upstream's zipball to targz automatically without having
>> to specify --repack every time.)

> actually, wouldn't a file under debian/source (or directly under
> debian/) be a better place than debian/copyright, given that there are
> reason for excluding files that are not related to copyright, such as
> for their size, or a safeguard against accidentally building against
> convenience code copies.

It would be nice to keep all of the information about the provenance of
the upstream source in one location.  Currently, that's (mostly)
debian/copyright.  If we start moving that data, perhaps we could move
*all* of it, rather than deciding every new piece of data piecemeal?

The argument for having that data in debian/copyright is, so far as I
understand it, two-fold: identifying the upstream location allows people
to review and confirm license information, and some upstream licenses may
require pointers to the original source location.  However, the latter
should, I hope, be dealt with by the license text, which we have to
reproduce anyway, and in any case are a minority.  And the former seems
like a relatively weak association with debian/copyright to me.

Personally, I care more about it being in one place than about what that
place is.  As long as people aren't going to move this information out of
debian/copyright, I want new, related information, such as excluded files,
to go in the same place so that the information isn't split.  But I'm fine
with moving it entirely.

-- 
Russ Allbery (r...@debian.org)   


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87lih7n9y3@windlord.stanford.edu

Re: Enabling uscan to simply remove files from upstream source

[Charles Plessy]

Le Tue, Aug 21, 2012 at 03:48:33PM +0200, Andrew Shadura a écrit :
> 
> By the way, how about adding something under debian/source/... to allow
> automatic repacking regardless of Files-Excluded? (Or please tell me
> how to repack upstream's zipball to targz automatically without having
> to specify --repack every time.)

Hi all,

actually, wouldn't a file under debian/source (or directly under debian/) be a
better place than debian/copyright, given that there are reason for excluding
files that are not related to copyright, such as for their size, or a safeguard
against accidentally building against convenience code copies.

Have a nice day,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120821223811.gc11...@falafel.plessy.net

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

On Tue, Aug 21, 2012 at 03:40:41PM +0200, olivier sallou wrote:
> 
> Just a remark, it would be nice in this case to get a warning/log that
> some/all glob do not match to track changes in upstream tarballs (exclusion
> in previous release, then not needed anymore).

Yes.  Uscan does print several messages in verbose mode.  Something like
this should be printed (that's what I planed but forgot to mention). 

Kind regards

   Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120821141406.gc23...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Stefano Zacchiroli]

On Tue, Aug 21, 2012 at 12:21:21PM +0200, Andreas Tille wrote:
> Any further hints / remarks?

... just a big THANKS for helping the discussion in this thread and for
the draft code! I totally agree with Jakub that the main issue here is
that repackaging is a PITA, fixing the tools is the way to go.

Zack, enjoying pop-corns while reading a productive -devel thrad.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: Digital signature

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

On Tue, Aug 21, 2012 at 03:48:33PM +0200, Andrew Shadura wrote:
> 
> By the way, how about adding something under debian/source/... to allow
> automatic repacking regardless of Files-Excluded? (Or please tell me
> how to repack upstream's zipball to targz automatically without having
> to specify --repack every time.)

I admit I fail to see the connection to the proposal if we are actually
are talking about Files-Excluded.  This somehow goes in line with
specifying the arget compression method and as I said I would like to
see this discussed under a different topic.

Kind regards

   Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120821141205.gb23...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Andrew Shadura]

Hello,

On Tue, 21 Aug 2012 12:21:21 +0200
Andreas Tille  wrote:

>  2. If files matching are contained in the source tarball this will
> be repackaged except if the option --no-exclusion is given at
> uscan command line or if USCAN_NO_EXCLUSION is set in
> /etc/devscripts.conf or ~/.devscripts.

>  3. If the tarball did not contained any of the globs in
> debian/copyright::Files-Excluded it should be left untouched
> (except if the repackaging is needed because of compression method
> anyway if the user forces --repack).

By the way, how about adding something under debian/source/... to allow
automatic repacking regardless of Files-Excluded? (Or please tell me
how to repack upstream's zipball to targz automatically without having
to specify --repack every time.)

-- 
WBR, Andrew


signature.asc
Description: PGP signature

Re: Enabling uscan to simply remove files from upstream source

[olivier sallou]

2012/8/21 Andreas Tille 

> Hi,
>
> third summary of the proposal
>
>  1. The new field Files-Excluded in debian/copyright contains a space
> separated list of globs (as used by find and for specifying file
> lists in machine readable debian/control files). The deletion
> process will loop over every expression
>
>   rm -rf ${MAIN_SOURCE_DIR}/
>
> An example copyright file would look like this:
>
> Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
> Source: http://susy.oddbird.net/
>   Repackaged, excluding non-DFSG licensed fonts and source-less
>   JavaScript
> Files-Excluded:
>   docs/source/fonts/*
>   docs/source/javascripts/jquery-1.7.1.min.js
>   docs/source/javascripts/modernizr-2.5.3.min.js
>
>  2. If files matching are contained in the source tarball this will
> be repackaged except if the option --no-exclusion is given at
> uscan command line or if USCAN_NO_EXCLUSION is set in
> /etc/devscripts.conf or ~/.devscripts.
>
>  3. If the tarball did not contained any of the globs in
> debian/copyright::Files-Excluded it should be left untouched
> (except if the repackaging is needed because of compression method
> anyway if the user forces --repack).
>

Just a remark, it would be nice in this case to get a warning/log that
some/all glob do not match to track changes in upstream tarballs (exclusion
in previous release, then not needed anymore).

Olivier


>  4. In case something was removed the version string will be appended by
> '+dfsg' to express the fact that the content of the original source
> was changed.
>
> This discussion brought up additional new wishlist features for uscan:
>
>  a) Configurable option when repacking (this is somehow related to
> the suggestion above but I would like to split up this to a
> different task).
>
>  b) Uscan should be enabled to download VCS repositories (and once
> it does deletion of files should be possible according to the same
> method above (this is an interesting feature in principle but once
> uscan is able to delete files it can do it for any download method).
>
>  c) The suggested repackaging method was requested for non-uscan
> based downloads (for instance from VCS) which might have an
> influence for the final implementation as a separate tool which
> could simply called by uscan (and others).
>
>  d) Enable confirguration of compression method.  I'd consider this
> an unrelated feature which also could be useful for --repack.
> I admit once we are repackaging anyway it might be reasonable to
> be able to influence the compression method but I also would like
> to split this up to a different task.
>
> Regarding the implementation there was some uncertainity about the
> actual Perl module to use.  In the attached example script I decided to
> stick to Dpkg::Control and left the code for Parse::DebControl as a
> comment which could pretty easily could replace the other parser.  The
> code works for me however, there might be some remaining empty
> directories which I'm tempted to delete these as well via an "educated"
>
>find tmp -type d -empty -delete
>
> which means I would care for deleting only those directories that became
> empty by the previous removal process and not those directories which
> were originally empty in the tarball.  On the other hand we might simply
> go with those empty dirs that finally do not harm.
>
> Any further hints / remarks?
>
> Kind regards
>
>   Andreas.
>
> --
> http://fam-tille.de
>



-- 

gpg key id: 4096R/326D8438  (keyring.debian.org)

Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438

Re: Enabling uscan to simply remove files from upstream source

[Simon Josefsson]

Andreas Tille  writes:

> On Tue, Aug 21, 2012 at 01:25:26PM +0200, Simon Josefsson wrote:
>> How about resolving the empty directory problem by permitting the
>> Files-Excluded to match directories?  Thus, if you want to remove the
>> docs/source/fonts/ hierarchy, you would instead write:
>> 
>> Files-Excluded:
>>docs/source/fonts/
>
> You can certainly do this but this would leave you with the chance
> that docs/source/ remains as empty directory.

Ah, right, I see.  Then if somebody finds that to be a problem, the
maintainer can change Files-Excluded to be docs/source/, surely?  I'm
just trying to keep the complexity low.

/Simon


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87txvwh22g@latte.josefsson.org

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

Hi Charles,

On Tue, Aug 21, 2012 at 09:11:47PM +0900, Charles Plessy wrote:
> As suggested in http://lists.debian.org/2012081822.gt5...@jones.dk,
> something similar to the following two commands would do be compatible with 
> the
> syntax of the Files field (although I am still unsure if a prefix to
>  needs to be added or not).
> 
> find ${MAIN_SOURCE_DIR}/* -path  -delete
> 
> find ${MAIN_SOURCE_DIR}/* type f -name  -delete
> 
> The key limitation of the solution you propose, apart from the complexity of
> introducing a new syntax, is that it can not recognise expressions such as
> */Makefile.in, as in the example of the Files field.

I wonder what you and Jacub mean with this.  Did any of you read the
example script I attached to my mail[1].  It implements what you are
writing above.  Some testing would be nice.

I somehow afraid that my cut-n-pasto in item 1. where I said

   'rm -rf ${MAIN_SOURCE_DIR}/'  (=old, outdated, wrong)

might have lead to the confusion.  Please forget this and read

   'find ... -delete'(=what I intended to write)

(as it is realised inside the code ...)

Kind regards

Andreas.

[1] http://lists.debian.org/debian-devel/2012/08/msg00600.html

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120821122216.ga23...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Charles Plessy]

Le Tue, Aug 21, 2012 at 12:21:21PM +0200, Andreas Tille a écrit :
> 
>  1. The new field Files-Excluded in debian/copyright contains a space
> separated list of globs (as used by find and for specifying file
> lists in machine readable debian/control files). The deletion
> process will loop over every expression
> 
>   rm -rf ${MAIN_SOURCE_DIR}/
 
>  3. If the tarball did not contained any of the globs in
> debian/copyright::Files-Excluded it should be left untouched
> (except if the repackaging is needed because of compression method
> anyway if the user forces --repack).
 
> Regarding the implementation there was some uncertainity about the
> actual Perl module to use.  In the attached example script I decided to
> stick to Dpkg::Control and left the code for Parse::DebControl as a
> comment which could pretty easily could replace the other parser.  The
> code works for me however, there might be some remaining empty
> directories which I'm tempted to delete these as well via an "educated"
> 
>find tmp -type d -empty -delete

Hi Andreas,

I would suggest to stick to the format of the Files field, and to abort with
error when files are not found, so that the Files-Excluded list stays accurate.

As suggested in http://lists.debian.org/2012081822.gt5...@jones.dk,
something similar to the following two commands would do be compatible with the
syntax of the Files field (although I am still unsure if a prefix to
 needs to be added or not).

find ${MAIN_SOURCE_DIR}/* -path  -delete

find ${MAIN_SOURCE_DIR}/* type f -name  -delete

The key limitation of the solution you propose, apart from the complexity of
introducing a new syntax, is that it can not recognise expressions such as
*/Makefile.in, as in the example of the Files field.

Have a nice day,

-- 
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120821121147.gc27...@falafel.plessy.net

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

On Tue, Aug 21, 2012 at 01:25:26PM +0200, Simon Josefsson wrote:
> How about resolving the empty directory problem by permitting the
> Files-Excluded to match directories?  Thus, if you want to remove the
> docs/source/fonts/ hierarchy, you would instead write:
> 
> Files-Excluded:
>docs/source/fonts/

You can certainly do this but this would leave you with the chance
that docs/source/ remains as empty directory.
 
> I'm worried that empty directories may be present for other reasons, and
> removing all of them would have bad side-effects.  I would prefer to not
> remove empty directories over using the find-approach above, if my
> proposal is not adopted.

Sure I would never remove simply all empty directories.  This would be
insane and finally an empty directory does not really harm.

> Thanks for doing this, I believe that having an easy way to remove files
> from upstream packages will save Debian package maintainers time and
> frustration.

I'm quite positive that I get back my time I spend into this considering
how promising it is to safe a lot of time in the future. :-)

Kind regards

   Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120821120644.gf14...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Simon Josefsson]

Andreas Tille  writes:

> Files-Excluded:
>   docs/source/fonts/*
>   docs/source/javascripts/jquery-1.7.1.min.js
>   docs/source/javascripts/modernizr-2.5.3.min.js
...
> Regarding the implementation there was some uncertainity about the
> actual Perl module to use.  In the attached example script I decided to
> stick to Dpkg::Control and left the code for Parse::DebControl as a
> comment which could pretty easily could replace the other parser.  The
> code works for me however, there might be some remaining empty
> directories which I'm tempted to delete these as well via an "educated"
>
>find tmp -type d -empty -delete
>
> which means I would care for deleting only those directories that became
> empty by the previous removal process and not those directories which
> were originally empty in the tarball.  On the other hand we might simply
> go with those empty dirs that finally do not harm.
>
> Any further hints / remarks?

How about resolving the empty directory problem by permitting the
Files-Excluded to match directories?  Thus, if you want to remove the
docs/source/fonts/ hierarchy, you would instead write:

Files-Excluded:
   docs/source/fonts/
   docs/source/javascripts/jquery-1.7.1.min.js
   docs/source/javascripts/modernizr-2.5.3.min.js

I'm worried that empty directories may be present for other reasons, and
removing all of them would have bad side-effects.  I would prefer to not
remove empty directories over using the find-approach above, if my
proposal is not adopted.

Thanks for doing this, I believe that having an easy way to remove files
from upstream packages will save Debian package maintainers time and
frustration.

/Simon


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ehn0ij15@latte.josefsson.org

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

On Tue, Aug 21, 2012 at 12:32:18PM +0200, Jakub Wilk wrote:
> * Andreas Tille , 2012-08-21, 12:21:
> >The new field Files-Excluded in debian/copyright contains a space
> >separated list of globs (as used by find and for specifying file
> >lists in machine readable debian/control files). The deletion
> >process will loop over every expression
> >
> > rm -rf ${MAIN_SOURCE_DIR}/
> 
> So wildcards in the Files-Excluded field would have different
> semantics than in the Files field. That's confusing.

In how far?

  http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/#files-field

   Filename patterns in the Files field are specified using a simplified
   shell glob syntax. Patterns are separated by whitespace.

Please propose a better wording that does not confuse you.

Kind regards

Andreas.

-- 
http://fam-tille.de


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120821110238.ge14...@an3as.eu

Re: Enabling uscan to simply remove files from upstream source

[Jakub Wilk]

* Andreas Tille , 2012-08-21, 12:21:
The new field Files-Excluded in debian/copyright contains a space 
separated list of globs (as used by find and for specifying file lists 
in machine readable debian/control files). The deletion process will 
loop over every expression


 rm -rf ${MAIN_SOURCE_DIR}/


So wildcards in the Files-Excluded field would have different semantics 
than in the Files field. That's confusing.


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120821103218.ga7...@jwilk.net

Re: Enabling uscan to simply remove files from upstream source

[Andreas Tille]

Hi,

third summary of the proposal

 1. The new field Files-Excluded in debian/copyright contains a space
separated list of globs (as used by find and for specifying file
lists in machine readable debian/control files). The deletion
process will loop over every expression

  rm -rf ${MAIN_SOURCE_DIR}/

An example copyright file would look like this:

Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Source: http://susy.oddbird.net/
  Repackaged, excluding non-DFSG licensed fonts and source-less
  JavaScript
Files-Excluded:
  docs/source/fonts/*
  docs/source/javascripts/jquery-1.7.1.min.js
  docs/source/javascripts/modernizr-2.5.3.min.js

 2. If files matching are contained in the source tarball this will
be repackaged except if the option --no-exclusion is given at
uscan command line or if USCAN_NO_EXCLUSION is set in
/etc/devscripts.conf or ~/.devscripts.

 3. If the tarball did not contained any of the globs in
debian/copyright::Files-Excluded it should be left untouched
(except if the repackaging is needed because of compression method
anyway if the user forces --repack).

 4. In case something was removed the version string will be appended by
'+dfsg' to express the fact that the content of the original source
was changed.

This discussion brought up additional new wishlist features for uscan:

 a) Configurable option when repacking (this is somehow related to
the suggestion above but I would like to split up this to a
different task).

 b) Uscan should be enabled to download VCS repositories (and once
it does deletion of files should be possible according to the same
method above (this is an interesting feature in principle but once
uscan is able to delete files it can do it for any download method).

 c) The suggested repackaging method was requested for non-uscan
based downloads (for instance from VCS) which might have an
influence for the final implementation as a separate tool which
could simply called by uscan (and others).

 d) Enable confirguration of compression method.  I'd consider this
an unrelated feature which also could be useful for --repack.
I admit once we are repackaging anyway it might be reasonable to
be able to influence the compression method but I also would like
to split this up to a different task.

Regarding the implementation there was some uncertainity about the
actual Perl module to use.  In the attached example script I decided to
stick to Dpkg::Control and left the code for Parse::DebControl as a
comment which could pretty easily could replace the other parser.  The
code works for me however, there might be some remaining empty
directories which I'm tempted to delete these as well via an "educated"

   find tmp -type d -empty -delete

which means I would care for deleting only those directories that became
empty by the previous removal process and not those directories which
were originally empty in the tarball.  On the other hand we might simply
go with those empty dirs that finally do not harm.

Any further hints / remarks?

Kind regards

  Andreas.

--
http://fam-tille.de
#!/usr/bin/perl   
use strict;
use warnings;

my $parsefile = 'debian/copyright';

# Dpkg::Control::Hash prefered by James McCoy (who did the last three uscan.pl edits using a debian.org e-mail address)
use Dpkg::Control::Hash;
my $data = Dpkg::Control::Hash->new();
$data->load($parsefile);

# Parse::DebControl suggested by Jonas Smedegaard
# use Parse::DebControl;
# my $parser = new Parse::DebControl(1);
# my $data = $parser->parse_file($parsefile, {discardCase=>1,singleBlock=>1,});

my $okformat = qr'http://www.debian.org/doc/packaging-manuals/copyright-format/1.0';
my $main_source_dir = $ARGV[0] ;
die unless ($data->{'format'} =~ m{^$okformat/?$});
if ( $data->{'files-excluded'} ) {
my $nfiles_before = `find $main_source_dir | wc -l`;
foreach (grep {/\//} split /\s+/, $data->{"files-excluded"}) {
`find $main_source_dir -path "$main_source_dir/$_" -delete`;
};
foreach (grep {/^[^\/]+$/} split /\s+/, $data->{"files-excluded"}) {
`find $main_source_dir -type f -name $_ -delete`;
};
my $nfiles_after = `find $main_source_dir | wc -l`;
if ( $nfiles_before == $nfiles_after ) {
	print "Source tree remains identical - no need for repacking.\n"
}
}