Re: Automatic trimming of changelogs in binary packages

[Didier 'OdyX' Raboud]

19 septembre 2022 23:19 "Bill Allombert"  a écrit:
> Le Tue, Sep 06, 2022 at 07:13:30AM +0200, Gioele Barabucci a écrit :
> 
>> On 18/08/22 21:18, Gioele Barabucci wrote:
>> Does anybody have objective objections against activating automatic
>> changelog trimming in binary packages?
>> 
>> Hi,
>> 
>> a couple of weeks since the initial email (thanks everybody for the input),
>> my reading is that there is now consensus in going ahead with the proposed
>> automatic trimming of changelogs.
> 
> Count me out. changelog embbed Debian history. The very first entries
> are often very important.

They can be important _in the source package_. I have yet to see binary packages
for which having access to early debian/changelog entries matters at all.
( debian/NEWS is another story of course, and these are not affected).

Re: Automatic trimming of changelogs in binary packages

[Bill Allombert]

Le Tue, Sep 06, 2022 at 07:13:30AM +0200, Gioele Barabucci a écrit :
> On 18/08/22 21:18, Gioele Barabucci wrote:
> > Does anybody have objective objections against activating automatic
> > changelog trimming in binary packages?
> 
> Hi,
> 
> a couple of weeks since the initial email (thanks everybody for the input),
> my reading is that there is now consensus in going ahead with the proposed
> automatic trimming of changelogs.

Count me out. changelog embbed Debian history. The very first entries
are often very important. It is much more convenient to have them as
file you can grep to that as output of some commands that depend
on network availability.

Network access is not universal.  There are better way to save diskspace.

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Re: Automatic trimming of changelogs in binary packages

[Wouter Verhelst]

On Wed, Sep 14, 2022 at 10:40:28AM +0300, Hakan Bayındır wrote:
> 
> 
> > On 14 Sep 2022, at 10:37, Wouter Verhelst  wrote:
> > 
> > On Sun, Sep 11, 2022 at 03:09:07PM +0300, Hakan Bayındır wrote:
> >> Yes, you’re right. However, my reservation is whether dpkg is more prone to
> >> breaking in disaster recovery scenarios. Reading a gzipped file is always
> >> simpler than querying a DB via more abstraction.
> > 
> > Honestly though, the way to track down a regression is to read
> > /var/log/dpkg.log, not changelogs.
> 
> Does /var/log/dpkg.log contain why my VPNs suddenly don’t connect or my
> daemon behaves differently after the last update? I thought they’re delivered
> through news and changelogs.

They are. And once you know which package is likely to be the culprit,
you can look at changelogs.

But in order to do that, you would have to look at which packages were
upgraded first. That's what dpkg.log helps you with.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

Re: Automatic trimming of changelogs in binary packages

[Marvin Renich]

* Hakan Bayındır  [220914 03:41]:
> 
> 
> > On 14 Sep 2022, at 10:37, Wouter Verhelst  wrote:
> > 
> > On Sun, Sep 11, 2022 at 03:09:07PM +0300, Hakan Bayındır wrote:
> >> Yes, you’re right. However, my reservation is whether dpkg is more prone to
> >> breaking in disaster recovery scenarios. Reading a gzipped file is always
> >> simpler than querying a DB via more abstraction.
> > 
> > Honestly though, the way to track down a regression is to read
> > /var/log/dpkg.log, not changelogs.
> 
> Does /var/log/dpkg.log contain why my VPNs suddenly don’t connect or
> my daemon behaves differently after the last update? I thought they’re
> delivered through news and changelogs.
> 
> Actually, OpenVPN’s changing defaults are nicely delivered through
> NEWS and changelogs, so I know why things broke at the first place.

And the trimmed changelogs are unlikely to alter that at all.  The
probability that the change that caused your VPN to break will have been
trimmed are close to zero.  If it has been trimmed, it means you haven't
upgraded in more than two releases, and yes, you will have to get the
untrimmed changelog using apt changelog.

...Marvin

Re: Automatic trimming of changelogs in binary packages

[Hakan Bayındır]

> On 14 Sep 2022, at 10:37, Wouter Verhelst  wrote:
> 
> On Sun, Sep 11, 2022 at 03:09:07PM +0300, Hakan Bayındır wrote:
>> Yes, you’re right. However, my reservation is whether dpkg is more prone to
>> breaking in disaster recovery scenarios. Reading a gzipped file is always
>> simpler than querying a DB via more abstraction.
> 
> Honestly though, the way to track down a regression is to read
> /var/log/dpkg.log, not changelogs.

Does /var/log/dpkg.log contain why my VPNs suddenly don’t connect or my daemon 
behaves differently after the last update? I thought they’re delivered through 
news and changelogs.

Actually, OpenVPN’s changing defaults are nicely delivered through NEWS and 
changelogs, so I know why things broke at the first place.

Cheers,

H.

> -- 
> w@uter.{be,co.za}
> wouter@{grep.be,fosdem.org,debian.org}
> 
> I will have a Tin-Actinium-Potassium mixture, thanks.

Re: Automatic trimming of changelogs in binary packages

[Wouter Verhelst]

On Sun, Sep 11, 2022 at 03:09:07PM +0300, Hakan Bayındır wrote:
> Yes, you’re right. However, my reservation is whether dpkg is more prone to
> breaking in disaster recovery scenarios. Reading a gzipped file is always
> simpler than querying a DB via more abstraction.

Honestly though, the way to track down a regression is to read
/var/log/dpkg.log, not changelogs.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

Re: Automatic trimming of changelogs in binary packages

[Andrey Rahmatullin]

On Sun, Sep 11, 2022 at 03:09:07PM +0300, Hakan Bayındır wrote:
>  Stuffing them behind a command, possibly making them online only in the
>  process will arguably make system troubleshooting and administration 
>  harder,
>  esp. if the system has connectivity issues.
>  
>  If something critical breaks, I can boot to recovery and look at the logs
>  and changelogs of recently updated packages. Having recent-ish 
>  changelogs on
>  the disk arguably accelerates the fixing process.
> >>> I don't think removing recent-ish changelogs from the disk is proposed
> >>> here.
> >> 
> >> Again, I may have misread, then. Because what I understood is
> >> 
> >> Trim changelogs:
> >> 1. Convert them to metadata
> >> 2. Ship untrimmed part in package DB
> >> 3. Get remaining part from network
> >> 
> >> Effectively eliminating /usr/share/doc/*changelog* files.
> > Even this isn't "making them online only".
> 
> Yes, you’re right. However, my reservation is whether dpkg is more prone to 
> breaking in disaster recovery scenarios. Reading a gzipped file is always 
> simpler than querying a DB via more abstraction.
I assume that cases when your dpkg database is broken and cases when you
need to read changelogs of updated packages shouldn't intersect.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[Hakan Bayındır]

> On 11 Sep 2022, at 14:59, Andrey Rahmatullin  wrote:
> 
> On Sun, Sep 11, 2022 at 02:41:24PM +0300, Hakan Bayındır wrote:
>>> On Tue, Sep 06, 2022 at 12:11:38PM +0300, Hakan Bayındır wrote:
 While all looks good and feels sound from many aspects, I have some
 reservations against treating changelogs as metadata.
 
 Current changelogs as files have a well known place, can be used by 
 anything
 and everything, and they are local.
>>> Assuming you are talking about making changelogs available at a dpkg
>>> command, as in the RPM world, it's actually making the way to get a
>>> package changelog more well-known, not less.
>> 
>> If this created the opposite effect in RPM world w.r.t. my theory, then I 
>> stand corrected.
> It didn't create anything because that was how it worked from the
> beginning.
> But yes, it's easier to run -q --changelog than write a full file path.

While I manage RPM based systems too, I’m not that familiar with depths of RPM.

> 
 Stuffing them behind a command, possibly making them online only in the
 process will arguably make system troubleshooting and administration 
 harder,
 esp. if the system has connectivity issues.
 
 If something critical breaks, I can boot to recovery and look at the logs
 and changelogs of recently updated packages. Having recent-ish changelogs 
 on
 the disk arguably accelerates the fixing process.
>>> I don't think removing recent-ish changelogs from the disk is proposed
>>> here.
>> 
>> Again, I may have misread, then. Because what I understood is
>> 
>> Trim changelogs:
>> 1. Convert them to metadata
>> 2. Ship untrimmed part in package DB
>> 3. Get remaining part from network
>> 
>> Effectively eliminating /usr/share/doc/*changelog* files.
> Even this isn't "making them online only".

Yes, you’re right. However, my reservation is whether dpkg is more prone to 
breaking in disaster recovery scenarios. Reading a gzipped file is always 
simpler than querying a DB via more abstraction.

Cheers,

H.

> -- 
> WBR, wRAR

Re: Automatic trimming of changelogs in binary packages

[Andrey Rahmatullin]

On Sun, Sep 11, 2022 at 02:41:24PM +0300, Hakan Bayındır wrote:
> > On Tue, Sep 06, 2022 at 12:11:38PM +0300, Hakan Bayındır wrote:
> >> While all looks good and feels sound from many aspects, I have some
> >> reservations against treating changelogs as metadata.
> >> 
> >> Current changelogs as files have a well known place, can be used by 
> >> anything
> >> and everything, and they are local.
> > Assuming you are talking about making changelogs available at a dpkg
> > command, as in the RPM world, it's actually making the way to get a
> > package changelog more well-known, not less.
> 
> If this created the opposite effect in RPM world w.r.t. my theory, then I 
> stand corrected.
It didn't create anything because that was how it worked from the
beginning.
But yes, it's easier to run -q --changelog than write a full file path.

> >> Stuffing them behind a command, possibly making them online only in the
> >> process will arguably make system troubleshooting and administration 
> >> harder,
> >> esp. if the system has connectivity issues.
> >> 
> >> If something critical breaks, I can boot to recovery and look at the logs
> >> and changelogs of recently updated packages. Having recent-ish changelogs 
> >> on
> >> the disk arguably accelerates the fixing process.
> > I don't think removing recent-ish changelogs from the disk is proposed
> > here.
> 
> Again, I may have misread, then. Because what I understood is
> 
> Trim changelogs:
> 1. Convert them to metadata
> 2. Ship untrimmed part in package DB
> 3. Get remaining part from network
> 
> Effectively eliminating /usr/share/doc/*changelog* files.
Even this isn't "making them online only".

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[Hakan Bayındır]

Hi Andrey,

> On 6 Sep 2022, at 12:42, Andrey Rahmatullin  wrote:
> 
> On Tue, Sep 06, 2022 at 12:11:38PM +0300, Hakan Bayındır wrote:
>> While all looks good and feels sound from many aspects, I have some
>> reservations against treating changelogs as metadata.
>> 
>> Current changelogs as files have a well known place, can be used by anything
>> and everything, and they are local.
> Assuming you are talking about making changelogs available at a dpkg
> command, as in the RPM world, it's actually making the way to get a
> package changelog more well-known, not less.

If this created the opposite effect in RPM world w.r.t. my theory, then I stand 
corrected.

> 
>> Stuffing them behind a command, possibly making them online only in the
>> process will arguably make system troubleshooting and administration harder,
>> esp. if the system has connectivity issues.
>> 
>> If something critical breaks, I can boot to recovery and look at the logs
>> and changelogs of recently updated packages. Having recent-ish changelogs on
>> the disk arguably accelerates the fixing process.
> I don't think removing recent-ish changelogs from the disk is proposed
> here.

Again, I may have misread, then. Because what I understood is

Trim changelogs:
1. Convert them to metadata
2. Ship untrimmed part in package DB
3. Get remaining part from network

Effectively eliminating /usr/share/doc/*changelog* files.

Thanks for clarifying both,

Cheers,

H.

> -- 
> WBR, wRAR

Re: Automatic trimming of changelogs in binary packages

[Gioele Barabucci]

On 06/09/22 11:42, Andrey Rahmatullin wrote:

On Tue, Sep 06, 2022 at 12:11:38PM +0300, Hakan Bayındır wrote:

Stuffing them behind a command, possibly making them online only in the
process will arguably make system troubleshooting and administration harder,
esp. if the system has connectivity issues.

If something critical breaks, I can boot to recovery and look at the logs
and changelogs of recently updated packages. Having recent-ish changelogs on
the disk arguably accelerates the fixing process.

I don't think removing recent-ish changelogs from the disk is proposed
here.



Andrey is correct: what is being proposed is just trimming the installed 
changelogs, not removing them completely. And the full logs will remain 
in the source packages.


Regards,

--
Gioele Barabucci

Re: Automatic trimming of changelogs in binary packages

[Gioele Barabucci]

On 06/09/22 10:25, Paul Wise wrote:

On Tue, 2022-09-06 at 07:13 +0200, Gioele Barabucci wrote:


* Packages not meant to be included in Debian (and without access to a
changelog server): Creators that want to preserve the full history can
use the `--no-trim` option to disable the trimming.


Most derivatives aren't going to have a changelog server so I don't
think it makes sense to enable it by default on all derivatives.

Perhaps the dpkg vendor could be used as guidance for when the trimming
should occur by default? Unless Debian/Ubuntu, disable by default.


I have searched the debhelper's source code and its documentation for 
how it handles derivative distros, and it seems to me that


1) derivative distros are not handled at all,

2) the implicit policy is that vendors will modify the default 
parameters to suit their needs.


In addition (or in alternative) to what Niels suggested in the other 
reply, derivatives could add to their patchsets a 1-line patch to make 
`--no-trim` the default. That would be in line with the kind of 
modifications that derivs to fit Debian to their needs, wouldn't it?


Regards,

--
Gioele Barabucci

Re: Automatic trimming of changelogs in binary packages

[Niels Thykier]

Paul Wise:

On Tue, 2022-09-06 at 07:13 +0200, Gioele Barabucci wrote:


* Packages not meant to be included in Debian (and without access to a
changelog server): Creators that want to preserve the full history can
use the `--no-trim` option to disable the trimming.


Most derivatives aren't going to have a changelog server so I don't
think it makes sense to enable it by default on all derivatives.

Perhaps the dpkg vendor could be used as guidance for when the trimming
should occur by default? Unless Debian/Ubuntu, disable by default.

Then have a way for vendors to enable it by default if they want.


* Third-party repositories: Their administrators can provide access to
the full changelogs by setting `Changelogs:` in the `Release` file, or
can use `--no-trim` in the packages that they distribute.


Modifying every single package to add --no-trim would be tedious for
some repositories, perhaps allow a config option to disable it?



Sounds like a job for (not yet implemented) DEB_BUILD_OPTIONS=notrimdch 
(or some other name).


Thanks,
~Niels

Re: Automatic trimming of changelogs in binary packages

[Andrey Rahmatullin]

On Tue, Sep 06, 2022 at 12:11:38PM +0300, Hakan Bayındır wrote:
> While all looks good and feels sound from many aspects, I have some
> reservations against treating changelogs as metadata.
> 
> Current changelogs as files have a well known place, can be used by anything
> and everything, and they are local.
Assuming you are talking about making changelogs available at a dpkg
command, as in the RPM world, it's actually making the way to get a
package changelog more well-known, not less.

> Stuffing them behind a command, possibly making them online only in the
> process will arguably make system troubleshooting and administration harder,
> esp. if the system has connectivity issues.
> 
> If something critical breaks, I can boot to recovery and look at the logs
> and changelogs of recently updated packages. Having recent-ish changelogs on
> the disk arguably accelerates the fixing process.
I don't think removing recent-ish changelogs from the disk is proposed
here.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[Hakan Bayındır]

Hello all,

While all looks good and feels sound from many aspects, I have some 
reservations against treating changelogs as metadata.


Current changelogs as files have a well known place, can be used by 
anything and everything, and they are local.


Stuffing them behind a command, possibly making them online only in the 
process will arguably make system troubleshooting and administration 
harder, esp. if the system has connectivity issues.


If something critical breaks, I can boot to recovery and look at the 
logs and changelogs of recently updated packages. Having recent-ish 
changelogs on the disk arguably accelerates the fixing process.


Not expanding the tools and capabilities required to do system 
maintenance and disaster recovery is a good target to have if you ask me.


Cheers,

Hakan

On 6.09.2022 08:13, Gioele Barabucci wrote:

On 18/08/22 21:18, Gioele Barabucci wrote:
Does anybody have objective objections against activating automatic 
changelog trimming in binary packages?


Hi,

a couple of weeks since the initial email (thanks everybody for the 
input), my reading is that there is now consensus in going ahead with 
the proposed automatic trimming of changelogs.


## Addressed contrary objections

In particular, I understand that all contrary objections that were 
raised have been satisfyingly addressed:


* Packages not meant to be included in Debian (and without access to a 
changelog server): Creators that want to preserve the full history can 
use the `--no-trim` option to disable the trimming.


* Third-party repositories: Their administrators can provide access to 
the full changelogs by setting `Changelogs:` in the `Release` file, or 
can use `--no-trim` in the packages that they distribute.


* Users should be made aware that changelogs have been trimmed: 
`dh_installchangelogs` has been modified to add a comment at the end of 
the trimmed changelogs.


* Source packages should retain full changelogs: This is, in fact, 
already the case: only the changelogs included in the binary packages 
are trimmed.


* Reproducible output: Once trimming will be enabled in 
`dh_installchangelogs`, either the changelogs will be trimmed (default) 
or not (due to the explicit use of `--no-trim`). This ensures the 
reproducibility of packages.


## Non-blocking concerns to be addressed in the future

In addition, there were concerns and ideas that were raised as non 
blocking and that will be discussed and addressed in the future:


* Perhaps have dpkg treat changelogs as metadata: There are pros 
(deduplication) and cons (the need to have a new command like `dpkg 
--show-changelog`).


* d-security has no online changelogs: Fixing this has been proposed in 
2008 (https://bugs.debian.org/490848) but it did not seem to sparkle 
enough interest at the time (and users of d-security are unlikely to be 
interested in changelog entries older than old-old-stable).


* apt changelog could be instructed to choose between the local and the 
remote changelogs, perhaps via options like `Changelogs::PreferOnline` 
(to complement `AlwaysOnline`) and `--full`.


* Maybe the URLs used by the tracker to link to the changelogs and the 
URLs used by apt to fetch the changelogs could be aligned.


Regards,

--
Gioele Barabucci


OpenPGP_signature
Description: OpenPGP digital signature

Re: Automatic trimming of changelogs in binary packages

[Paul Wise]

On Tue, 2022-09-06 at 07:13 +0200, Gioele Barabucci wrote:

> * Packages not meant to be included in Debian (and without access to a 
> changelog server): Creators that want to preserve the full history can 
> use the `--no-trim` option to disable the trimming.

Most derivatives aren't going to have a changelog server so I don't
think it makes sense to enable it by default on all derivatives.

Perhaps the dpkg vendor could be used as guidance for when the trimming
should occur by default? Unless Debian/Ubuntu, disable by default.

Then have a way for vendors to enable it by default if they want.

> * Third-party repositories: Their administrators can provide access to 
> the full changelogs by setting `Changelogs:` in the `Release` file, or 
> can use `--no-trim` in the packages that they distribute.

Modifying every single package to add --no-trim would be tedious for
some repositories, perhaps allow a config option to disable it?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Automatic trimming of changelogs in binary packages

[Gioele Barabucci]

On 18/08/22 21:18, Gioele Barabucci wrote:
Does anybody have objective objections against activating automatic 
changelog trimming in binary packages?


Hi,

a couple of weeks since the initial email (thanks everybody for the 
input), my reading is that there is now consensus in going ahead with 
the proposed automatic trimming of changelogs.


## Addressed contrary objections

In particular, I understand that all contrary objections that were 
raised have been satisfyingly addressed:


* Packages not meant to be included in Debian (and without access to a 
changelog server): Creators that want to preserve the full history can 
use the `--no-trim` option to disable the trimming.


* Third-party repositories: Their administrators can provide access to 
the full changelogs by setting `Changelogs:` in the `Release` file, or 
can use `--no-trim` in the packages that they distribute.


* Users should be made aware that changelogs have been trimmed: 
`dh_installchangelogs` has been modified to add a comment at the end of 
the trimmed changelogs.


* Source packages should retain full changelogs: This is, in fact, 
already the case: only the changelogs included in the binary packages 
are trimmed.


* Reproducible output: Once trimming will be enabled in 
`dh_installchangelogs`, either the changelogs will be trimmed (default) 
or not (due to the explicit use of `--no-trim`). This ensures the 
reproducibility of packages.


## Non-blocking concerns to be addressed in the future

In addition, there were concerns and ideas that were raised as non 
blocking and that will be discussed and addressed in the future:


* Perhaps have dpkg treat changelogs as metadata: There are pros 
(deduplication) and cons (the need to have a new command like `dpkg 
--show-changelog`).


* d-security has no online changelogs: Fixing this has been proposed in 
2008 (https://bugs.debian.org/490848) but it did not seem to sparkle 
enough interest at the time (and users of d-security are unlikely to be 
interested in changelog entries older than old-old-stable).


* apt changelog could be instructed to choose between the local and the 
remote changelogs, perhaps via options like `Changelogs::PreferOnline` 
(to complement `AlwaysOnline`) and `--full`.


* Maybe the URLs used by the tracker to link to the changelogs and the 
URLs used by apt to fetch the changelogs could be aligned.


Regards,

--
Gioele Barabucci

Re: Automatic trimming of changelogs in binary packages

[Andrey Rahmatullin]

On Fri, Aug 19, 2022 at 11:44:03AM +, Bastien Roucariès wrote:
> Le jeudi 18 août 2022, 19:18:35 UTC Gioele Barabucci a écrit :
> > Hello,
> > 
> > in 2020 there was a brief discussion on debian-devel@ about trimming
> > changelogs [1,2].
> > 
> > Now there is a working implementation of said functionality in
> > `dh_installchangelogs` [3].
> Could you stress on the mapage that some license ask to document the change 
> done by downstream in binary distribution and that in this case this tool 
> should be disable
Like GPLv2's "You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change"? We already
don't do this.


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[Sean Whitton]

Hello,

On Fri 19 Aug 2022 at 11:44AM GMT, Bastien Roucariès wrote:

>
> Le jeudi 18 août 2022, 19:18:35 UTC Gioele Barabucci a écrit :
>> Hello,
>>
>> in 2020 there was a brief discussion on debian-devel@ about trimming
>> changelogs [1,2].
>>
>> Now there is a working implementation of said functionality in
>> `dh_installchangelogs` [3].
> Could you stress on the mapage that some license ask to document the change
> done by downstream in binary distribution and that in this case this tool
> should be disable

I think our separation of orig.tar and diff.gz/whatever has this
covered?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[Bastien Roucariès]

Le jeudi 18 août 2022, 19:18:35 UTC Gioele Barabucci a écrit :
> Hello,
> 
> in 2020 there was a brief discussion on debian-devel@ about trimming
> changelogs [1,2].
> 
> Now there is a working implementation of said functionality in
> `dh_installchangelogs` [3].
Could you stress on the mapage that some license ask to document the change 
done by downstream in binary distribution and that in this case this tool 
should be disable

bastien
> 
> This implementation, combined with the evolution of the apt/dpkg
> ecosystem that happened in the meantime, provides now all the benefits
> of changelog trimming (less wasted space and bandwidth worldwide,
> reduced processing time) without the downsides discussed at the time.
> 
> ## In detail
> 
> * `dh_installchangelogs` is modified install in binary packages the
> trimmed changelogs, i.e. changelogs that contain only entries more
> recent than the release date of oldstable.
> 
> * The trimming is done automatically in compat >= 14.
> 
> * The `--no-trim` option allows package maintainers that want to ship
> the whole changelog a way to do so.
> 
> * The full changelogs are preserved in the source packages and thus
> available via `apt changelog` and similar mechanisms.
> 
> * The trimming process happens at build time and requires no
> modification to the changelogs stored in the VCS repos, nor changes to
> the packaging.
> 
> ## Data on file size reduction
> 
> On a sample of ~13.000 packages, the median reduction in size of
> gzip-9'ed changelogs is 56% (mean 50%).
> 
> Ancient packages or heavily developed packages gain a lot from trimming
> the changelogs. Some examples (gzipped → trimmed+gzipped):
> 
> * debian-keyring: 664k  → 47k (-92%)
> * dpkg (essential): 223k → 22k (-90%)
> * apt (essential): 156k → 14k (-90%)
> * systemd: 110k → 23k (-78%)
> * gcc-12: 189k → 18k (-90%)
> * python3.9: 48k → 4k (-90%)
> * e2fsprogs: 68k → 7k (-89%)
> 
> ## Consensus
> 
> Does anybody have objective objections against activating automatic
> changelog trimming in binary packages?
> 
> Regards,
> 
> [1] https://lists.debian.org/debian-devel/2020/03/msg00299.html
> [2] https://bugs.debian.org/954313
> [3] https://salsa.debian.org/debian/debhelper/-/merge_requests/80
> 
> --
> Gioele Barabucci



signature.asc
Description: This is a digitally signed message part.

Re: Automatic trimming of changelogs in binary packages

[Ansgar]

On Fri, 2022-08-19 at 12:44 +0200, Philip Hands wrote:
> Ansgar  writes:
> 
> >  - Having to spawn an external command ("dpkg --show-changelog") just
> >    to access a file is more complicated.
> 
> The fact that it currently needs to dug out of the main data archive
> rather than the control archive to show the user changes at install time
> seemed like a decent reason to move it, but that was assuming that the
> file would still end up being visible in the same place in the end.

As far as I understand the metadata proposal, how the changelog is
stored would be a (private) implementation detail of dpkg. It could be
just a file in /var/lib/dpkg/info/${pkg}.changelog, it could be stored
by hash (for deduplication), in a sqlite database, ...

If the goal was just to provide faster access for apt-listchanges and
similar tools, it might be enough to just shuffle the order of files in
data.tar.* so that /usr/share/doc/*/changelog.Debian.* and similar are
the first members. This should even be backwards-compatible.

> If that were not the case then I think it would be more trouble than
> it's worth.

That is my opinion too.

Ansgar

Re: Automatic trimming of changelogs in binary packages

[Philip Hands]

Michael Biebl  writes:

> Am 19.08.22 um 10:35 schrieb Philip Hands:
>> Paul Wise  writes:
>> 
>>> On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:
>>>
 Does anybody have objective objections against activating automatic
 changelog trimming in binary packages?
>>>
>>> Before we consider enabling this by default, first we need a way for
>>> `apt changelog` to download the full changelog rather than loading the
>>> changelog from /usr/share/doc in the currently installed package.
>>>
>>> Otherwise people who want to look at the full changelog for the
>>> currently installed version of the package will have no easy way to do
>>> so. They will have to manually find it instead, which isn't exactly an
>>> easy process if you do not know where the changelogs are stored online.
>> 
>> How about making the end of the trimmed file be a standard footer
>> including a hint about how one can get hold of the rest of the
>> changelog, and then have `apt changelog` look out for that footer in the
>> local copies in order to know that they've been trimmed, and that it
>> should therefore try to download the full version.
>
> If we enable trimming by default (for all packages), then apt can just 
> go directly to query the changelog online.
> What would be the benefit for "apt changelog" to look for such a footer?

I was assuming that life's a bit more messy than that, and that one
might want apt to use the local copy if it's complete, and download
otherwise -- if that's not the case, fine.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[Philip Hands]

Ansgar  writes:

>  - Having to spawn an external command ("dpkg --show-changelog") just
>to access a file is more complicated.

The fact that it currently needs to dug out of the main data archive
rather than the control archive to show the user changes at install time
seemed like a decent reason to move it, but that was assuming that the
file would still end up being visible in the same place in the end.

If that were not the case then I think it would be more trouble than
it's worth.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[David Kalnischkies]

On Fri, Aug 19, 2022 at 09:01:22AM +0800, Paul Wise wrote:
> Before we consider enabling this by default, first we need a way for
> `apt changelog` to download the full changelog rather than loading the
> changelog from /usr/share/doc in the currently installed package.

You can tell apt to ignore the local changelogs completely with:
Acquire::Changelogs::AlwaysOnline "true";

This is set by default on ubuntu-vendor apts, not sure why as for
a repository identifying as "Origin: Ubuntu" apt knows that it should
grab the online changelog, regardless of the vendor apt is built for
(Acquire::Changelogs::AlwaysOnline::Origin::Ubuntu).


That is an all or nothing setting through as Ubuntu does trimming
unconditionally for all their packages. I don't see a logic¹ that would
be able to detect "builds with dh >= 14 (and hasn't opted out)" given
only binary package metadata, but I guess for most needless downloads
are not much of a concern.

¹ I guess we could implement looking at the free-form text in trimmed
  changelogs, but that feels a bit brittle. If the last line of the
  changelog doesn't start with " -- " … except that this isn't always
  the case as e.g. shown by dpkg.



As previously said, another problem is that not all repositories have
online changelogs – and most tools building repositories have no option
for it –, but a dh change either effects them all or we get into the
problem of wanting to know if a package will end up in a repository
which has them or not while building (build-profile?).


Also note that e.g. d-security has no online changelogs as far as apt
is concerned as they are not to be found on metadata.f-m.d.o (#490848).

The tracker uses a different URI which seemingly has them (and other
files for download apt doesn't know/offer), but I have no idea who
maintains that, if it should be used by others and the URI scheme is
slightly different (it doesn't contain the component the package
belongs to) so apt can't be told to use it anyway.
(And IF apt should use it, it should be told via the Release files,
 which only stable does currently, stable-updates and stable-security
 rely on apts built-in fallback which is sad)


Best regards

David Kalnischkies

P.S.: As I was quoted already, as a side note: I am not against nor in
favour of trimming; I am just pointing out potential problems and
sometimes even their solutions as far as apt is concerned.


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[Gioele Barabucci]

On 19/08/22 11:31, Michael Biebl wrote:
I guess this could be solved by dpkg creating symlinks from the "master 
copy" which is per-source to /usr/share/doc/$binpkg/


Maybe the "master copy" could be in `/usr/share/doc/src:$pkg/`?

/usr/share/doc/openssh-{client,server}/X → /usr/share/doc/src:openssh/X

Regards,

--
Gioele Barabucci

Re: Automatic trimming of changelogs in binary packages

[Paul Wise]

On Fri, 2022-08-19 at 09:06 +0200, Michael Biebl wrote:

> If we turn on changelog trimming by default, we should probably also 
> turn on apt downloading the changelogs by default.

I think the default apt behaviour should be as it is now, show the
installed changelog (with the footer suggested by Phil), but provide an
option (shorter than the existing one) to download the full changelog.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Automatic trimming of changelogs in binary packages

[Michael Biebl]

Am 19.08.22 um 10:42 schrieb Ansgar:

On Fri, 2022-08-19 at 10:35 +0200, Philip Hands wrote:

P.S. BTW the change Guillem suggests seems like a good idea anyway:
    treating changelogs as control files.


I'm interested: why?

What makes Debian's changelog different from other documentation such
as the upstream changelog, release notes, man pages, ...?

Most things proposed (deduplication, "dpkg --show-changelog" command,
even faster extraction, etc.) can also be implemented with the existing
way of installing them as regular files; the binNMU problem was also
solved 10 years or so ago.

There are however many downsides to special treatment of changelog:

  - Upstream and Debian changelog are suddenly in totally different
places. Same for release notes (NEWS.Debian, NEWS). Unless one
makes all of them special data ("dpkg --show-upstream-changelog",
"dpkg --show-upstream-news", ...?).


I guess this could be solved by dpkg creating symlinks from the "master 
copy" which is per-source to /usr/share/doc/$binpkg/



There is dh_installdocs --link-doc as prior art, which I considered 
hacky though, as it requires one main binary package, which holds those 
files. I don't like this approach, as it is not always possible to have 
such a "main" package.

It shows though that there is a need for it.

Michael


OpenPGP_signature
Description: OpenPGP digital signature

Re: Automatic trimming of changelogs in binary packages

[Michael Biebl]

Am 19.08.22 um 10:35 schrieb Philip Hands:

Paul Wise  writes:


On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:


Does anybody have objective objections against activating automatic
changelog trimming in binary packages?


Before we consider enabling this by default, first we need a way for
`apt changelog` to download the full changelog rather than loading the
changelog from /usr/share/doc in the currently installed package.

Otherwise people who want to look at the full changelog for the
currently installed version of the package will have no easy way to do
so. They will have to manually find it instead, which isn't exactly an
easy process if you do not know where the changelogs are stored online.


How about making the end of the trimmed file be a standard footer
including a hint about how one can get hold of the rest of the
changelog, and then have `apt changelog` look out for that footer in the
local copies in order to know that they've been trimmed, and that it
should therefore try to download the full version.


If we enable trimming by default (for all packages), then apt can just 
go directly to query the changelog online.

What would be the benefit for "apt changelog" to look for such a footer?



OpenPGP_signature
Description: OpenPGP digital signature

Re: Automatic trimming of changelogs in binary packages

[Gioele Barabucci]

On 19/08/22 10:35, Philip Hands wrote:

How about making the end of the trimmed file be a standard footer
including a hint about how one can get hold of the rest of the
changelog, and then have `apt changelog` look out for that footer in the
local copies in order to know that they've been trimmed, and that it
should therefore try to download the full version.



This two-line footer is added to the trimmed changelogs:

# Older entries have been removed from this changelog.
# To read the complete changelog use `apt changelog foobar`.

Regards,

--
Gioele Barabucci

Re: Automatic trimming of changelogs in binary packages

[Ansgar]

On Fri, 2022-08-19 at 10:35 +0200, Philip Hands wrote:
> P.S. BTW the change Guillem suggests seems like a good idea anyway:
>    treating changelogs as control files.

I'm interested: why?

What makes Debian's changelog different from other documentation such
as the upstream changelog, release notes, man pages, ...?

Most things proposed (deduplication, "dpkg --show-changelog" command,
even faster extraction, etc.) can also be implemented with the existing
way of installing them as regular files; the binNMU problem was also
solved 10 years or so ago.

There are however many downsides to special treatment of changelog:

 - Upstream and Debian changelog are suddenly in totally different
   places. Same for release notes (NEWS.Debian, NEWS). Unless one
   makes all of them special data ("dpkg --show-upstream-changelog",
   "dpkg --show-upstream-news", ...?).
 - It's harder to discover a Debian-specific command than regular 
   files. And you already might want to look in /usr/share/doc for
   other documentation anyway.
 - Having to spawn an external command ("dpkg --show-changelog") just
   to access a file is more complicated.

Ansgar

Re: Automatic trimming of changelogs in binary packages

[Philip Hands]

Paul Wise  writes:

> On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:
>
>> Does anybody have objective objections against activating automatic 
>> changelog trimming in binary packages?
>
> Before we consider enabling this by default, first we need a way for
> `apt changelog` to download the full changelog rather than loading the
> changelog from /usr/share/doc in the currently installed package.
>
> Otherwise people who want to look at the full changelog for the
> currently installed version of the package will have no easy way to do
> so. They will have to manually find it instead, which isn't exactly an
> easy process if you do not know where the changelogs are stored online.

How about making the end of the trimmed file be a standard footer
including a hint about how one can get hold of the rest of the
changelog, and then have `apt changelog` look out for that footer in the
local copies in order to know that they've been trimmed, and that it
should therefore try to download the full version.

Cheers, Phil.

P.S. BTW the change Guillem suggests seems like a good idea anyway:
   treating changelogs as control files.

 debootstrap being able to exclude paths during installs (#811269)
 is also a good idea (as mentioned in the second thread Guillem
 referenced.) -- I'll see if I can nudge that towards fruition.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Automatic trimming of changelogs in binary packages

[Fabio Fantoni]

Il 19/08/2022 09:43, julien.pu...@gmail.com ha scritto:

Le vendredi 19 août 2022 à 09:04 +0200, Fabio Fantoni a écrit :

Il 19/08/2022 03:01, Paul Wise ha scritto:

On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:


Does anybody have objective objections against activating
automatic
changelog trimming in binary packages?

Before we consider enabling this by default, first we need a way
for
`apt changelog` to download the full changelog rather than loading
the
changelog from /usr/share/doc in the currently installed package.

Otherwise people who want to look at the full changelog for the
currently installed version of the package will have no easy way to
do
so. They will have to manually find it instead, which isn't exactly
an
easy process if you do not know where the changelogs are stored
online.


Hi, I also used the changelog many times both in the packages I
maintain
and in the one I only use, in some cases a very old changelog entries
was needed.

As long as "apt-get source" gives the whole d/changelog as well as
d/rules, d/control and everything, I think this objection is out.

Why put rarely-used information in each and every installation of our
_users_?


Thanks for reply, I also think that trimming changelog in binaries is 
useful to save disk space and a little bit of bandwidth/time (when 
downloading thanks to smaller packages)


What I mean was only have a fast/easy/know to view the full changelog 
before make trimming the default.


About "apt-get source" need add of deb-src in source.list, download the 
full source of the package and some operations so is not optimal, "apt 
changelog" that download and view the full changelog by default as I saw 
in previous replies seems a good solution




Cheers,

J.Puydt





OpenPGP_signature
Description: OpenPGP digital signature

Re: Automatic trimming of changelogs in binary packages

[Michael Biebl]

Am 19.08.22 um 09:04 schrieb Fabio Fantoni:


I also use many times the changelog view on packages.debian.org, it show 
the full changelog from source and will still show the full changelog?




Correct. The changelogs linked from packages.debian.org are from 
https://metadata.ftp-master.debian.org so will always show the complete 
changelog file as shipped in the source package.






OpenPGP_signature
Description: OpenPGP digital signature

Re: Automatic trimming of changelogs in binary packages

[julien . puydt]

Le vendredi 19 août 2022 à 09:04 +0200, Fabio Fantoni a écrit :
> Il 19/08/2022 03:01, Paul Wise ha scritto:
> > On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:
> > 
> > > Does anybody have objective objections against activating
> > > automatic
> > > changelog trimming in binary packages?
> > Before we consider enabling this by default, first we need a way
> > for
> > `apt changelog` to download the full changelog rather than loading
> > the
> > changelog from /usr/share/doc in the currently installed package.
> > 
> > Otherwise people who want to look at the full changelog for the
> > currently installed version of the package will have no easy way to
> > do
> > so. They will have to manually find it instead, which isn't exactly
> > an
> > easy process if you do not know where the changelogs are stored
> > online.
> > 
> Hi, I also used the changelog many times both in the packages I
> maintain 
> and in the one I only use, in some cases a very old changelog entries
> was needed.

As long as "apt-get source" gives the whole d/changelog as well as
d/rules, d/control and everything, I think this objection is out.

Why put rarely-used information in each and every installation of our
_users_?

Cheers,

J.Puydt

Re: Automatic trimming of changelogs in binary packages

[Fabio Fantoni]

Il 19/08/2022 03:01, Paul Wise ha scritto:

On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:


Does anybody have objective objections against activating automatic
changelog trimming in binary packages?

Before we consider enabling this by default, first we need a way for
`apt changelog` to download the full changelog rather than loading the
changelog from /usr/share/doc in the currently installed package.

Otherwise people who want to look at the full changelog for the
currently installed version of the package will have no easy way to do
so. They will have to manually find it instead, which isn't exactly an
easy process if you do not know where the changelogs are stored online.

Hi, I also used the changelog many times both in the packages I maintain 
and in the one I only use, in some cases a very old changelog entries 
was needed.


I also think a simply, fast and know way to look that full changelog is 
needed.


I also use many times the changelog view on packages.debian.org, it show 
the full changelog from source and will still show the full changelog?




OpenPGP_signature
Description: OpenPGP digital signature

Re: Automatic trimming of changelogs in binary packages

[Gioele Barabucci]

On Fri, 19 Aug 2022 00:46:21 +0200, Guillem Jover wrote:

My objections from that time still stand:

  

I would also like to highlight David Kalnischkies reply:

  


Dear Guillem,

I believe all factual objections raised in those two email are well 
handled by the proposed `dh_installchangelogs` MR.


Here are the factual objections in those emails, reordered and 
reformatted for sake of readability:


Doing that trimming globally would also mean that this applies even 
to packages that are for local or derived use where something like 
«apt changelog» will in most cases not work.
Maintainers of packages _meant_ to be installed locally (i.e., 
distributed outside repos) can use `--no-trim`.


Packages distributed by Debian derivatives can (and should) set the 
`Changelogs` field in their Release files [1], or provide an appropriate 
`Acquire::Changelogs::URI::Origin::DISTRO` apt.conf.d snippet.



Assuming the repository supports it. I have yet to encounter a
third-party which does, so if Debian would trim e.g. in debhelper by 
default some care might need to be applied so that this happens only

to packages which end up in Debians repositories… which could
complicate reproducibility as its clear for a buildd, but my local
sbuild…

Similar to the point above, if the third-party developers are

1) willingly distributing their packages via a repository that does not 
set `Changelogs:` and

2) also really want to make the full changelogs available,

then they can pass `--no-trim` to `dh_installchangelogs`.

The explicit use of `--no-trim` will make the packages reproducible.


The benefit of treating all packages the same is that tools working
with changelogs can handle the grunt work: "apt{,-get} changelog pkg"
prefers the changelog on disk if available – except for repositories
which identify as "Ubuntu" for which it will always download the
online changelog for display.
(I read this as a comment _in favor_ of automatic trimming, but maybe 
I'm just reading it through rose-tinted glasses.)


Once automatic changelog trimming is the default, then fixing `apt 
changelog` is a matter of setting 
`Acquire::Changelogs::AlwaysOnline::Origin::Debian` to true in apt.conf.d/


A proposal I've been floating around from time to time has been to 
instead make the changelog and copyright files deb/dpkg metadata 
(which they really are anyway IMO).
That is indeed a good idea. And, by implementing it in addition to 
`dh_installchangelogs` auto-trimming, it will even further reduce the 
amount of wasted space and bandwidth.


Regards,

[1] 
https://salsa.debian.org/apt-team/apt/-/blob/main/doc/apt.conf.5.xml#L628


--
Gioele Barabucci

Re: Automatic trimming of changelogs in binary packages

[Michael Biebl]

Hi Paul

Am 19.08.22 um 03:01 schrieb Paul Wise:

On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:


Does anybody have objective objections against activating automatic
changelog trimming in binary packages?


Before we consider enabling this by default, first we need a way for
`apt changelog` to download the full changelog rather than loading the
changelog from /usr/share/doc in the currently installed package.

Otherwise people who want to look at the full changelog for the
currently installed version of the package will have no easy way to do
so. They will have to manually find it instead, which isn't exactly an
easy process if you do not know where the changelogs are stored online.



apt has the necessary infrastructure already in place and we have 
https://metadata.ftp-master.debian.org/changelogs


So all one needs to do is to turn it on via

# cat /etc/apt/apt.conf.d/01-changelogs
Acquire::Changelogs::AlwaysOnline "true";

If we turn on changelog trimming by default, we should probably also 
turn on apt downloading the changelogs by default.



Regards,
Michael


OpenPGP_signature
Description: OpenPGP digital signature

Re: Automatic trimming of changelogs in binary packages

[Paul Wise]

On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:

> Does anybody have objective objections against activating automatic 
> changelog trimming in binary packages?

Before we consider enabling this by default, first we need a way for
`apt changelog` to download the full changelog rather than loading the
changelog from /usr/share/doc in the currently installed package.

Otherwise people who want to look at the full changelog for the
currently installed version of the package will have no easy way to do
so. They will have to manually find it instead, which isn't exactly an
easy process if you do not know where the changelogs are stored online.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Automatic trimming of changelogs in binary packages

[Guillem Jover]

Hi!

On Thu, 2022-08-18 at 21:18:35 +0200, Gioele Barabucci wrote:
> in 2020 there was a brief discussion on debian-devel@ about trimming
> changelogs [1,2].

My objections from that time still stand:

  

I would also like to highlight David Kalnischkies reply:

  

Thanks,
Guillem

Re: Automatic trimming of changelogs in binary packages

[M. Zhou]

On Thu, 2022-08-18 at 21:18 +0200, Gioele Barabucci wrote:
> * The `--no-trim` option allows package maintainers that want to ship 
> the whole changelog a way to do so.
> 
> * The full changelogs are preserved in the source packages and thus 
> available via `apt changelog` and similar mechanisms.
> 
> Does anybody have objective objections against activating automatic 
> changelog trimming in binary packages?

Thank you for working on this.

My original concern about automatically trimming logs was about convenience
of debugging -- I sometimes need to search among the full history to see
what happened to the package of interest in the past, or to quickly figure
out what change has been made at what time.

Since `apt changelog` can still retrieval the full history, my concerns
are gone.

Re: Automatic trimming of changelogs in binary packages

[Michael Biebl]

Hi Gioele,

thanks for working on this!

Am 18.08.22 um 21:18 schrieb Gioele Barabucci:


Hello,

in 2020 there was a brief discussion on debian-devel@ about trimming 
changelogs [1,2].


Now there is a working implementation of said functionality in 
`dh_installchangelogs` [3].


This implementation, combined with the evolution of the apt/dpkg 
ecosystem that happened in the meantime, provides now all the benefits 
of changelog trimming (less wasted space and bandwidth worldwide, 
reduced processing time) without the downsides discussed at the time.


## In detail

* `dh_installchangelogs` is modified install in binary packages the 
trimmed changelogs, i.e. changelogs that contain only entries more 
recent than the release date of oldstable.


* The trimming is done automatically in compat >= 14.



I think we should do the trimming by default, so I'm all in favour of 
your proposal. Having the last changelog entries dating back to 
oldstable seeems like a reasonably chosen time frame and older changelog 
entries can be queried easily enough via "apt changelog".
If you need more details about a specific package, you are most likely 
best served anyway by looking at the VCS.



I'd even go as far and enable the trimming unconditionally and not tie 
it to a compat bump.


Regards,
Michael


OpenPGP_signature
Description: OpenPGP digital signature