Re: DEP-4: The TDeb specification.

[Neil Williams]

On Sun, 3 Apr 2011 21:09:55 +0200
Helge Kreutzmann  wrote:

> Hello Neil,
> On Tue, Mar 17, 2009 at 04:13:15PM +, Neil Williams wrote:
> > Time to launch the debate on the Draft TDeb Specification - DEP-4.
> > 
> > Current HTML form: http://people.debian.org/~codehelp/tdeb/
> 
> ...
> 
> > Discuss.
> 
> just out of curiosity, is this still an active proposal, especially
> since Lenny+1 has now been released?

It is in active usage in Emdebian in a slightly modified form but I'm
currently looking at what can be done in Debian in relation to the move
to having -data and -common packages, source-only uploads and exactly
how TDebs can be made to work with more than just gettext translations.

I'm hoping to pick up on this a bit later this year.

-- 


Neil Williams
=
http://www.linux.codehelp.co.uk/



pgpDrWthdsTxe.pgp
Description: PGP signature

Re: DEP-4: The TDeb specification.

[Helge Kreutzmann]

Hello Neil,
On Tue, Mar 17, 2009 at 04:13:15PM +, Neil Williams wrote:
> Time to launch the debate on the Draft TDeb Specification - DEP-4.
> 
> Current HTML form: http://people.debian.org/~codehelp/tdeb/

...

> Discuss.

just out of curiosity, is this still an active proposal, especially
since Lenny+1 has now been released?

Greetings

  Helge

-- 
  Dr. Helge Kreutzmann deb...@helgefjell.de
   Dipl.-Phys.   http://www.helgefjell.de/debian.php
64bit GNU powered gpg signed mail preferred
   Help keep free software "libre": http://www.ffii.de/


signature.asc
Description: Digital signature

Re: Re: .tdeb format (DEP-4: The TDeb specification.)

[Filipus Klutiero]

Neil Williams wrote:

On Tue, 07 Apr 2009 10:23:37 -0400
Filipus Klutiero  wrote:

> > In a similar way to udebs. The .tdeb needs to be handled differently by
> > package management tools (things like reprepro and dak) so that uploads
> > of TDebs can be made by translation teams, so that the existing source
> > package is not affected, so that TDeb uploads can more easily be made
> > during a release freeze without causing problems for archive management
> > software and without needing the tools to look inside the TDeb or rely
> > solely on a version suffix.
> >   
> Yes. I think a translation package could simply be identified by a 
> control field.


Then every file-based operation has to query the contents of the file
to know how to handle it. TDebs are sufficiently different internally
that other tools can handle them more easily if the filename is clearly
indicative.

This is more important where there are lots of TDebs, so that is how
.tdeb came into the Specification. In Emdebian, a package can have
dozens of .tdeb files, obscuring the .deb files in a simple directory
listing. With the modifications for Debian TDebs after DebConf and the
resulting single TDeb per source package, this is not a problem for
archive listings but it could still be a problem for other tools.

> Say
> Translates: foo
> or, if Translates would only list the source package,
> Translation: Yes

I'm working on the patches for dpkg at the moment, I'm extending
Thomas' code to support 'dpkg -c' etc. and it is looking like a field
in DEBIAN/control will be necessary. I'm looking at putting a Languages:
field inside DEBIAN/control that lists the locale roots supported by
that TDeb. (The name of the field isn't material, Locale-roots: is more
accurate than Languages: but not as suitable IMHO.) This field will
be generated by dpkg-deb --tdeb -b so that it is always up to date.
However, if I find a better way of obtaining the data that can be
gleaned from using: `ar -t` with relevant parsing, then there will be no
need for the field, at least on the part of dpkg.

There is also a Package-Type: tdeb field which is used by
dpkg-genchanges in the same way as Package-Type: udeb is done currently.
  


Ah, I never heard about that. That sounds better than my suggestions.

I don't see that either of those predicate against a .tdeb extension
when .udeb exists for the same reasons.
  


Neither do I. I'm sorry if I've been unclear, but I never meant to say 
anything about the naming of translation deb files. I was only 
questioning the introduction of a new package format.


[...]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Re: tdiff (DEP-4: The TDeb specification.)

[Filipus Klutiero]

On Tue, 07 Apr 2009 09:57:30 -0400
Filipus Klutiero  wrote:

(Could you add a blank line between the quoted reply and your content?
It makes the content easier for me to read. Thanks.)
  


I'll try to.

> Neil Williams wrote:
> > On Mon, 06 Apr 2009 01:13:19 -0400
> > Filipus Klutiero  wrote:
> >
> > > > > > > That would be a nice improvement, but let me suggest another 
> > > > > > > implementation. To avoid introducing a second diff, why not updating the 
> > > > > > > regular diff (in the case of non-native packages) but indicating that 
> > > > > > > the package shouldn't be entirely rebuilt when uploading? This seems 
> > > > > > > simpler.

> >
> > That also involves separate handling for native vs non-native packages.
> > The DEP does not require that - the +t1.diff.gz is used for native and
> > non-native. Any changes to the .diff.gz (or .tar.gz for native
> > packages) uploaded by the maintainer would require a source rebuild in
> > order to maintain the integrity of the archive - that is precisely what
> > the DEP is trying to avoid. You can't have a source package in a state
> > that says "this isn't the source we actually used to build the binary".
> > The source to build the .deb needs to be unchanged.
>
> Yes, but the source package contains more than the source used to build 
> the "traditional" binary packages. It also contains (potentially) source 
> for translation debs. As long as the only source changed is the source 
> for the translation debs, the "traditional" binary packages don't need 
> to be rebuilt.


But the source package would still be the wrong source for the binaries
that exist. The archive needs to contain the unique source package for
the binaries that exist, modifying the source without modifying the
binaries is not an option for legal reasons, as I understand it.
  


I don't understand. If you have a source package which contains the 
source for a set of binary packages A, and the source for a set of 
binary packages B. You can change B's source without affecting A's 
source. This allows an infinity of source packages for the same source 
of A. Changing the source package does not imply a change in A's source.

> > The changes made to
> > generate the .tdeb need to sit alongside the existing source package
> > and have absolutely no effect on anything related to the binary
> > packages or the source package uploaded by the maintainer.
>
> Per what I wrote above, I don't see anything preventing the source 
> changes for translation debs to be integrated in the traditional diff 
> (again, in the case of non-native packages).


Why treat the two differently?


I'm not sure I understand your question. In the case of a native 
package, there is no diff, so you can't integrate the source changes for 
translation debs there. In the case of a non-native package, orig is 
upstream's business, so you can't integrate the source changes for 
translation debs there. Unless you introduce a new file avoiding that, 
you need to integrate changes to native packages in the source tarball, 
and to integrate changes to non-native packages in the diff.

> > > > > > That breaks the existing source package in Debian - the .dsc 
referring
> > > > > > to that .diff.gz has been signed by the maintainer and any changes 
will
> > > > > > break that signature.
> > > >
> > > > > Right - unless the .dsc is also updated. And if there is to be a tdiff, 
> > > > > I don't see why .dsc-s wouldn't include the tdiff's digest.

> > > >
> > > > There will be a complementary dsc using the +t1 version suffix.
> >
> > > That can work, though I don't see what you were trying to say.
> >
> > That there should be no update to the .dsc because the existing source
> > package must not be altered, therefore the second .dsc with the +t1
> > version suffix. In the case of a TDeb update during a release
> > freeze, the .dsc signed by the maintainer is already part of testing so
> > the TDeb upload to unstable must not change that .dsc.
>
> I don't see a problem preventing the traditional .dsc to be changed. 
> Changing the .dsc does not imply that the existing source package must 
> be altered. The .dsc could be updated in testing.


If the .dsc is modified, the source package no longer matches the
binaries that are in the archive that were built from the old .dsc.
  


Why?

Just who/what is supposed to modify the existing .dsc anyway? The
archive software?
  


That's indeed one way to do it.

[...]

The +t1.dsc is likely
to need to be signed but merging that with the existing .dsc could give
the utterly misleading impression that the nominated l10n person doing
the upload is somehow responsible for the binary packages that have
not been touched but are affected by the change to the .dsc.
  



I'm now aware that l10n NMUs caused such issues. It's possible they do, 
but that impression would have a tiny impact.



--
To UNSUBSCRIBE, email to 

Re: tdiff (DEP-4: The TDeb specification.)

[Neil Williams]

On Tue, 07 Apr 2009 09:57:30 -0400
Filipus Klutiero  wrote:

(Could you add a blank line between the quoted reply and your content?
It makes the content easier for me to read. Thanks.)

> Neil Williams wrote:
> > On Mon, 06 Apr 2009 01:13:19 -0400
> > Filipus Klutiero  wrote:
> >
> > > > > > > That would be a nice improvement, but let me suggest another 
> > > > > > > implementation. To avoid introducing a second diff, why not 
> > > > > > > updating the 
> > > > > > > regular diff (in the case of non-native packages) but indicating 
> > > > > > > that 
> > > > > > > the package shouldn't be entirely rebuilt when uploading? This 
> > > > > > > seems 
> > > > > > > simpler.
> >
> > That also involves separate handling for native vs non-native packages.
> > The DEP does not require that - the +t1.diff.gz is used for native and
> > non-native. Any changes to the .diff.gz (or .tar.gz for native
> > packages) uploaded by the maintainer would require a source rebuild in
> > order to maintain the integrity of the archive - that is precisely what
> > the DEP is trying to avoid. You can't have a source package in a state
> > that says "this isn't the source we actually used to build the binary".
> > The source to build the .deb needs to be unchanged.
>
> Yes, but the source package contains more than the source used to build 
> the "traditional" binary packages. It also contains (potentially) source 
> for translation debs. As long as the only source changed is the source 
> for the translation debs, the "traditional" binary packages don't need 
> to be rebuilt.

But the source package would still be the wrong source for the binaries
that exist. The archive needs to contain the unique source package for
the binaries that exist, modifying the source without modifying the
binaries is not an option for legal reasons, as I understand it.

> > The changes made to
> > generate the .tdeb need to sit alongside the existing source package
> > and have absolutely no effect on anything related to the binary
> > packages or the source package uploaded by the maintainer.
>
> Per what I wrote above, I don't see anything preventing the source 
> changes for translation debs to be integrated in the traditional diff 
> (again, in the case of non-native packages).

Why treat the two differently? Whether the package is native or
non-native makes no difference to how the TDeb is generated or handled.
It has a bearing on which packages will be handled at which stage of
the transition to TDebs but nothing beyond that.

> > > > > > That breaks the existing source package in Debian - the .dsc 
> > > > > > referring
> > > > > > to that .diff.gz has been signed by the maintainer and any changes 
> > > > > > will
> > > > > > break that signature.
> > > >
> > > > > Right - unless the .dsc is also updated. And if there is to be a 
> > > > > tdiff, 
> > > > > I don't see why .dsc-s wouldn't include the tdiff's digest.
> > > >
> > > > There will be a complementary dsc using the +t1 version suffix.
> >
> > > That can work, though I don't see what you were trying to say.
> >
> > That there should be no update to the .dsc because the existing source
> > package must not be altered, therefore the second .dsc with the +t1
> > version suffix. In the case of a TDeb update during a release
> > freeze, the .dsc signed by the maintainer is already part of testing so
> > the TDeb upload to unstable must not change that .dsc.
>
> I don't see a problem preventing the traditional .dsc to be changed. 
> Changing the .dsc does not imply that the existing source package must 
> be altered. The .dsc could be updated in testing.

If the .dsc is modified, the source package no longer matches the
binaries that are in the archive that were built from the old .dsc.

Just who/what is supposed to modify the existing .dsc anyway? The
archive software? TDebs are not necessarily built from an unpacked
source directory, they can be built by tools working just from the POT
file and PO contributions.

Yes, the +t0 TDeb, the one created by the maintainer, will have the
normal source and the normal tools, there is nothing to say that +t1
would have the unpacked source available necessarily. Translators will
often have the source in order to test the translation but the final
+t1 TDeb will collate the contributions of several translators and
several translation teams, via several sources. The +t1.dsc is likely
to need to be signed but merging that with the existing .dsc could give
the utterly misleading impression that the nominated l10n person doing
the upload is somehow responsible for the binary packages that have
not been touched but are affected by the change to the .dsc.

This is another area where TDebs are quite different to normal .deb -
there is nothing to say that dpkg-buildpackage has to be involved, none
of the standard Debian build tools need exist other than dpkg itself
(not even dpkg-dev, although that would be unusual).

> > (Whether the
> > +t1.dsc is retained 

Re: .tdeb format (DEP-4: The TDeb specification.)

[Neil Williams]

On Tue, 07 Apr 2009 10:23:37 -0400
Filipus Klutiero  wrote:

> > In a similar way to udebs. The .tdeb needs to be handled differently by
> > package management tools (things like reprepro and dak) so that uploads
> > of TDebs can be made by translation teams, so that the existing source
> > package is not affected, so that TDeb uploads can more easily be made
> > during a release freeze without causing problems for archive management
> > software and without needing the tools to look inside the TDeb or rely
> > solely on a version suffix.
> >   
> Yes. I think a translation package could simply be identified by a 
> control field.

Then every file-based operation has to query the contents of the file
to know how to handle it. TDebs are sufficiently different internally
that other tools can handle them more easily if the filename is clearly
indicative.

This is more important where there are lots of TDebs, so that is how
.tdeb came into the Specification. In Emdebian, a package can have
dozens of .tdeb files, obscuring the .deb files in a simple directory
listing. With the modifications for Debian TDebs after DebConf and the
resulting single TDeb per source package, this is not a problem for
archive listings but it could still be a problem for other tools.

> Say
> Translates: foo
> or, if Translates would only list the source package,
> Translation: Yes

I'm working on the patches for dpkg at the moment, I'm extending
Thomas' code to support 'dpkg -c' etc. and it is looking like a field
in DEBIAN/control will be necessary. I'm looking at putting a Languages:
field inside DEBIAN/control that lists the locale roots supported by
that TDeb. (The name of the field isn't material, Locale-roots: is more
accurate than Languages: but not as suitable IMHO.) This field will
be generated by dpkg-deb --tdeb -b so that it is always up to date.
However, if I find a better way of obtaining the data that can be
gleaned from using: `ar -t` with relevant parsing, then there will be no
need for the field, at least on the part of dpkg.

There is also a Package-Type: tdeb field which is used by
dpkg-genchanges in the same way as Package-Type: udeb is done currently.

I don't see that either of those predicate against a .tdeb extension
when .udeb exists for the same reasons.

> > > If you're only talking about the tdeb format per se, 
> > > I don't understand your reasoning.
> >
> > Other tools may need to support the locale roots in the .tdeb - these
> > changes will be easier if the tool can rely on these only existing in
> > a .tdeb instead of occurring in random .debs but not in others.
> >   
> But these locale roots would only exist in .tdeb-s. In .deb-s, there 
> wouldn't be such locale roots. If we're wondering whether we should 
> introduce a .tdeb format, we shouldn't reason that that there will be a 
> need for .tdeb if .tdeb is introduced.

Having .tdeb makes it simpler to handle TDebs on a variety of fronts
but I really don't think that a control field is comparable in terms of
functionality.

We have .udeb, .tdeb is sufficiently clear and aids the handling
of .tdeb by other packages - why restrict every operation to having to
inspect the DEBIAN/control file every time? `dpkg -f $file Languages`
is going to get tiresome every time, not to mention slow. (Package-Type
is just a different string for the same thing, except that will require
additional parsing for tdeb vs udeb.)

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgpyIJpOI03ny.pgp
Description: PGP signature

.tdeb format (DEP-4: The TDeb specification.)

[Filipus Klutiero]

> > Package management tools need a way to tell a .deb from a .tdeb - the
> > two need to be handled differently by tools like dak, britney, apt,
> > dpkg, reprepro, deb-gview and others.

> Do you mean that package management tools need a way to tell a 
> traditional/current .deb from a package containing what the DEP proposes 
> to put in a .tdeb? 


In a similar way to udebs. The .tdeb needs to be handled differently by
package management tools (things like reprepro and dak) so that uploads
of TDebs can be made by translation teams, so that the existing source
package is not affected, so that TDeb uploads can more easily be made
during a release freeze without causing problems for archive management
software and without needing the tools to look inside the TDeb or rely
solely on a version suffix.
  
Yes. I think a translation package could simply be identified by a 
control field.

Say
Translates: foo
or, if Translates would only list the source package,
Translation: Yes
> If you're only talking about the tdeb format per se, 
> I don't understand your reasoning.


Other tools may need to support the locale roots in the .tdeb - these
changes will be easier if the tool can rely on these only existing in
a .tdeb instead of occurring in random .debs but not in others.
  
But these locale roots would only exist in .tdeb-s. In .deb-s, there 
wouldn't be such locale roots. If we're wondering whether we should 
introduce a .tdeb format, we shouldn't reason that that there will be a 
need for .tdeb if .tdeb is introduced.



--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

tdiff (DEP-4: The TDeb specification.)

[Filipus Klutiero]

Neil Williams wrote:

On Mon, 06 Apr 2009 01:13:19 -0400
Filipus Klutiero  wrote:

> > > > > That would be a nice improvement, but let me suggest another 
> > > > > implementation. To avoid introducing a second diff, why not updating the 
> > > > > regular diff (in the case of non-native packages) but indicating that 
> > > > > the package shouldn't be entirely rebuilt when uploading? This seems 
> > > > > simpler.


That also involves separate handling for native vs non-native packages.
The DEP does not require that - the +t1.diff.gz is used for native and
non-native. Any changes to the .diff.gz (or .tar.gz for native
packages) uploaded by the maintainer would require a source rebuild in
order to maintain the integrity of the archive - that is precisely what
the DEP is trying to avoid. You can't have a source package in a state
that says "this isn't the source we actually used to build the binary".
The source to build the .deb needs to be unchanged.
Yes, but the source package contains more than the source used to build 
the "traditional" binary packages. It also contains (potentially) source 
for translation debs. As long as the only source changed is the source 
for the translation debs, the "traditional" binary packages don't need 
to be rebuilt.

The changes made to
generate the .tdeb need to sit alongside the existing source package
and have absolutely no effect on anything related to the binary
packages or the source package uploaded by the maintainer.
  
Per what I wrote above, I don't see anything preventing the source 
changes for translation debs to be integrated in the traditional diff 
(again, in the case of non-native packages).

> > > > That breaks the existing source package in Debian - the .dsc referring
> > > > to that .diff.gz has been signed by the maintainer and any changes will
> > > > break that signature.
> >
> > > Right - unless the .dsc is also updated. And if there is to be a tdiff, 
> > > I don't see why .dsc-s wouldn't include the tdiff's digest.

> >
> > There will be a complementary dsc using the +t1 version suffix.

> That can work, though I don't see what you were trying to say.

That there should be no update to the .dsc because the existing source
package must not be altered, therefore the second .dsc with the +t1
version suffix. In the case of a TDeb update during a release
freeze, the .dsc signed by the maintainer is already part of testing so
the TDeb upload to unstable must not change that .dsc.
I don't see a problem preventing the traditional .dsc to be changed. 
Changing the .dsc does not imply that the existing source package must 
be altered. The .dsc could be updated in testing.

(Whether the
+t1.dsc is retained is a different issue - I suspect that we only
actually need to retain the +t1.diff.gz but as any .dsc is tiny, it
isn't a big issue either way.)

I think it should be retained; small issue, indeed.


To sum up, I still don't see anything that imposes introducing a tdiff, 
even to avoid rebuilding the traditional binary packages. But, I'm 
starting to see tdiff is probably a good idea anyway for security reasons.



--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Mon, 06 Apr 2009 01:13:19 -0400
Filipus Klutiero  wrote:

> > > > > That would be a nice improvement, but let me suggest another 
> > > > > implementation. To avoid introducing a second diff, why not updating 
> > > > > the 
> > > > > regular diff (in the case of non-native packages) but indicating that 
> > > > > the package shouldn't be entirely rebuilt when uploading? This seems 
> > > > > simpler.

That also involves separate handling for native vs non-native packages.
The DEP does not require that - the +t1.diff.gz is used for native and
non-native. Any changes to the .diff.gz (or .tar.gz for native
packages) uploaded by the maintainer would require a source rebuild in
order to maintain the integrity of the archive - that is precisely what
the DEP is trying to avoid. You can't have a source package in a state
that says "this isn't the source we actually used to build the binary".
The source to build the .deb needs to be unchanged. The changes made to
generate the .tdeb need to sit alongside the existing source package
and have absolutely no effect on anything related to the binary
packages or the source package uploaded by the maintainer.

> > > > That breaks the existing source package in Debian - the .dsc referring
> > > > to that .diff.gz has been signed by the maintainer and any changes will
> > > > break that signature.
> >
> > > Right - unless the .dsc is also updated. And if there is to be a tdiff, 
> > > I don't see why .dsc-s wouldn't include the tdiff's digest.
> >
> > There will be a complementary dsc using the +t1 version suffix.

> That can work, though I don't see what you were trying to say.

That there should be no update to the .dsc because the existing source
package must not be altered, therefore the second .dsc with the +t1
version suffix. In the case of a TDeb update during a release
freeze, the .dsc signed by the maintainer is already part of testing so
the TDeb upload to unstable must not change that .dsc. (Whether the
+t1.dsc is retained is a different issue - I suspect that we only
actually need to retain the +t1.diff.gz but as any .dsc is tiny, it
isn't a big issue either way.)

Note that many packages containing translatable content won't need a
+t1.diff.gz at all, if the current version in testing was made after
making a call for translations and waiting to receive updates before
the upload (subject to RC bug fixes requiring a change during the
freeze). The +t1.diff.gz is a way to introduce new translations and
translation updates for packages that could not be translated prior to
the maintainer upload of the original '+t0' TDeb which itself is
described in the source package uploaded by the maintainer.

Outside a release freeze, it will be up to debian-i18n to coordinate
TDeb uploads for each source package amongst the translation teams and
possibly with the maintainer to make arrangements such that translation
updates can be uploaded in a collective manner (typically using a
deadline of some sort) or as part of the next maintainer upload. Whilst
the DEP supports +t[0-9], it is envisaged that the need for +t2 and +t3
etc. should be rare.

> > Package management tools need a way to tell a .deb from a .tdeb - the
> > two need to be handled differently by tools like dak, britney, apt,
> > dpkg, reprepro, deb-gview and others.

> Do you mean that package management tools need a way to tell a 
> traditional/current .deb from a package containing what the DEP proposes 
> to put in a .tdeb? 

In a similar way to udebs. The .tdeb needs to be handled differently by
package management tools (things like reprepro and dak) so that uploads
of TDebs can be made by translation teams, so that the existing source
package is not affected, so that TDeb uploads can more easily be made
during a release freeze without causing problems for archive management
software and without needing the tools to look inside the TDeb or rely
solely on a version suffix. 

udebs have different requirements for migration into testing (specific
unblocks required with d-i approval), different locations under
pool/main/ and different handling by archive software compared to
standard .debs. TDebs have different requirements for migration into
testing (TDebs migrate even if the package itself is frozen, as long as
the version in testing matches that in unstable or the TDeb upload goes
via testing-proposed-updates etc.), different handling by archive tools
to retain the .diff.gz and the +t1.diff.gz and different upload
restrictions.

> If you're only talking about the tdeb format per se, 
> I don't understand your reasoning.

Other tools may need to support the locale roots in the .tdeb - these
changes will be easier if the tool can rely on these only existing in
a .tdeb instead of occurring in random .debs but not in others.

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgp3AHIXo2PQI.pgp
Description: PGP signature

Re: Re: DEP-4: The TDeb specification.

[Filipus Klutiero]

Neil Williams wrote:

On Fri, 03 Apr 2009 16:45:46 -0400
Filipus Klutiero  wrote:

[...] 

> > > That would be a nice improvement, but let me suggest another 
> > > implementation. To avoid introducing a second diff, why not updating the 
> > > regular diff (in the case of non-native packages) but indicating that 
> > > the package shouldn't be entirely rebuilt when uploading? This seems 
> > > simpler.

> >
> > That breaks the existing source package in Debian - the .dsc referring
> > to that .diff.gz has been signed by the maintainer and any changes will
> > break that signature.

> Right - unless the .dsc is also updated. And if there is to be a tdiff, 
> I don't see why .dsc-s wouldn't include the tdiff's digest.


There will be a complementary dsc using the +t1 version suffix.

That can work, though I don't see what you were trying to say.

[...]

> > > > > What is the purpose of creating a new binary package format for this (as 
> > > > > opposed to reusing, say, the deb format)?

> > > >
> > > > To support easier management of the translations, including allowing
> > > > users to only install the translations that are needed for one
> > > > particular installation, instead of every user getting every
> > > > translation for every package they install, whether those translations
> > > > are even supported or not.
> >
> > > There are two ways to achieve this. One is per-language packages, the 
> > > other is localization packages with multiple languages but using 
> > > something like dpkg class support. Currently, the only way supported is 
> > > language packages, but introducing a new package format will not 
> > > necessarily allow the second way. Implementing this would take about the 
> > > same amout of work as implementing something like dpkg class support for 
> > > the deb file format.

> >
> > ? It's already implemented via a patch devised by Thomas Viehmann.
> >   
> Great. Then it should be little work to implement the same for .deb.


Package management tools need a way to tell a .deb from a .tdeb - the
two need to be handled differently by tools like dak, britney, apt,
dpkg, reprepro, deb-gview and others.
Do you mean that package management tools need a way to tell a 
traditional/current .deb from a package containing what the DEP proposes 
to put in a .tdeb? If you're only talking about the tdeb format per se, 
I don't understand your reasoning.



--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Fri, 03 Apr 2009 16:45:46 -0400
Filipus Klutiero  wrote:

Please note:

All of the implementation details can wait until after Squeeze - all
that is needed for this release cycle is that apt and dpkg can handle a
migration from Squeeze to Squeeze+1 where packages being upgraded may
use TDebs for the debconf translations. There's quite enough work there
without trying to think beyond Squeeze into issues that may or may not
arise around how the actual TDebs that won't exist until after Squeeze
get handled by other tools not actually involved in allowing migrations
from Squeeze to Squeeze+1.

Please bear this in mind when discussing DEP-4 - I'll expand on the
current Timeline section of the DEP to include some clearer indications
of which stages get done when. (For those who want to know before I
make the update, it will be based on the timescales from my
presentation at Fosdem 2009, PDF and video available.)

http://www.emdebian.org/media/debian-media/slides/TDebs_draft_specification/

http://www.emdebian.org/media/debian-media/

> > > That would be a nice improvement, but let me suggest another 
> > > implementation. To avoid introducing a second diff, why not updating the 
> > > regular diff (in the case of non-native packages) but indicating that 
> > > the package shouldn't be entirely rebuilt when uploading? This seems 
> > > simpler.
> >
> > That breaks the existing source package in Debian - the .dsc referring
> > to that .diff.gz has been signed by the maintainer and any changes will
> > break that signature.

> Right - unless the .dsc is also updated. And if there is to be a tdiff, 
> I don't see why .dsc-s wouldn't include the tdiff's digest.

There will be a complementary dsc using the +t1 version suffix.
The .dsc uploaded by the maintainer cannot be updated without the
maintainer signing it again. However, this is an implementation issue
for after Squeeze.

> > TDebs are not related to source changes and
> > should not affect the existing source package. A TDeb upload needs to
> > sit alongside the existing files in the archive, not replace files
> > uploaded by the maintainer. This allows TDeb uploads during a release
> > freeze - even very late in a release cycle (which is the very time
> > that many debconf translations get updated currently).

> That's the question. I don't see why adding a new diff would ease 
> testing transition compared to updating the existing diff - given that 
> the "traditional"/current binary packages aren't rebuilt when uploading 
> translation updates.

The existing diff is not to be changed because that makes it easier to
be sure that we have the relevant source for the relevant binaries.
ftp-master made it clear at DebConf8 that the source package (the .dsc,
the .diff.gz and .orig.tar.gz or .tar.gz in the case of native
packages) was not to be touched by a TDeb upload. Changes made to
generate the TDeb go into +t1.diff.gz.

Joerg? Mark? Do you want to add some comments on that?

> > > > > What is the purpose of creating a new binary package format for this 
> > > > > (as 
> > > > > opposed to reusing, say, the deb format)?
> > > >
> > > > To support easier management of the translations, including allowing
> > > > users to only install the translations that are needed for one
> > > > particular installation, instead of every user getting every
> > > > translation for every package they install, whether those translations
> > > > are even supported or not.
> >
> > > There are two ways to achieve this. One is per-language packages, the 
> > > other is localization packages with multiple languages but using 
> > > something like dpkg class support. Currently, the only way supported is 
> > > language packages, but introducing a new package format will not 
> > > necessarily allow the second way. Implementing this would take about the 
> > > same amout of work as implementing something like dpkg class support for 
> > > the deb file format.
> >
> > ? It's already implemented via a patch devised by Thomas Viehmann.
> >   
> Great. Then it should be little work to implement the same for .deb.

Package management tools need a way to tell a .deb from a .tdeb - the
two need to be handled differently by tools like dak, britney, apt,
dpkg, reprepro, deb-gview and others.

> > The DEP does describe the organisation of the data.tar.gz:
> > http://dep.debian.net/deps/dep4/#index5h2
> >
> > Which details are missing from the DEP?
> >   
> The data.tar.gz is correctly described, yes. The details I was referring 
> to are the simplifications of "various stages of handling the resulting 
> packages in the repository, in upload rules and in other support tools" 
> you were talking about. These details don't need to be part of the 
> specification, but could be somewhere in the DEP or in its presentation.

Those are the bug reports that I need to start preparing but not
until after Squeeze. ;-)

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarep

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Fri, 3 Apr 2009 18:46:48 +0100
Ian Jackson  wrote:

> Neil Williams writes ("Re: DEP-4: The TDeb specification."):
> > On Tue, 31 Mar 2009 13:06:35 -0400
> > Filipus Klutiero  wrote:
> > > What is the purpose of creating a new binary package format for this (as 
> > > opposed to reusing, say, the deb format)?
> > 
> > To support easier management of the translations, including allowing
> > users to only install the translations that are needed for one
> > particular installation, instead of every user getting every
> > translation for every package they install, whether those translations
> > are even supported or not.
> 
> This would be easier to analyse if the reasoning behind the tdeb
> specific changes was clearly explained.
> 
> Also, are we supposed to be reading this:
>  http://dep.debian.net/deps/dep4/

The DEP, please, yes.

I'll update the DEP with the answers to any questions that arise.

> or this:
>  http://wiki.debian.org/i18n/TranslationDebs

Ah, good point - that's a very old page so not that one. There is some
useful stuff on that so I don't think we should just remove the entire
thing or replace it with the DEP (defeats the point of the DEP to turn
it back into a wiki page with all problems of duplication).

I'll work on some edits tomorrow unless someone else takes the
opportunity.
 
> I think rebuilding the whole package as a binNMU just because of a
> translation change is something that should be avoided.  Translation
> uploads should not be able to change non-translation content.
> 
> One problem that I think you need to address is the
> /var/lib/dpkg/.list files.

Yes, that needs to be folded into the changes within dpkg so that when
a TDeb is installed, the actual filename installed on the system goes
into the /var/lib/dpkg/info/$package-tdeb.list file - the $package.list
file would already have had the relevant content removed because the
version of $package using the TDeb would not contain translated content.

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgpySx6tNr9YS.pgp
Description: PGP signature

Re: Re: DEP-4: The TDeb specification.

[Filipus Klutiero]

On Fri, 03 Apr 2009 04:21:59 -0400
Filipus Klutiero  wrote:

> That would be a nice improvement, but let me suggest another 
> implementation. To avoid introducing a second diff, why not updating the 
> regular diff (in the case of non-native packages) but indicating that 
> the package shouldn't be entirely rebuilt when uploading? This seems 
> simpler.


That breaks the existing source package in Debian - the .dsc referring
to that .diff.gz has been signed by the maintainer and any changes will
break that signature.
  
Right - unless the .dsc is also updated. And if there is to be a tdiff, 
I don't see why .dsc-s wouldn't include the tdiff's digest.

TDebs are not related to source changes and
should not affect the existing source package. A TDeb upload needs to
sit alongside the existing files in the archive, not replace files
uploaded by the maintainer. This allows TDeb uploads during a release
freeze - even very late in a release cycle (which is the very time
that many debconf translations get updated currently).
  
That's the question. I don't see why adding a new diff would ease 
testing transition compared to updating the existing diff - given that 
the "traditional"/current binary packages aren't rebuilt when uploading 
translation updates.
> > > What is the purpose of creating a new binary package format for this (as 
> > > opposed to reusing, say, the deb format)?

> >
> > To support easier management of the translations, including allowing
> > users to only install the translations that are needed for one
> > particular installation, instead of every user getting every
> > translation for every package they install, whether those translations
> > are even supported or not.

> There are two ways to achieve this. One is per-language packages, the 
> other is localization packages with multiple languages but using 
> something like dpkg class support. Currently, the only way supported is 
> language packages, but introducing a new package format will not 
> necessarily allow the second way. Implementing this would take about the 
> same amout of work as implementing something like dpkg class support for 
> the deb file format.


? It's already implemented via a patch devised by Thomas Viehmann.
  

Great. Then it should be little work to implement the same for .deb.

[...]

> > TDebs are based on the .deb format, it is only a small change in the
> > organisation of the data.tar.gz but it simplifies various stages of
> > handling the resulting packages in the repository, in upload rules and
> > in other support tools.

> Well, without details, I can't approve wholeheartedly, but if this isn't 
> reinventing the wheel for no reason, the specification looks fine.


The DEP does describe the organisation of the data.tar.gz:
http://dep.debian.net/deps/dep4/#index5h2

Which details are missing from the DEP?
  
The data.tar.gz is correctly described, yes. The details I was referring 
to are the simplifications of "various stages of handling the resulting 
packages in the repository, in upload rules and in other support tools" 
you were talking about. These details don't need to be part of the 
specification, but could be somewhere in the DEP or in its presentation.

If you're actually asking to see the code, the changes were also put
into http://git.debian.org/?p=users/codehelp/dpkg.git;a=summary
  

No, I'm not asking to see the code...

[...]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: DEP-4: The TDeb specification.

[Ian Jackson]

Neil Williams writes ("Re: DEP-4: The TDeb specification."):
> On Tue, 31 Mar 2009 13:06:35 -0400
> Filipus Klutiero  wrote:
> > What is the purpose of creating a new binary package format for this (as 
> > opposed to reusing, say, the deb format)?
> 
> To support easier management of the translations, including allowing
> users to only install the translations that are needed for one
> particular installation, instead of every user getting every
> translation for every package they install, whether those translations
> are even supported or not.

This would be easier to analyse if the reasoning behind the tdeb
specific changes was clearly explained.

Also, are we supposed to be reading this:
 http://dep.debian.net/deps/dep4/
or this:
 http://wiki.debian.org/i18n/TranslationDebs

I think rebuilding the whole package as a binNMU just because of a
translation change is something that should be avoided.  Translation
uploads should not be able to change non-translation content.

One problem that I think you need to address is the
/var/lib/dpkg/.list files.

Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Fri, 03 Apr 2009 04:21:59 -0400
Filipus Klutiero  wrote:

> That would be a nice improvement, but let me suggest another 
> implementation. To avoid introducing a second diff, why not updating the 
> regular diff (in the case of non-native packages) but indicating that 
> the package shouldn't be entirely rebuilt when uploading? This seems 
> simpler.

That breaks the existing source package in Debian - the .dsc referring
to that .diff.gz has been signed by the maintainer and any changes will
break that signature. TDebs are not related to source changes and
should not affect the existing source package. A TDeb upload needs to
sit alongside the existing files in the archive, not replace files
uploaded by the maintainer. This allows TDeb uploads during a release
freeze - even very late in a release cycle (which is the very time
that many debconf translations get updated currently).

> > > What is the purpose of creating a new binary package format for this (as 
> > > opposed to reusing, say, the deb format)?
> >
> > To support easier management of the translations, including allowing
> > users to only install the translations that are needed for one
> > particular installation, instead of every user getting every
> > translation for every package they install, whether those translations
> > are even supported or not.

> There are two ways to achieve this. One is per-language packages, the 
> other is localization packages with multiple languages but using 
> something like dpkg class support. Currently, the only way supported is 
> language packages, but introducing a new package format will not 
> necessarily allow the second way. Implementing this would take about the 
> same amout of work as implementing something like dpkg class support for 
> the deb file format.

? It's already implemented via a patch devised by Thomas Viehmann.
Sadly, due to other pressures, his tree on git.debian.org has been
removed.

> > TDebs are based on the .deb format, it is only a small change in the
> > organisation of the data.tar.gz but it simplifies various stages of
> > handling the resulting packages in the repository, in upload rules and
> > in other support tools.

> Well, without details, I can't approve wholeheartedly, but if this isn't 
> reinventing the wheel for no reason, the specification looks fine.

The DEP does describe the organisation of the data.tar.gz:
http://dep.debian.net/deps/dep4/#index5h2

Which details are missing from the DEP?

If you're actually asking to see the code, the changes were also put
into http://git.debian.org/?p=users/codehelp/dpkg.git;a=summary

However, I'm useless with git and my git 'thing' appears broken as far
as I can tell. I can't see how or why or how to fix it, so I'll update
the TDeb changes to the latest dpkg source next weekend and just stick
with patches or subversion - at least that works.

(Please, git-fanboys, don't start a sub-thread/flamewar abusing me for
my obvious idiocy in being unable to comprehend git - I hate it, I
apparently can't use it and I would simply prefer to actually get
things done without having to fight it.)

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgpqF28fIeSlJ.pgp
Description: PGP signature

Re: Re: DEP-4: The TDeb specification.

[Filipus Klutiero]

Neil Williams wrote:


On Tue, 31 Mar 2009 13:06:35 -0400
Filipus Klutiero  wrote:

> Neil Williams wrote:
> > Primary Motivations (in order):
> >1. Updates to translations should not require source NMU's.
> >   
> I guess that means avoiding to NMU with new diff.gz -s? If so, what are 
> the underlying motivations?


To prevent translation updates (particularly debconf ones) from
interfering with existing transitions and release cycles. There is no
need to rebuild the source of the package merely to update or add a new
translation. The current barriers to such rebuilds can be overcome with
adapted tools, as described in the DEP.

http://dep.debian.net/deps/dep4/#index10h2

There is no good reason to permit translation updates to cause the
binary packages to be changed by recompiling the source. Dependency
versions change, new bugs can be introduced, existing transitions get
more complicated and the one time when translation updates are most
useful is during a release freeze when the binaries must not change.
The whole point of i18n (the process of making a package support
localisation by internationalising the codebase) is to ensure that the
same compiled binary can operate with any compatible translation. This
allows users to set LANG= and get the localised version at runtime.
Requiring that translations can only be added or updated in Debian by
recompiling the source is a complete waste of effort and is something
that gettext and other l10n middle-ware are expressly designed to
avoid. Debian, currently, is simply not getting the best out of l10n
and is making life difficult for everyone by putting the translated
files in the same packages as the compiled binaries. Each release
cycle, this comes back to bite us. TDebs will remove this burden.
  
That would be a nice improvement, but let me suggest another 
implementation. To avoid introducing a second diff, why not updating the 
regular diff (in the case of non-native packages) but indicating that 
the package shouldn't be entirely rebuilt when uploading? This seems 
simpler.
> What is the purpose of creating a new binary package format for this (as 
> opposed to reusing, say, the deb format)?


To support easier management of the translations, including allowing
users to only install the translations that are needed for one
particular installation, instead of every user getting every
translation for every package they install, whether those translations
are even supported or not.
There are two ways to achieve this. One is per-language packages, the 
other is localization packages with multiple languages but using 
something like dpkg class support. Currently, the only way supported is 
language packages, but introducing a new package format will not 
necessarily allow the second way. Implementing this would take about the 
same amout of work as implementing something like dpkg class support for 
the deb file format.

 The current model is based on server
installations where lots of locales may be configured to support
localisation of web content. Installing all 90 locales does not
translate well to desktop installations with only a handful of users
and maybe 5 or 6 languages. On a typical Debian box, /usr/share/locale/
takes up 250Mb when each language only requires a maximum of 9Mb.
  
I'm not sure why you write that the current model is based on server 
installations. The current "model" I see is just the "natural" state of 
things, with no modularization WRT translations in the vast majority of 
packages. But yeah, it would be great to save this space.

TDebs are based on the .deb format, it is only a small change in the
organisation of the data.tar.gz but it simplifies various stages of
handling the resulting packages in the repository, in upload rules and
in other support tools.
Well, without details, I can't approve wholeheartedly, but if this isn't 
reinventing the wheel for no reason, the specification looks fine.



--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Tue, 31 Mar 2009 13:06:35 -0400
Filipus Klutiero  wrote:

> Neil Williams wrote:
> > Primary Motivations (in order):
> >1. Updates to translations should not require source NMU's.
> >   
> I guess that means avoiding to NMU with new diff.gz -s? If so, what are 
> the underlying motivations?

To prevent translation updates (particularly debconf ones) from
interfering with existing transitions and release cycles. There is no
need to rebuild the source of the package merely to update or add a new
translation. The current barriers to such rebuilds can be overcome with
adapted tools, as described in the DEP.

http://dep.debian.net/deps/dep4/#index10h2

There is no good reason to permit translation updates to cause the
binary packages to be changed by recompiling the source. Dependency
versions change, new bugs can be introduced, existing transitions get
more complicated and the one time when translation updates are most
useful is during a release freeze when the binaries must not change.
The whole point of i18n (the process of making a package support
localisation by internationalising the codebase) is to ensure that the
same compiled binary can operate with any compatible translation. This
allows users to set LANG= and get the localised version at runtime.
Requiring that translations can only be added or updated in Debian by
recompiling the source is a complete waste of effort and is something
that gettext and other l10n middle-ware are expressly designed to
avoid. Debian, currently, is simply not getting the best out of l10n
and is making life difficult for everyone by putting the translated
files in the same packages as the compiled binaries. Each release
cycle, this comes back to bite us. TDebs will remove this burden.

> What is the purpose of creating a new binary package format for this (as 
> opposed to reusing, say, the deb format)?

To support easier management of the translations, including allowing
users to only install the translations that are needed for one
particular installation, instead of every user getting every
translation for every package they install, whether those translations
are even supported or not. The current model is based on server
installations where lots of locales may be configured to support
localisation of web content. Installing all 90 locales does not
translate well to desktop installations with only a handful of users
and maybe 5 or 6 languages. On a typical Debian box, /usr/share/locale/
takes up 250Mb when each language only requires a maximum of 9Mb.

TDebs are based on the .deb format, it is only a small change in the
organisation of the data.tar.gz but it simplifies various stages of
handling the resulting packages in the repository, in upload rules and
in other support tools.

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgpXwK3DnwMRG.pgp
Description: PGP signature

Re: DEP-4: The TDeb specification.

[Filipus Klutiero]

Neil Williams wrote:

Primary Motivations (in order):
   1. Updates to translations should not require source NMU's.
  
I guess that means avoiding to NMU with new diff.gz -s? If so, what are 
the underlying motivations?


What is the purpose of creating a new binary package format for this (as 
opposed to reusing, say, the deb format)?



--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Thu, 19 Mar 2009 10:35:59 +0100
Jan Hauke Rahm  wrote:

> On Wed, Mar 18, 2009 at 10:31:29AM +, Neil Williams wrote:
> > Why should 3.0 be any more difficult than 1.0 or anything that follows?
> > (Not that I have any particular desire to use 3.0 or quilt myself.) 3.0
> > has to deal with incorporating patches and changes from the BTS, so
> > +t1.diff.gz is no different. In Squeeze+1, the changes are restricted
> > to debian/po or po/ for native Debian packages so there is no need to
> > do anything with 3.0 until Squeeze+2.
> 
> So you already see a need to change TDebs again?

No, just extending the use case from an initial debconf-only to other
packages in subsequent releases.
 
> I'm definitely not on your level of dpkg knowledge, so please forgive me
> if I'm talking nonsense here. But I feel like introducing a new diff.gz
> after finally getting rid of those with 3.0 (quilt) is a bit of a
> regression. 

The whole point of TDebs is that there is *no change* to the source
package. The generation of TDebs needs to be separate from whatever
methods are used to generate the source package or rebuild the package.
It therefore has to use a separate diff/tarball that is outside the
existing Debian source package - otherwise, the archive becomes
inconsistent.

This isn't an issue to be fixed in dpkg, it's more of an issue for
ftp-master.

> Can't dpkg-source sort out the po files [1] and put them in
> an additional package-version.tdebian.tar.(gz|bz2|lzma)?

How does that differ from +t1.diff.gz ? - bearing in mind that nothing
about generated +t1.diff.gz is allowed to modify the source package
already in Debian. Translators deserve a full diff, not some tarball of
PO files without context.

> That way i18n
> and translators can always download all translation files without any
> code, then change them if needed and upload again (the tdeb would have
> to be built before, of course).

That would happen anyway with +t1.diff.gz - however - it is not good to
only package the PO or POT file and expect translators to be happy with
their lot. Translations often need to be tested and translators often
benefit from having the source code around to understand messages that
the upstream has not made particularly clear from their perspective.
Translators always need to have the full source available. Forcing
translators to only work from the PO or POT is tantamount to assuming
that translators never have any understanding of the mysteries of source
code and shouldn't need to see it. Many translators are also upstream
developers for other projects, there is no reason to deny them the full
source code to provide context and identify bugs. (e.g. strings that
contain translatable content but were not marked for translation and
therefore never appear in the PO or POT file.)

> That would make tdebs only work with
> source format 3.0 but since it's going to be default soon I don't see a
> problem (but maybe the entire suggestion is nonsense :)).

I don't see that there is a need for a change simply because of 3.0.

> > What matters is that the maintainer gets the +t1.diff.gz and applies it
> > onto the source package prior to the next upload. It's no different to
> > how the same maintainer would handle a patch or new translations
> > file sent to the BTS. I'm quite sure various tools and scriptage will
> > be devised to help with different workflow patterns before Squeeze+2.
> 
> How is the maintainer supposed to get the +t1.diff.gz? Have dget, apt
> et alii have to deal with multiple +tX.diff.gz or should the maintainer
> look those up at PTS? I don't see the workflow here (and again I'd see a
> win in a seperate tarball as suggested above since it's always one
> tdebian.tar.compr).

There's no need to see the workflow for this in any detail at this stage
- it doesn't apply until after Squeeze. However, the +t1.diff.gz will be
in the archive as normal, the various tools can be extended to look for
and apply the +t1.diff.gz

> Huh, thinking of it... A new tdeb uploaded by a translator overwrites
> the old one? Then the +tX.diff.gz can be regenerated from the tdeb,
> right?

It's not regenerated, foo_1.2.3-4+t2_all.tdeb replaces foo_1.2.3-4
+t1_all.tdeb in the same upload that replaces foo_1.2.3-4+t1.diff.gz
with foo_1.2.3-4+t2.diff.gz but the +t2.diff.gz is built by the
nominated person doing the upload on behalf of one or more translation
teams.

> [1] You have a pretty specific pattern to work on, right?
> ${top_srcdir}/po/
> ${top_srcdir}/po-*/

The first target is debian/po/ for debconf but the point is moot, the
'target' to which you refer should be '-R *' for reasons explained
above. 

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgplJVggaMjQ3.pgp
Description: PGP signature

Re: DEP-4: The TDeb specification.

[Jan Hauke Rahm]

On Wed, Mar 18, 2009 at 10:31:29AM +, Neil Williams wrote:
> Why should 3.0 be any more difficult than 1.0 or anything that follows?
> (Not that I have any particular desire to use 3.0 or quilt myself.) 3.0
> has to deal with incorporating patches and changes from the BTS, so
> +t1.diff.gz is no different. In Squeeze+1, the changes are restricted
> to debian/po or po/ for native Debian packages so there is no need to
> do anything with 3.0 until Squeeze+2.

So you already see a need to change TDebs again?

I'm definitely not on your level of dpkg knowledge, so please forgive me
if I'm talking nonsense here. But I feel like introducing a new diff.gz
after finally getting rid of those with 3.0 (quilt) is a bit of a
regression. Can't dpkg-source sort out the po files [1] and put them in
an additional package-version.tdebian.tar.(gz|bz2|lzma)? That way i18n
and translators can always download all translation files without any
code, then change them if needed and upload again (the tdeb would have
to be built before, of course). That would make tdebs only work with
source format 3.0 but since it's going to be default soon I don't see a
problem (but maybe the entire suggestion is nonsense :)).

> What matters is that the maintainer gets the +t1.diff.gz and applies it
> onto the source package prior to the next upload. It's no different to
> how the same maintainer would handle a patch or new translations
> file sent to the BTS. I'm quite sure various tools and scriptage will
> be devised to help with different workflow patterns before Squeeze+2.

How is the maintainer supposed to get the +t1.diff.gz? Have dget, apt
et alii have to deal with multiple +tX.diff.gz or should the maintainer
look those up at PTS? I don't see the workflow here (and again I'd see a
win in a seperate tarball as suggested above since it's always one
tdebian.tar.compr).
Huh, thinking of it... A new tdeb uploaded by a translator overwrites
the old one? Then the +tX.diff.gz can be regenerated from the tdeb,
right?

I better stop here and wait for you to rip me up.

Hauke

[1] You have a pretty specific pattern to work on, right?
${top_srcdir}/po/
${top_srcdir}/po-*/


signature.asc
Description: Digital signature

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Wed, 18 Mar 2009 19:13:03 +0900
Charles Plessy  wrote:

> Le Wed, Mar 18, 2009 at 07:25:40AM +, Neil Williams a écrit :
> > 
> > Maintainers will be creating TDebs in Squeeze+1, using debian/rules,
> > using debhelper calls and uploading TDebs each time they would
> > currently upload any package that contains /usr/share/locale/LC_*/ etc.
> > Those TDebs are, effectively, +t0 - only updates by translators start
> > the +t1 sequence.
> > 
> > Maintainer uploads:
> > foo_1.2.3-4_amd64.deb
> > foo-tdeb_1.2.3-4_all.tdeb
> > foo-bar_1.2.3-4_amd64.deb
> > foo_1.2.3-4.diff.gz
> > foo_1.2.3.orig.tar.gz
> > foo_1.2.3-4.dsc
> > foo_1.2.3-4_amd64.changes
> 
> Hi Neil,
> 
> how do you coordinate your action with the project of using the source format
> '3.0 (quilt)' by default in Squeeze +1?

Why should 3.0 be any more difficult than 1.0 or anything that follows?
(Not that I have any particular desire to use 3.0 or quilt myself.) 3.0
has to deal with incorporating patches and changes from the BTS, so
+t1.diff.gz is no different. In Squeeze+1, the changes are restricted
to debian/po or po/ for native Debian packages so there is no need to
do anything with 3.0 until Squeeze+2.

What matters is that the maintainer gets the +t1.diff.gz and applies it
onto the source package prior to the next upload. It's no different to
how the same maintainer would handle a patch or new translations
file sent to the BTS. I'm quite sure various tools and scriptage will
be devised to help with different workflow patterns before Squeeze+2.

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgpQnfJp71RjA.pgp
Description: PGP signature

Re: DEP-4: The TDeb specification.

[Charles Plessy]

Le Wed, Mar 18, 2009 at 07:25:40AM +, Neil Williams a écrit :
> 
> Maintainers will be creating TDebs in Squeeze+1, using debian/rules,
> using debhelper calls and uploading TDebs each time they would
> currently upload any package that contains /usr/share/locale/LC_*/ etc.
> Those TDebs are, effectively, +t0 - only updates by translators start
> the +t1 sequence.
> 
> Maintainer uploads:
> foo_1.2.3-4_amd64.deb
> foo-tdeb_1.2.3-4_all.tdeb
> foo-bar_1.2.3-4_amd64.deb
> foo_1.2.3-4.diff.gz
> foo_1.2.3.orig.tar.gz
> foo_1.2.3-4.dsc
> foo_1.2.3-4_amd64.changes

Hi Neil,

how do you coordinate your action with the project of using the source format
'3.0 (quilt)' by default in Squeeze +1?

Have a nice day,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Wed, 18 Mar 2009 00:26:30 -0500
Manoj Srivastava  wrote:

> On Tue, Mar 17 2009, Neil Williams wrote:
> 
> >>  Do you prevent mixing old and new .debs and .tdebs?
> >
> > Changes to translations use +t1.diff.gz etc.
> >
> >>  How do you merge data from a new package into the tdeb data?
> >
> > The real question is how to get apt to understand getting the
> > +t1.diff.gz when asked for apt-get source and for dpkg to expect to
> > unpack +t1.diff.gz etc.
> 
> Also, currently one may put up a new version of a package into
>  experimental or people.debian.org; and it carries with it the older
>  translations. If the translations are separated, then the people.d.o
>  package will be useful only for testing by the speakers of the original
>  langiage the debconf templates were written in.  This seems like a step
>  back

No, the initial TDeb will be generated by the maintainer, effectively
+t0, so the maintainer uploads the initial TDeb containing whatever
translations are currently supported, putting the TDeb wherever you
put the binary and .dsc. It is up to the maintainer to incorporate any
+t1.diff.gz containing updated or new translations that may exist
already into each new Debian version. The other repository could also
simply copy any existing TDeb that does already exist alongside the test
package. (Being Arch:all, the TDeb is only included in the maintainer
upload or subsequent translator updates, buildd's don't need to worry
about TDebs at all.)

If the new version has changed translated strings then those will only
available in English until the +t1 TDeb arrives anyway - that doesn't
change. (English is always the language in which the templates must be
written initially - en_US at that.) There remains the current advice
that changes to the templates should always seek translation updates
prior to the upload of the initial TDeb by the maintainer. If
maintainers implement a string freeze and wait for translation updates
before uploading, the chances of a +t1.diff.gz being required by time
of the next release by the maintainer are lower.

Just like the dependencies of a package in experimental, the TDeb is
still available and can be updated by translators, the +t1.diff.gz is
available to the maintainer to incorporate into any new version and
debian/rules needs to include support for generating the TDeb for each
new version.

Maintainers will be creating TDebs in Squeeze+1, using debian/rules,
using debhelper calls and uploading TDebs each time they would
currently upload any package that contains /usr/share/locale/LC_*/ etc.
Those TDebs are, effectively, +t0 - only updates by translators start
the +t1 sequence.

Maintainer uploads:
foo_1.2.3-4_amd64.deb
foo-tdeb_1.2.3-4_all.tdeb
foo-bar_1.2.3-4_amd64.deb
foo_1.2.3-4.diff.gz
foo_1.2.3.orig.tar.gz
foo_1.2.3-4.dsc
foo_1.2.3-4_amd64.changes

The foo-tdeb package will be listed in the .changes anyway so existing
tools will simply add it to the list of files to be uploaded to
ftp-master or wherever. foo-tdeb_1.2.3-4_all.tdeb is, effectively,
foo-tdeb_1.2.3-4+t0_all.tdeb

Translator updates arrive:
foo-tdeb_1.2.3-4+t1_all.tdeb
foo_1.2.3-4+t1.diff.gz
foo_1.2.3-4+t1.dsc
foo_1.2.3-4+t1_all.changes

Maintainer makes a new release foo_1.2.3-5 which incorporates the TDeb
changes in a similar manner to how an NMU is included today. All files
matching foo*1.2.3-4* are removed by dak when the new version is
uploaded. The updated translations now exist in
foo-tdeb_1.2.3-5_all.tdeb - uploaded by the maintainer and there is no
+t1.diff.gz or +t1_all.tdeb until the package translations need to be
touched again.

The key point is that a +t1 revision can happen *during a release
freeze* without touching the source, without changing any of the
binaries. Once the release is out and unstable is accessible again, the
maintainer adds +t1.diff.gz to their next upload. Done.

(Interesting point, if the current Debian version has already migrated
to testing, the +t1 TDeb will also need to migrate with the +t1.diff.gz
as source. Whether the rules should change for a TDeb migration needs
to be decided - it could be that TDebs have a shorter time in unstable
if the relevant version of the package is already in testing or even
be allowed to migrate immediately. That's all for Squeeze+1.)

The system is quite similar to how an NMU is currently done - without
the problems of a source rebuild - and maintainers can handle the +t1
in much the same manner to fit into their workflow. Whether a changelog
entry is needed is undecided, it would be courteous but not necessarily
mandatory, especially if no bug exists to be closed.

What remains to be resolved is how best to fit that
"maintainer incorporates the changes from the TDeb" stage is handled so
that it doesn't disturb maintainer workflow too much. The changes will
only be in the po/ files and consist of simply unpacking the
+t1.diff.gz on top of your existing checkout (and committing any
changes if you use an RCS).

None of this

Re: DEP-4: The TDeb specification.

[Manoj Srivastava]

On Tue, Mar 17 2009, Neil Williams wrote:


>>  Do you prevent mixing old and new .debs and .tdebs?
>
> Changes to translations use +t1.diff.gz etc.
>
>>  How do you merge data from a new package into the tdeb data?
>
> The real question is how to get apt to understand getting the
> +t1.diff.gz when asked for apt-get source and for dpkg to expect to
> unpack +t1.diff.gz etc.

Also, currently one may put up a new version of a package into
 experimental or people.debian.org; and it carries with it the older
 translations. If the translations are separated, then the people.d.o
 package will be useful only for testing by the speakers of the original
 langiage the debconf templates were written in.  This seems like a step
 back

manoj
-- 
Keep your eyes wide open before marriage, half shut afterwards. Benjamin
Franklin
Manoj Srivastava    
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: DEP-4: The TDeb specification.

[Neil Williams]

On Wed, 18 Mar 2009 00:22:19 +0100
Loïc Minier  wrote:

> > Current HTML form: http://people.debian.org/~codehelp/tdeb/
> 
>  So do I understand correctly that there might be one .tdeb per package
>  and per language unless maintainers merge regularly the contents into
>  their source packages?  How many .tdebs files does that imply in the
>  main archive?

No, updates to translations will update the existing TDeb, creating
+t2.diff.gz and +t3.diff.gz etc. All supported languages go into the
existing TDeb, organised by locale root.

Unless a package needs more than one TDeb for the debconf+large_docs
corner case, each source package should only expect to have one TDeb
for all binary packages and all locales.

Translation teams can work together to make uploads in a coordinated
manner - similar to the current method of requesting deadlines for i18n
bugs, a nominated person can collate the various translations prior to
a deadline chosen by the teams themselves, according to the needs of
that particular package.

Updated TDebs use the +t1, +t2 version suffix etc.

http://people.debian.org/~codehelp/tdeb/ch2.html

>  It seems to me fetching multiple .tdeb for perhaps every .deb to update
>  has the potential to slow down an update by a huge factor, not only in
>  size of the tdeb index but for the amount of roundtrips.

Generally, there will be only one TDeb.

>  Do you prevent mixing old and new .debs and .tdebs?

Changes to translations use +t1.diff.gz etc.

>  How do you merge data from a new package into the tdeb data?

The real question is how to get apt to understand getting the
+t1.diff.gz when asked for apt-get source and for dpkg to expect to
unpack +t1.diff.gz etc.

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgpMQEc6KICuJ.pgp
Description: PGP signature

Re: DEP-4: The TDeb specification.

[Loïc Minier]

> Current HTML form: http://people.debian.org/~codehelp/tdeb/

 So do I understand correctly that there might be one .tdeb per package
 and per language unless maintainers merge regularly the contents into
 their source packages?  How many .tdebs files does that imply in the
 main archive?


 It seems to me fetching multiple .tdeb for perhaps every .deb to update
 has the potential to slow down an update by a huge factor, not only in
 size of the tdeb index but for the amount of roundtrips.


 Do you prevent mixing old and new .debs and .tdebs?


 How do you merge data from a new package into the tdeb data?

-- 
Loïc Minier


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

DEP-4: The TDeb specification.

[Neil Williams]

(No need to CC: me on either debian-devel or debian-i18n, thanks.)

Time to launch the debate on the Draft TDeb Specification - DEP-4.

Current HTML form: http://people.debian.org/~codehelp/tdeb/

SGML form for modification: svn://svn.debian.org/dep/web/deps/
(Draft.sgml - please notify me if the SGML is modified so that I can
update the HTML generated from the SGML.)

DEP form: http://dep.debian.net/deps/dep4/
(not yet fully automated from SVN commits, hence not yet the preferred
form for modification - can be mostly generated from the SGML.)

Date: March 2009
Drivers: Neil Williams 
 Joerg Jaspert 
 Thomas Viehmann 
 Mark Hymers 
 Frank Lichtenheld 

Abstract: This document provides an overview of the TDeb format, TDeb
 design and usage. This specification should be considered as a work in
 progress. 

Purpose: The aim of the DEP is to improve and ratify sufficient
portions of the specification to allow Squeeze to be released with
support for TDebs in core packaging tools, ready for the creation of
the first TDebs in Squeeze+1. 

Primary focus: Tool support in Squeeze. Initially, only debconf
translations will be targeted for TDebs in Squeeze+1 - package
translations for native packages should also be possible in Squeeze+1
but other translations are not likely to be implemented until 
Squeeze+2. Please keep the DEP concentrating on the tools, not the later
implementation. There will be time to improve support in Squeeze+1 for
corner cases arising from packages looking for TDebs in Squeeze+2.

Primary Motivations (in order):
   1. Updates to translations should not require source NMU's.
   2. Translation data should not be distributed in
architecture-dependent packages.
   3. Translators should have a common interface for getting updates
into Debian (possibly with automated TDeb generation after i18n team
review).

Secondary Benefits:
   a) users will be able to selectively install package localisation
according to the supported locales on individual machines and Debian
will keep all such selective installations updated automatically,
including adding/removing translations for existing packages when the
locale configuration is modified.
   b) easier, quicker and more thoroughly
reviewed translations of debconf, native package strings and other
translations.

Resources:
 i:
http://meetings-archive.debian.net/pub/debian-meetings/2009/fosdem/slides/TDebs_draft_specification/tdeb-fosdem-2009.pdf
 ii:
http://meetings-archive.debian.net/pub/debian-meetings/2009/fosdem/low/TDebs_draft_specification.ogg
 
 iii:
http://meetings-archive.debian.net/pub/debian-meetings/2009/fosdem/high/TDebs_draft_specification.ogg

Code: git://git.debian.org/git/users/tviehmann/dpkg
git://git.debian.org/git/users/codehelp/debhelper

ToDo: +t1.diff.gz support - i.e. allowing dpkg and apt to look for and
apply the changes made by translators - only some elements of this need
to be implemented in Squeeze. apt also needs to understand locales and
download the relevant TDeb alongside the package so that
apt-extracttemplates can offer translated debconf questions upon first
installation.

Other packages: reprepro, langupdate (in NEW), ...

Limitations: At this stage, there is no need to discuss the finer
points of TDebs as they pertain to implementations not possible until
*after* the release of Squeeze, EXCEPT where those points have a direct
bearing on how tools like dpkg and apt handle tdebs in the
Squeeze->Squeeze+1 migrations. For this reason, there doesn't need to
be any definitive agreement about how TDebs will work with upstream
translations, KDE or non-gettext translations or anything else relating
to the actual use of TDebs after Squeeze during the ratification of
DEP-4 itself.

Relation to Emdebian: Emdebian has thousands of TDebs with a slightly
different emphasis - although you are welcome to use the
http://www.emdebian.org/locale/ repository as a test case, bear in mind
that Debian TDebs will differ internally and in numbers. Note also that
reprepro needs to support TDebs - it currently forces a renaming
to .deb - a bug report will follow in due course.

Numbers: The DebConf8 and Extremadura meetings established that the
number of TDebs does need to be careully managed, therefore TDebs will
be architecture-independent and source-package based. Some TDebs will
be very, very small - others could be quite large indeed. There is
scope in the specification for multiple TDebs where source packages
have debconf support and large amounts of translated documentation.

Policy: Elements of DEP-4 can be migrated into Policy discussions and
patches in due course - how that is actually done is up to maintainers
of debian-policy. 

Discuss.

:-)

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/



pgpMq2ktJZKwO.pgp
Description: PGP signature