Re: Thoughts about tdebs
On Wed, 3 Dec 2008 16:32:17 +0100 Raphael Hertzog <[EMAIL PROTECTED]> wrote: (I'm subscribed, no need to CC me) > Anyway, having translation outside of the original package also > represent a cost: > - manually installing a software requires download of 2 packages with > the risk of badly mixing them > - in term of preparation for the packager, he has to say what's part > of the tdeb and what's not > - in term of administration for the repository manager True, but the Debian repository manager (ftp-master) is already happy with the benefits of having translations completely outside the original package. (Yes, ftp-master recognises the TDebs are a lot of work and I got quite a few complaints from Joerg and Mark about how much work was involved but still, after all that, TDebs are still worthwhile for ftp-master.) With regard to NEW, I'm sure we can arrange that during Squeeze+1, uploads that merely add a TDeb have a fast-track through NEW - as long as that this track is clearly identified and not abused. Plenty of time to arrange this. http://people.debian.org/~codehelp/tdeb/ch9.html#s9.1 In terms of the packager, dh_gentdeb includes sufficient support for 95% of all packages using gettext to not have to care about what goes into the tdeb, it is done automatically. This has been done specifically to remove the need to modify the source package. See http://people.debian.org/~codehelp/tdeb/ch6.html http://git.debian.org/?p=users/codehelp/debhelper.git;a=summary The other support can be developed during Squeeze. The intention is that migrating any package to TDeb support is mostly trivial: http://people.debian.org/~codehelp/tdeb/ch6.html#s6.3 (Just noticed that there is a typo there, the first two list items need to be collapsed into one.) Manually installing is not the way it will work. Think about debconf templates - the translations need to exist before the package is configured, therefore tdeb support implicitly means that apt will need to be taught to get the TDeb alongside the package so that apt-extracttemplates (from debconf) can locate the templates file and use it with the maintainer scripts that are retained in the original package. That is a relatively simple change - the TDeb is already listed in debian/control and apt just needs to understand that Package-Type: tdeb means "get this TDeb alongside anything else from this source package". Whether untranslated debconf questions are retained in the original package is a matter to be decided (for dpkg -i support). "Badly mixing them?" - that is a matter for apt to reconcile and dependencies to resolve. I don't see how the current spec would allow for packages to be badly mixed but there is plenty of time to find the corner cases as TDebs themselves will not arrive in the archive until after the release of Squeeze. Quite a lot of work was done on how the packages would combine during Extremadura. I don't foresee any problems that the existing support cannot resolve. (Is that not in the spec? If it isn't, I'll need to add something on that.) http://people.debian.org/~codehelp/tdeb/ch7.html#s7.2 If that is unclear, please tell me what might need to be added for clarification. > So while I agree that it would be good to resolve the problem of the > translators, I don't think that it should be done at all costs. ftp-master agrees, as do I. However, TDebs are still desirable, as outlined in the spec. > In > particular when the Emdebian-specific size-problem gets less and less > real over the years (for my own semi-embedded projects, I have been > forced to buy larger and larger DOM over the years as the smaller > capacity are simply not produced any more). I come across this sentiment a lot but it just isn't true. The NSLU2 has 6.4Mb available for the operating system so that it can finally have the ability to change the external hard drive without rebooting and also being able to use external storage that has not had to be (extensively) reconfigured to support the OS. There are always situations where the size of the installed OS is critical - hence Emdebian Crush. Other devices out there do exist and there are developers on the embedded mailing lists who are crying out for Debian to support their machines - machines with generally between 10Mb and 64Mb available storage and *no* way of extending that storage without completely redesigning the board. Debian is not the universal OS until it can be adapted to such systems. Current Debian packaging insists that the hardware be expanded to meet the requirements of the software which, to me, is a Microsoft approach. I cannot meet the demands of these users and developers currently because of the Lenny freeze - TDebs are a small but vital part of this solution. For intermediate installations, Emdebian Grip is more suitable - anything with more than about 250Mb total storage. What you might call semi-embedded, again using TDebs as the key instrument to reduce the installation size and download s
Re: Thoughts about tdebs
On Wed, 03 Dec 2008, Neil Williams wrote: > On Wed, 3 Dec 2008 14:28:58 + > Mark Brown <[EMAIL PROTECTED]> wrote: > > > > >From what I understand of your idea, you seem to think about > > > translations mainly as updates. While it is one of the goals to > > > enable translation-only updates, it is not quite obvious to me what > > > your proposal has to offer in terms of splitting translations out > > > of the debs, which is an explicit goal. Also, the proposal > > > specifically aims at limiting the amount of data that the archive > > > and apt have to handle. > > > > Surely translations can be modeled as updates to translationless .deb > > files (ie, you have a .deb with no translations and then patch that > > package to add the translations)? > > Why patch the package? Why not simply put a TDeb alongside the > "translationless" Debian package? The internal layout of the TDeb is > important to the overall usefulness of TDebs in general. Why? If you have rules to generate the internal layout, you can most certainly adapt those rules to apply them at some other point of the distribution mechanism. Anyway, having translation outside of the original package also represent a cost: - manually installing a software requires download of 2 packages with the risk of badly mixing them - in term of preparation for the packager, he has to say what's part of the tdeb and what's not - in term of administration for the repository manager So while I agree that it would be good to resolve the problem of the translators, I don't think that it should be done at all costs. In particular when the Emdebian-specific size-problem gets less and less real over the years (for my own semi-embedded projects, I have been forced to buy larger and larger DOM over the years as the smaller capacity are simply not produced any more). Whatever experiment you might have done within Emdebian, I will not trust any design decision that lacks a (serious) rationale and explanations of why the alternatives are not viable. (That said, it's only my opinion and I'm not the one that maintains the C part of dpkg) Cheers, -- Raphaël Hertzog Le best-seller français mis à jour pour Debian Etch : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Thoughts about tdebs
On Wed, 3 Dec 2008 14:28:58 + Mark Brown <[EMAIL PROTECTED]> wrote: > > >From what I understand of your idea, you seem to think about > > translations mainly as updates. While it is one of the goals to > > enable translation-only updates, it is not quite obvious to me what > > your proposal has to offer in terms of splitting translations out > > of the debs, which is an explicit goal. Also, the proposal > > specifically aims at limiting the amount of data that the archive > > and apt have to handle. > > Surely translations can be modeled as updates to translationless .deb > files (ie, you have a .deb with no translations and then patch that > package to add the translations)? Why patch the package? Why not simply put a TDeb alongside the "translationless" Debian package? The internal layout of the TDeb is important to the overall usefulness of TDebs in general. Updates to the TDeb then simply replace the previous TDeb, leaving the .dsc and the rest of the (signed) Debian files completely unchanged. AFAICT it is not possible to "patch" a binary debian package - you lose all the benefit of signing the .dsc in the first place. -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ pgpxlqSVXll3i.pgp Description: PGP signature
Re: Thoughts about tdebs
On Tue, Dec 02, 2008 at 05:51:30PM +0100, Thomas Viehmann wrote: > Raphael Hertzog wrote: > > I guess you understand better the idea by now. This needs more thoughts > > but I wanted to share it so that you can think about it too. > >From what I understand of your idea, you seem to think about > translations mainly as updates. While it is one of the goals to enable > translation-only updates, it is not quite obvious to me what your > proposal has to offer in terms of splitting translations out of the > debs, which is an explicit goal. Also, the proposal specifically aims at > limiting the amount of data that the archive and apt have to handle. Surely translations can be modeled as updates to translationless .deb files (ie, you have a .deb with no translations and then patch that package to add the translations)? -- "You grabbed my hand and we fell into it, like a daydream - or a fever." -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Thoughts about tdebs
On Wed, 3 Dec 2008 09:12:46 +0100 Raphael Hertzog <[EMAIL PROTECTED]> wrote: BTW, it was Eddy Petrisor, not Eric - typo. Sorry Eddy. (Thanks, Christian.) > On Tue, 02 Dec 2008, Neil Williams wrote: > > > It is a very sizable special case, though, and growing thanks to > > > the tireless translation efforts within and beyond Debian. This > > > is why the consensus at the meeting seemed to be to get this > > > right even if it means having to implement some special casing. > > > > Raphael - I think you've missed some of the purpose behind TDebs > > and if that is a fault of the current draft TDeb spec, then I would > > value your comments on how to fix it. > > I hear the size problem that Emdebian is facing but Tdeb is not the > right solution for this either IMO. We should rather aim at having > some DEB_BUILD_OPTIONS or something similar to instruct the build > process to create the minimalist package that you need instead of > dropping the translation entirely from the original .deb. We already have DEB_BUILD_OPTIONS to instruct the build process to create the minimalist package but those don't meet the need for TDebs. It isn't "the translation" it is dozens of translations per package - a single package can contain 30 or 40 .mo files, between 7k and 30kb each. An embedded device does not need all the translations, it needs one, two, maybe four or five at most, carefully matched against the locales supported on that one specific device. The rest are a complete waste of storage because storage space is not cheap. Emdebian users need control over which translations are installed - absolute control - without bloating the download sizes of the packages themselves. Therefore, Emdebian Crush splits out each .mo file into a dedicated .tdeb package and provides tools to match the package against the supported locales and the installed package list. No other component of a Debian package needs as much specialised code - manpages, other docs, all those can be dumped out of the package without any other considerations. Package names can be changed to reflect changes in --enable-foo to --disable-foo using DEB_BUILD_OPTIONS to attain reductions in the sizes of the actual binaries and (more importantly) the dependency chain of those binaries, other DEB_BUILD_OPTIONS can drop README and TODO etc., but translations are special. The current TDeb specification also allows for Debian users to have a similar level of control over what actually gets installed - albeit without the ability to reduce the size of the total download. That's OK because Debian doesn't need to care about the size of the download via apt-get etc. - Emdebian does. TDebs are a specialised solution to one side of the need for smaller packages but Emdebian has other ways of reducing the size of the rest of the package. I don't want to handle translations as "just another component of a package", I want a solution that works as a translator. I have a host of other methods for reducing the package size, that is only part of the goal for tdebs. > You can always continue to use Tdebs to distribute the translation > that you need while installing the minimalistic .deb. That is precisely what we already provide. Minimalised .debs (rebuilt using cross-build support or repackaged using dpkg support) with TDebs in a separate repository and tools that allow users to manage the actual translations that get installed. TDebs are a solution for translations, not for the creation of the minimalistic .deb - however, once you have a set of TDebs for a package, it makes no sense to retain any .mo files in the .deb. Actually making the remainder of the .deb smaller then comes down to use of DEB_BUILD_OPTIONS. > > If the idea becomes separated from translations, automated creation > > of TDebs by translation review teams would be compromised, > > difficulties in re-assembling the source to ensure that subsequent > > uploads retain the intervening translation updates become even more > > complex and the drive to deliver TDebs starts to wane. (Personally, > > I am also not at all convinced that ftp-master would consider > > partial debs as a viable option.) > > I'm sorry but the technical choice has little to do with the ability > of translation teams to create .tdeb. Then what is the point? TDebs are all about the translation teams creating .tdeb - without that the whole thing is pointless. The technical choice must rely on the needs of the translation teams. A significant reason for adopting TDebs in Debian is to make it easier and less intrusive for translations to be updated in a release freeze - to allow Christian a better method than doing source NMU's on every package using debconf when everything else is meant to be frozen. It isn't a good idea to keep rebuilding binaries just to update the templates file - TDebs solve that problem. It is a specialised problem and TDebs are a specialised solution. If the technical choice does not consider the needs of t
Re: Thoughts about tdebs
On Tue, 02 Dec 2008, Neil Williams wrote: > > It is a very sizable special case, though, and growing thanks to the > > tireless translation efforts within and beyond Debian. This is why the > > consensus at the meeting seemed to be to get this right even if it > > means having to implement some special casing. > > Raphael - I think you've missed some of the purpose behind TDebs and if > that is a fault of the current draft TDeb spec, then I would value > your comments on how to fix it. I hear the size problem that Emdebian is facing but Tdeb is not the right solution for this either IMO. We should rather aim at having some DEB_BUILD_OPTIONS or something similar to instruct the build process to create the minimalist package that you need instead of dropping the translation entirely from the original .deb. You can always continue to use Tdebs to distribute the translation that you need while installing the minimalistic .deb. > If the idea becomes separated from translations, automated creation of > TDebs by translation review teams would be compromised, difficulties in > re-assembling the source to ensure that subsequent uploads retain the > intervening translation updates become even more complex and the drive > to deliver TDebs starts to wane. (Personally, I am also not at all > convinced that ftp-master would consider partial debs as a viable > option.) I'm sorry but the technical choice has little to do with the ability of translation teams to create .tdeb. > A key point of agreement with ftp-master is that the TDeb can be > updated entirely separately from the rest of the package - this is > absolutely essential to all concerned. If the idea gets diluted to > include security updates (which almost inevitably change the compiled > binaries of any compiled package involved in a security bug) then this > update method becomes impossible and we start talking about binary > patches or mass-replacement of binaries by partial debs. I find that > idea more than a little insane. ;-) It would be a sizeable regression > even from where we are now with binNMUs and source NMUs. This is about > removing the code churn from i18n updates pre-release, it is about > preventing translation updates during a release freeze from > ever rebuilding any binaries. You're making assumptions that you shouldn't do at this point. Whatever you manage to do with .tdeb can be done with partial .deb… I propose an idea so that people can think about it and you're trying to argue that it can't work and that it's insane and that we shouldn't even think about it. > However, updates to existing packages is only half the story - the > drive for TDebs has come from Emdebian. Emdebian must have TDebs in > order to meet any of the objectives for Emdebian itself. Partial debs > are not a solution, embedded systems need intelligently handled > translations and lots of smaller packages. The Emdebian solution > for TDebs cannot work within Debian unmodified, everyone accepts this, > but the Emdebian requirements do need to be achievable directly from the > Debian implementation. And can you explain why "Partial debs are not a solution" ? I see no justification in your paragraph. Partial .deb could be used to add translations to any package in theory. > The current specification supports this - partially via the premise > that TDeb updates only contain changes to translations and partially > because the individual localisations can be easily split out to create > the extra packages for Emdebian (as described on the Emdebian website > at the link above). You could do the same with the set of partial deb dedicated to localization. Don't mix technical choices with policy choices. > There are use-cases for such things but I am not the one to take on > that task. Well, when someone ask me to modify dpkg, I try to look at the bigger picture because that's what makes sense. I know that you have a lot of energy and enthusiasim for Emdebian and related projects but please do not use that energy to reject outright any suggestion that you don't like at first sight. > Actually, from my perspective, your idea is far from simple and seems > to be a lot more complex than anything so far agreed with ftp-master. >From a dpkg point of view, I find my idea simpler. :) We all have biases but we should try to step back and see further in general. Cheers, -- Raphaël Hertzog Le best-seller français mis à jour pour Debian Etch : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Thoughts about tdebs
On Tue, 02 Dec 2008 17:51:30 +0100 Thomas Viehmann <[EMAIL PROTECTED]> wrote: (I'm subscribed to the dpkg list, Thomas, thanks for the prompt though.) > Raphael Hertzog wrote: > > I just read the specs drafted at the meeting this WE and I'm not > > really excited about the prospective implementation of TDebs. > > Thanks for your appreciation and taking the time to comment. I've been wondering for a while about just when this should get more people involved. So far, it has been very gradual and I'm very cautious about taking this to a high volume/high membership list like debian-devel until Lenny is released. It's fair enough to discuss it here (and on debian-embedded). > > IMO we're heading in the wrong direction by focussing only on > > translation and by special casing everything to meet this expected > > usage. > > It is a very sizable special case, though, and growing thanks to the > tireless translation efforts within and beyond Debian. This is why the > consensus at the meeting seemed to be to get this right even if it > means having to implement some special casing. Raphael - I think you've missed some of the purpose behind TDebs and if that is a fault of the current draft TDeb spec, then I would value your comments on how to fix it. TDebs need to remain focused on l10n support to remain useful to the i18n teams and to the embedded developers. If the idea becomes too generalised, I would not be able to use it as the basis for Emdebian TDebs which are a level on from Debian TDebs due to scalability issues. http://www.emdebian.org/emdebian/langupdate.html If the idea becomes separated from translations, automated creation of TDebs by translation review teams would be compromised, difficulties in re-assembling the source to ensure that subsequent uploads retain the intervening translation updates become even more complex and the drive to deliver TDebs starts to wane. (Personally, I am also not at all convinced that ftp-master would consider partial debs as a viable option.) A key point of agreement with ftp-master is that the TDeb can be updated entirely separately from the rest of the package - this is absolutely essential to all concerned. If the idea gets diluted to include security updates (which almost inevitably change the compiled binaries of any compiled package involved in a security bug) then this update method becomes impossible and we start talking about binary patches or mass-replacement of binaries by partial debs. I find that idea more than a little insane. ;-) It would be a sizeable regression even from where we are now with binNMUs and source NMUs. This is about removing the code churn from i18n updates pre-release, it is about preventing translation updates during a release freeze from ever rebuilding any binaries. However, updates to existing packages is only half the story - the drive for TDebs has come from Emdebian. Emdebian must have TDebs in order to meet any of the objectives for Emdebian itself. Partial debs are not a solution, embedded systems need intelligently handled translations and lots of smaller packages. The Emdebian solution for TDebs cannot work within Debian unmodified, everyone accepts this, but the Emdebian requirements do need to be achievable directly from the Debian implementation. The current specification supports this - partially via the premise that TDeb updates only contain changes to translations and partially because the individual localisations can be easily split out to create the extra packages for Emdebian (as described on the Emdebian website at the link above). > > I guess you understand better the idea by now. This needs more > > thoughts but I wanted to share it so that you can think about it > > too. It certainly does need lots of thought - but this isn't about splitting the archive or just making uploads easier for certain kinds of packages. This is about making i18n and l10n genuinely functional within Debian. It is also about bringing the (stable) technology of translation packages out of the embedded world (OpenEmbedded and, from that, Emdebian) into mainstream Debian and all the scalability issues that result. Emdebian can afford to have over 40 TDebs per source package (we can afford to have over 40 TDebs per source package per architecture too and already do so). > > I'd like to suggest something simpler: partial .deb. The idea is > > that we would provide debian packages (.pdeb?) that contain only > > some parts of the original package (the part that we want to > > update), in our case the translation. Each partial package would > > state on which original version he can be applied and would have an > > identifier that define the part that it touches (with a version > > number so that we can have several updates of the same part). We > > could apply updates coming from several partial packages (security > > update, translation update, misc data update, …). There are use-cases for such things but I am not the one to ta
Re: Thoughts about tdebs
Hello, Raphael Hertzog wrote: > I just read the specs drafted at the meeting this WE and I'm not really > excited about the prospective implementation of TDebs. Thanks for your appreciation and taking the time to comment. > IMO we're heading in the wrong direction by focussing only on translation > and by special casing everything to meet this expected usage. It is a very sizable special case, though, and growing thanks to the tireless translation efforts within and beyond Debian. This is why the consensus at the meeting seemed to be to get this right even if it means having to implement some special casing. > I guess you understand better the idea by now. This needs more thoughts > but I wanted to share it so that you can think about it too. >From what I understand of your idea, you seem to think about translations mainly as updates. While it is one of the goals to enable translation-only updates, it is not quite obvious to me what your proposal has to offer in terms of splitting translations out of the debs, which is an explicit goal. Also, the proposal specifically aims at limiting the amount of data that the archive and apt have to handle. I'm not quite sure how useful the notes about design considerations are, but you could always ask Neil whether they are in a shape to publish them for further discussion. It seems that at Casar de Cáceres we found quite a few corner-cases that had not been previously considered. With some distance, it might well be that we could generalize (at least parts of) the tdeb-implementation to class support (with different members of the ar archive per class[1]). We would still want to separate them out, but that might be a good way to arrive at a better mechanism. In summary, yes, there might be room for generalization, but taking your suggestions as an indication, maybe part of your dissatisfaction with the direction that the tdeb proposal takes stems from a disparity in the goals that each is concerned with. Kind regards T. P.S.: While looking at dpkg-deb: Would it be reasonable to drop old-style deb support by now? 1. Given appropriate protocol support, this would allow retrieving only certain translations. -- Thomas Viehmann, http://thomas.viehmann.net/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Thoughts about tdebs
Hello, I just read the specs drafted at the meeting this WE and I'm not really excited about the prospective implementation of TDebs. IMO we're heading in the wrong direction by focussing only on translation and by special casing everything to meet this expected usage. I'd like to suggest something simpler: partial .deb. The idea is that we would provide debian packages (.pdeb?) that contain only some parts of the original package (the part that we want to update), in our case the translation. Each partial package would state on which original version he can be applied and would have an identifier that define the part that it touches (with a version number so that we can have several updates of the same part). We could apply updates coming from several partial packages (security update, translation update, misc data update, …). Optionaly the package update could specify if his application should change the version of the package once installed. This is particularly important for security updates where the update need to change the version number so that users can track if they are up-to-date. This also means that we might want to be able to say that a translation update could apply on top of several versions. Installed updates would be tracked in a new field in /v/l/dpkg/status, maybe like this: Installed-Updates: security (1), translation (3) Installing a full package would clear the Installed-Updates field. Each partial .deb would be a normal .deb but without any configuration script. The preinst/postinst of the package (already stored on the system) would still be ran during installation of the update because the package effectively changes and daemon might need to be restarted or new documentation might need to be registered. The partial .deb would have some special control fields: Partial: 1 Version: 1.4-2 Target-Version: - Supports-Versions: (>= 1.4-2), (<< 1.4-3) "Partial: 1" states: the package is partial and thus files of the package can replace existing files, but deletion of removed files should not happen since this doesn't contain all files. "Version: 1.4-2" states: the update has been created based on this specific version. "Target-Version: -" states: the version should not be modified once the update has been installed. "Supports-Versions: (>= 1.4-2), (<< 1.4-3)" states: the update can be applied on top of any version in that range. I guess you understand better the idea by now. This needs more thoughts but I wanted to share it so that you can think about it too. Cheers, -- Raphaël Hertzog Le best-seller français mis à jour pour Debian Etch : http://www.ouaza.com/livre/admin-debian/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
