Bug#375829: locales-all: compress locales
On Sun, Jul 02, 2006 at 02:38:20PM +0200, Javier Serrano Polo wrote: En/na Denis Barbier ha escrit: Here it is, but this is way too slow when generating many locales. Yeah, space vs speed. Actually, it's quicker to uncompress the whole tar even for one single locale (may not be the case with locales without hard links). Modifying files shipped by packages is usually a bad idea because md5sums may be altered. 72 MB of temporary space should be affordable. Here it is the modified script. In fact, there is another way: one may ask localepurge to remove unwanted locale files from /usr/lib/locales-all. IMO this is the best solution for your specific case. I am also hacking localedef so that it runs faster and consumes much less memory with UTF-8 locales. First results are promising, but I will not have time to finish before mid August. Denis -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#375829: locales-all: compress locales
En/na Denis Barbier ha escrit: Modifying files shipped by packages is usually a bad idea because md5sums may be altered. Well, my idea was having this /usr/lib/locales-all/locales.tar.gz packaged instead of the uncompressed files. In fact, there is another way: one may ask localepurge to remove unwanted locale files from /usr/lib/locales-all. IMO this is the best solution for your specific case. But then I wouldn't be able to reconfigure without downloading the package again. I am also hacking localedef so that it runs faster and consumes much less memory with UTF-8 locales. First results are promising, but I will not have time to finish before mid August. Uh... compiling took about... more than two hours? You'll need a miracle to beat the one minute uncompressing process. Don't worry about finishing this. I'm quite happy with my present scheme and it's well described in this bug if anyone else interested. Bye. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#375829: locales-all: compress locales
On Fri, Jun 30, 2006 at 01:02:39AM +0200, Javier Serrano Polo wrote: En/na Denis Barbier ha escrit: So now I consider adding a debconf template like: The following packages can configure the locales to use on your system. Only one of these packages have to be configured. Please select the package which will configure your locales: 1. locales 2. locales-all 3. belocs-locales-data Of course, only the packages which are installed are displayed. I was planning to use the alternatives system to avoid that kind of interaction. That way, dpkg-reconfigure locales would be the same as dpkg-reconfigure locales-all. In this case priorities shouldn't be based on popularity contest, as suggested on some discussions. But when you upgrade locales and locales-all, locale-gen will be run twice; this is precisely what I wanted to avoid, locales should be generated by a single package. Oh my, I completely forgot belocs. In order to save space, similar files are currently hardlinked (by localedef). When the package is generated, one has to hardlink (or symlink) compressed files instead, this is quite tricky. Now I see what you mean. I didn't thought of compressing single files. My idea was: - Delete everything under /usr/lib/locales-all - Uncompress the selected languages, usually one or two (in my case only one) from this big archive that has all locales compressed. - When you find a link, uncompress the real file and try again This means ~300KB per locale. If this works I wouldn't mind the extra space used by the compressed locale. I do not understand. Can you please send a script to explain what you have in mind? Denis -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#375829: locales-all: compress locales
On Wed, Jun 28, 2006 at 09:42:29PM +0200, Denis Barbier wrote:
On Wed, Jun 28, 2006 at 09:17:52PM +0200, Javier Serrano Polo wrote:
En/na Denis Barbier ha escrit:
Ok, that looks like a good idea, reopening this bug. But please note
that this is not trivial, one has to think about interferences between
locales and locales-all.
Would it be enough to divert locale-gen? The debconf part could be the
same as in locales,
Another option is to have shared templates to specify locales to
generate, and the package which generates those locales is chosen
via another template. This is similar to kdm/gdm/xdm, or to the
various dictionaries.
To be more precise, I did not want to ship locales-all files under
/usr/lib/locale/ to prevent configuration problems when both locales
and locales-all are installed. Setting LOCPATH is only a workaround,
setuid programs do not use $LOCPATH for security reasons.
I was going to run
localedef --add-to-archive /usr/lib/locales-all/*
in locales-all.postinst, but your bug report has good points.
So now I consider adding a debconf template like:
The following packages can configure the locales to use on your
system. Only one of these packages have to be configured.
Please select the package which will configure your locales:
1. locales 2. locales-all 3. belocs-locales-data
Of course, only the packages which are installed are displayed.
then follow the second method described in README.Debian. The
compressed file'd be in /usr/share/locales-all.
In order to save space, similar files are currently hardlinked (by
localedef). When the package is generated, one has to hardlink
(or symlink) compressed files instead, this is quite tricky.
Here are 2 scripts:
* locales-compress compress files under /usr/share/locales-all/
and take care of hard links. It can be run when building the
locales-all package.
* locales-uncompress runs 'localedef --add-to-archive' qfter
uncompressing locale files, it can be easily customized to
only add the requested locales in locales-all.postinst.
The ideal solution would be to patch localedef so that it can
read .gz files directly, but this should be done only in conjunction
with --add-to-archive (or in other words, this feature should be
implemented within localedef but not within libc itself). No idea
yet whether this is feasible, so please let me know if above scripts
are useful for low resource machines.
Denis
#! /bin/sh
set -e
dir=/usr/lib/locales-all
tmp=$(tempfile)
find $dir -type f $tmp
for f in $(cat $tmp | awk '{printf %s , $1'})
do
[ -e $f ] || continue
links=$(ls -l $f | awk '{print $2}')
if [ $links -eq 1 ]; then
gzip -9 $f
else
inode=$(ls -i $f | awk '{print $1}')
tmp2=$(tempfile)
find $dir -inum $inode $tmp2
for g in $(cat $tmp2 | awk '{printf %s , $1'})
do
[ $f = $g ] || unlink $g
done
gzip -9 $f
for g in $(cat $tmp2 | awk '{printf %s , $1'})
do
[ $f = $g ] || ln $f.gz $g.gz
done
rm -f $tmp2
fi
done
rm -f $tmp
#! /bin/sh
set -e
dir=/usr/lib/locales-all
tmp=$(mktemp -d)
rm -f /usr/lib/locale/locale-archive
for f in /usr/lib/locales-all/*
do
rm -rf $tmp/*
cp -a $f $tmp/
find $tmp -name \*.gz | xargs gunzip
localedef --add-to-archive $tmp/*
done
rm -rf $tmp
Bug#375829: locales-all: compress locales
En/na Denis Barbier ha escrit: So now I consider adding a debconf template like: The following packages can configure the locales to use on your system. Only one of these packages have to be configured. Please select the package which will configure your locales: 1. locales 2. locales-all 3. belocs-locales-data Of course, only the packages which are installed are displayed. I was planning to use the alternatives system to avoid that kind of interaction. That way, dpkg-reconfigure locales would be the same as dpkg-reconfigure locales-all. In this case priorities shouldn't be based on popularity contest, as suggested on some discussions. Oh my, I completely forgot belocs. In order to save space, similar files are currently hardlinked (by localedef). When the package is generated, one has to hardlink (or symlink) compressed files instead, this is quite tricky. Now I see what you mean. I didn't thought of compressing single files. My idea was: - Delete everything under /usr/lib/locales-all - Uncompress the selected languages, usually one or two (in my case only one) from this big archive that has all locales compressed. - When you find a link, uncompress the real file and try again This means ~300KB per locale. If this works I wouldn't mind the extra space used by the compressed locale. Bye. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#375829: locales-all: compress locales
Package: locales-all Version: 2.3.6-15 Severity: wishlist All locales could be compressed in a single archive and, using a dialog like in locales package, uncompress the selected ones instead of compiling them. -- System Information: Debian Release: testing/unstable APT prefers testing APT policy: (500, 'testing') Architecture: i386 (i686) Shell: /bin/sh linked to /bin/bash Kernel: Linux 2.6.15-1-686 Locale: LANG=ca_ES.UTF-8, LC_CTYPE=ca_ES.UTF-8 (charmap=UTF-8) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#375829: locales-all: compress locales
En/na Denis Barbier ha escrit: On Wed, Jun 28, 2006 at 01:26:58PM +0200, Javier Serrano Polo wrote: Package: locales-all Version: 2.3.6-15 Severity: wishlist All locales could be compressed in a single archive and, using a dialog like in locales package, uncompress the selected ones instead of compiling them. Locales are not compressed when put into a single archive, the main benefit is that, at startup, applications do not have to search for a bunch of files, only this archive. I am thus closing your report. Denis Let me explain better. Now we have all locales under /usr/lib/locales-all, ~72MB. We could have a gz-compressed file, ~15M, and use dpkg-reconfigure to uncompress/delete the selected locale folders. I think this is a better trade-off in low resource machines. Bye. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#375829: locales-all: compress locales
reopen 375829 thanks On Wed, Jun 28, 2006 at 03:53:59PM +0200, Javier Serrano Polo wrote: En/na Denis Barbier ha escrit: On Wed, Jun 28, 2006 at 01:26:58PM +0200, Javier Serrano Polo wrote: Package: locales-all Version: 2.3.6-15 Severity: wishlist All locales could be compressed in a single archive and, using a dialog like in locales package, uncompress the selected ones instead of compiling them. Locales are not compressed when put into a single archive, the main benefit is that, at startup, applications do not have to search for a bunch of files, only this archive. I am thus closing your report. Let me explain better. Now we have all locales under /usr/lib/locales-all, ~72MB. We could have a gz-compressed file, ~15M, and use dpkg-reconfigure to uncompress/delete the selected locale folders. I think this is a better trade-off in low resource machines. Ok, that looks like a good idea, reopening this bug. But please note that this is not trivial, one has to think about interferences between locales and locales-all. Denis -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Processed: Re: Bug#375829: locales-all: compress locales
Processing commands for [EMAIL PROTECTED]: reopen 375829 Bug#375829: locales-all: compress locales Bug reopened, originator not changed. thanks Stopping processing here. Please contact me if you need assistance. Debian bug tracking system administrator (administrator, Debian Bugs database) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#375829: locales-all: compress locales
En/na Denis Barbier ha escrit: Ok, that looks like a good idea, reopening this bug. But please note that this is not trivial, one has to think about interferences between locales and locales-all. Would it be enough to divert locale-gen? The debconf part could be the same as in locales, then follow the second method described in README.Debian. The compressed file'd be in /usr/share/locales-all. I can give it a try this weekend if you don't mind. Bye. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#375829: locales-all: compress locales
On Wed, Jun 28, 2006 at 09:17:52PM +0200, Javier Serrano Polo wrote: En/na Denis Barbier ha escrit: Ok, that looks like a good idea, reopening this bug. But please note that this is not trivial, one has to think about interferences between locales and locales-all. Would it be enough to divert locale-gen? The debconf part could be the same as in locales, Another option is to have shared templates to specify locales to generate, and the package which generates those locales is chosen via another template. This is similar to kdm/gdm/xdm, or to the various dictionaries. then follow the second method described in README.Debian. The compressed file'd be in /usr/share/locales-all. In order to save space, similar files are currently hardlinked (by localedef). When the package is generated, one has to hardlink (or symlink) compressed files instead, this is quite tricky. I can give it a try this weekend if you don't mind. Sure, no problem. Denis -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
