Bug#1037198: locales: please parallelise locale-gen
Hi! On Thu, Jun 15, 2023 at 09:26:43PM +0200, Aurelien Jarno wrote: > On 2023-06-07 16:04, наб wrote: > > Posting as a bug per comment from Andrej; originally posted 2022-05-06 as > > https://salsa.debian.org/glibc-team/glibc/-/merge_requests/7 > > > > Patch based on current Salsa HEAD attached, incl. analysis. > > Thanks for the patch. I looks good, I have a comment though. > > MemFree: in /proc/meminfo is available on all supported Debian kernels, > > and, indeed, exactly what procps free(1) uses > What is the reason to use MemFree instead of MemAvailable. That's what procps free(1) used, and all Debian kernels (kFreeBSD, Hurd, Linux) supported it. > The Linux > kernel tends to maintain MemFree close to 0 by using the free RAM as > cache. MemAvailable also includes reclaimable memory blocks like cache > or inactive pages and therefore sounds better suited. Since I first posted this, procps free(1) started using MemAvailable to evaluate free/used, so sure. I don't feel strongly either way. A Hurd image from 2021 I have (bullseye branding) and the 2023 release (bookworm branding) don't have MemAvailable, neither does kFreeBSD 10 (from the 2017 installer ISO; appears to be the latest from https://wiki.debian.org/Debian_GNU/kFreeBSD). I've updated the Salsa revision and am including an updated patch here, which overrides MemFree with MemAvailable if available. Best, наб From d64e6b551948726dbe5cc6800e93a2d7b25d3f89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=BD=D0=B0=D0=B1?= Date: Fri, 6 May 2022 01:22:10 +0200 Subject: [PATCH] Parallelise locale-gen if possible MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Mutt-PGP: OS Assuming a very generous 200M free/localedef (because I saw a max RSS of 147M w/time(1)), this will attempt to keep all jobs saturated, and usually succeed. There's little starvation, since the vast majority of time is spent in gzip(1) ‒ 1:14 user vs 27:55 sys At 2.2ish seconds per locale, even on a low-end system of today with 4 CPUs (and 800 free MB), we can generate up to 4 locales at once for 6.6s' speed-up. Assuming no super-pathological cases, this globally scales in roughly ceil(locales/ncpus)*2.2s chunks, which is a massive win The only user-visible change is that, with nproc>1, the output is en_GB.UTF-8... instead of en_GB.UTF-8... MemFree: in /proc/meminfo is available on all supported Debian kernels, MemAvailable: only on Linux; procps free(1) uses MemAvailable to estimate "used" space where available. --- debian/local/usr_sbin/locale-gen | 31 +-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/debian/local/usr_sbin/locale-gen b/debian/local/usr_sbin/locale-gen index 7fa3d772..30f70f5e 100755 --- a/debian/local/usr_sbin/locale-gen +++ b/debian/local/usr_sbin/locale-gen @@ -23,6 +23,19 @@ is_entry_ok() { fi } +nproc="$(nproc 2>/dev/null)" || nproc=1 +if [ "$nproc" -gt 1 ]; then + mem_free=0 + while read -r k v _; do + [ "$k" = "MemFree:" ] && mem_free="$v" || : + [ "$k" = "MemAvailable:" ] && mem_free="$v" && break || : # Prefer using MemAvailable on Linux; other Debian kernels only have MemFree + done < /proc/meminfo || : + mem_free=$(( mem_free / 1024 / 200 )) + [ "$mem_free" -lt 1 ] && mem_free=1 || : + [ "$mem_free" -lt "$nproc" ] && nproc="$mem_free" || : + jobs=0; pids= +fi 2>/dev/null + echo "Generating locales (this might take a while)..." while read -r locale charset; do if [ -z "$locale" ] || [ "${locale#\#}" != "$locale" ]; then continue; fi @@ -35,6 +48,7 @@ while read -r locale charset; do locale_at="${locale#*@}" [ "$locale_at" = "$locale" ] && locale_at= || locale_at="@$locale_at" printf " %s.%s%s..." "$locale_base" "$charset" "$locale_at" + [ "$nproc" -gt 1 ] && echo || : if [ -e "$USER_LOCALES/$locale" ]; then input="$USER_LOCALES/$locale" @@ -46,7 +60,20 @@ while read -r locale charset; do input="$USER_LOCALES/$input" fi fi - localedef -i "$input" -c -f "$charset" -A /usr/share/locale/locale.alias "$locale" || : - echo " done" + localedef -i "$input" -c -f "$charset" -A /usr/share/locale/locale.alias "$locale" & + if [ "$nproc" -gt 1 ]; then + pids="$pids$! " + jobs=$(( jobs + 1 )) + + if [ "$jobs" -ge "$nproc" ]; then + wait "${pids%% *}" || : + jobs=$(( jobs - 1 )) + pids="${pids#* }" + fi + else + wait + echo " done" + fi done < "$LOCALEGEN" +wait echo "Generation complete." -- 2.39.2 signature.asc Description: PGP signature
Bug#1037198: locales: please parallelise locale-gen
Package: locales Version: 2.36-9 Severity: wishlist Tags: patch Dear Maintainer, Posting as a bug per comment from Andrej; originally posted 2022-05-06 as https://salsa.debian.org/glibc-team/glibc/-/merge_requests/7 Patch based on current Salsa HEAD attached, incl. analysis. Best, наб -- System Information: Debian Release: 12.0 APT prefers unstable APT policy: (500, 'unstable') Architecture: x32 (x86_64) Foreign Architectures: amd64, i386 Kernel: Linux 6.1.0-2-amd64 (SMP w/2 CPU threads; PREEMPT) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages locales depends on: ii debconf [debconf-2.0] 1.5.82 ii libc-bin 2.36-9 ii libc-l10n 2.36-9 locales recommends no packages. locales suggests no packages. -- debconf information: * locales/locales_to_be_generated: en_GB.UTF-8 UTF-8 * locales/default_environment_locale: en_GB.UTF-8 From b6af0ad83f5517fd1987f9c7ac0493565bc0976d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=BD=D0=B0=D0=B1?= Date: Fri, 6 May 2022 01:22:10 +0200 Subject: [PATCH] Parallelise locale-gen if possible MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Mutt-PGP: OS Assuming a very generous 200M free/localedef (because I saw a max RSS of 147M w/time(1)), this will attempt to keep all jobs saturated, and usually succeed. There's little starvation, since the vast majority of time is spent in gzip(1) ‒ 1:14 user vs 27:55 sys At 2.2ish seconds per locale, even on a low-end system of today with 4 CPUs (and 800 free MB), we can generate up to 4 locales at once for 6.6s' speed-up. Assuming no super-pathological cases, this globally scales in roughly ceil(locales/ncpus)*2.2s chunks, which is a massive win The only user-visible change is that, with nproc>1, the output is en_GB.UTF-8... instead of en_GB.UTF-8... MemFree: in /proc/meminfo is available on all supported Debian kernels, and, indeed, exactly what procps free(1) uses --- debian/local/usr_sbin/locale-gen | 30 -- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/debian/local/usr_sbin/locale-gen b/debian/local/usr_sbin/locale-gen index 7fa3d772..f1632f4e 100755 --- a/debian/local/usr_sbin/locale-gen +++ b/debian/local/usr_sbin/locale-gen @@ -23,6 +23,18 @@ is_entry_ok() { fi } +nproc="$(nproc 2>/dev/null)" || nproc=1 +if [ "$nproc" -gt 1 ]; then + mem_free=0 + while read -r k v _; do + [ "$k" = "MemFree:" ] && mem_free="$v" && break || : + done < /proc/meminfo || : + mem_free=$(( mem_free / 1024 / 200 )) + [ "$mem_free" -lt 1 ] && mem_free=1 || : + [ "$mem_free" -lt "$nproc" ] && nproc="$mem_free" || : + jobs=0; pids= +fi 2>/dev/null + echo "Generating locales (this might take a while)..." while read -r locale charset; do if [ -z "$locale" ] || [ "${locale#\#}" != "$locale" ]; then continue; fi @@ -35,6 +47,7 @@ while read -r locale charset; do locale_at="${locale#*@}" [ "$locale_at" = "$locale" ] && locale_at= || locale_at="@$locale_at" printf " %s.%s%s..." "$locale_base" "$charset" "$locale_at" + [ "$nproc" -gt 1 ] && echo || : if [ -e "$USER_LOCALES/$locale" ]; then input="$USER_LOCALES/$locale" @@ -46,7 +59,20 @@ while read -r locale charset; do input="$USER_LOCALES/$input" fi fi - localedef -i "$input" -c -f "$charset" -A /usr/share/locale/locale.alias "$locale" || : - echo " done" + localedef -i "$input" -c -f "$charset" -A /usr/share/locale/locale.alias "$locale" & + if [ "$nproc" -gt 1 ]; then + pids="$pids$! " + jobs=$(( jobs + 1 )) + + if [ "$jobs" -ge "$nproc" ]; then + wait "${pids%% *}" || : + jobs=$(( jobs - 1 )) + pids="${pids#* }" + fi + else + wait + echo " done" + fi done < "$LOCALEGEN" +wait echo "Generation complete." -- 2.30.2 signature.asc Description: PGP signature
Bug#444589: strftime(%Y) overflows into the negatives for very positive years
In my haste I forgot to note, that, unsurprisingly, the same holds for %G and %C: -- >8 -- $ TZ=UTC0 ./a.out '%F %G %C %s' 67768036191676799 -2147481749-12-31 -2147481748 -21474818 67768036191676799 -- >8 -- наб signature.asc Description: PGP signature
Bug#444589: strftime(%Y) overflows into the negatives for very positive years: the most positive and the most negative times are 1 second apart
Control: retitle -1 strftime(%Y) overflows into the negatives for very positive years Control: found -1 2.33-7 Under TZ=UTC0, the most positive and most negative times glibc accepts (i.e. "doesn't NULL, EOVERFLOW for") are 67768036191676799 (0x00F0C2AB7C54A97F) -67768040609740800 (0xFF0F3D537C550800) which are, respectively 2147485547-12-31 23:59:59 -2147481748-01-01 00:00:00 however, glibc strftime() apparently forgets to widen the intermediate representation, because strftime("%F %T") yields, respectively: -2147481749-12-31 23:59:59 -2147481748-01-01 00:00:00 these are 1 second apart, not ~2^32 years. As expected, 2147481748+1900 is 0x8000`, and 2147485547-1900 is 0x7FFF`. Measured via -- >8 -- #include #include #include #include int main(int, const char ** argv) { std::setlocale(LC_ALL, ""); auto fmt = argv[1] ?: "no format!"; struct timespec ts {}; if(argv[1] && argv[2]) ts.tv_sec = strtoll(argv[2], nullptr, 0); else clock_gettime(CLOCK_REALTIME, ); char buf[4096]; std::strftime(buf, sizeof(buf), fmt, localtime(_sec)); std::puts(buf); } -- >8 -- Which produces (notably, coreutils date does /not/ have this issue): -- >8 -- $ TZ=UTC0 date +'%F %T' -d@67768036191676799 +2147485547-12-31 23:59:59 $ TZ=UTC0 date +'%F %T' -d@-67768040609740800 -2147481748-01-01 00:00:00 $ TZ=UTC0 ./strdate '%F %T' 67768036191676799 -2147481749-12-31 23:59:59 $ TZ=UTC0 ./strdate '%F %T' -67768040609740800 -2147481748-01-01 00:00:00 -- >8 -- I can reproduce this on bullseye and sid (2.33-7); substituting experimental (2.34) glibc unrelatedly segfaults. Best, наб signature.asc Description: PGP signature