Bug#539617: dpkg: Add option to suppress progress display while reading database

[Thorsten Glaser]

Package: dpkg
Version: 1.21.22
Followup-For: Bug #539617
X-Debbugs-Cc: t...@mirbsd.de

This issue is still present, cluttering logs of CI jobs by a lot.


-- Package-specific info:
This system uses merged-usr-via-aliased-dirs, going behind dpkg's
back, breaking its core assumptions. This can cause silent file
overwrites and disappearances, and its general tools misbehavior.
See .

-- System Information:
Debian Release: trixie/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'buildd-unstable'), (500, 
'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.10.0-23-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/lksh
Init: sysvinit (via /sbin/init)

Versions of packages dpkg depends on:
ii  libbz2-1.0   1.0.8-5+b1
ii  libc62.37-7
ii  liblzma5 5.4.1-0.2
ii  libmd0   1.1.0-1
ii  libselinux1  3.5-1
ii  libzstd1 1.5.5+dfsg2-1
ii  tar  1.34+dfsg-1.2
ii  zlib1g   1:1.2.13.dfsg-1

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt2.7.3
pn  debsig-verify  

-- no debconf information

Bug#539617: dpkg: Add option to suppress progress display while reading database

[Yuri Kanivetsky]

Hi,

stdout is not tty, but dpkg still produces a lot of output:


$ docker stop apt-quiet \
; docker run --rm --name apt-quiet -d cypress/base:12 sleep infinity \
&& docker exec apt-quiet sh -euxc '
[ -t 1 ] && echo tty || echo non-tty
&& ps -efH
&& export DEBIAN_FRONTEND=noninteractive
&& apt-get -qq update
&& apt-get -qq install less
'

apt-quiet
10ff09dfe813df80917d6a4b180f616c54ecd78267739eb5e0c79927410e55a4
non-tty
+ [ -t 1 ]
+ echo non-tty
+ ps -efH
UID  PIDPPID  C STIME TTY  TIME CMD
root   7   0  0 13:08 ?00:00:00 sh -euxc [ -t 1 ]
&& echo tty || echo non-tty && ps -efH && export
DEBIAN_FRONTEND=noninteractive && apt-get -qq update && apt-get -qq
install less
root  12   7  0 13:08 ?00:00:00   ps -efH
root   1   0 28 13:08 ?00:00:00 sleep infinity
+ export DEBIAN_FRONTEND=noninteractive
+ apt-get -qq update
+ apt-get -qq install less
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ...  (Reading database ... 5% (Reading database ...
10% (Reading database ... 15% (Reading database ... 20% (Reading
database ... 25% (Reading database ... 30% (Reading database ... 35%
(Reading database ... 40% (Reading database ... 45% (Reading database
... 50% (Reading database ... 55% (Reading database ... 60% (Reading
database ... 65% (Reading database ... 70% (Reading database ... 75%
(Reading database ... 80% (Reading database ... 85% (Reading database
... 90% (Reading database ... 95% (Reading database ... 100% (Reading
database ... 33231 files and directories currently installed.)
Preparing to unpack .../less_487-0.1+b1_amd64.deb ...
Unpacking less (487-0.1+b1) ...
Setting up less (487-0.1+b1) ...
Processing triggers for mime-support (3.62) ...


Regards,
Yuri

Bug#539617: dpkg: Add option to suppress progress display while reading database

[Frans Pop]

Package: dpkg
Version: 1.15.3.1
Severity: wishlist
Tags: patch

When the output of package installations are logged to a file, the 
progress counter updates while reading the database are distracting.

This is especially the case in Debian Installer where multiple package 
installations are started and all are logged to the syslog. For an 
example of the result, see the syslog attached in [1].

Another example is buildd logs, for example [2].

The attached patch adds an option --no-db-progress that allows to suppress 
the progress updates in such situations.

If you'd like any changes (maybe a more generic name for the option?) or 
if I've forgotten anything, then please let me know.

The patch is based on current git:
commit 173652e75fad004c42906608fe0ff4556be29b19
Author: Helge Kreutzmann deb...@helgefjell.de
Date:   Thu Jul 30 18:38:57 2009 +0200
Fix errors in a paragraph in the German man page translation.

[1]http://bugs.debian.org/538344#15 (see lines starting at 00:26:43)
[2]https://buildd.debian.org/fetch.cgi?pkg=clock-setuparch=i386ver=0.99stamp=1249206304file=logas=raw
diff --git a/man/dpkg.1 b/man/dpkg.1
index 665519f..e925a2e 100644
--- a/man/dpkg.1
+++ b/man/dpkg.1
@@ -556,6 +556,10 @@ may leave packages in the improper \fBtriggers\-awaited\fP and
 .TP
 \fB\-\-triggers\fP
 Cancels a previous \fB\-\-no\-triggers\fP.
+.TP
+\fB\-\-no-db-progress\fP
+Do not report progress while reading database. This can be useful when
+logging the output of dpkg to a file.
 .
 .SH FILES
 .TP
diff --git a/src/filesdb.c b/src/filesdb.c
index da8cde2..dc0bfad 100644
--- a/src/filesdb.c
+++ b/src/filesdb.c
@@ -219,21 +219,25 @@ void ensure_allinstfiles_available(void) {
 int max = countpackages();
 
 saidread=1;
-progress_init(progress, _((Reading database ... ), max);
+if (!f_nodbprogr)
+  progress_init(progress, _((Reading database ... ), max);
+else
+  printf(_((Reading database ... ));
   }
 
   it= iterpkgstart();
   while ((pkg = iterpkgnext(it)) != NULL) {
 ensure_packagefiles_available(pkg);
 
-if (saidread == 1)
+if (!f_nodbprogr  saidread == 1)
   progress_step(progress);
   }
   iterpkgend(it);
   allpackagesdone= 1;
 
   if (saidread==1) {
-progress_done(progress);
+if (!f_nodbprogr)
+  progress_done(progress);
 printf(_(%d files and directories currently installed.)\n),nfiles);
 saidread=2;
   }
diff --git a/src/main.c b/src/main.c
index ad9b4b8..6a5fed8 100644
--- a/src/main.c
+++ b/src/main.c
@@ -136,6 +136,7 @@ usage(void)
   --no-force-...|--refuse-...\n
  Stop when problems encountered.\n
   --abort-after n  Abort after encountering n errors.\n
+  --no-db-progress   Do not report progress while reading database.\n
 \n), ADMINDIR)  0) werr (stdout);
 
   if (fprintf (stdout, _(
@@ -164,7 +165,7 @@ const char printforhelp[]= N_(
 
 const struct cmdinfo *cipaction = NULL;
 int f_pending=0, f_recursive=0, f_alsoselect=1, f_skipsame=0, f_noact=0;
-int f_autodeconf=0, f_nodebsig=0;
+int f_autodeconf=0, f_nodebsig=0, f_nodbprogr=0;
 int f_triggers = 0;
 unsigned long f_debug=0;
 /* Change fc_overwrite to 1 to enable force-overwrite by default */
@@ -442,6 +443,7 @@ static const struct cmdinfo cmdinfos[]= {
   { selected-only, 'O', 0, f_alsoselect, NULL,  NULL,0 },
   { triggers,   0,  0, f_triggers,   NULL,  NULL,1 },
   { no-triggers,0,  0, f_triggers,   NULL,  NULL,   -1 },
+  { no-db-progress, 0,  0, f_nodbprogr,  NULL,  NULL,1 },
   /* FIXME: Remove ('N') sometime. */
   { no-also-select,'N', 0, f_alsoselect, NULL,  NULL,0 },
   { skip-same-version, 'E', 0, f_skipsame,   NULL,  NULL,1 },
diff --git a/src/main.h b/src/main.h
index cd8c7b6..d235ed4 100644
--- a/src/main.h
+++ b/src/main.h
@@ -84,7 +84,7 @@ extern const char *const statusstrings[];
 
 extern const struct cmdinfo *cipaction;
 extern int f_pending, f_recursive, f_alsoselect, f_skipsame, f_noact;
-extern int f_autodeconf, f_nodebsig;
+extern int f_autodeconf, f_nodebsig, f_nodbprogr;
 extern int f_triggers;
 extern unsigned long f_debug;
 extern int fc_downgrade, fc_configureany, fc_hold, fc_removereinstreq, fc_overwrite;
diff --git a/src/query.c b/src/query.c
index 48eacc0..2d06446 100644
--- a/src/query.c
+++ b/src/query.c
@@ -492,7 +492,7 @@ Use --license for copyright license and lack of warranty (GNU GPL).);
 
 const struct cmdinfo *cipaction = NULL;
 int f_pending=0, f_recursive=0, f_alsoselect=1, f_skipsame=0, f_noact=0;
-int f_autodeconf=0, f_nodebsig=0;
+int f_autodeconf=0, f_nodebsig=0, f_nodbprogr=0;
 unsigned long f_debug=0;
 /* Change fc_overwrite to 1 to enable force-overwrite by default */
 int fc_hold=0;

Bug#539617: dpkg: Add option to suppress progress display while reading database

[Guillem Jover]

Hi!

On Sun, 2009-08-02 at 13:54:24 +0200, Frans Pop wrote:
 Package: dpkg
 Version: 1.15.3.1
 Severity: wishlist
 Tags: patch
 
 When the output of package installations are logged to a file, the 
 progress counter updates while reading the database are distracting.

It should only get updated if stdout is a tty. In the same way most other
programs do it, including apt for example.

 This is especially the case in Debian Installer where multiple package 
 installations are started and all are logged to the syslog. For an 
 example of the result, see the syslog attached in [1].
 
 Another example is buildd logs, for example [2].

I'm guessing this logging is taken from the tty directly instead of
redirecting stdout? If so, OOC what was the reason to do it that way?

Also you'll notice that in normal conditions this works just fine,
see 00:10:32, 00:10:39, etc. And when it does not work (on in-target) it
does not work only for dpkg but for other stuff as well, see 01:12:27.

On the build logs the same happens, some parts work ok, others not, I'm
guessing a similar issue with not just redirecting stdout.

 The attached patch adds an option --no-db-progress that allows to suppress 
 the progress updates in such situations.
 
 If you'd like any changes (maybe a more generic name for the option?) or 
 if I've forgotten anything, then please let me know.

Hmm, this options is too specific, and I'd like dpkg to automatically do
the right thing, OTOH most apps do fancier stuff when they know they are
running on a tty, and if one reads from the tty directly it's expected
to get garbage. That does not mean we could not add something like a
--quiet option, though, but I'd rather understand first what and why
of the current situation.

regards,
guillem



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#539617: dpkg: Add option to suppress progress display while reading database

[Julien Cristau]

On Sun, Aug  2, 2009 at 13:54:24 +0200, Frans Pop wrote:

 When the output of package installations are logged to a file, the 
 progress counter updates while reading the database are distracting.
 
 This is especially the case in Debian Installer where multiple package 
 installations are started and all are logged to the syslog. For an 
 example of the result, see the syslog attached in [1].
 
 Another example is buildd logs, for example [2].
 
As far as I can tell, progress_step() isn't supposed to do anything if
stdout isn't a tty.  I'd expect that to be the case for buildd logs, at
least (don't know how this is handled by d-i).

Cheers,
Julien



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#539617: dpkg: Add option to suppress progress display while reading database

[Frans Pop]

Hi Guillem,

On Sunday 02 August 2009, Guillem Jover wrote:
 I'm guessing this logging is taken from the tty directly instead of
 redirecting stdout? If so, OOC what was the reason to do it that way?

The reason is that we're running in a fairly complex environment there. 
We're installing in a chroot and getting progress info through debconf, 
with debconf info being passed from debconf in the chroot to cdebconf in 
the D-I environment. So we already have endless fun redirections.

A program called log-output is used to do the eventual logging.

See debian-installer-utils [1] if you'd like to have a look at them.
A typical installation of a single package would use first apt-install, 
which calls apt-get (through debconf-apt-progress with various options) 
using in-target, which in turn sources chroot-setup.sh and uses 
log-output...
At some point we also call tasksel, but the principle remains the same.

So, it's not all that strange that dpkg isn't able to tell where the 
output ends up in the end. Colin Watson will be better able than me to 
really explain the intricacies.

 Also you'll notice that in normal conditions this works just fine,
 see 00:10:32, 00:10:39, etc.

Yes, that's debootstrap which runs in a very different environment (though 
also with a D-I wrapper).

 And when it does not work (on in-target) it does not work only for dpkg
 but for other stuff as well, see 01:12:27.

True and it would be nice to get rid of that as well, but there's quite a 
difference between 4 and 20 lines :-)

  If you'd like any changes (maybe a more generic name for the option?)
  or if I've forgotten anything, then please let me know.

 Hmm, this options is too specific, and I'd like dpkg to automatically
 do the right thing, OTOH most apps do fancier stuff when they know they
 are running on a tty, and if one reads from the tty directly it's
 expected to get garbage. That does not mean we could not add something
 like a --quiet option, though, but I'd rather understand first what and
 why of the current situation.

Yeah, I already thought you might prefer something like that. A 
general --quiet option would be fine with me as long as we don't lose any 
of the real info needed to debug installation failures.

Thanks,
FJP

[1]http://svn.debian.org/wsvn/d-i/trunk/packages/debian-installer-utils/#_trunk_packages_debian-installer-utils_



-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org