[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
This bug was fixed in the package audit - 1:2.8.2-1ubuntu1.1
---
audit (1:2.8.2-1ubuntu1.1) bionic; urgency=medium
* d/p/04-startup-hang.patch: fix hang on startup (LP: #1848330)
-- Andreas Hasenack Fri, 08 Jan 2021 14:23:06
-0300
** Changed in: audit (Ubuntu Bionic)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Released
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
TL;DR verification-succeeded
Ok, so here are the details.
I have two vms: one called orig-audit-bionic, the other called sru-
audit-bionic, where I ran the script from comment #23 over the weekend
in multiple scenarios. With auditd-1:2.8.2-1ubuntu1, the bug is
reproduced after a few hours, whereas with 1:2.8.2-1ubuntu1.1 I had it
running over 36h in one case with no failure.
a) orig-audit-bionic
Installed with the original auditd-1:2.8.2-1ubuntu1, I had two runs to verify
the failure:
a.1) First run
started at Fri Jan 22 19:20:29 UTC 2021
failed at Fri Jan 22 22:43:51 UTC 2021
Jan 22 22:43:51 orig-audit-bionic systemd[1]: Starting Security Auditing
Service...
Jan 22 22:43:51 orig-audit-bionic auditd[24058]: Started dispatcher:
/sbin/audispd pid: 24060
Jan 22 22:43:51 orig-audit-bionic audispd: No plugins found, exiting
Jan 22 22:45:21 orig-audit-bionic systemd[1]: auditd.service: Start operation
timed out. Terminating.
a.2) Second run, same package
started at Sat Jan 23 14:30:11 UTC 2021
failed at Sat Jan 23 21:35:20 UTC 2021
Jan 23 21:35:20 orig-audit-bionic systemd[1]: Starting Security Auditing
Service...
Jan 23 21:35:20 orig-audit-bionic auditd[7794]: Started dispatcher:
/sbin/audispd pid: 7796
Jan 23 21:35:20 orig-audit-bionic audispd: No plugins found, exiting
Jan 23 21:36:50 orig-audit-bionic systemd[1]: auditd.service: Start operation
timed out. Terminating.
I then upgraded the auditd package to 1:2.8.2-1ubuntu1.1, and started another
run:
started at Sat Jan 23 23:54:35 UTC 2021
manually aborted at Mon Jan 25 12:23:42 UTC 2021
No failure.
b) sru-audit-bionic
Installed the original auditd-1:2.8.2-1ubuntu1, and upgraded it straight away
to 1:2.8.2-1ubuntu1.1. Then started the script.
started at Fri Jan 22 19:23:38 UTC 2021
manually aborted at Sun Jan 24 18:53:09 UTC 2021
No failure.
I then downgraded the auditd package back to auditd-1:2.8.2-1ubuntu1 and ran
the script again.
started at Sun Jan 24 19:00:56 UTC 2021
failed at Sun Jan 24 23:32:58 UTC 2021
Jan 24 23:32:58 sru-audit-bionic systemd[1]: Starting Security Auditing
Service...
Jan 24 23:32:58 sru-audit-bionic auditd[11439]: Started dispatcher:
/sbin/audispd pid: 11441
Jan 24 23:32:58 sru-audit-bionic audispd: No plugins found, exiting
Jan 24 23:34:28 sru-audit-bionic systemd[1]: auditd.service: Start operation
timed out. Terminating.
Full logs attached as tarballs for, heh, audit purposes :)
** Attachment added: "audit-sru-1848330.tar.xz"
https://bugs.launchpad.net/ubuntu/+source/audit/+bug/1848330/+attachment/5456640/+files/audit-sru-1848330.tar.xz
** Tags removed: verification-needed-bionic
** Tags added: verification-done-bionic
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
I prepared two bionic instances to run over the weekend.
One is running auditd from bionic, and the other is running the SRU
proposed package.
I have auditd being restarted via this script in both (just the email message
is different, to say which package it was):
#!/bin/bash
result=0
while /bin/true; do
date
sudo systemctl restart auditd || result=$?
if [ "$result" -ne "0" ]; then
echo "FAILED, result=$result"
break
fi
pid=$(pidof auditd) || result=$?
if [ "$result" -ne "0" ]; then
echo "FAILED, auditd not running"
break
fi
echo "auditd pid = $pid"
sleep 2
echo
done
mail -s "ALERT: audit orig test failed" andr...@canonical.com < reaped" isn't shown, which
is exactly the bug: auditd hangs while trying to log that message inside
a signal handler.
So, looking good. Let's see if I can get another failure.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
I think it's important to distinguish:
a) merely failing to reproduce the issue; versus
b) confirming reproduction of the issue against the previous version of
the package (version 1:2.8.2-1ubuntu1 in this case), and then confirming
that the proposed version (version 1:2.8.2-1ubuntu1.1 in this case)
resolves the issue _in the same environment_.
The distinction is important because if the test method and environment
used is unable to reproduce against the previous version, then
confirming that it's not possible to reproduce the issue with the
proposed version is effectively no testing at all.
Please could you confirm if the testing was actually case b, and if so
detail what testing was performed?
Thanks!
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
Hi, all.
I obtained feedback from one Canonical customer (with a Support Case
opened just for that), that hit the very same bug and, when testing the
package sitting in -proposed, didn't reproduce the issue anymore.
>From his own words:
"I wasn't able to recreate the auditd problem with the updated auditd
package from proposed. Once we have the release package we can test
that in the production environment."
Thanks!
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Kil
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
Dr. Harbott, would you be able to test the new audit packages in bionic-
proposed? The SRU team is reluctant to approve this update without some
sort of confirmation that it fixes the bug, and I haven't been able to
reproduce it myself.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process
exited, code=killed status=9
Sep 17 18:43:06 compute-node21
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
** Tags removed: verification-done-bionic
** Tags added: verification-needed-bionic
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process
exited, code=killed status=9
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result
'timeout'.
Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing
Service.
dp
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
I'd like to see some testing from someone actually affected please.
Otherwise we'd be releasing a change that risks regression to users not
affected, for a bug that we're not even sure if we've fixed.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process
exited, code=killed status=9
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
Since it's difficult to reproduce the bug, what I'm going to do is setup a system with the previous auditd, setup some rules, confirm they are working, then upgrade, and confirm it keeps working, also after a reboot. # Bionic verification auditd from bionic: auditd: Installed: 1:2.8.2-1ubuntu1 Candidate: 1:2.8.2-1ubuntu1 Version table: *** 1:2.8.2-1ubuntu1 500 500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages Created a simple rule: # cat /etc/audit/rules.d/30-shadow.rules -w /etc/shadow -p wa -k shadow-changed Loaded after restart: # auditctl -l -w /etc/shadow -p wa -k shadow-changed Confirmed a change to the file gets logged: # chmod 0400 /etc/shadow # /var/log/audit/auditd.log (parsed with ausearch -i): type=PROCTITLE msg=audit(01/18/21 17:49:31.077:32) : proctitle=chmod 0400 /etc/shadow type=PATH msg=audit(01/18/21 17:49:31.077:32) : item=0 name=/etc/shadow inode=64070 dev=fc:01 mode=file,640 ouid=root ogid=shadow rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 type=CWD msg=audit(01/18/21 17:49:31.077:32) : cwd=/root type=SYSCALL msg=audit(01/18/21 17:49:31.077:32) : arch=x86_64 syscall=fchmodat success=yes exit=0 a0=0xff9c a1=0x5577580dc1c0 a2=0400 a3=0x0 items=1 ppid=1499 pid=1992 auid=ubuntu uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts1 ses=3 comm=chmod exe=/bin/chmod key=shadow-changed Now updating the package: # apt-cache policy auditd auditd: Installed: 1:2.8.2-1ubuntu1.1 Candidate: 1:2.8.2-1ubuntu1.1 Version table: *** 1:2.8.2-1ubuntu1.1 500 500 http://br.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages 100 /var/lib/dpkg/status 1:2.8.2-1ubuntu1 500 500 http://br.archive.ubuntu.com/ubuntu bionic/main amd64 Packages (and its deps, like libaudit1, etc). The same rule continues loaded: # auditctl -l -w /etc/shadow -p wa -k shadow-changed Also after a manual restart: # systemctl restart auditd # auditctl -l -w /etc/shadow -p wa -k shadow-changed And changing /etc/shadow is logged (let's use 0640 this time): # chmod 0640 /etc/shadow # log: type=PROCTITLE msg=audit(01/18/21 17:54:51.942:56) : proctitle=chmod 0640 /etc/shadow type=PATH msg=audit(01/18/21 17:54:51.942:56) : item=0 name=/etc/shadow inode=64070 dev=fc:01 mode=file,400 ouid=root ogid=shadow rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 type=CWD msg=audit(01/18/21 17:54:51.942:56) : cwd=/root type=SYSCALL msg=audit(01/18/21 17:54:51.942:56) : arch=x86_64 syscall=fchmodat success=yes exit=0 a0=0xff9c a1=0x563ae04471c0 a2=0640 a3=0x0 items=1 ppid=1499 pid=2845 auid=ubuntu uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts1 ses=3 comm=chmod exe=/bin/chmod key=shadow-changed I then rebooted the system, performed the same tests, and got the same results with the updated package. It would be great if people who were affected by this bug, and can reasonably reproduce it, could test the packages from proposed. In the meantime, I'll mark this as verification succeeded. ** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to audit in Ubuntu. https://bugs.launchpad.net/bugs/1848330 Title: Installing auditd sometimes fails in post-inst Status in audit package in Ubuntu: Fix Released Status in audit source package in Bionic: Fix Committed Status in audit package in Debian: New Bug description: [Impact] Sometimes, auditd will get stuck when starting up, causing systemd to kill it after a while since it (systemd) never got the start notification. Upstream troubleshooted this to be caused by calling a syslog() function inside a signal handler. [Test Case] There is no reliable test case to reproduce the bug, other than trying the fixed packages on an affected system where the hang occurs more frequently. Basically: sudo systemctl stop auditd sudo systemctl start auditd should work reliably. Do not run that in a tight loop, however, as that will trigger a it's-restarting-too-frequently failure. [Where problems could occur] - if auditd fails to start, then the first fallback is syslog, and if that is not picking up the audit messages, the last resort is the kernel buffer, which can fill up. In the case it fills up, audit logs will be lost. - it's possible to configure the audit system to panic() the machine if audit messages are lost or otherwise not able to be recorded (auditctl -f 2; default is 1 which is printk()) - the update restarts auditd as expected. Misconfiguration on very very busy systems could mean that audit logs would be lost during the brief moment the service is restarted. If that's the case, this update would just be one more way to trigger it, but not be the root cause of t
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
All regressions have been resolved after some retries.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process
exited, code=killed status=9
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result
'timeout'.
Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing
Service.
dpkg: error processing package
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
I'm going over the DEP8 failures
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process
exited, code=killed status=9
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result
'timeout'.
Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing
Service.
dpkg: error processing package auditd (--configure):
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
I verified that this is fixed in both Focal and Hirsute by examining the
source (so presumably Groovy too).
** Also affects: audit (Ubuntu Bionic)
Importance: Undecided
Status: New
** Changed in: audit (Ubuntu)
Status: In Progress => Fix Released
** Changed in: audit (Ubuntu Bionic)
Status: New => Fix Committed
** Tags added: verification-needed verification-needed-bionic
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Fix Released
Status in audit source package in Bionic:
Fix Committed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (au
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
Package uploaded to the SRU queue
** Changed in: audit (Ubuntu)
Status: Confirmed => In Progress
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
In Progress
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process
exited, code=killed status=9
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result
'timeout'.
Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing
Service.
dpkg: error processing package auditd (-
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
** Description changed:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog() function
inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as that
will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine if
audit messages are lost or otherwise not able to be recorded (auditctl
-f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very very
busy systems could mean that audit logs would be lost during the brief
moment the service is restarted. If that's the case, this update would
just be one more way to trigger it, but not be the root cause of the
problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but was
never loaded before. The restart will load the config, and will fail in
such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
+ The real fix for this bug is just dropping the audit_msg() call in the signal
handler code. But the original reporter of the bug, who is also who came up
with the fix (see https://bugzilla.redhat.com/show_bug.cgi?id=1587995#c4)
stated that with the 3 changes in the patch the startup hang didn't happen to
him anymore. Since this bug is difficult to reproduce elsewhere (either you
have it, or you don't), I chose to keep the 3 changes instead of just the
removal of the audit_msg() call.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from the
failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process
exited, code=killed status=9
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result
'timeout'.
Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing
Service.
dpkg: error processing package auditd (--configure):
installed auditd package post-installation script subprocess returned error
exit status 1
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Confirmed
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
** Merge proposal linked:
https://code.launchpad.net/~ahasenack/ubuntu/+source/audit/+git/audit/+merge/396028
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to audit in Ubuntu.
https://bugs.launchpad.net/bugs/1848330
Title:
Installing auditd sometimes fails in post-inst
Status in audit package in Ubuntu:
Confirmed
Status in audit package in Debian:
New
Bug description:
[Impact]
Sometimes, auditd will get stuck when starting up, causing systemd to
kill it after a while since it (systemd) never got the start
notification.
Upstream troubleshooted this to be caused by calling a syslog()
function inside a signal handler.
[Test Case]
There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
Basically:
sudo systemctl stop auditd
sudo systemctl start auditd
should work reliably. Do not run that in a tight loop, however, as
that will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
- if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- it's possible to configure the audit system to panic() the machine
if audit messages are lost or otherwise not able to be recorded
(auditctl -f 2; default is 1 which is printk())
- the update restarts auditd as expected. Misconfiguration on very
very busy systems could mean that audit logs would be lost during the
brief moment the service is restarted. If that's the case, this update
would just be one more way to trigger it, but not be the root cause of
the problem
- similarly, as is usual with updates that restart services, it's
possible than an incorrect configuration for auditd is present, but
was never loaded before. The restart will load the config, and will
fail in such a case.
- this update removes a logging statement that occurs during startup:
("dispatcher %d reaped", pid)
It's unlikely, but possible, that some monitoring software could be
looking for that message in the logs. It won't be there anymore after
this update.
[Other Info]
The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
[Original Description]
This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also
fixes the issue, but the failure breaks our automation. The log from
the failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9703 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process
exited, code=killed status=9
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result
'timeout'.
Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing
Service.
dpkg: error processing package auditd (--configure):
installed auditd package post-installation script subprocess returned error
exit status 1
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/audit/+bug/1848330/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
** Description changed:
[Impact]
- * An explanation of the effects of the bug on users and
+ Sometimes, auditd will get stuck when starting up, causing systemd to
+ kill it after a while since it (systemd) never got the start
+ notification.
- * justification for backporting the fix to the stable release.
-
- * In addition, it is helpful, but not required, to include an
-explanation of how the upload fixes this bug.
+ Upstream troubleshooted this to be caused by calling a syslog() function
+ inside a signal handler.
[Test Case]
+ There is no reliable test case to reproduce the bug, other than trying the
fixed packages on an affected system where the hang occurs more frequently.
- * detailed instructions how to reproduce the bug
+ Basically:
+ sudo systemctl stop auditd
+ sudo systemctl start auditd
- * these should allow someone who is not familiar with the affected
-package to reproduce the bug and verify that the updated package fixes
-the problem.
+ should work reliably. Do not run that in a tight loop, however, as that
+ will trigger a it's-restarting-too-frequently failure.
[Where problems could occur]
+ - if auditd fails to start, then the first fallback is syslog, and if that is
not picking up the audit messages, the last resort is the kernel buffer, which
can fill up. In the case it fills up, audit logs will be lost.
- * Think about what the upload changes in the software. Imagine the change is
-wrong or breaks something else: how would this show up?
+ - it's possible to configure the audit system to panic() the machine if
+ audit messages are lost or otherwise not able to be recorded (auditctl
+ -f 2; default is 1 which is printk())
- * It is assumed that any SRU candidate patch is well-tested before
-upload and has a low overall risk of regression, but it's important
-to make the effort to think about what ''could'' happen in the
-event of a regression.
+ - the update restarts auditd as expected. Misconfiguration on very very
+ busy systems could mean that audit logs would be lost during the brief
+ moment the service is restarted. If that's the case, this update would
+ just be one more way to trigger it, but not be the root cause of the
+ problem
- * This must '''never''' be "None" or "Low", or entirely an argument as to why
-your upload is low risk.
+ - this update removes a logging statement that occurs during startup:
- * This both shows the SRU team that the risks have been considered,
-and provides guidance to testers in regression-testing the SRU.
+ ("dispatcher %d reaped", pid)
+
+ It's unlikely, but possible, that some monitoring software could be
+ looking for that message in the logs. It won't be there anymore after
+ this update.
+
[Other Info]
-
- * Anything else you think is useful to include
- * Anticipate questions from users, SRU, +1 maintenance, security teams and
the Technical Board
- * and address these questions in advance
+ The patch is committed upstream and part of the 2.8.5 release, which is
present in Focal and later.
[Original Description]
-
- This happens sometimes when installing auditd on Ubuntu 18.04.2, most
installations work successfully, though. Re-running the install also fixes the
issue, but the failure breaks our automation. The log from the failure looks
like this:
+ This happens sometimes when installing auditd on Ubuntu 18.04.2, most
+ installations work successfully, though. Re-running the install also
+ fixes the issue, but the failure breaks our automation. The log from the
+ failure looks like this:
# apt install auditd
...
Setting up auditd (1:2.8.2-1ubuntu1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service →
/lib/systemd/system/auditd.service.
Job for auditd.service failed because a timeout was exceeded.
See "systemctl status auditd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript auditd, action "start" failed.
● auditd.service - Security Auditing Service
Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor
preset: enabled)
Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms
ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL)
Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing
Service...
Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher:
/sbin/audispd pid: 9705
Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting
Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation
timed out. Terminating.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State
'stop-sigterm' timed out. Killing.
Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process
9702 (auditd) with signal SIGKILL.
Sep 17 18:43:06 compute
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
Yikes @Kodiak, sounds painful :( -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to audit in Ubuntu. https://bugs.launchpad.net/bugs/1848330 Title: Installing auditd sometimes fails in post-inst Status in audit package in Ubuntu: Confirmed Status in audit package in Debian: New Bug description: [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Case] * detailed instructions how to reproduce the bug * these should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem. [Where problems could occur] * Think about what the upload changes in the software. Imagine the change is wrong or breaks something else: how would this show up? * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a regression. * This must '''never''' be "None" or "Low", or entirely an argument as to why your upload is low risk. * This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU. [Other Info] * Anything else you think is useful to include * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board * and address these questions in advance [Original Description] This happens sometimes when installing auditd on Ubuntu 18.04.2, most installations work successfully, though. Re-running the install also fixes the issue, but the failure breaks our automation. The log from the failure looks like this: # apt install auditd ... Setting up auditd (1:2.8.2-1ubuntu1) ... Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service → /lib/systemd/system/auditd.service. Job for auditd.service failed because a timeout was exceeded. See "systemctl status auditd.service" and "journalctl -xe" for details. invoke-rc.d: initscript auditd, action "start" failed. ● auditd.service - Security Auditing Service Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms ago Docs: man:auditd(8) https://github.com/linux-audit/audit-documentation Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL) Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing Service... Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher: /sbin/audispd pid: 9705 Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation timed out. Terminating. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State 'stop-sigterm' timed out. Killing. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9702 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9703 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process exited, code=killed status=9 Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result 'timeout'. Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing Service. dpkg: error processing package auditd (--configure): installed auditd package post-installation script subprocess returned error exit status 1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/audit/+bug/1848330/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
** Description changed: - This happens sometimes when installing auditd on Ubuntu 18.04.2, most - installations work successfully, though. Re-running the install also - fixes the issue, but the failure breaks our automation. The log from the - failure looks like this: + [Impact] + + * An explanation of the effects of the bug on users and + + * justification for backporting the fix to the stable release. + + * In addition, it is helpful, but not required, to include an +explanation of how the upload fixes this bug. + + [Test Case] + + * detailed instructions how to reproduce the bug + + * these should allow someone who is not familiar with the affected +package to reproduce the bug and verify that the updated package fixes +the problem. + + [Where problems could occur] + + * Think about what the upload changes in the software. Imagine the change is +wrong or breaks something else: how would this show up? + + * It is assumed that any SRU candidate patch is well-tested before +upload and has a low overall risk of regression, but it's important +to make the effort to think about what ''could'' happen in the +event of a regression. + + * This must '''never''' be "None" or "Low", or entirely an argument as to why +your upload is low risk. + + * This both shows the SRU team that the risks have been considered, +and provides guidance to testers in regression-testing the SRU. + + [Other Info] + + * Anything else you think is useful to include + * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board + * and address these questions in advance + + + [Original Description] + + + This happens sometimes when installing auditd on Ubuntu 18.04.2, most installations work successfully, though. Re-running the install also fixes the issue, but the failure breaks our automation. The log from the failure looks like this: # apt install auditd ... Setting up auditd (1:2.8.2-1ubuntu1) ... Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service → /lib/systemd/system/auditd.service. Job for auditd.service failed because a timeout was exceeded. See "systemctl status auditd.service" and "journalctl -xe" for details. invoke-rc.d: initscript auditd, action "start" failed. ● auditd.service - Security Auditing Service -Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) -Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms ago - Docs: man:auditd(8) -https://github.com/linux-audit/audit-documentation - Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL) + Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) + Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms ago + Docs: man:auditd(8) + https://github.com/linux-audit/audit-documentation + Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL) Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing Service... Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher: /sbin/audispd pid: 9705 Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation timed out. Terminating. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State 'stop-sigterm' timed out. Killing. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9702 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9703 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process exited, code=killed status=9 Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result 'timeout'. Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing Service. dpkg: error processing package auditd (--configure): - installed auditd package post-installation script subprocess returned error exit status 1 + installed auditd package post-installation script subprocess returned error exit status 1 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to audit in Ubuntu. https://bugs.launchpad.net/bugs/1848330 Title: Installing auditd sometimes fails in post-inst Status in audit package in Ubuntu: Confirmed Status in audit package in Debian: New Bug description: [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Case] * detailed instructions how to reproduce the bug * these should allow someone who is not familiar with the affected pac
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
Apologies - I don't remember the full specifics here but IIRC, the Mandiant FireEye HX agent foolishly re-implements Linux Audit, and it's every bit as terrible as that sounds. We discovered that we needed to basically purge both FireEye HX and Auditd on the system, then install Auditd, then disable it entirely, then allow the travesty of having the FireEye HX agent run it's own auditd daemon which clobbers the OS auditd. Yikes. And yes we filed a "please don't do such ugly things with core Linux services" request with their support. :/ -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to audit in Ubuntu. https://bugs.launchpad.net/bugs/1848330 Title: Installing auditd sometimes fails in post-inst Status in audit package in Ubuntu: Confirmed Status in audit package in Debian: New Bug description: This happens sometimes when installing auditd on Ubuntu 18.04.2, most installations work successfully, though. Re-running the install also fixes the issue, but the failure breaks our automation. The log from the failure looks like this: # apt install auditd ... Setting up auditd (1:2.8.2-1ubuntu1) ... Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service → /lib/systemd/system/auditd.service. Job for auditd.service failed because a timeout was exceeded. See "systemctl status auditd.service" and "journalctl -xe" for details. invoke-rc.d: initscript auditd, action "start" failed. ● auditd.service - Security Auditing Service Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms ago Docs: man:auditd(8) https://github.com/linux-audit/audit-documentation Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL) Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing Service... Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher: /sbin/audispd pid: 9705 Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation timed out. Terminating. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State 'stop-sigterm' timed out. Killing. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9702 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9703 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process exited, code=killed status=9 Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result 'timeout'. Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing Service. dpkg: error processing package auditd (--configure): installed auditd package post-installation script subprocess returned error exit status 1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/audit/+bug/1848330/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
I'm having difficulties reproducing the bug, to validate the patch. I build bionic test packages with the patch mentioned earlier, if someone wants to test: https://launchpad.net/~ahasenack/+archive/ubuntu/audit- startup-hang-1848330 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to audit in Ubuntu. https://bugs.launchpad.net/bugs/1848330 Title: Installing auditd sometimes fails in post-inst Status in audit package in Ubuntu: Confirmed Status in audit package in Debian: New Bug description: This happens sometimes when installing auditd on Ubuntu 18.04.2, most installations work successfully, though. Re-running the install also fixes the issue, but the failure breaks our automation. The log from the failure looks like this: # apt install auditd ... Setting up auditd (1:2.8.2-1ubuntu1) ... Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service → /lib/systemd/system/auditd.service. Job for auditd.service failed because a timeout was exceeded. See "systemctl status auditd.service" and "journalctl -xe" for details. invoke-rc.d: initscript auditd, action "start" failed. ● auditd.service - Security Auditing Service Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms ago Docs: man:auditd(8) https://github.com/linux-audit/audit-documentation Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL) Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing Service... Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher: /sbin/audispd pid: 9705 Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation timed out. Terminating. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State 'stop-sigterm' timed out. Killing. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9702 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9703 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process exited, code=killed status=9 Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result 'timeout'. Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing Service. dpkg: error processing package auditd (--configure): installed auditd package post-installation script subprocess returned error exit status 1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/audit/+bug/1848330/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
** Changed in: audit (Debian) Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to audit in Ubuntu. https://bugs.launchpad.net/bugs/1848330 Title: Installing auditd sometimes fails in post-inst Status in audit package in Ubuntu: Confirmed Status in audit package in Debian: New Bug description: This happens sometimes when installing auditd on Ubuntu 18.04.2, most installations work successfully, though. Re-running the install also fixes the issue, but the failure breaks our automation. The log from the failure looks like this: # apt install auditd ... Setting up auditd (1:2.8.2-1ubuntu1) ... Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service → /lib/systemd/system/auditd.service. Job for auditd.service failed because a timeout was exceeded. See "systemctl status auditd.service" and "journalctl -xe" for details. invoke-rc.d: initscript auditd, action "start" failed. ● auditd.service - Security Auditing Service Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms ago Docs: man:auditd(8) https://github.com/linux-audit/audit-documentation Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL) Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing Service... Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher: /sbin/audispd pid: 9705 Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation timed out. Terminating. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State 'stop-sigterm' timed out. Killing. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9702 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9703 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process exited, code=killed status=9 Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result 'timeout'. Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing Service. dpkg: error processing package auditd (--configure): installed auditd package post-installation script subprocess returned error exit status 1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/audit/+bug/1848330/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1848330] Re: Installing auditd sometimes fails in post-inst
correct source package name is audit, not auditd (apparently we have/had both) ** Changed in: auditd (Ubuntu) Assignee: (unassigned) => Andreas Hasenack (ahasenack) ** Bug watch added: Debian Bug tracker #962451 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962451 ** Also affects: audit (Debian) via https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962451 Importance: Unknown Status: Unknown ** Package changed: auditd (Ubuntu) => audit (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to audit in Ubuntu. https://bugs.launchpad.net/bugs/1848330 Title: Installing auditd sometimes fails in post-inst Status in audit package in Ubuntu: Confirmed Status in audit package in Debian: Unknown Bug description: This happens sometimes when installing auditd on Ubuntu 18.04.2, most installations work successfully, though. Re-running the install also fixes the issue, but the failure breaks our automation. The log from the failure looks like this: # apt install auditd ... Setting up auditd (1:2.8.2-1ubuntu1) ... Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service → /lib/systemd/system/auditd.service. Job for auditd.service failed because a timeout was exceeded. See "systemctl status auditd.service" and "journalctl -xe" for details. invoke-rc.d: initscript auditd, action "start" failed. ● auditd.service - Security Auditing Service Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Tue 2019-09-17 18:43:06 UTC; 11ms ago Docs: man:auditd(8) https://github.com/linux-audit/audit-documentation Process: 9702 ExecStart=/sbin/auditd (code=killed, signal=KILL) Sep 17 18:40:06 compute-node21 systemd[1]: Starting Security Auditing Service... Sep 17 18:40:06 compute-node21 auditd[9703]: Started dispatcher: /sbin/audispd pid: 9705 Sep 17 18:40:06 compute-node21 audispd[9705]: No plugins found, exiting Sep 17 18:41:36 compute-node21 systemd[1]: auditd.service: Start operation timed out. Terminating. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: State 'stop-sigterm' timed out. Killing. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9702 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Killing process 9703 (auditd) with signal SIGKILL. Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Control process exited, code=killed status=9 Sep 17 18:43:06 compute-node21 systemd[1]: auditd.service: Failed with result 'timeout'. Sep 17 18:43:06 compute-node21 systemd[1]: Failed to start Security Auditing Service. dpkg: error processing package auditd (--configure): installed auditd package post-installation script subprocess returned error exit status 1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/audit/+bug/1848330/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
