** Attachment added: "Output of force-reexec.sh on a trusty system with fixes
for both bugs, showing the problem has been resolved."
https://bugs.launchpad.net/upstart/+bug/1338637/+attachment/4207317/+files/typescript.good
** Description changed:
= PROBLEM =
The version of Upstart currently in trusty (1.12.1-0ubuntu4.2) suffers
from a couple of problems which in combination could make upgrading
difficult on systems with high uptimes.
The 2 issues are:
== bug 901038 ==
The issue here is that telinit in trusty is not fully synchronous. This
is not normally a problem. However if Upstart is re-exec'ed using
'telinit u', as is done by the maintainer scripts for the following
packages...
- - libc6
- - libdbus-1-3
- - libjson-c2
- - libnih1
- - libnih-dbus1
- - libselinux1
- - libsepol1
- - upstart
+ - libc6
+ - libdbus-1-3
+ - libjson-c2
+ - libnih1
+ - libnih-dbus1
+ - libselinux1
+ - libsepol1
+ - upstart
... and if that operation is slow (it is normally extremely fast),
subsequent package upgrades (part of the same apt-get run) which *also*
need to restart Upstart may fail since 'telinit u' is unable to connect
to PID 1 as a result of Upstart still being in the process of
re-exec'ing from the last call to 'telinit u'.
-
== bug 1338637 ==
This bug only really affects server systems which have high uptimes. If
a re-exec is triggered via 'telinit u' as part of a package upgrade (see
above), Upstart will consume 2 additional inotify watches. Although this
doesn't affect the correct behaviour of Upstart, it does have two
repercussions:
a) It wastes inotify watches.
b) It slows down an Upstart re-exec ('telinit u').
Note that the slow-down for a vanilla Trusty server may not be
detectable unless the value of /proc/sys/fs/inotify/max_user_instances
has been raised above the default value of 128.
-
= FIX DETAILS =
== bug 901038 ==
The fix for this bug now makes 'telinit u' block until the re-exec
operation has completed fully.
Technically, it is not possible to make 'telinit u' synchronous since
because D-Bus connections cannot be serialised and since the 'telinit u'
request is made via a D-Bus connection, when Upstart re-exec's, it has
to sever all D-Bus connections, including the 'telinit u' D-Bus
connection. As such, 'telinit u' now performs the following operations:
- - Requests synchronously that Upstart re-exec itself.
- - Polls Upstart "forever" by attempting to connect to PID 1 and if
- that operation fails, waiting for a period, the retrying.
+ - Requests synchronously that Upstart re-exec itself.
+ - Polls Upstart "forever" by attempting to connect to PID 1 and if
+ that operation fails, waiting for a period, the retrying.
The code is well commented to explain this less-than-ideal but
nonetheless essential poll operation:
- http://bazaar.launchpad.net/~upstart-
+ http://bazaar.launchpad.net/~upstart-
devel/upstart/trunk/view/head:/util/telinit.c#L171
== bug 1338637 ==
This was a simple bug fix.
-
= FIX AVAILABILITY =
TBD.
-
= IMPACT =
On a high uptime server, the more times that 'telinit u' is called (as
the result of normal apt-get updates to any of the packages listed under
bug 901038 above), the slower the operation will take to complete, and
thus the likelihood of bug 901038 being seen will increase.
-
= JUSTIFICATION =
System updates need to "just work". However, as outlined above, the
longer a systems uptime, the more likely it is to be affected by this
issue which increases the likelihood of a system update failure.
This issue needs to be fixed as soon as possible to minimise issues for
trusty users and administrators, particularly before any potential
upgrade to Utopic.
-
= TEST CASE =
== To demonstrate bug 1338637 ==
$ sudo telinit u
$ sudo ls -l /proc/1/fd|grep inotify|wc -l
The correct output from the above 2 commands should be simply "2".
However, on trusty systems, if the commands above are run after a fresh
boot, the value will in all likelihood be "4". Also, every subsequent
'telinit u' will increase the number displayed by 2.
== To demonstrate bug 901038 ==
for i in $(seq 10); do telinit u; done
There should be no output from the command-line above, but on a trusty
system the chances are that one or more lines will display like this:
$ for i in $(seq 5); do sudo telinit u;done
telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused
telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused
telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused
== To prove the overall problem has been fixed ==
1) Download the attached "force-reexec.sh" onto a trusty system.
2) Make executable:
- $ chmod 755 ./force-reexec.sh
+ $ chmod 755 ./force-reexec.sh
3) Run script as root, specifying a number of iterations (100) and
- saving output to file "./typescript.bad":
+ saving output to file "./typescript.bad":
- $ script -c 'sudo ./force-reexec.sh 100' typescript.bad
+ $ script -c 'sudo ./force-reexec.sh 100' typescript.bad
4) Upgrade to latest version of Upstart that fixes the 2 bugs.
5) Reboot.
6) Re-run the script
- $ script -c 'sudo ./force-reexec.sh 100' typescript.good
+ $ script -c 'sudo ./force-reexec.sh 100' typescript.good
7) Check the results:
- - file "typescript.bad" will show an increasing number of watches
- and *MAY* show a slowly increasing restart time (depending on
- the value of /proc/sys/fs/inotify/max_user_instances (see above)).
+ - file "typescript.bad" will show an increasing number of watches
+ and *MAY* show a slowly increasing restart time (depending on
+ the value of /proc/sys/fs/inotify/max_user_instances (see above)).
- - file "typescript.good" should consistently show 2 watches and a
- restart time of 0 on average.
+ - file "typescript.good" should consistently show 2 watches and a
+ restart time of 0 on average.
+
+ = TEST RESULTS =
+
+ See the 2 attached typescript files showing a run of force-reexec.sh on
+ an affected system, and the other showing the output of force-reexec.sh
+ on a system where the problem has been resolved.
[REGRESSION POTENTIAL]
The only theoretically potential issue is what happens if the continuous
poll performed by 'telinit u' never completes. However, this should
never happen since:
1) Upstart checks to ensure that it can serialise its own state *before*
- it actually performs the re-exec. If it cannot for some reason (the
- only possibility here is critically low memory), it will
- automatically degrade to a stateless re-exec. A stateless re-exec can
- only fail if the on-disk "/sbin/init" binary or associated libraries
- are somehow corrupted.
+ it actually performs the re-exec. If it cannot for some reason (the
+ only possibility here is critically low memory), it will
+ automatically degrade to a stateless re-exec. A stateless re-exec can
+ only fail if the on-disk "/sbin/init" binary or associated libraries
+ are somehow corrupted.
2) If, after checking that its own state can be serialised, the actual
- re-exec operation fails "mid-flight", again, Upstart will
- automatically revert to performing a simple stateless re-exec which
- can only fail if the on-disk "/sbin/init" binary or associated
- libraries are somehow corrupted.
-
+ re-exec operation fails "mid-flight", again, Upstart will
+ automatically revert to performing a simple stateless re-exec which
+ can only fail if the on-disk "/sbin/init" binary or associated
+ libraries are somehow corrupted.
= OTHER INFORMATION =
== Upgrade Procedure ==
Note that updating a system to the latest version of Upstart that fixes
the two bugs outlined above will stop the number of inotify watches
growing, but will NOT bring the value down to the expected "2".
As such, to correct the problem fully, it is necessary to reboot the
system after successfully upgrading the Upstart package.
== D-Bus Daemon ==
Note that although Upstart makes heavy use of D-Bus, it does not require
a D-Bus daemon to be running. Specifically, 'telinit u' communicates
with PID 1 via a private abstract D-Bus socket, so is immune from issues
with dbus-daemon(1).
-
-
= Original Description =
$ sudo ls -al /proc/1/fd|grep anon|wc -l
2
$ i=0; while [ $i -lt 1024 ]; do sudo telinit u; i=$((i+1)); done
$ sudo ls -al /proc/1/fd|grep anon|wc -l
106
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to upstart in Ubuntu.
https://bugs.launchpad.net/bugs/1338637
Title:
continuous re-exec can result in a build-up of inotify fds [SRU]
Status in Upstart:
Fix Released
Status in “upstart” package in Ubuntu:
New
Status in “upstart” source package in Trusty:
New
Bug description:
= PROBLEM =
The version of Upstart currently in trusty (1.12.1-0ubuntu4.2) suffers
from a couple of problems which in combination could make upgrading
difficult on systems with high uptimes.
The 2 issues are:
== bug 901038 ==
The issue here is that telinit in trusty is not fully synchronous. This
is not normally a problem. However if Upstart is re-exec'ed using
'telinit u', as is done by the maintainer scripts for the following
packages...
- libc6
- libdbus-1-3
- libjson-c2
- libnih1
- libnih-dbus1
- libselinux1
- libsepol1
- upstart
... and if that operation is slow (it is normally extremely fast),
subsequent package upgrades (part of the same apt-get run) which *also*
need to restart Upstart may fail since 'telinit u' is unable to connect
to PID 1 as a result of Upstart still being in the process of
re-exec'ing from the last call to 'telinit u'.
== bug 1338637 ==
This bug only really affects server systems which have high uptimes. If
a re-exec is triggered via 'telinit u' as part of a package upgrade (see
above), Upstart will consume 2 additional inotify watches. Although this
doesn't affect the correct behaviour of Upstart, it does have two
repercussions:
a) It wastes inotify watches.
b) It slows down an Upstart re-exec ('telinit u').
Note that the slow-down for a vanilla Trusty server may not be
detectable unless the value of /proc/sys/fs/inotify/max_user_instances
has been raised above the default value of 128.
= FIX DETAILS =
== bug 901038 ==
The fix for this bug now makes 'telinit u' block until the re-exec
operation has completed fully.
Technically, it is not possible to make 'telinit u' synchronous since
because D-Bus connections cannot be serialised and since the 'telinit u'
request is made via a D-Bus connection, when Upstart re-exec's, it has
to sever all D-Bus connections, including the 'telinit u' D-Bus
connection. As such, 'telinit u' now performs the following operations:
- Requests synchronously that Upstart re-exec itself.
- Polls Upstart "forever" by attempting to connect to PID 1 and if
that operation fails, waiting for a period, the retrying.
The code is well commented to explain this less-than-ideal but
nonetheless essential poll operation:
http://bazaar.launchpad.net/~upstart-
devel/upstart/trunk/view/head:/util/telinit.c#L171
== bug 1338637 ==
This was a simple bug fix.
= FIX AVAILABILITY =
TBD.
= IMPACT =
On a high uptime server, the more times that 'telinit u' is called (as
the result of normal apt-get updates to any of the packages listed under
bug 901038 above), the slower the operation will take to complete, and
thus the likelihood of bug 901038 being seen will increase.
= JUSTIFICATION =
System updates need to "just work". However, as outlined above, the
longer a systems uptime, the more likely it is to be affected by this
issue which increases the likelihood of a system update failure.
This issue needs to be fixed as soon as possible to minimise issues for
trusty users and administrators, particularly before any potential
upgrade to Utopic.
= TEST CASE =
== To demonstrate bug 1338637 ==
$ sudo telinit u
$ sudo ls -l /proc/1/fd|grep inotify|wc -l
The correct output from the above 2 commands should be simply "2".
However, on trusty systems, if the commands above are run after a fresh
boot, the value will in all likelihood be "4". Also, every subsequent
'telinit u' will increase the number displayed by 2.
== To demonstrate bug 901038 ==
for i in $(seq 10); do telinit u; done
There should be no output from the command-line above, but on a trusty
system the chances are that one or more lines will display like this:
$ for i in $(seq 5); do sudo telinit u;done
telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused
telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused
telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused
== To prove the overall problem has been fixed ==
1) Download the attached "force-reexec.sh" onto a trusty system.
2) Make executable:
$ chmod 755 ./force-reexec.sh
3) Run script as root, specifying a number of iterations (100) and
saving output to file "./typescript.bad":
$ script -c 'sudo ./force-reexec.sh 100' typescript.bad
4) Upgrade to latest version of Upstart that fixes the 2 bugs.
5) Reboot.
6) Re-run the script
$ script -c 'sudo ./force-reexec.sh 100' typescript.good
7) Check the results:
- file "typescript.bad" will show an increasing number of watches
and *MAY* show a slowly increasing restart time (depending on
the value of /proc/sys/fs/inotify/max_user_instances (see above)).
- file "typescript.good" should consistently show 2 watches and a
restart time of 0 on average.
= TEST RESULTS =
See the 2 attached typescript files showing a run of force-reexec.sh
on an affected system, and the other showing the output of force-
reexec.sh on a system where the problem has been resolved.
[REGRESSION POTENTIAL]
The only theoretically potential issue is what happens if the continuous
poll performed by 'telinit u' never completes. However, this should
never happen since:
1) Upstart checks to ensure that it can serialise its own state *before*
it actually performs the re-exec. If it cannot for some reason (the
only possibility here is critically low memory), it will
automatically degrade to a stateless re-exec. A stateless re-exec can
only fail if the on-disk "/sbin/init" binary or associated libraries
are somehow corrupted.
2) If, after checking that its own state can be serialised, the actual
re-exec operation fails "mid-flight", again, Upstart will
automatically revert to performing a simple stateless re-exec which
can only fail if the on-disk "/sbin/init" binary or associated
libraries are somehow corrupted.
= OTHER INFORMATION =
== Upgrade Procedure ==
Note that updating a system to the latest version of Upstart that fixes
the two bugs outlined above will stop the number of inotify watches
growing, but will NOT bring the value down to the expected "2".
As such, to correct the problem fully, it is necessary to reboot the
system after successfully upgrading the Upstart package.
== D-Bus Daemon ==
Note that although Upstart makes heavy use of D-Bus, it does not require
a D-Bus daemon to be running. Specifically, 'telinit u' communicates
with PID 1 via a private abstract D-Bus socket, so is immune from issues
with dbus-daemon(1).
= Original Description =
$ sudo ls -al /proc/1/fd|grep anon|wc -l
2
$ i=0; while [ $i -lt 1024 ]; do sudo telinit u; i=$((i+1)); done
$ sudo ls -al /proc/1/fd|grep anon|wc -l
106
To manage notifications about this bug go to:
https://bugs.launchpad.net/upstart/+bug/1338637/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
