** Attachment added: "Output of force-reexec.sh on a trusty system with fixes for both bugs, showing the problem has been resolved." https://bugs.launchpad.net/upstart/+bug/1338637/+attachment/4207317/+files/typescript.good
** Description changed: = PROBLEM = The version of Upstart currently in trusty (1.12.1-0ubuntu4.2) suffers from a couple of problems which in combination could make upgrading difficult on systems with high uptimes. The 2 issues are: == bug 901038 == The issue here is that telinit in trusty is not fully synchronous. This is not normally a problem. However if Upstart is re-exec'ed using 'telinit u', as is done by the maintainer scripts for the following packages... - - libc6 - - libdbus-1-3 - - libjson-c2 - - libnih1 - - libnih-dbus1 - - libselinux1 - - libsepol1 - - upstart + - libc6 + - libdbus-1-3 + - libjson-c2 + - libnih1 + - libnih-dbus1 + - libselinux1 + - libsepol1 + - upstart ... and if that operation is slow (it is normally extremely fast), subsequent package upgrades (part of the same apt-get run) which *also* need to restart Upstart may fail since 'telinit u' is unable to connect to PID 1 as a result of Upstart still being in the process of re-exec'ing from the last call to 'telinit u'. - == bug 1338637 == This bug only really affects server systems which have high uptimes. If a re-exec is triggered via 'telinit u' as part of a package upgrade (see above), Upstart will consume 2 additional inotify watches. Although this doesn't affect the correct behaviour of Upstart, it does have two repercussions: a) It wastes inotify watches. b) It slows down an Upstart re-exec ('telinit u'). Note that the slow-down for a vanilla Trusty server may not be detectable unless the value of /proc/sys/fs/inotify/max_user_instances has been raised above the default value of 128. - = FIX DETAILS = == bug 901038 == The fix for this bug now makes 'telinit u' block until the re-exec operation has completed fully. Technically, it is not possible to make 'telinit u' synchronous since because D-Bus connections cannot be serialised and since the 'telinit u' request is made via a D-Bus connection, when Upstart re-exec's, it has to sever all D-Bus connections, including the 'telinit u' D-Bus connection. As such, 'telinit u' now performs the following operations: - - Requests synchronously that Upstart re-exec itself. - - Polls Upstart "forever" by attempting to connect to PID 1 and if - that operation fails, waiting for a period, the retrying. + - Requests synchronously that Upstart re-exec itself. + - Polls Upstart "forever" by attempting to connect to PID 1 and if + that operation fails, waiting for a period, the retrying. The code is well commented to explain this less-than-ideal but nonetheless essential poll operation: - http://bazaar.launchpad.net/~upstart- + http://bazaar.launchpad.net/~upstart- devel/upstart/trunk/view/head:/util/telinit.c#L171 == bug 1338637 == This was a simple bug fix. - = FIX AVAILABILITY = TBD. - = IMPACT = On a high uptime server, the more times that 'telinit u' is called (as the result of normal apt-get updates to any of the packages listed under bug 901038 above), the slower the operation will take to complete, and thus the likelihood of bug 901038 being seen will increase. - = JUSTIFICATION = System updates need to "just work". However, as outlined above, the longer a systems uptime, the more likely it is to be affected by this issue which increases the likelihood of a system update failure. This issue needs to be fixed as soon as possible to minimise issues for trusty users and administrators, particularly before any potential upgrade to Utopic. - = TEST CASE = == To demonstrate bug 1338637 == $ sudo telinit u $ sudo ls -l /proc/1/fd|grep inotify|wc -l The correct output from the above 2 commands should be simply "2". However, on trusty systems, if the commands above are run after a fresh boot, the value will in all likelihood be "4". Also, every subsequent 'telinit u' will increase the number displayed by 2. == To demonstrate bug 901038 == for i in $(seq 10); do telinit u; done There should be no output from the command-line above, but on a trusty system the chances are that one or more lines will display like this: $ for i in $(seq 5); do sudo telinit u;done telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused == To prove the overall problem has been fixed == 1) Download the attached "force-reexec.sh" onto a trusty system. 2) Make executable: - $ chmod 755 ./force-reexec.sh + $ chmod 755 ./force-reexec.sh 3) Run script as root, specifying a number of iterations (100) and - saving output to file "./typescript.bad": + saving output to file "./typescript.bad": - $ script -c 'sudo ./force-reexec.sh 100' typescript.bad + $ script -c 'sudo ./force-reexec.sh 100' typescript.bad 4) Upgrade to latest version of Upstart that fixes the 2 bugs. 5) Reboot. 6) Re-run the script - $ script -c 'sudo ./force-reexec.sh 100' typescript.good + $ script -c 'sudo ./force-reexec.sh 100' typescript.good 7) Check the results: - - file "typescript.bad" will show an increasing number of watches - and *MAY* show a slowly increasing restart time (depending on - the value of /proc/sys/fs/inotify/max_user_instances (see above)). + - file "typescript.bad" will show an increasing number of watches + and *MAY* show a slowly increasing restart time (depending on + the value of /proc/sys/fs/inotify/max_user_instances (see above)). - - file "typescript.good" should consistently show 2 watches and a - restart time of 0 on average. + - file "typescript.good" should consistently show 2 watches and a + restart time of 0 on average. + + = TEST RESULTS = + + See the 2 attached typescript files showing a run of force-reexec.sh on + an affected system, and the other showing the output of force-reexec.sh + on a system where the problem has been resolved. [REGRESSION POTENTIAL] The only theoretically potential issue is what happens if the continuous poll performed by 'telinit u' never completes. However, this should never happen since: 1) Upstart checks to ensure that it can serialise its own state *before* - it actually performs the re-exec. If it cannot for some reason (the - only possibility here is critically low memory), it will - automatically degrade to a stateless re-exec. A stateless re-exec can - only fail if the on-disk "/sbin/init" binary or associated libraries - are somehow corrupted. + it actually performs the re-exec. If it cannot for some reason (the + only possibility here is critically low memory), it will + automatically degrade to a stateless re-exec. A stateless re-exec can + only fail if the on-disk "/sbin/init" binary or associated libraries + are somehow corrupted. 2) If, after checking that its own state can be serialised, the actual - re-exec operation fails "mid-flight", again, Upstart will - automatically revert to performing a simple stateless re-exec which - can only fail if the on-disk "/sbin/init" binary or associated - libraries are somehow corrupted. - + re-exec operation fails "mid-flight", again, Upstart will + automatically revert to performing a simple stateless re-exec which + can only fail if the on-disk "/sbin/init" binary or associated + libraries are somehow corrupted. = OTHER INFORMATION = == Upgrade Procedure == Note that updating a system to the latest version of Upstart that fixes the two bugs outlined above will stop the number of inotify watches growing, but will NOT bring the value down to the expected "2". As such, to correct the problem fully, it is necessary to reboot the system after successfully upgrading the Upstart package. == D-Bus Daemon == Note that although Upstart makes heavy use of D-Bus, it does not require a D-Bus daemon to be running. Specifically, 'telinit u' communicates with PID 1 via a private abstract D-Bus socket, so is immune from issues with dbus-daemon(1). - - = Original Description = $ sudo ls -al /proc/1/fd|grep anon|wc -l 2 $ i=0; while [ $i -lt 1024 ]; do sudo telinit u; i=$((i+1)); done $ sudo ls -al /proc/1/fd|grep anon|wc -l 106 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to upstart in Ubuntu. https://bugs.launchpad.net/bugs/1338637 Title: continuous re-exec can result in a build-up of inotify fds [SRU] Status in Upstart: Fix Released Status in “upstart” package in Ubuntu: New Status in “upstart” source package in Trusty: New Bug description: = PROBLEM = The version of Upstart currently in trusty (1.12.1-0ubuntu4.2) suffers from a couple of problems which in combination could make upgrading difficult on systems with high uptimes. The 2 issues are: == bug 901038 == The issue here is that telinit in trusty is not fully synchronous. This is not normally a problem. However if Upstart is re-exec'ed using 'telinit u', as is done by the maintainer scripts for the following packages... - libc6 - libdbus-1-3 - libjson-c2 - libnih1 - libnih-dbus1 - libselinux1 - libsepol1 - upstart ... and if that operation is slow (it is normally extremely fast), subsequent package upgrades (part of the same apt-get run) which *also* need to restart Upstart may fail since 'telinit u' is unable to connect to PID 1 as a result of Upstart still being in the process of re-exec'ing from the last call to 'telinit u'. == bug 1338637 == This bug only really affects server systems which have high uptimes. If a re-exec is triggered via 'telinit u' as part of a package upgrade (see above), Upstart will consume 2 additional inotify watches. Although this doesn't affect the correct behaviour of Upstart, it does have two repercussions: a) It wastes inotify watches. b) It slows down an Upstart re-exec ('telinit u'). Note that the slow-down for a vanilla Trusty server may not be detectable unless the value of /proc/sys/fs/inotify/max_user_instances has been raised above the default value of 128. = FIX DETAILS = == bug 901038 == The fix for this bug now makes 'telinit u' block until the re-exec operation has completed fully. Technically, it is not possible to make 'telinit u' synchronous since because D-Bus connections cannot be serialised and since the 'telinit u' request is made via a D-Bus connection, when Upstart re-exec's, it has to sever all D-Bus connections, including the 'telinit u' D-Bus connection. As such, 'telinit u' now performs the following operations: - Requests synchronously that Upstart re-exec itself. - Polls Upstart "forever" by attempting to connect to PID 1 and if that operation fails, waiting for a period, the retrying. The code is well commented to explain this less-than-ideal but nonetheless essential poll operation: http://bazaar.launchpad.net/~upstart- devel/upstart/trunk/view/head:/util/telinit.c#L171 == bug 1338637 == This was a simple bug fix. = FIX AVAILABILITY = TBD. = IMPACT = On a high uptime server, the more times that 'telinit u' is called (as the result of normal apt-get updates to any of the packages listed under bug 901038 above), the slower the operation will take to complete, and thus the likelihood of bug 901038 being seen will increase. = JUSTIFICATION = System updates need to "just work". However, as outlined above, the longer a systems uptime, the more likely it is to be affected by this issue which increases the likelihood of a system update failure. This issue needs to be fixed as soon as possible to minimise issues for trusty users and administrators, particularly before any potential upgrade to Utopic. = TEST CASE = == To demonstrate bug 1338637 == $ sudo telinit u $ sudo ls -l /proc/1/fd|grep inotify|wc -l The correct output from the above 2 commands should be simply "2". However, on trusty systems, if the commands above are run after a fresh boot, the value will in all likelihood be "4". Also, every subsequent 'telinit u' will increase the number displayed by 2. == To demonstrate bug 901038 == for i in $(seq 10); do telinit u; done There should be no output from the command-line above, but on a trusty system the chances are that one or more lines will display like this: $ for i in $(seq 5); do sudo telinit u;done telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused telinit: Failed to connect to socket /com/ubuntu/upstart: Connection refused == To prove the overall problem has been fixed == 1) Download the attached "force-reexec.sh" onto a trusty system. 2) Make executable: $ chmod 755 ./force-reexec.sh 3) Run script as root, specifying a number of iterations (100) and saving output to file "./typescript.bad": $ script -c 'sudo ./force-reexec.sh 100' typescript.bad 4) Upgrade to latest version of Upstart that fixes the 2 bugs. 5) Reboot. 6) Re-run the script $ script -c 'sudo ./force-reexec.sh 100' typescript.good 7) Check the results: - file "typescript.bad" will show an increasing number of watches and *MAY* show a slowly increasing restart time (depending on the value of /proc/sys/fs/inotify/max_user_instances (see above)). - file "typescript.good" should consistently show 2 watches and a restart time of 0 on average. = TEST RESULTS = See the 2 attached typescript files showing a run of force-reexec.sh on an affected system, and the other showing the output of force- reexec.sh on a system where the problem has been resolved. [REGRESSION POTENTIAL] The only theoretically potential issue is what happens if the continuous poll performed by 'telinit u' never completes. However, this should never happen since: 1) Upstart checks to ensure that it can serialise its own state *before* it actually performs the re-exec. If it cannot for some reason (the only possibility here is critically low memory), it will automatically degrade to a stateless re-exec. A stateless re-exec can only fail if the on-disk "/sbin/init" binary or associated libraries are somehow corrupted. 2) If, after checking that its own state can be serialised, the actual re-exec operation fails "mid-flight", again, Upstart will automatically revert to performing a simple stateless re-exec which can only fail if the on-disk "/sbin/init" binary or associated libraries are somehow corrupted. = OTHER INFORMATION = == Upgrade Procedure == Note that updating a system to the latest version of Upstart that fixes the two bugs outlined above will stop the number of inotify watches growing, but will NOT bring the value down to the expected "2". As such, to correct the problem fully, it is necessary to reboot the system after successfully upgrading the Upstart package. == D-Bus Daemon == Note that although Upstart makes heavy use of D-Bus, it does not require a D-Bus daemon to be running. Specifically, 'telinit u' communicates with PID 1 via a private abstract D-Bus socket, so is immune from issues with dbus-daemon(1). = Original Description = $ sudo ls -al /proc/1/fd|grep anon|wc -l 2 $ i=0; while [ $i -lt 1024 ]; do sudo telinit u; i=$((i+1)); done $ sudo ls -al /proc/1/fd|grep anon|wc -l 106 To manage notifications about this bug go to: https://bugs.launchpad.net/upstart/+bug/1338637/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp