[Touch-packages] [Bug 1691901] Re: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems
This bug was fixed in the package base-files - 9.6ubuntu13.1 --- base-files (9.6ubuntu13.1) zesty-proposed; urgency=medium * update-motd.d/50-news: Fix LP: #1691901 - when the curl fails (perhaps due to a firewall issue), we should create an empty cache file, so that subsequent runs are not penalized; ensure that we *only* print when we're not forced -- Dustin KirklandThu, 29 Jun 2017 09:22:56 -0500 ** Changed in: base-files (Ubuntu Zesty) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to base-files in Ubuntu. https://bugs.launchpad.net/bugs/1691901 Title: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems Status in base-files package in Ubuntu: Fix Released Status in base-files source package in Zesty: Fix Released Status in base-files source package in Artful: Fix Released Bug description: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. === SRU === [ IMPACT ] This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually. [ TEST CASE ] You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately. Failure case: $ lxc launch ubuntu:17.04 LP1691901 $ lxc exec LP1691901 bash # ssh-import-id kirkland # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts # rm -f /var/cache/motd-news # exit $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m5.333s user0m0.016s sys 0m0.000s Apply the fix. $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m0.316s user0m0.008s sys 0m0.008s [ REGRESSION ] This is a simple, safe fix with minimal regression potential: diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.0 -0600 +++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.0 -0500 @@ -51,9 +51,13 @@ # If we're not forcing an update, and we have a cached motd-news file, # then just print it and exit as quickly as possible, for login performance. # Note that systemd should keep this cache file up to date, asynchronously -if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then - echo - safe_print $CACHE +if [ "$FORCED" != "1" ]; then + if [ -r $CACHE ]; then + echo + safe_print $CACHE + else + : > $CACHE + fi exit 0 fi @@ -111,7 +115,9 @@ safe_print "$NEWS" # Try to update the cache safe_print "$NEWS" 2>/dev/null >$CACHE || true - fi + else + : > "$CACHE" + fi done rm -f "$NEWS" "$NEWS.err" exit 0 -- [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108 ProblemType: Bug DistroRelease: Ubuntu 17.04 Package: base-files 9.6ubuntu13 ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11 Uname: Linux 4.10.0-21-generic x86_64 ApportVersion: 2.20.4-0ubuntu4 Architecture: amd64 Date: Fri May 19
[Touch-packages] [Bug 1691901] Re: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems
I've run through the test case as described, and confirm that this takes care of the login delay problem. ** Tags removed: verification-needed verification-needed-zesty ** Tags added: verification-done-zesty ** Tags removed: removal-candidate -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to base-files in Ubuntu. https://bugs.launchpad.net/bugs/1691901 Title: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems Status in base-files package in Ubuntu: Fix Released Status in base-files source package in Zesty: Fix Committed Status in base-files source package in Artful: Fix Released Bug description: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. === SRU === [ IMPACT ] This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually. [ TEST CASE ] You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately. Failure case: $ lxc launch ubuntu:17.04 LP1691901 $ lxc exec LP1691901 bash # ssh-import-id kirkland # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts # rm -f /var/cache/motd-news # exit $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m5.333s user0m0.016s sys 0m0.000s Apply the fix. $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m0.316s user0m0.008s sys 0m0.008s [ REGRESSION ] This is a simple, safe fix with minimal regression potential: diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.0 -0600 +++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.0 -0500 @@ -51,9 +51,13 @@ # If we're not forcing an update, and we have a cached motd-news file, # then just print it and exit as quickly as possible, for login performance. # Note that systemd should keep this cache file up to date, asynchronously -if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then - echo - safe_print $CACHE +if [ "$FORCED" != "1" ]; then + if [ -r $CACHE ]; then + echo + safe_print $CACHE + else + : > $CACHE + fi exit 0 fi @@ -111,7 +115,9 @@ safe_print "$NEWS" # Try to update the cache safe_print "$NEWS" 2>/dev/null >$CACHE || true - fi + else + : > "$CACHE" + fi done rm -f "$NEWS" "$NEWS.err" exit 0 -- [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108 ProblemType: Bug DistroRelease: Ubuntu 17.04 Package: base-files 9.6ubuntu13 ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11 Uname: Linux 4.10.0-21-generic x86_64 ApportVersion: 2.20.4-0ubuntu4 Architecture: amd64 Date: Fri May 19 01:11:30 2017 Ec2AMI: ami-004f Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova Ec2InstanceType: m1.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash
[Touch-packages] [Bug 1691901] Re: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems
Hello Scott, or anyone else affected, Accepted base-files into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base- files/9.6ubuntu13.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty.If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: base-files (Ubuntu Zesty) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-zesty -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to base-files in Ubuntu. https://bugs.launchpad.net/bugs/1691901 Title: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems Status in base-files package in Ubuntu: Fix Released Status in base-files source package in Zesty: Fix Committed Status in base-files source package in Artful: Fix Released Bug description: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. === SRU === [ IMPACT ] This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually. [ TEST CASE ] You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately. Failure case: $ lxc launch ubuntu:17.04 LP1691901 $ lxc exec LP1691901 bash # ssh-import-id kirkland # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts # rm -f /var/cache/motd-news # exit $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m5.333s user0m0.016s sys 0m0.000s Apply the fix. $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m0.316s user0m0.008s sys 0m0.008s [ REGRESSION ] This is a simple, safe fix with minimal regression potential: diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.0 -0600 +++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.0 -0500 @@ -51,9 +51,13 @@ # If we're not forcing an update, and we have a cached motd-news file, # then just print it and exit as quickly as possible, for login performance. # Note that systemd should keep this cache file up to date, asynchronously -if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then - echo - safe_print $CACHE +if [ "$FORCED" != "1" ]; then + if [ -r $CACHE ]; then + echo + safe_print $CACHE + else + : > $CACHE + fi exit 0 fi @@ -111,7 +115,9 @@ safe_print "$NEWS"
[Touch-packages] [Bug 1691901] Re: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems
This bug was fixed in the package base-files - 9.6ubuntu101 --- base-files (9.6ubuntu101) artful; urgency=medium * update-motd.d/50-news: Fix LP: #1691901 - when the curl fails (perhaps due to a firewall issue), we should create an empty cache file, so that subsequent runs are not penalized; ensure that we *only* print when we're not forced -- Dustin KirklandThu, 29 Jun 2017 10:35:06 -0500 ** Changed in: base-files (Ubuntu Artful) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to base-files in Ubuntu. https://bugs.launchpad.net/bugs/1691901 Title: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems Status in base-files package in Ubuntu: Fix Released Status in base-files source package in Zesty: In Progress Status in base-files source package in Artful: Fix Released Bug description: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. === SRU === [ IMPACT ] This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually. [ TEST CASE ] You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately. Failure case: $ lxc launch ubuntu:17.04 LP1691901 $ lxc exec LP1691901 bash # ssh-import-id kirkland # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts # rm -f /var/cache/motd-news # exit $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m5.333s user0m0.016s sys 0m0.000s Apply the fix. $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m0.316s user0m0.008s sys 0m0.008s [ REGRESSION ] This is a simple, safe fix with minimal regression potential: diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.0 -0600 +++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.0 -0500 @@ -51,9 +51,13 @@ # If we're not forcing an update, and we have a cached motd-news file, # then just print it and exit as quickly as possible, for login performance. # Note that systemd should keep this cache file up to date, asynchronously -if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then - echo - safe_print $CACHE +if [ "$FORCED" != "1" ]; then + if [ -r $CACHE ]; then + echo + safe_print $CACHE + else + : > $CACHE + fi exit 0 fi @@ -111,7 +115,9 @@ safe_print "$NEWS" # Try to update the cache safe_print "$NEWS" 2>/dev/null >$CACHE || true - fi + else + : > "$CACHE" + fi done rm -f "$NEWS" "$NEWS.err" exit 0 -- [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108 ProblemType: Bug DistroRelease: Ubuntu 17.04 Package: base-files 9.6ubuntu13 ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11 Uname: Linux 4.10.0-21-generic x86_64 ApportVersion: 2.20.4-0ubuntu4 Architecture: amd64 Date: Fri May 19 01:11:30 2017
[Touch-packages] [Bug 1691901] Re: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems
** Description changed: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. - === SRU === [ IMPACT ] This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually. [ TEST CASE ] You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately. Failure case: $ lxc launch ubuntu:17.04 LP1691901 $ lxc exec LP1691901 bash # ssh-import-id kirkland # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts # rm -f /var/cache/motd-news # exit $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m5.333s user0m0.016s sys 0m0.000s - Apply the fix. Note that it'll take one run (might be a login, a - reboot, or the 12-hourly timer) to create the file: + Apply the fix. - $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true - real0m5.357s - user0m0.016s - sys 0m0.004s $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m0.316s user0m0.008s sys 0m0.008s [ REGRESSION ] - This is a super simple, safe, 2-line fix with minimal regression - potential: + This is a simple, safe fix with minimal regression potential: - diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.1/update-motd.d/50-motd-news - --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.0 -0600 - +++ base-files-9.6ubuntu13.1/update-motd.d/50-motd-news 2017-06-29 09:22:31.0 -0500 - @@ -111,6 +111,8 @@ - safe_print "$NEWS" - # Try to update the cache - safe_print "$NEWS" 2>/dev/null >$CACHE || true - + else - + : > "$CACHE" - fi + diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news + --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.0 -0600 + +++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.0 -0500 + @@ -51,9 +51,13 @@ + # If we're not forcing an update, and we have a cached motd-news file, + # then just print it and exit as quickly as possible, for login performance. + # Note that systemd should keep this cache file up to date, asynchronously + -if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then + - echo + - safe_print $CACHE + +if [ "$FORCED" != "1" ]; then + + if [ -r $CACHE ]; then + + echo + + safe_print $CACHE + + else + + : > $CACHE + + fi + exit 0 + fi + + @@ -111,7 +115,9 @@ + safe_print "$NEWS" + # Try to update the cache + safe_print "$NEWS" 2>/dev/null >$CACHE || true + - fi + + else + + : > "$CACHE" + + fi done rm -f "$NEWS" "$NEWS.err" - + exit 0 -- [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108 ProblemType: Bug DistroRelease: Ubuntu 17.04 Package: base-files 9.6ubuntu13 ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11 Uname: Linux 4.10.0-21-generic x86_64 ApportVersion: 2.20.4-0ubuntu4 Architecture: amd64 Date: Fri May 19 01:11:30 2017 Ec2AMI: ami-004f Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova
[Touch-packages] [Bug 1691901] Re: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems
16:03 kirkland: ^ what happens on first login? That'll still be delayed, right? So the bug is being reduced in scope, but not being fixed entirely? 16:03 If the first login is before the timer has run for the first time, that is, which seems likely. 16:04 rbasak: true 16:04 rbasak: that's an easy fix, though, just one more line to add 16:05 rbasak: would you like me to fix that? 16:05 I think it would be a good idea, yeah 16:05 kirkland: yes please. Probably better for the SRU process to do it fewer times :) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to base-files in Ubuntu. https://bugs.launchpad.net/bugs/1691901 Title: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems Status in base-files package in Ubuntu: Fix Committed Status in base-files source package in Zesty: Fix Committed Status in base-files source package in Artful: Fix Committed Bug description: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. === SRU === [ IMPACT ] This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually. [ TEST CASE ] You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately. Failure case: $ lxc launch ubuntu:17.04 LP1691901 $ lxc exec LP1691901 bash # ssh-import-id kirkland # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts # rm -f /var/cache/motd-news # exit $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m5.333s user0m0.016s sys 0m0.000s Apply the fix. Note that it'll take one run (might be a login, a reboot, or the 12-hourly timer) to create the file: $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m5.357s user0m0.016s sys 0m0.004s $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real0m0.316s user0m0.008s sys 0m0.008s [ REGRESSION ] This is a super simple, safe, 2-line fix with minimal regression potential: diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.1/update-motd.d/50-motd-news --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.0 -0600 +++ base-files-9.6ubuntu13.1/update-motd.d/50-motd-news 2017-06-29 09:22:31.0 -0500 @@ -111,6 +111,8 @@ safe_print "$NEWS" # Try to update the cache safe_print "$NEWS" 2>/dev/null >$CACHE || true + else + : > "$CACHE" fi done rm -f "$NEWS" "$NEWS.err" -- [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108 ProblemType: Bug DistroRelease: Ubuntu 17.04 Package: base-files 9.6ubuntu13 ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11 Uname: Linux 4.10.0-21-generic x86_64 ApportVersion: 2.20.4-0ubuntu4 Architecture: amd64 Date: Fri May 19 01:11:30 2017 Ec2AMI: ami-004f Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova Ec2InstanceType: m1.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: base-files UpgradeStatus: No upgrade log
[Touch-packages] [Bug 1691901] Re: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems
** Description changed: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: - curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com + curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. + + + === SRU === + + [ IMPACT ] + + This bug affects Ubuntu 17.04 systems which cannot reach the internet + (more specifically, https://motd.ubuntu.com). There is a bug in the + implementation of /etc/update-motd.d/50-news whereby the failure of the + curl fetch of the motd news causes a 5-second delay. This should not be + the case. By design, the motd is fetched in the background, by a + systemd timer, and refreshed every ~12 hours, also in the background. + On login, the script should just cat the cache file. If the cache file + isn't found, then it will try to fetch a new motd. In the firewalled + system case, the background fetch failed, AND didn't create an empty + file. The fix as attached simply ensures that an empty file is created + if the curl fetch fails. This eliminates the login delay and solves the + bug. Every 12 hours, the curl will be retried, quietly in the + background, so if the curl failure happened for temporary reasons, it'll + recover, gracefully, eventually. + + [ TEST CASE ] + + You can either test this on a firewalled system. Or, if you can hack an + entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP + address. Without the fix, you'll experience a 5 second delay on login. + With the fix, you'll login immediately. + + Failure case: + + $ lxc launch ubuntu:17.04 LP1691901 + $ lxc exec LP1691901 bash + # ssh-import-id kirkland + # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts + # rm -f /var/cache/motd-news + # exit + $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true + real0m5.333s + user0m0.016s + sys 0m0.000s + + Apply the fix. Note that it'll take one run (might be a login, a + reboot, or the 12-hourly timer) to create the file: + + $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true + real0m5.357s + user0m0.016s + sys 0m0.004s + $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true + real0m0.316s + user0m0.008s + sys 0m0.008s + + [ REGRESSION ] + + This is a super simple, safe, 2-line fix with minimal regression + potential: + + diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.1/update-motd.d/50-motd-news + --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.0 -0600 + +++ base-files-9.6ubuntu13.1/update-motd.d/50-motd-news 2017-06-29 09:22:31.0 -0500 + @@ -111,6 +111,8 @@ + safe_print "$NEWS" + # Try to update the cache + safe_print "$NEWS" 2>/dev/null >$CACHE || true + + else + + : > "$CACHE" + fi + done + rm -f "$NEWS" "$NEWS.err" + -- [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108 ProblemType: Bug DistroRelease: Ubuntu 17.04 Package: base-files 9.6ubuntu13 ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11 Uname: Linux 4.10.0-21-generic x86_64 ApportVersion: 2.20.4-0ubuntu4 Architecture: amd64 Date: Fri May 19 01:11:30 2017 Ec2AMI: ami-004f Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova Ec2InstanceType: m1.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable ProcEnviron: - TERM=xterm-256color - PATH=(custom, no user) - XDG_RUNTIME_DIR= - LANG=en_US.UTF-8 - SHELL=/bin/bash + TERM=xterm-256color + PATH=(custom, no user) + XDG_RUNTIME_DIR= + LANG=en_US.UTF-8 + SHELL=/bin/bash SourcePackage: base-files UpgradeStatus: No upgrade log present (probably fresh install) mtime.conffile..etc.update-motd.d.50-motd-news: 2017-05-19T01:10:25.572110 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to base-files in Ubuntu. https://bugs.launchpad.net/bugs/1691901 Title: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems Status in base-files package in Ubuntu: Fix Committed Status in base-files source package in Zesty: Fix Committed Status in base-files source package in Artful: Fix Committed Bug description: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file