** Description changed: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. - ======= SRU ======= [ IMPACT ] This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually. [ TEST CASE ] You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately. Failure case: $ lxc launch ubuntu:17.04 LP1691901 $ lxc exec LP1691901 bash # ssh-import-id kirkland # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts # rm -f /var/cache/motd-news # exit $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real 0m5.333s user 0m0.016s sys 0m0.000s - Apply the fix. Note that it'll take one run (might be a login, a - reboot, or the 12-hourly timer) to create the file: + Apply the fix. - $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true - real 0m5.357s - user 0m0.016s - sys 0m0.004s $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real 0m0.316s user 0m0.008s sys 0m0.008s [ REGRESSION ] - This is a super simple, safe, 2-line fix with minimal regression - potential: + This is a simple, safe fix with minimal regression potential: - diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.1/update-motd.d/50-motd-news - --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.000000000 -0600 - +++ base-files-9.6ubuntu13.1/update-motd.d/50-motd-news 2017-06-29 09:22:31.000000000 -0500 - @@ -111,6 +111,8 @@ - safe_print "$NEWS" - # Try to update the cache - safe_print "$NEWS" 2>/dev/null >$CACHE || true - + else - + : > "$CACHE" - fi + diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news + --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.000000000 -0600 + +++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.000000000 -0500 + @@ -51,9 +51,13 @@ + # If we're not forcing an update, and we have a cached motd-news file, + # then just print it and exit as quickly as possible, for login performance. + # Note that systemd should keep this cache file up to date, asynchronously + -if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then + - echo + - safe_print $CACHE + +if [ "$FORCED" != "1" ]; then + + if [ -r $CACHE ]; then + + echo + + safe_print $CACHE + + else + + : > $CACHE + + fi + exit 0 + fi + + @@ -111,7 +115,9 @@ + safe_print "$NEWS" + # Try to update the cache + safe_print "$NEWS" 2>/dev/null >$CACHE || true + - fi + + else + + : > "$CACHE" + + fi done rm -f "$NEWS" "$NEWS.err" - + exit 0 -- [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108 ProblemType: Bug DistroRelease: Ubuntu 17.04 Package: base-files 9.6ubuntu13 ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11 Uname: Linux 4.10.0-21-generic x86_64 ApportVersion: 2.20.4-0ubuntu4 Architecture: amd64 Date: Fri May 19 01:11:30 2017 Ec2AMI: ami-0000004f Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova Ec2InstanceType: m1.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: base-files UpgradeStatus: No upgrade log present (probably fresh install) mtime.conffile..etc.update-motd.d.50-motd-news: 2017-05-19T01:10:25.572110
** Changed in: base-files (Ubuntu Zesty) Status: Fix Committed => In Progress -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to base-files in Ubuntu. https://bugs.launchpad.net/bugs/1691901 Title: [SRU] 50-motd-news costs 5 seconds every login on firewalled systems Status in base-files package in Ubuntu: Fix Committed Status in base-files source package in Zesty: In Progress Status in base-files source package in Artful: Fix Committed Bug description: I noticed quite a long time to login to some of my systems via ssh (or scp). Investigating lead me to find out that the '50-motd-news' file (/etc/update-motd.d/50-motd-news) was the primary cost. In line 108 [1] it does something like: curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com. The way the lab is configured, they just end up timing out. So every scp or ssh connection or other path to trigger update-motd will cost 5 seconds. ======= SRU ======= [ IMPACT ] This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually. [ TEST CASE ] You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately. Failure case: $ lxc launch ubuntu:17.04 LP1691901 $ lxc exec LP1691901 bash # ssh-import-id kirkland # echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts # rm -f /var/cache/motd-news # exit $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real 0m5.333s user 0m0.016s sys 0m0.000s Apply the fix. $ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true real 0m0.316s user 0m0.008s sys 0m0.008s [ REGRESSION ] This is a simple, safe fix with minimal regression potential: diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news --- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.000000000 -0600 +++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.000000000 -0500 @@ -51,9 +51,13 @@ # If we're not forcing an update, and we have a cached motd-news file, # then just print it and exit as quickly as possible, for login performance. # Note that systemd should keep this cache file up to date, asynchronously -if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then - echo - safe_print $CACHE +if [ "$FORCED" != "1" ]; then + if [ -r $CACHE ]; then + echo + safe_print $CACHE + else + : > $CACHE + fi exit 0 fi @@ -111,7 +115,9 @@ safe_print "$NEWS" # Try to update the cache safe_print "$NEWS" 2>/dev/null >$CACHE || true - fi + else + : > "$CACHE" + fi done rm -f "$NEWS" "$NEWS.err" exit 0 -- [1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108 ProblemType: Bug DistroRelease: Ubuntu 17.04 Package: base-files 9.6ubuntu13 ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11 Uname: Linux 4.10.0-21-generic x86_64 ApportVersion: 2.20.4-0ubuntu4 Architecture: amd64 Date: Fri May 19 01:11:30 2017 Ec2AMI: ami-0000004f Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova Ec2InstanceType: m1.small Ec2Kernel: unavailable Ec2Ramdisk: unavailable ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: base-files UpgradeStatus: No upgrade log present (probably fresh install) mtime.conffile..etc.update-motd.d.50-motd-news: 2017-05-19T01:10:25.572110 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/1691901/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp