** Description changed: - We run into transient network issues where index files fail to download. - The deployment ends up being marked as failed. Then subsequent - deployment succeeds but test has already failed. Curtin should be able - to retry when such error happens. + === Begin SRU Template === + [Description] + During installation, curtin will run 'apt-get update' on in the target root. That is done as a requirement to installing new packages in the target. + + 'apt-get update' is widely known to fail as a result of transient + network failures. This is commonly worked around by simply sleeping and + re-trying the operation. + + The solution implemented is to improve the 'subp' (subprocess) helper in curtin/util to take a 'retries' argument. + If provided that is a iterator that contains a time to sleep before trying again. If no retries is provided, then only one try is done. + Then, the curtin/util.py helper apt_update invokes subp with retries=(1, 2, 3). + + [Impact] + Installation fails when a simple retry of 'apt-get update' would have succeeded. + + [Test Case] + As this is a transient failure, it is hard to catch and hard to test for. + + Installation should be more reliable now, with any 'apt-get update' + operation that returned non-zero being retried 3 times. + + [Regression Potential] + The only really likely regression path here would be retrying 'apt update' on its successful return. That seems fairly unlikely as the code in subp to check exit status has not changed. + + [Other] + Related bugs: + * bug 972077: apt repository disk format has race conditions + === End SRU Template === + + + We run into transient network issues where index files fail to download. The deployment ends up being marked as failed. Then subsequent deployment succeeds but test has already failed. Curtin should be able to retry when such error happens. Here's console output: ======================================================================== Get:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B] Get:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B] Get:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB] Fetched 13.8 MB in 5s (2426 kB/s) W: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch E: Some index files failed to download. They have been ignored, or old ones used instead. Unexpected error while running command. Command: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet'] Exit code: 100 Reason: - Stdout: '' Stderr: '' Installation failed with exception: Unexpected error while running command. Command: ['curtin', 'curthooks'] Exit code: 3 Reason: - Stdout: "Ign http://archive.ubuntu.com trusty InRelease\nIgn http://archive.ubuntu.com trusty-updates InRelease\nIgn http://archive.ubuntu.com trusty-security InRelease\nGet:1 http://archive.ubuntu.com trusty Release.gpg [933 B]\nGet:2 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]\nGet:3 http://archive.ubuntu.com trusty-security Release.gpg [933 B]\nGet:4 http://archive.ubuntu.com trusty Release [58.5 kB]\nGet:5 http://archive.ubuntu.com trusty-updates Release [62.0 kB]\nGet:6 http://archive.ubuntu.com trusty-security Release [62.0 kB]\nGet:7 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]\nGet:8 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]\nGet:9 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]\nGet:10 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]\nGet:11 http://archive.ubuntu.com trusty/main Translation-en [762 kB]\nGet:12 http://archive.ubuntu.com trusty/multiverse Translation-en [102 k B]\nGet:13 http://archive.ubuntu.com trusty/restricted Translation-en [3457 B]\nGet:14 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]\nGet:15 http://archive.ubuntu.com trusty-updates/main amd64 Packages [384 kB]\nGet:16 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [8861 B]\nGet:17 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [228 kB]\nGet:18 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [9356 B]\nGet:19 http://archive.ubuntu.com trusty-updates/main Translation-en [179 kB]\nGet:20 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [4719 B]\nGet:21 http://archive.ubuntu.com trusty-updates/restricted Translation-en [2266 B]\nGet:22 http://archive.ubuntu.com trusty-updates/universe Translation-en [117 kB]\nGet:23 http://archive.ubuntu.com trusty-security/main amd64 Packages [181 kB]\nGet:24 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [8861 B]\nGet:25 http://archive. ubuntu.com trusty-security/universe amd64 Packages [76.0 kB]\nGet:26 http://archive.ubuntu.com trusty-security/multiverse amd64 Packages [1143 B]\nGet:27 http://archive.ubuntu.com trusty-security/main Translation-en [90.8 kB]\nGet:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]\nGet:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]\nGet:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]\nFetched 13.8 MB in 5s (2426 kB/s)\nW: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch\n\nE: Some index files failed to download. They have been ignored, or old ones used instead.\nUnexpected error while running command.\nCommand: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']\nExit code: 100\nReason: -\nStdout: ''\nStderr: ''\n" Stderr: '' Success ci-info: +++++++Authorized keys ========================================================================
-- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1403133 Title: curtin should retry fetching from archives after transient failure To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1403133/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
