[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
Just saw this on some field units that are iOT type devices and sometimes get shutdown non-cleanly. We run apt-update on startup and we noticed one machine would not update and had no knowledge of any packages in one of the repos in the sources.list. We noticed the Packages file has 0 size, and no amount of apt update or apt-cache clean could fix it besides deleting/removing the corrupt file. Deleting it resolved the issue: ``` $ ls -lat /var/lib/apt/lists/ total 71252 drwx-- 2 _apt root 4096 Apr 23 16:18 partial drwxr-xr-x 5 root root 4096 Apr 23 15:40 .. drwxr-xr-x 4 root root 4096 Apr 22 21:13 . -rw-r--r-- 1 root root 4342 Apr 22 20:12 [redacted-package-server]-experimental_dists_focal_InRelease -rw-r--r-- 1 root root0 Apr 22 20:12 [redacted-package-server]-experimental_dists_focal_main_binary-amd64_Packages -rw-r--r-- 1 root root 4362 Jan 27 2023 [redacted-package-server]-ubuntu-experimental_dists_focal_InRelease ``` Note the 0 file size on the `[redacted-package- server]-experimental_dists_focal_main_binary-amd64_Packages` file. During this time, it would not Get any new information from the package server. Even with updates pushed to the package server. After removing the file, it does a `Get` and now can see all the packages again: $ sudo mv /var/lib/apt/lists/[redacted-package-server]-experimental_dists_focal_main_binary-amd64_Packages $ sudo apt update Hit:1 [redacted-server]-experimental focal InRelease Hit:2 [redacted-server]-ubuntu-experimental focal InRelease Get:3 [redacted-server]-experimental focal/main amd64 Packages [3524 kB] Fetched 3524 kB in 6s (571 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 27 packages can be upgraded. Run 'apt list --upgradable' to see them. ``` Server software is aptly. apt version is `apt 2.0.6 (amd64)` Happy to track this down and submit a patch for checking the file size of 0, or the algorithm as suggested above. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: Triaged Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
** Tags added: fr-281 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: Triaged Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filing one here. The situation occurred on an Ubuntu 16.04 system, but is 100% reproducible with Google's chrome repository on my Ubuntu 18.04.1 laptop. I can provide a set of reproduction steps if needed, but it's fairly straight-forward. The fact that this corruption appears to be "everything working okay" to the end user, except that apt doesn't know about packages it says it knows about, and there is no error logging for any sort, is partly why I'm filing this. Note the "if one of two things happens" case a) above: if the repository
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
** Tags added: id-5c336e5b216dc852b7a80d86 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: Triaged Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filing one here. The situation occurred on an Ubuntu 16.04 system, but is 100% reproducible with Google's chrome repository on my Ubuntu 18.04.1 laptop. I can provide a set of reproduction steps if needed, but it's fairly straight-forward. The fact that this corruption appears to be "everything working okay" to the end user, except that apt doesn't know about packages it says it knows about, and there is no error logging for any sort, is partly why I'm filing this. Note the "if one of two things happens" case a) above
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
"reporting bugs to the effect that a no-change update should finish instantly." By definition, if the on-disk copy is corrupt, there is an change-update available. The algorithm for update should look like this: - does the local copy exist? No -> update available - is the local copy valid (checksums match)? No -> update available - does the remote repo report a change? Yes -> update available "That said, files of size 0 could be made always invalid" My case happened to be corruption of the form of an empty file. Checking the checksum will detect all forms of corruption. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: Triaged Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filin
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
Note that the file we have in lists/ is not what we downloaded as we have downloaded a highly compressed version of the content (e.g. xz), but store it either uncompressed (for which we have a checksum) or lightly compressed (e.g. lz4 for which we have no checksum and can not as different versions of a compressor could produce different files). So such a check is not exactly free as we need to potentially uncompress the content we want to check – we can't even do a size check in the general case "for free". It might be worth it paying the price in "update", but there are a bunch of people who believe we shouldn't, reporting bugs to the effect that a no-change update should finish instantly. That said, files of size 0 could be made always invalid: We don't download such files nowadays (as an empty file compressed has a [small] size), so files in lists/ should have at least some content and if they don't something is absolutely fishy. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: Triaged Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
I agree. ** Changed in: apt (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: Triaged Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filing one here. The situation occurred on an Ubuntu 16.04 system, but is 100% reproducible with Google's chrome repository on my Ubuntu 18.04.1 laptop. I can provide a set of reproduction steps if needed, but it's fairly straight-forward. The fact that this corruption appears to be "everything working okay" to the end user, except that apt doesn't know about packages it says it knows about, and there is no error logging for any sort, is partly why I'm filing this. Note the "if one of two thin
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
I was picturing scenario (1); if there are updates or the checksum on the existing doesn't match, download the index. apt has trained users to "apt update" before doing anything else, so this would be okay. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: New Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filing one here. The situation occurred on an Ubuntu 16.04 system, but is 100% reproducible with Google's chrome repository on my Ubuntu 18.04.1 laptop. I can provide a set of reproduction steps if needed, but it's fairly straight-forward. The fact that this corruption appears to be "everything working okay" to the end user, except that apt doesn't know about package
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
So, 0.166s is very expensive in the sense that we can't do it every time we open the cache. We're literally optimizing every 10ms we can get there - apt-cache show would be twice as fast. There are two more points where we could do those checks, though: (1) in update - we can check if the lists we have are correct before deciding whether to download new ones (2) in install - we can abort with an error before installing packages The first one should essentially be free, and is probably the best way to fix the issue, as you don't see any upgrades, run update, and see them. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: New Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filing on
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
Filesystem is ext4 ubuntu@T00-tx2b:~$ mount /dev/mmcblk0p1 on / type ext4 (rw,relatime,data=ordered) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: New Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filing one here. The situation occurred on an Ubuntu 16.04 system, but is 100% reproducible with Google's chrome repository on my Ubuntu 18.04.1 laptop. I can provide a set of reproduction steps if needed, but it's fairly straight-forward. The fact that this corruption appears to be "everything working okay" to the end user, except that apt doesn't know about packages it says it knows about, and there is no error logging for any sort, is partly why I'm filing thi
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
The problem with deferring the situation to the clean command is that the system appears to be working properly with no problem, and the user will have to _guess_ that cleanup is required. Running md5sum on the packages is not expensive: ubuntu@T00-tx2b:/var/lib/apt/lists$ time md5sum ports.ubuntu.com_ubuntu-ports_dists_xenial_universe_binary-arm64_Packages 005695bf761c4719325927b5a236a77e ports.ubuntu.com_ubuntu-ports_dists_xenial_universe_binary-arm64_Packages real0m0.166s user0m0.136s sys 0m0.036s Certainly it's much faster than the downloading of the indexes in the first place. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: New Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
Verifying the checksums in normal operation is too expensive. I think it would be reasonable to make the clean command do that, and remove broken files. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: New Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filing one here. The situation occurred on an Ubuntu 16.04 system, but is 100% reproducible with Google's chrome repository on my Ubuntu 18.04.1 laptop. I can provide a set of reproduction steps if needed, but it's fairly straight-forward. The fact that this corruption appears to be "everything working okay" to the end user, except that apt doesn't know about packages it says it knows about, and there is no error loggi
[Touch-packages] [Bug 1809174] Re: apt doesn't detect file corruption in /var/lib/apt/lists
I'm just curious, but what filesystem are those filesystems on? My understanding is that 0 byte files should not happen on ext4 at least. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1809174 Title: apt doesn't detect file corruption in /var/lib/apt/lists Status in apt package in Ubuntu: New Bug description: The Problem == /var/lib/apt/lists contains the repository index caches or similar; I'm not sure what the correct apt-terminology is. I've installed Chrome on my laptop, so I have: smacdonald@L247:/var/lib/apt/lists$ dir *goog* -rw-r--r-- 1 root root 943 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release -rw-r--r-- 1 root root 819 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_Release.gpg -rw-r--r-- 1 root root 4457 Dec 19 14:02 dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages for example. dl.google.com_linux_chrome_deb_dists_stable_Release contains checksums for the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file: smacdonald@L247:/var/lib/apt/lists$ cat dl.google.com_linux_chrome_deb_dists_stable_Release Origin: Google LLC Label: Google Suite: stable Codename: stable Version: 1.0 Date: Wed, 19 Dec 2018 18:51:54 UTC Architectures: amd64 Components: main Description: Google chrome-linux software repository MD5Sum: 9e0d0ad6a4f5ccf8e3971c32e9bb22d3 4457 main/binary-amd64/Packages a17f6de0ef487b82af58ccd91df52d04 1109 main/binary-amd64/Packages.gz 156e5ea7a0c6bed5973a68a45e546dc9 151 main/binary-amd64/Release SHA1: 4c2cde4f71476d7881262d9a07e33cf4506232a7 4457 main/binary-amd64/Packages e002924c9ddfe41ee2033594ec768ed9e4545909 1109 main/binary-amd64/Packages.gz 0f4348c2d4d7cc1f8e59b5934d87f1ca872f6e34 151 main/binary-amd64/Release SHA256: fb0e586c2b5ec5afa17965d0bbc6bd46c2071336f75e2b0f0c7f3e7b090a7844 4457 main/binary-amd64/Packages 2462cff732765679a56373a7ca9a5b8b029fdb445e707b1aba10d01fbdb853b3 1109 main/binary-amd64/Packages.gz c1e3c9318381862306adcdc4fd4fe2d85be8aa4c4f3dcbb40fce80413f588286 151 main/binary-amd64/Release If the dl.google.com_linux_chrome_deb_dists_stable_main_binary-amd64_Packages file has become corrupt in the specific manner of being 0 bytes in length, apt does not detect this, and the repository is effectively unreachable until one of two things occurs: a) the repository has an update causing apt to re-fetch the repository information and accidentally fix-by-over-writing the corrupt 0 byte file, or, b) the user removes the corrupt 0-byte file and does an apt update to refetch the repository information. The Context == Our IoT devices run Ubuntu 16.04, and their main storage is eMMC. Sometimes there are catastrophic power cuts, and, despite other precautions, files are occasionally corrupted in the manner of becoming 0 bytes in length. We're not sure exactly why or how. Today a deployed device suffered the above scenario. We maintain a debian package repository for updating our devices in the field, and we suddenly couldn't install packages from it. A bit of investigation turned up the 0 byte *_Packages file for our repo, and we worked around the problem. Part of the situation is our debian repository doesn't have updates very often, so 'sudo apt-get update' was giving a Hit: instead of a Get: result all the time, and everything from the "normal user command line" side of things looked okay. There were no logs in /var/log/syslog either. We just could not see our packages from our repo, despite 'apt-get update' looking good. What I Expected to Happen == Given that the the *_Release file contains checksums for the *_Package file, I would expect that apt verifies the checksum, and if it fails, refetches the repository information even if there hasn't been an update, during any given 'apt update' operation. Further Information == I checked apt's project in Debian at https://bugs.debian.org/cgi- bin/pkgreport.cgi?pkg=apt and there don't appear to be any bugs about this filed already, so I'm starting by filing one here. The situation occurred on an Ubuntu 16.04 system, but is 100% reproducible with Google's chrome repository on my Ubuntu 18.04.1 laptop. I can provide a set of reproduction steps if needed, but it's fairly straight-forward. The fact that this corruption appears to be "everything working okay" to the end user, except that apt doesn't know about packages it says it knows about, and there is no error logging for any sort