Re: [gentoo-user] Possible heads up: Digest verification failed
On 10/29/2016 03:11 AM, Rich Freeman wrote: > On Sat, Oct 29, 2016 at 5:42 AM, Neil Bothwick wrote: >> On Fri, 28 Oct 2016 23:53:03 -0700, Daniel Campbell wrote: >> Anyone seeing this again? I've just sync'd to two servers in Australia, and then, for the hell of it, one in Canada and am getting it for: dev-libs/botan app-arch/tar media-video/libav app-crypt/qca net-print/cups-filters I suppose time will sort it out. Andrew >>> This shouldn't happen unless the distfiles aren't found, or someone (a >>> dev) didn't use repoman to commit, leaving an old Manifest around. >> >> It looks like repoman is the culprit >> >> https://bugs.gentoo.org/show_bug.cgi?id=598376 >> > > This is probably not the issue here, since Gentoo uses thin manifests > (there is nothing for repoman to update). The manifests that are > causing the problem aren't created by regular Gentoo developers. > They're created by a script that runs as a part of the rsync mirror > process. > > This is a fairly well-known problem that has been around for over a year. > > You will only run into this problem if you use rsync to update your > repository, since the problem is created when creating the master > rsync mirror. The original git repository doesn't contain the error, > and the git mirror on github doesn't mess with the Manifests. > > The issue apparently has to do with Changelog generation. In April > the Council gave Infra the option to stop generating Changelogs, which > would eliminate the problem. I suspect those maintaining the scripts > prefer to keep them around, and I don't think anybody on the Council > has access to change the scripts. > > I switched to git syncing eons ago, so I've never seen this bug. I > recognize it has been a source of frustration for a lot of users, and > a bit of frustration for the Council, since there doesn't seem to be a > lot we can do to change it in practice. > > zlg is of course right that these kinds of problems can also be caused > by maintainer failure to use repoman/etc or if an upstream distfile > changes. If that is the problem then you'll see it no matter how you > sync your repo. However, when you get a bunch of these after syncing > it is almost always a result of the mirror creation process. I can't > remember the last time I saw a manifest error (granted, I'm also using > mgorny's stable mirror branch, which I think screens for these kinds > of errors). > > While there can be some latency I do in general recommend syncing from > https://github.com/gentoo-mirror/gentoo . This is a mirror of the > Gentoo developer git repository with two changes: > 1. Metadata is added to the mirror, which greatly speeds things up > compared to using the raw git repository (you can do this yourself, it > is one of the steps done by the rsync generation process as well, but > this one is not buggy). > 2. The default stable branch of this mirror screens for numerous > issues before accepting commits. That means it is generally a little > behind the main branch (at this moment the main branch is 2 minutes > old, and the default stable branch is 20min old), but a lot of the > really annoying issues that are caused by devs skipping repoman won't > be seen. Now, if a maintainer breaks a package then this mirror will > quickly get out of date until the problem is corrected, but Gentoo QA > gets warnings when this is happening and usually the maintainer is > being pestered or somebody else is fixing it. I suspect this process > has probably reduced the error rate for everybody. I have seen this > get a few days old though, which is something to keep in mind. > > It does not contain Changelogs, though if you use it you'll have a > full history so you can just run git whatchanged to get > something pretty close to a changelog. > > To use it just put this in /etc/portage/repos.conf/gentoo.conf: > [DEFAULT] > main-repo = gentoo > > [gentoo] > location = /usr/portage > sync-type = git > sync-uri = https://github.com/gentoo-mirror/gentoo.git > auto-sync = yes > > If you want to git-sync from some other mirror, just change the url > accordingly. > > If you switch your mirror I suggest just renaming /usr/portage and > letting portage re-create it. > > The other big benefit of git syncing is that if you sync every day it > is a lot faster. If you sync less often it will become slower > compared to rsync. git is much more efficient at finding what has > changed, but rsync is not burdened with transferring a complete > history. If you only sync once every few months rsync will be a lot > faster. > Wow, quite the mail there! I actually wasn't aware of thin vs. thick manifest issues (or mgorny's work on a stable mirror). Thanks for sharing and teaching. -- Daniel Campbell - Gentoo Developer OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net fpr: AE03 9064 AE00 053C 270C 1DE4 6F7A 9091 1EA0 55D6 signature.asc Descr
Re: [gentoo-user] Possible heads up: Digest verification failed
On Sat, Oct 29, 2016 at 5:42 AM, Neil Bothwick wrote: > On Fri, 28 Oct 2016 23:53:03 -0700, Daniel Campbell wrote: > >> > Anyone seeing this again? I've just sync'd to two servers in >> > Australia, and then, for the hell of it, one in Canada and am getting >> > it for: >> > >> > dev-libs/botan >> > app-arch/tar >> > media-video/libav >> > app-crypt/qca >> > net-print/cups-filters >> > >> > I suppose time will sort it out. >> > >> > Andrew >> > >> This shouldn't happen unless the distfiles aren't found, or someone (a >> dev) didn't use repoman to commit, leaving an old Manifest around. > > It looks like repoman is the culprit > > https://bugs.gentoo.org/show_bug.cgi?id=598376 > This is probably not the issue here, since Gentoo uses thin manifests (there is nothing for repoman to update). The manifests that are causing the problem aren't created by regular Gentoo developers. They're created by a script that runs as a part of the rsync mirror process. This is a fairly well-known problem that has been around for over a year. You will only run into this problem if you use rsync to update your repository, since the problem is created when creating the master rsync mirror. The original git repository doesn't contain the error, and the git mirror on github doesn't mess with the Manifests. The issue apparently has to do with Changelog generation. In April the Council gave Infra the option to stop generating Changelogs, which would eliminate the problem. I suspect those maintaining the scripts prefer to keep them around, and I don't think anybody on the Council has access to change the scripts. I switched to git syncing eons ago, so I've never seen this bug. I recognize it has been a source of frustration for a lot of users, and a bit of frustration for the Council, since there doesn't seem to be a lot we can do to change it in practice. zlg is of course right that these kinds of problems can also be caused by maintainer failure to use repoman/etc or if an upstream distfile changes. If that is the problem then you'll see it no matter how you sync your repo. However, when you get a bunch of these after syncing it is almost always a result of the mirror creation process. I can't remember the last time I saw a manifest error (granted, I'm also using mgorny's stable mirror branch, which I think screens for these kinds of errors). While there can be some latency I do in general recommend syncing from https://github.com/gentoo-mirror/gentoo . This is a mirror of the Gentoo developer git repository with two changes: 1. Metadata is added to the mirror, which greatly speeds things up compared to using the raw git repository (you can do this yourself, it is one of the steps done by the rsync generation process as well, but this one is not buggy). 2. The default stable branch of this mirror screens for numerous issues before accepting commits. That means it is generally a little behind the main branch (at this moment the main branch is 2 minutes old, and the default stable branch is 20min old), but a lot of the really annoying issues that are caused by devs skipping repoman won't be seen. Now, if a maintainer breaks a package then this mirror will quickly get out of date until the problem is corrected, but Gentoo QA gets warnings when this is happening and usually the maintainer is being pestered or somebody else is fixing it. I suspect this process has probably reduced the error rate for everybody. I have seen this get a few days old though, which is something to keep in mind. It does not contain Changelogs, though if you use it you'll have a full history so you can just run git whatchanged to get something pretty close to a changelog. To use it just put this in /etc/portage/repos.conf/gentoo.conf: [DEFAULT] main-repo = gentoo [gentoo] location = /usr/portage sync-type = git sync-uri = https://github.com/gentoo-mirror/gentoo.git auto-sync = yes If you want to git-sync from some other mirror, just change the url accordingly. If you switch your mirror I suggest just renaming /usr/portage and letting portage re-create it. The other big benefit of git syncing is that if you sync every day it is a lot faster. If you sync less often it will become slower compared to rsync. git is much more efficient at finding what has changed, but rsync is not burdened with transferring a complete history. If you only sync once every few months rsync will be a lot faster. -- Rich
Re: [gentoo-user] Possible heads up: Digest verification failed
On Fri, 28 Oct 2016 23:53:03 -0700, Daniel Campbell wrote: > > Anyone seeing this again? I've just sync'd to two servers in > > Australia, and then, for the hell of it, one in Canada and am getting > > it for: > > > > dev-libs/botan > > app-arch/tar > > media-video/libav > > app-crypt/qca > > net-print/cups-filters > > > > I suppose time will sort it out. > > > > Andrew > > > This shouldn't happen unless the distfiles aren't found, or someone (a > dev) didn't use repoman to commit, leaving an old Manifest around. It looks like repoman is the culprit https://bugs.gentoo.org/show_bug.cgi?id=598376 -- Neil Bothwick Scrotum is a small planet near Uranus. True/False? pgpba3PJP0BZP.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Possible heads up: Digest verification failed
Checked rsync11.us.gentoo.org and its the same as rsync.au.gentoo.org/ftp.cc.swin.edu.au.
Re: [gentoo-user] Possible heads up: Digest verification failed
On Sat, Oct 29, 2016 at 6:20 PM, Andrew Lowe wrote: > On 29/10/16 14:53, Daniel Campbell wrote: > >> On 10/28/2016 09:56 PM, Andrew Lowe wrote: >> >>> Hi all, >>> Anyone seeing this again? I've just sync'd to two servers in >>> Australia, and then, for the hell of it, one in Canada and am getting it >>> for: >>> >>> dev-libs/botan >>> app-arch/tar >>> media-video/libav >>> app-crypt/qca >>> net-print/cups-filters >>> >>> I suppose time will sort it out. >>> >>> Andrew >>> >>> This shouldn't happen unless the distfiles aren't found, or someone (a >> dev) didn't use repoman to commit, leaving an old Manifest around. >> >> Which servers in question? Have you popped in IRC to ask about it? I'm >> not involved with our mirrors, and I sync directly from git, so I can't >> really help on that front, but it seems to me that it's a simple >> oversight that is sure to be fixed once someone knows about it. >> >> > Servers: Swinburne, I use this one too. Emerge world gave me; !!! Digest verification failed: !!! /fast/portage/sys-kernel/gentoo-sources/ChangeLog !!! Reason: Filesize does not match recorded size !!! Got: 66345 !!! Expected: 66162 !!! Digest verification failed: !!! /fast/portage/net-print/hplip/ChangeLog !!! Reason: Filesize does not match recorded size !!! Got: 8466 !!! Expected: 8315 And the rsync output shows many/all the ChangeLog and Manifest files were just updated net-mail/Freemail/ChangeLog net-mail/altermime/ChangeLog net-mail/altermime/Manifest net-mail/amavis-logwatch/ChangeLog net-mail/amavis-logwatch/Manifest net-mail/archivemail/ChangeLog net-mail/archivemail/Manifest net-mail/asmail/ChangeLog net-mail/asmail/Manifest net-mail/automx/ChangeLog net-mail/automx/Manifest net-mail/autorespond/ChangeLog net-mail/autorespond/Manifest net-mail/bincimap/ChangeLog net-mail/bincimap/Manifest net-mail/checkpassword-pam/ChangeLog net-mail/checkpassword-pam/Manifest net-mail/checkpassword/ChangeLog net-mail/checkpassword/Manifest net-mail/cmd5checkpw/ChangeLog net-mail/cmd5checkpw/Manifest net-mail/courier-imap/ChangeLog net-mail/courier-imap/Manifest net-mail/courierpassd/ChangeLog net-mail/courierpassd/Manifest net-mail/cyrus-imap-admin/ChangeLog net-mail/cyrus-imap-admin/Manifest
Re: [gentoo-user] Possible heads up: Digest verification failed
On 29/10/16 14:53, Daniel Campbell wrote: On 10/28/2016 09:56 PM, Andrew Lowe wrote: Hi all, Anyone seeing this again? I've just sync'd to two servers in Australia, and then, for the hell of it, one in Canada and am getting it for: dev-libs/botan app-arch/tar media-video/libav app-crypt/qca net-print/cups-filters I suppose time will sort it out. Andrew This shouldn't happen unless the distfiles aren't found, or someone (a dev) didn't use repoman to commit, leaving an old Manifest around. Which servers in question? Have you popped in IRC to ask about it? I'm not involved with our mirrors, and I sync directly from git, so I can't really help on that front, but it seems to me that it's a simple oversight that is sure to be fixed once someone knows about it. Servers: Swinburne, iiNet, terra-byte.com (Canada) Sorry, wouldn't know IRC if I tripped over it Time will heal all wounds :) Andrew
Re: [gentoo-user] Possible heads up: Digest verification failed
On 10/28/2016 09:56 PM, Andrew Lowe wrote: > Hi all, > Anyone seeing this again? I've just sync'd to two servers in > Australia, and then, for the hell of it, one in Canada and am getting it > for: > > dev-libs/botan > app-arch/tar > media-video/libav > app-crypt/qca > net-print/cups-filters > > I suppose time will sort it out. > > Andrew > This shouldn't happen unless the distfiles aren't found, or someone (a dev) didn't use repoman to commit, leaving an old Manifest around. Which servers in question? Have you popped in IRC to ask about it? I'm not involved with our mirrors, and I sync directly from git, so I can't really help on that front, but it seems to me that it's a simple oversight that is sure to be fixed once someone knows about it. -- Daniel Campbell - Gentoo Developer OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net fpr: AE03 9064 AE00 053C 270C 1DE4 6F7A 9091 1EA0 55D6 signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] Possible heads up: Digest verification failed
On October 29, 2016 6:56:46 AM GMT+02:00, Andrew Lowe wrote: >Hi all, > Anyone seeing this again? I've just sync'd to two servers in >Australia, >and then, for the hell of it, one in Canada and am getting it for: > >dev-libs/botan >app-arch/tar >media-video/libav >app-crypt/qca >net-print/cups-filters > >I suppose time will sort it out. > > Andrew I haven't seen this in a while. Ever since switching over to syncing with git. Would that be an option for you? -- Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
[gentoo-user] Possible heads up: Digest verification failed
Hi all, Anyone seeing this again? I've just sync'd to two servers in Australia, and then, for the hell of it, one in Canada and am getting it for: dev-libs/botan app-arch/tar media-video/libav app-crypt/qca net-print/cups-filters I suppose time will sort it out. Andrew