Re: [gentoo-user] Possible heads up: Digest verification failed

[Daniel Campbell]

On 10/29/2016 03:11 AM, Rich Freeman wrote:
> On Sat, Oct 29, 2016 at 5:42 AM, Neil Bothwick  wrote:
>> On Fri, 28 Oct 2016 23:53:03 -0700, Daniel Campbell wrote:
>>
 Anyone seeing this again? I've just sync'd to two servers in
 Australia, and then, for the hell of it, one in Canada and am getting
 it for:

 dev-libs/botan
 app-arch/tar
 media-video/libav
 app-crypt/qca
 net-print/cups-filters

 I suppose time will sort it out.

 Andrew

>>> This shouldn't happen unless the distfiles aren't found, or someone (a
>>> dev) didn't use repoman to commit, leaving an old Manifest around.
>>
>> It looks like repoman is the culprit
>>
>> https://bugs.gentoo.org/show_bug.cgi?id=598376
>>
> 
> This is probably not the issue here, since Gentoo uses thin manifests
> (there is nothing for repoman to update).  The manifests that are
> causing the problem aren't created by regular Gentoo developers.
> They're created by a script that runs as a part of the rsync mirror
> process.
> 
> This is a fairly well-known problem that has been around for over a year.
> 
> You will only run into this problem if you use rsync to update your
> repository, since the problem is created when creating the master
> rsync mirror.  The original git repository doesn't contain the error,
> and the git mirror on github doesn't mess with the Manifests.
> 
> The issue apparently has to do with Changelog generation.  In April
> the Council gave Infra the option to stop generating Changelogs, which
> would eliminate the problem.  I suspect those maintaining the scripts
> prefer to keep them around, and I don't think anybody on the Council
> has access to change the scripts.
> 
> I switched to git syncing eons ago, so I've never seen this bug.  I
> recognize it has been a source of frustration for a lot of users, and
> a bit of frustration for the Council, since there doesn't seem to be a
> lot we can do to change it in practice.
> 
> zlg is of course right that these kinds of problems can also be caused
> by maintainer failure to use repoman/etc or if an upstream distfile
> changes.  If that is the problem then you'll see it no matter how you
> sync your repo.  However, when you get a bunch of these after syncing
> it is almost always a result of the mirror creation process.  I can't
> remember the last time I saw a manifest error (granted, I'm also using
> mgorny's stable mirror branch, which I think screens for these kinds
> of errors).
> 
> While there can be some latency I do in general recommend syncing from
> https://github.com/gentoo-mirror/gentoo .  This is a mirror of the
> Gentoo developer git repository with two changes:
> 1.  Metadata is added to the mirror, which greatly speeds things up
> compared to using the raw git repository (you can do this yourself, it
> is one of the steps done by the rsync generation process as well, but
> this one is not buggy).
> 2.  The default stable branch of this mirror screens for numerous
> issues before accepting commits.  That means it is generally a little
> behind the main branch (at this moment the main branch is 2 minutes
> old, and the default stable branch is 20min old), but a lot of the
> really annoying issues that are caused by devs skipping repoman won't
> be seen.   Now, if a maintainer breaks a package then this mirror will
> quickly get out of date until the problem is corrected, but Gentoo QA
> gets warnings when this is happening and usually the maintainer is
> being pestered or somebody else is fixing it.  I suspect this process
> has probably reduced the error rate for everybody.  I have seen this
> get a few days old though, which is something to keep in mind.
> 
> It does not contain Changelogs, though if you use it you'll have a
> full history so you can just run git whatchanged  to get
> something pretty close to a changelog.
> 
> To use it just put this in /etc/portage/repos.conf/gentoo.conf:
> [DEFAULT]
> main-repo = gentoo
> 
> [gentoo]
> location = /usr/portage
> sync-type = git
> sync-uri = https://github.com/gentoo-mirror/gentoo.git
> auto-sync = yes
> 
> If you want to git-sync from some other mirror, just change the url 
> accordingly.
> 
> If you switch your mirror I suggest just renaming /usr/portage and
> letting portage re-create it.
> 
> The other big benefit of git syncing is that if you sync every day it
> is a lot faster.  If you sync less often it will become slower
> compared to rsync.  git is much more efficient at finding what has
> changed, but rsync is not burdened with transferring a complete
> history.  If you only sync once every few months rsync will be a lot
> faster.
> 
Wow, quite the mail there! I actually wasn't aware of thin vs. thick
manifest issues (or mgorny's work on a stable mirror). Thanks for
sharing and teaching.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6

Re: [gentoo-user] Possible heads up: Digest verification failed

[Rich Freeman]

On Sat, Oct 29, 2016 at 5:42 AM, Neil Bothwick  wrote:
> On Fri, 28 Oct 2016 23:53:03 -0700, Daniel Campbell wrote:
>
>> > Anyone seeing this again? I've just sync'd to two servers in
>> > Australia, and then, for the hell of it, one in Canada and am getting
>> > it for:
>> >
>> > dev-libs/botan
>> > app-arch/tar
>> > media-video/libav
>> > app-crypt/qca
>> > net-print/cups-filters
>> >
>> > I suppose time will sort it out.
>> >
>> > Andrew
>> >
>> This shouldn't happen unless the distfiles aren't found, or someone (a
>> dev) didn't use repoman to commit, leaving an old Manifest around.
>
> It looks like repoman is the culprit
>
> https://bugs.gentoo.org/show_bug.cgi?id=598376
>

This is probably not the issue here, since Gentoo uses thin manifests
(there is nothing for repoman to update).  The manifests that are
causing the problem aren't created by regular Gentoo developers.
They're created by a script that runs as a part of the rsync mirror
process.

This is a fairly well-known problem that has been around for over a year.

You will only run into this problem if you use rsync to update your
repository, since the problem is created when creating the master
rsync mirror.  The original git repository doesn't contain the error,
and the git mirror on github doesn't mess with the Manifests.

The issue apparently has to do with Changelog generation.  In April
the Council gave Infra the option to stop generating Changelogs, which
would eliminate the problem.  I suspect those maintaining the scripts
prefer to keep them around, and I don't think anybody on the Council
has access to change the scripts.

I switched to git syncing eons ago, so I've never seen this bug.  I
recognize it has been a source of frustration for a lot of users, and
a bit of frustration for the Council, since there doesn't seem to be a
lot we can do to change it in practice.

zlg is of course right that these kinds of problems can also be caused
by maintainer failure to use repoman/etc or if an upstream distfile
changes.  If that is the problem then you'll see it no matter how you
sync your repo.  However, when you get a bunch of these after syncing
it is almost always a result of the mirror creation process.  I can't
remember the last time I saw a manifest error (granted, I'm also using
mgorny's stable mirror branch, which I think screens for these kinds
of errors).

While there can be some latency I do in general recommend syncing from
https://github.com/gentoo-mirror/gentoo .  This is a mirror of the
Gentoo developer git repository with two changes:
1.  Metadata is added to the mirror, which greatly speeds things up
compared to using the raw git repository (you can do this yourself, it
is one of the steps done by the rsync generation process as well, but
this one is not buggy).
2.  The default stable branch of this mirror screens for numerous
issues before accepting commits.  That means it is generally a little
behind the main branch (at this moment the main branch is 2 minutes
old, and the default stable branch is 20min old), but a lot of the
really annoying issues that are caused by devs skipping repoman won't
be seen.   Now, if a maintainer breaks a package then this mirror will
quickly get out of date until the problem is corrected, but Gentoo QA
gets warnings when this is happening and usually the maintainer is
being pestered or somebody else is fixing it.  I suspect this process
has probably reduced the error rate for everybody.  I have seen this
get a few days old though, which is something to keep in mind.

It does not contain Changelogs, though if you use it you'll have a
full history so you can just run git whatchanged  to get
something pretty close to a changelog.

To use it just put this in /etc/portage/repos.conf/gentoo.conf:
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
sync-type = git
sync-uri = https://github.com/gentoo-mirror/gentoo.git
auto-sync = yes

If you want to git-sync from some other mirror, just change the url accordingly.

If you switch your mirror I suggest just renaming /usr/portage and
letting portage re-create it.

The other big benefit of git syncing is that if you sync every day it
is a lot faster.  If you sync less often it will become slower
compared to rsync.  git is much more efficient at finding what has
changed, but rsync is not burdened with transferring a complete
history.  If you only sync once every few months rsync will be a lot
faster.

-- 
Rich

Re: [gentoo-user] Possible heads up: Digest verification failed

[Neil Bothwick]

On Fri, 28 Oct 2016 23:53:03 -0700, Daniel Campbell wrote:

> > Anyone seeing this again? I've just sync'd to two servers in
> > Australia, and then, for the hell of it, one in Canada and am getting
> > it for:
> > 
> > dev-libs/botan
> > app-arch/tar
> > media-video/libav
> > app-crypt/qca
> > net-print/cups-filters
> > 
> > I suppose time will sort it out.
> > 
> > Andrew
> >   
> This shouldn't happen unless the distfiles aren't found, or someone (a
> dev) didn't use repoman to commit, leaving an old Manifest around.

It looks like repoman is the culprit

https://bugs.gentoo.org/show_bug.cgi?id=598376


-- 
Neil Bothwick

Scrotum is a small planet near Uranus. True/False?


pgpba3PJP0BZP.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] Possible heads up: Digest verification failed

[Adam Carter]

Checked rsync11.us.gentoo.org and its the same as
rsync.au.gentoo.org/ftp.cc.swin.edu.au.

Re: [gentoo-user] Possible heads up: Digest verification failed

[Adam Carter]

On Sat, Oct 29, 2016 at 6:20 PM, Andrew Lowe  wrote:

> On 29/10/16 14:53, Daniel Campbell wrote:
>
>> On 10/28/2016 09:56 PM, Andrew Lowe wrote:
>>
>>> Hi all,
>>> Anyone seeing this again? I've just sync'd to two servers in
>>> Australia, and then, for the hell of it, one in Canada and am getting it
>>> for:
>>>
>>> dev-libs/botan
>>> app-arch/tar
>>> media-video/libav
>>> app-crypt/qca
>>> net-print/cups-filters
>>>
>>> I suppose time will sort it out.
>>>
>>> Andrew
>>>
>>> This shouldn't happen unless the distfiles aren't found, or someone (a
>> dev) didn't use repoman to commit, leaving an old Manifest around.
>>
>> Which servers in question? Have you popped in IRC to ask about it? I'm
>> not involved with our mirrors, and I sync directly from git, so I can't
>> really help on that front, but it seems to me that it's a simple
>> oversight that is sure to be fixed once someone knows about it.
>>
>>
> Servers: Swinburne,


I use this one too. Emerge world gave me;

!!! Digest verification failed:
!!! /fast/portage/sys-kernel/gentoo-sources/ChangeLog
!!! Reason: Filesize does not match recorded size
!!! Got: 66345
!!! Expected: 66162

!!! Digest verification failed:
!!! /fast/portage/net-print/hplip/ChangeLog
!!! Reason: Filesize does not match recorded size
!!! Got: 8466
!!! Expected: 8315

And the rsync output shows many/all the ChangeLog and Manifest files were
just updated


net-mail/Freemail/ChangeLog
net-mail/altermime/ChangeLog
net-mail/altermime/Manifest
net-mail/amavis-logwatch/ChangeLog
net-mail/amavis-logwatch/Manifest
net-mail/archivemail/ChangeLog
net-mail/archivemail/Manifest
net-mail/asmail/ChangeLog
net-mail/asmail/Manifest
net-mail/automx/ChangeLog
net-mail/automx/Manifest
net-mail/autorespond/ChangeLog
net-mail/autorespond/Manifest
net-mail/bincimap/ChangeLog
net-mail/bincimap/Manifest
net-mail/checkpassword-pam/ChangeLog
net-mail/checkpassword-pam/Manifest
net-mail/checkpassword/ChangeLog
net-mail/checkpassword/Manifest
net-mail/cmd5checkpw/ChangeLog
net-mail/cmd5checkpw/Manifest
net-mail/courier-imap/ChangeLog
net-mail/courier-imap/Manifest
net-mail/courierpassd/ChangeLog
net-mail/courierpassd/Manifest
net-mail/cyrus-imap-admin/ChangeLog
net-mail/cyrus-imap-admin/Manifest

Re: [gentoo-user] Possible heads up: Digest verification failed

[Andrew Lowe]

On 29/10/16 14:53, Daniel Campbell wrote:

On 10/28/2016 09:56 PM, Andrew Lowe wrote:

Hi all,
Anyone seeing this again? I've just sync'd to two servers in
Australia, and then, for the hell of it, one in Canada and am getting it
for:

dev-libs/botan
app-arch/tar
media-video/libav
app-crypt/qca
net-print/cups-filters

I suppose time will sort it out.

Andrew


This shouldn't happen unless the distfiles aren't found, or someone (a
dev) didn't use repoman to commit, leaving an old Manifest around.

Which servers in question? Have you popped in IRC to ask about it? I'm
not involved with our mirrors, and I sync directly from git, so I can't
really help on that front, but it seems to me that it's a simple
oversight that is sure to be fixed once someone knows about it.



Servers: Swinburne, iiNet, terra-byte.com (Canada)

Sorry, wouldn't know IRC if I tripped over it

Time will heal all wounds :)

Andrew

Re: [gentoo-user] Possible heads up: Digest verification failed

[Daniel Campbell]

On 10/28/2016 09:56 PM, Andrew Lowe wrote:
> Hi all,
> Anyone seeing this again? I've just sync'd to two servers in
> Australia, and then, for the hell of it, one in Canada and am getting it
> for:
> 
> dev-libs/botan
> app-arch/tar
> media-video/libav
> app-crypt/qca
> net-print/cups-filters
> 
> I suppose time will sort it out.
> 
> Andrew
> 
This shouldn't happen unless the distfiles aren't found, or someone (a
dev) didn't use repoman to commit, leaving an old Manifest around.

Which servers in question? Have you popped in IRC to ask about it? I'm
not involved with our mirrors, and I sync directly from git, so I can't
really help on that front, but it seems to me that it's a simple
oversight that is sure to be fixed once someone knows about it.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-user] Possible heads up: Digest verification failed

[J. Roeleveld]

On October 29, 2016 6:56:46 AM GMT+02:00, Andrew Lowe  wrote:
>Hi all,
>   Anyone seeing this again? I've just sync'd to two servers in
>Australia, 
>and then, for the hell of it, one in Canada and am getting it for:
>
>dev-libs/botan
>app-arch/tar
>media-video/libav
>app-crypt/qca
>net-print/cups-filters
>
>I suppose time will sort it out.
>
>   Andrew

I haven't seen this in a while. Ever since switching over to syncing with git.

Would that be an option for you?

--
Joost
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.