Re: [gentoo-dev] Re: Common rsync-gen errors, why they happen, and what you can do about it
On Wed, 18 Jan 2017 08:19:19 +0100 Dirkjan Ochtman wrote: > This sounds like a much better strategy to me. We're expecting people > to check things that should be easy to check for machines. Yes, some > people (like myself) will always use repoman to commit, but it would > be much better if something this important (because it basically > delays other updates to users everywhere) is checked by an automated > process for every push, and disallows pushes like this. That's not viable, mostly because the performance cost of doing so is significantly time consuming, that it would block `git push` for minutes at a time, and all users performing pushes would have to wait in a queue for several minutes per push. At scale, I'd expect the push'es to start timing out. That's why the gating is done between git-master and rsync-master, not between the user and git-master. Just imagine needing to wait for a travis test run to complete on the end of every commit, stacked with there being 20 different jobs per push and having to use the slower non-container architecture. Nobody would be working on *that* project, or people would simply make much larger push-queues, increasing the amount of conflicts and progressive de-coherence. > > > I can see if it's something I need to fix with my code. But it's been > > a while since that's been the case, so all the failures these days are > > primarily for the previously mentioned issues. > > That makes sense. My other comment initially reading your email would > be, send those emails to gentoo-core or -project or whatever. If > others don't get to feel the pain (of every half-hour error emails, > for example), they will be much less compelled to fix the problem. So > absorbing this "pain" into just you or infra makes us less scalable as > a distribution, and less likely that someone will feel motivated to > add the extra bits of automation (like a git hook) that will make this > problem go away. +1 , I was aware that "somebody" knew the tree was breaking, but I had no understanding of what was breaking, why it was breaking, or even how people knew it was breaking. And so upon reading the original email hearing that people could even be notified of this fact, made me immediately question what I should be doing to see said notifications, or if there was even a page that I could check periodically to know what the state of the sync was. As the saying goes, information and awareness of the problem is a critical first step to solving the problem. pgpzEz2qctT1T.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Common rsync-gen errors, why they happen, and what you can do about it
On 18/01/17 09:04, Kent Fredric wrote: > On Wed, 18 Jan 2017 08:19:19 +0100 > Dirkjan Ochtman wrote: > >> >> That makes sense. My other comment initially reading your email would >> be, send those emails to gentoo-core or -project or whatever. If >> others don't get to feel the pain (of every half-hour error emails, >> for example), they will be much less compelled to fix the problem. So >> absorbing this "pain" into just you or infra makes us less scalable as >> a distribution, and less likely that someone will feel motivated to >> add the extra bits of automation (like a git hook) that will make this >> problem go away. > +1 , I was aware that "somebody" knew the tree was breaking, but I had > no understanding of what was breaking, why it was breaking, or even how > people knew it was breaking. > > And so upon reading the original email hearing that people could > even be notified of this fact, made me immediately question what I should > be doing to see said notifications, or if there was even a page > that I could check periodically to know what the state of the sync was. > > As the saying goes, information and awareness of the problem is a critical > first step to solving the problem. Is this something that could be woven into the "gentoo-automated-testing" mailing-list ? Ever since I've been poking at mgorny+ 's efforts into CI I've been subscribed, and it would be a useful central place to put such reports. Is that viable? signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Common rsync-gen errors, why they happen, and what you can do about it
On Wed, Jan 18, 2017 at 10:04 AM, Kent Fredric wrote: > That's not viable, mostly because the performance cost of doing so is > significantly > time consuming, that it would block `git push` for minutes at a time, and all > users > performing pushes would have to wait in a queue for several minutes per push. This doesn't ring intuitively true for me. Maybe you're talking about running a "repoman full" for every committed package; I'm not. The checks Doug describes are all package-local. What makes this process so expensive? Even if it is true, the solution would be akin to more post-push check automation (like some of Michal's work) instead of blocking automation, not just saying "people should verify things before they push". Cheers, Dirkjan
[gentoo-dev] Last rites: x11-misc/ksuperkey
# Michael Palimaka (18 Jan 2017) # Obsolete - now a native feature in Plasma 5.8 # Masked for removal in 30 days x11-misc/ksuperkey
[gentoo-dev] Last rites: app-cdr/k9copy
# Michael Palimaka (18 Jan 2017) # Fails to build with ffmpeg-3. Dead upstream. # Masked for remvoal in 30 days. app-cdr/k9copy
[gentoo-dev] Last rites: app-admin/filebeat-bin
# Michael Palimaka (18 Jan 2017) # Obsolete. Use app-admin/filebeat instead. app-admin/filebeat-bin
[gentoo-dev] Last rites: media-video/flumotion:
# Michael Palimaka (18 Jan 2017) # Relies on dead gstreamer:0.10. Dead upstream. # Masked for removal in 30 days. Bug #603270. media-video/flumotion
Re: [gentoo-dev] Re: Common rsync-gen errors, why they happen, and what you can do about it
On Wed, 18 Jan 2017 10:48:18 +0100 Dirkjan Ochtman wrote: > This doesn't ring intuitively true for me. Maybe you're talking about > running a "repoman full" for every committed package; I'm not. The > checks Doug describes are all package-local. What makes this process > so expensive? Even on a per-package basis it can be expensive, and how expensive it is gets worse the more dependencies you have. And its really not worth it to me to penalise every developer with that overhead on every push simply because a subset of the developers don't want to execute that code locally. That seems like trading decentralised effort for a centralised and shared bottleneck. As it was, I had a push today take more than 15 seconds and I started to worry something was wrong. Centralised effort like that is only of benefit IME if it doesn't contribute to a bottleneck or impede the workflow. Though I admit a carefully constructed subset of repomans checks could be fast enough. eg: If at all possible, I'd want those tests we need done to be executable without needing to source the ebuild or any of the profiles. Things like MANIFEST checking and metadata.xml validation can probably be done without this as long as people do it right. /dev-perl/Dist-Zilla $ time repoman manifest-check real 0m1.120s user 0m1.020s sys0m0.100s cpu99.98% But actual checking is really bad with a cold cache ( and I seem to always have a cold cache because web browsers are made of bloat ) /dev-perl/Dist-Zilla $ time repoman full --include-arches amd64 real 0m51.698s user 0m8.890s sys0m7.898s cpu32.47% Warm cache: real 0m5.087s user 0m4.413s sys0m1.815s cpu122.42% Warm cache + multiple arches: real 0m7.469s user 0m6.566s sys0m1.893s cpu113.24% I guess it can't be slower than CVS but ideally, you want the git commit window to be as fast as possible to minimise the chance of contention. pgpVNG590AoCg.pgp Description: OpenPGP digital signature
[gentoo-dev] Last rites: dev-ml/ocamlgsl
# David Seifert (18 Jan 2017) # Dead upstream, spiritual successor is dev-ml/gsl-ocaml # Masked for removal in 30 days. Bug #574564, #593248, #601912. dev-ml/ocamlgsl
