On Sun, Apr 1, 2018 at 2:29 PM, Michał Górny <mgo...@gentoo.org> wrote:
> W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico > napisał: > > On 04/01/2018 03:57 AM, Michał Górny wrote: > > > W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico > > > napisał: > > > > Since key refresh is prone to failure, retry using exponential > > > > backoff with random jitter. This adds the following sync-openpgp-* > > > > configuration settings: > > > > > > > > sync-openpgp-key-refresh-retry-count = 40 > > > > > > > > Maximum number of times to retry key refresh if it fails. Between > > > > each key refresh attempt, there is an exponential delay with a > > > > constant multiplier and a uniform random multiplier between 0 and > 1. > > > > > > > > sync-openpgp-key-refresh-retry-delay-exp-base = 2 > > > > > > > > The base of the exponential expression. The exponent is the number > > > > of previous refresh attempts. > > > > > > > > sync-openpgp-key-refresh-retry-delay-max = 60 > > > > > > > > Maximum delay between each retry attempt, in units of seconds. > This > > > > places a limit on the length of the exponential delay. > > > > > > > > sync-openpgp-key-refresh-retry-delay-mult = 4 > > > > > > > > Multiplier for the exponential delay. > > > > > > > > sync-openpgp-key-refresh-retry-overall-timeout = 1200 > > > > > > > > Combined time limit for all refresh attempts, in units of seconds. > > > > > > > > Bug: https://bugs.gentoo.org/649276 > > > > > > > > Zac Medico (4): > > > > Add ForkExecutor (bug 649588) > > > > Add ExponentialBackoff and RandomExponentialBackoff > > > > Add retry decorator (API inspired by tenacity) > > > > rsync: add key refresh retry (bug 649276) > > > > > > > > cnf/repos.conf | 5 + > > > > man/portage.5 | 19 +++ > > > > pym/portage/repository/config.py | 22 ++++ > > > > pym/portage/sync/modules/rsync/rsync.py | 16 ++- > > > > pym/portage/sync/syncbase.py | 85 +++++++++++- > > > > pym/portage/tests/util/futures/test_retry.py | 147 > +++++++++++++++++++++ > > > > pym/portage/util/_eventloop/EventLoop.py | 45 ++++++- > > > > pym/portage/util/backoff.py | 48 +++++++ > > > > pym/portage/util/futures/executor/__init__.py | 0 > > > > pym/portage/util/futures/executor/fork.py | 130 > +++++++++++++++++++ > > > > pym/portage/util/futures/futures.py | 6 + > > > > pym/portage/util/futures/retry.py | 178 > ++++++++++++++++++++++++++ > > > > 12 files changed, 697 insertions(+), 4 deletions(-) > > > > create mode 100644 pym/portage/tests/util/futures/test_retry.py > > > > create mode 100644 pym/portage/util/backoff.py > > > > create mode 100644 pym/portage/util/futures/executor/__init__.py > > > > create mode 100644 pym/portage/util/futures/executor/fork.py > > > > create mode 100644 pym/portage/util/futures/retry.py > > > > > > > > > > This essentially looks like ~700 lines of code to try to workaround > > > broken networking. I would rather try to do that using 5 lines of code > > > but that's just me, and my programs aren't enterprise quality. I just > > > hope it actually solves as many problems as it's going to introduce. > > > > The vast majority of this code is generic and reusable, and I do intend > > to reuse it. For example, the executor support will be an essential > > piece for the asyncio.AbstractEventLoop for bug 649588. > > Sure it is and sure you will. But tell me: who is going to maintain it > all? Because as far as I can see, we're still dealing with a bus factor > of one and all you're doing is making it worse. More code, more > complexity, more creeping featurism and hacks. > I'll split this reply into two sorts of points. One is a brief point about this patchset. The other is a larger point about the project and its direction. About this patchset: I don't mind this code because its not domain specific and I can read it and understand it. Many apps need async ways to run code, and anyone who has used the futures API will find this code familiar. I reviewed it in about 10 minutes. I'm not heavily invested with the async-wagon that Zac is driving, but I also see how its an area where performance can be improved by doing more stuff at once using an async operations. I think if you have more generic concerns with the async frameworks I'd love to see some discussion about them (particularly around how we can improve perf by using tighter algorithms; often the 'perf' boost is just higher CPU utilization and driving multi-core systems which results in better walltime perf, but not necessarily cpu-perf.) Its fine to disagree here. I think the project has some requirements around minimized dependencies, so the choice is either to pull in some kind of retry-lib and add a dep, or add this code. Its also not immediately clear if the 3rd party deps would still not require some glue code to work with our async loops. > > Last time you went away, you left us with a horribly unmaintainable > package manager full of complexity, hacks and creeping featurism > and a Portage team whose members had barely any knowledge of the code. > Just when things started moving again, you came back and we're back to > square one. > > Today Portage once again is a one-developer project, full of more > complexity, more hacks and more creeping featurism. And we once again > have a bus factor of one -- one developer who apparently knows > everything, does everything and tries to be nice to everyone, except he > really ignores others, makes a lot of empty promises and consistently > makes the health of the project go from bad to worse. > > So, please tell me: what happens when you leave again? How have you used > your time in the project? What have you done to make sure that > the project stays alive without you? Because as far as I can see, adding > few thousand of lines of practically unreviewed code every month does > not help with that. > About the project: I'm not really sure if you have had conversations 1:1 or on the dev-portage alias (I've been on it for a month or two, but was off it for a while) but in general I haven't seen these conversations (and FWIW I'd appreciate if it there were on this list, so thanks for sending this email.) I'll try to cover a few points: 1) Unreviewed code. I think in general Zac posts things to the list and there is an opportunity for review. No one appears to be around to review the code (most code goes unreviewed; is my observation.) I don't think Zac not posting indicates a manner in which he is trying to 'sneak code by'; as this is unnecessary given the current structure. If you want more review, we need people to review the code. I'm fairly sure you have encountered similar when you post patchsets to the list and hear little back. One common problem when things are mailed to the list is that there is no explicit owner of the review, so the review never occurs. We could try a few things like: a) Have an explicit coding buddy; they are responsible for reviewing code for specific bugs / features? b) Get folks to signup for a rotation where they commit some time (e.g. 1h / w) on being the 'point person' for portage code reviews. So all reviews to the list would get 'assigned' to someone. In addition, I'd encourage people to 'ack' code that looks good; making reviews explicitly ack'd is helpful, as opposed to just waiting for some timeout period (e.g. "well no one said no to my code after 7 days, so I'll merge it"). I'd encourage 'active' reviews, where only 'good' code gets in, as opposed to mediocre code that no one bothered to review. 2) Feature-ism. I think this is the most contentious point. Ultimately your goal is to see portage perhaps bench feature development for a while and work on paying down some technical debt. But in practice that isn't happening. I'd like to learn more about why. a) Is Zac rejecting refactoring patches? b) Is Zac not working on refactoring (and to be clear, I don't see 'forcing' him to do this to be useful expenditure of effort.). 3) Zac disagrees with you on specific patchsets. E.g. the implementation of INSTALL_MASK seemed like there were some contentious points. One thing you can do is try to actually get support from other contributors to try to support your point. I think you will need to be ready to compromise on various issues though. > I forked Portage because I didn't want to fight with you. When I forked > it, I declared that I will merge mainline changes regularly for > the benefit of my users. But after a week, I really start feeling like > that's going to be a really bad idea. Like it's time to forget about > mainline Portage as a completely dead end, and go our separate ways. > I would consider thinking about this more. If a week sways you I'm somewhat concerned how committed you were to merging changes to begin with. Ultimately you not "fighting" with Zac means he will merge whatever (which you already said you didn't want) and then you merging it into portage[mgorny]. So if that was the extent of your plan I don't see how it could end otherwise. > I'm seriously worried about the future of Gentoo. I'd really appreciate > if you started focusing on that as well. I get that all this stuff looks > cool on paper but few months or years from now, someone will curse > 'whoever wrote that code' while trying to fix some nasty bug. Or get > things moving forward. Or implement EAPI 8. > I appreciate having this conversation. > > -- > Best regards, > Michał Górny > > >
