FYI https://github.com/apache/maven-wagon/pull/37
Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-12-05 6:54 GMT+01:00 Romain Manni-Bucau <[email protected]>: > > > Le 5 déc. 2017 06:20, "Eugene Kirpichov" <[email protected]> a écrit : > > > > On Mon, Dec 4, 2017 at 1:45 PM Romain Manni-Bucau <[email protected]> > wrote: >> >> >> >> Le 4 déc. 2017 21:45, "Eugene Kirpichov" <[email protected]> a écrit : >> >> Romain - as far as I understand, Maven *does* have a retry strategy, but >> it is a poor retry strategy and there is no way to tweak it. In particular, >> if it uses https://maven.apache.org/guides/mini/guide-http-settings.html , >> that means it uses Apache Http Client 4.1.2 whose default retry settings are >> extremely conservative >> >> https://github.com/apache/httpcomponents-client/blob/4.1.x/httpclient/src/main/java/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.java#L93 >> >> >> Right >> >> >> >> The errors we're hitting with Maven are some of those: "connection >> refused" and alike. >> >> >> >> Would a retry guarantee you it works. Should be quite easy to switch wagon >> with a configurable retry handler instance through mvnwrapper or custom mvn >> setup and test it but the logic looks sane to me. > > Retries indeed don't give any guarantees, but they're very good at reducing > error rate, and our builds could definitely use some of that. > > I tried to figure out how to make Maven use a configurable http wagon with a > custom retry handler, but I couldn't - if you know more, could you give some > pointers? I think we should definitely do that. > > > It is not configurable but to ensure it is worth a mvn patch we can add a > mvn wrapper script where we patch wagon and use that on the CI. > > > >> >> >> If it is on asf repo we must work with infra to fix it rather than working >> around the clients. If on a repo we dont control we can try to get rid of it >> maybe? > > It happens often enough with the maven central repo repo.maven.apache.org. > At Beam's (moderate) scale of hundreds of builds per day resolving hundreds > of dependencies each, even 99.99% availability with a poor retry strategy > would translate to builds failing regularly. > > > Hmm, this is an asf server and behing should be repo1 and repo2 so either > there is a huge client issue or the proxy can be better configure. If you > have some failure logs, do you mind pinging infra with them? > > There's also multiple parts that can fail - Maven Central itself, network > issues between Jenkins and Maven Central, network issues on the Jenkins > machine itself and so on - I doubt that ASF can fix all of this for us. > I think this sort of issue *has* to be resolved on the client. > > > Not all kind of failures, connection breakdown during the transfert yes but > other kind means the server is down so retry shouldnt help. > > Let me know if you nees help with wagon patching/testing. Can have a few > cycles end of the week to give some help. > > >> >> >> >> >> On Mon, Dec 4, 2017 at 12:15 PM Romain Manni-Bucau <[email protected]> >> wrote: >>> >>> 2017-12-04 18:58 GMT+01:00 Kenneth Knowles <[email protected]>: >>> > On Sat, Dec 2, 2017 at 3:06 AM, Romain Manni-Bucau >>> > <[email protected]> >>> > wrote: >>> >> >>> >> Need to check but if plugin dependencies were not tuned it should be >>> >> the >>> >> default with a retry of 3 IIRC. >>> >> >>> >> But arent you sure it was a repo/server issue any client cant solve? >>> > >>> > >>> > Not sure what you mean here. What we need is for mvn and gradle to >>> > succeed >>> > even though there are frequent transient failures of the repo. Anything >>> > that >>> > makes this happen is success. >>> >>> What I meant is you will likely never get it with whatever build >>> system until you go 100% local which wouldn't make the build >>> reproducible. >>> >>> Concretely you can do >>> >>> 1. >>> >>> for (int i = 0; i < 3; i++) { >>> if (download()) break; >>> } >>> >>> >>> 2. >>> >>> for (int i = 0; i < 3; i++) { >>> if (download()) break; >>> exponentialBackoff(); >>> } >>> >>> >>> Or other wait strategy with a timeout. >>> but you can't have: >>> >>> while (!download()) { >>> waitUntilItWorks(); >>> } >>> >>> >>> So concretely if your down time > reasonnable time you will see the >>> failure anyway. >>> >>> Maven default has a retry (you see it sometimes in the logs) so should >>> tolerante a restart or small downtime. Ivy (gradle) uses httpclient as >>> well so same kind of config is available but with the same limitation >>> by design. >>> >>> Having a local cache is the best way to avoid these issues but it >>> requires at least one green build for dependencies. >>> >>> What I often saw was to do a pre-build step just resolving >>> dependencies+plugins and use a .m2/repository caching. >>> >>> > >>> > Kenn >>> > >>> >> >>> >> >>> >> Le 1 déc. 2017 23:27, "Kenneth Knowles" <[email protected]> a écrit : >>> >>> >>> >>> How do you instruct maven or gradle to use that all the time? >>> >>> >>> >>> On Fri, Dec 1, 2017 at 1:58 PM, Romain Manni-Bucau >>> >>> <[email protected]> wrote: >>> >>>> >>> >>>> If you use wagon http - and not lightweigh - it should have retries >>> >>>> by >>> >>>> default. >>> >>>> >>> >>>> Le 1 déc. 2017 22:31, "Valentyn Tymofieiev" <[email protected]> a >>> >>>> écrit : >>> >>>>> >>> >>>>> Has this ever been brought up in Maven dev community? Perhaps they >>> >>>>> have >>> >>>>> some suggestions. It sounds like a reasonable feature request. >>> >>>>> >>> >>>>> On Fri, Dec 1, 2017 at 1:18 PM, Kenneth Knowles <[email protected]> >>> >>>>> wrote: >>> >>>>>> >>> >>>>>> I've repeatedly searched around for a way to just add proper retry >>> >>>>>> to >>> >>>>>> maven or gradle, and haven't found anything :-( >>> >>>>>> >>> >>>>>> I had thought that we altered our builds in such a way that the >>> >>>>>> .m2 >>> >>>>>> directory was permitted to survive across builds. True that it >>> >>>>>> isn't >>> >>>>>> hermetic, precisely, but it is pretty safe to treat as a cache of >>> >>>>>> immutable >>> >>>>>> data, which is no more dangerous than having a caching incremental >>> >>>>>> build >>> >>>>>> system. >>> >>>>>> >>> >>>>>> Kenn >>> >>>>>> >>> >>>>>> On Wed, Nov 29, 2017 at 3:39 PM, Eugene Kirpichov >>> >>>>>> <[email protected]> wrote: >>> >>>>>>> >>> >>>>>>> Our builds often hit transient Maven network issues, e.g. this >>> >>>>>>> one >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/5331/consoleFull >>> >>>>>>> >>> >>>>>>> 2017-11-29T02:18:02.936 [ERROR] Failed to execute goal on project >>> >>>>>>> beam-sdks-java-io-hadoop-jdk1.8-tests: Could not resolve >>> >>>>>>> dependencies for >>> >>>>>>> project >>> >>>>>>> >>> >>>>>>> org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:2.3.0-SNAPSHOT: >>> >>>>>>> Could not transfer artifact org.apache.derby:derby:jar:10.10.2.0 >>> >>>>>>> from/to >>> >>>>>>> central (https://repo.maven.apache.org/maven2): GET request of: >>> >>>>>>> org/apache/derby/derby/10.10.2.0/derby-10.10.2.0.jar from central >>> >>>>>>> failed: >>> >>>>>>> Connection reset -> [Help 1] >>> >>>>>>> >>> >>>>>>> It'd be good to increase reliability of our builds. >>> >>>>>>> repo.maven.apache.org seems quite unreliable. >>> >>>>>>> >>> >>>>>>> I tried finding a way to configure Maven to retry such network >>> >>>>>>> errors >>> >>>>>>> and it appears to be impossible [will be happy if someone proves >>> >>>>>>> me wrong]. >>> >>>>>>> >>> >>>>>>> Would this issue be resolved if we used multiple mirrors? >>> >>>>>>> https://maven.apache.org/guides/mini/guide-mirror-settings.html >>> >>>>>>> Any other >>> >>>>>>> suggestions? >>> >>>>>> >>> >>>>>> >>> >>>>> >>> >>> >>> > >> >> >
