Thank you! On Tue, Dec 5, 2017 at 3:11 AM Romain Manni-Bucau <[email protected]> wrote:
> FYI https://github.com/apache/maven-wagon/pull/37 > > Romain Manni-Bucau > @rmannibucau | Blog | Old Blog | Github | LinkedIn > > > 2017-12-05 6:54 GMT+01:00 Romain Manni-Bucau <[email protected]>: > > > > > > Le 5 déc. 2017 06:20, "Eugene Kirpichov" <[email protected]> a écrit > : > > > > > > > > On Mon, Dec 4, 2017 at 1:45 PM Romain Manni-Bucau <[email protected] > > > > wrote: > >> > >> > >> > >> Le 4 déc. 2017 21:45, "Eugene Kirpichov" <[email protected]> a > écrit : > >> > >> Romain - as far as I understand, Maven *does* have a retry strategy, but > >> it is a poor retry strategy and there is no way to tweak it. In > particular, > >> if it uses > https://maven.apache.org/guides/mini/guide-http-settings.html , > >> that means it uses Apache Http Client 4.1.2 whose default retry > settings are > >> extremely conservative > >> > >> > https://github.com/apache/httpcomponents-client/blob/4.1.x/httpclient/src/main/java/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.java#L93 > >> > >> > >> Right > >> > >> > >> > >> The errors we're hitting with Maven are some of those: "connection > >> refused" and alike. > >> > >> > >> > >> Would a retry guarantee you it works. Should be quite easy to switch > wagon > >> with a configurable retry handler instance through mvnwrapper or custom > mvn > >> setup and test it but the logic looks sane to me. > > > > Retries indeed don't give any guarantees, but they're very good at > reducing > > error rate, and our builds could definitely use some of that. > > > > I tried to figure out how to make Maven use a configurable http wagon > with a > > custom retry handler, but I couldn't - if you know more, could you give > some > > pointers? I think we should definitely do that. > > > > > > It is not configurable but to ensure it is worth a mvn patch we can add a > > mvn wrapper script where we patch wagon and use that on the CI. > > > > > > > >> > >> > >> If it is on asf repo we must work with infra to fix it rather than > working > >> around the clients. If on a repo we dont control we can try to get rid > of it > >> maybe? > > > > It happens often enough with the maven central repo > repo.maven.apache.org. > > At Beam's (moderate) scale of hundreds of builds per day resolving > hundreds > > of dependencies each, even 99.99% availability with a poor retry strategy > > would translate to builds failing regularly. > > > > > > Hmm, this is an asf server and behing should be repo1 and repo2 so either > > there is a huge client issue or the proxy can be better configure. If you > > have some failure logs, do you mind pinging infra with them? > > > > There's also multiple parts that can fail - Maven Central itself, network > > issues between Jenkins and Maven Central, network issues on the Jenkins > > machine itself and so on - I doubt that ASF can fix all of this for us. > > I think this sort of issue *has* to be resolved on the client. > > > > > > Not all kind of failures, connection breakdown during the transfert yes > but > > other kind means the server is down so retry shouldnt help. > > > > Let me know if you nees help with wagon patching/testing. Can have a few > > cycles end of the week to give some help. > > > > > >> > >> > >> > >> > >> On Mon, Dec 4, 2017 at 12:15 PM Romain Manni-Bucau < > [email protected]> > >> wrote: > >>> > >>> 2017-12-04 18:58 GMT+01:00 Kenneth Knowles <[email protected]>: > >>> > On Sat, Dec 2, 2017 at 3:06 AM, Romain Manni-Bucau > >>> > <[email protected]> > >>> > wrote: > >>> >> > >>> >> Need to check but if plugin dependencies were not tuned it should be > >>> >> the > >>> >> default with a retry of 3 IIRC. > >>> >> > >>> >> But arent you sure it was a repo/server issue any client cant solve? > >>> > > >>> > > >>> > Not sure what you mean here. What we need is for mvn and gradle to > >>> > succeed > >>> > even though there are frequent transient failures of the repo. > Anything > >>> > that > >>> > makes this happen is success. > >>> > >>> What I meant is you will likely never get it with whatever build > >>> system until you go 100% local which wouldn't make the build > >>> reproducible. > >>> > >>> Concretely you can do > >>> > >>> 1. > >>> > >>> for (int i = 0; i < 3; i++) { > >>> if (download()) break; > >>> } > >>> > >>> > >>> 2. > >>> > >>> for (int i = 0; i < 3; i++) { > >>> if (download()) break; > >>> exponentialBackoff(); > >>> } > >>> > >>> > >>> Or other wait strategy with a timeout. > >>> but you can't have: > >>> > >>> while (!download()) { > >>> waitUntilItWorks(); > >>> } > >>> > >>> > >>> So concretely if your down time > reasonnable time you will see the > >>> failure anyway. > >>> > >>> Maven default has a retry (you see it sometimes in the logs) so should > >>> tolerante a restart or small downtime. Ivy (gradle) uses httpclient as > >>> well so same kind of config is available but with the same limitation > >>> by design. > >>> > >>> Having a local cache is the best way to avoid these issues but it > >>> requires at least one green build for dependencies. > >>> > >>> What I often saw was to do a pre-build step just resolving > >>> dependencies+plugins and use a .m2/repository caching. > >>> > >>> > > >>> > Kenn > >>> > > >>> >> > >>> >> > >>> >> Le 1 déc. 2017 23:27, "Kenneth Knowles" <[email protected]> a écrit : > >>> >>> > >>> >>> How do you instruct maven or gradle to use that all the time? > >>> >>> > >>> >>> On Fri, Dec 1, 2017 at 1:58 PM, Romain Manni-Bucau > >>> >>> <[email protected]> wrote: > >>> >>>> > >>> >>>> If you use wagon http - and not lightweigh - it should have > retries > >>> >>>> by > >>> >>>> default. > >>> >>>> > >>> >>>> Le 1 déc. 2017 22:31, "Valentyn Tymofieiev" <[email protected]> > a > >>> >>>> écrit : > >>> >>>>> > >>> >>>>> Has this ever been brought up in Maven dev community? Perhaps > they > >>> >>>>> have > >>> >>>>> some suggestions. It sounds like a reasonable feature request. > >>> >>>>> > >>> >>>>> On Fri, Dec 1, 2017 at 1:18 PM, Kenneth Knowles <[email protected]> > >>> >>>>> wrote: > >>> >>>>>> > >>> >>>>>> I've repeatedly searched around for a way to just add proper > retry > >>> >>>>>> to > >>> >>>>>> maven or gradle, and haven't found anything :-( > >>> >>>>>> > >>> >>>>>> I had thought that we altered our builds in such a way that the > >>> >>>>>> .m2 > >>> >>>>>> directory was permitted to survive across builds. True that it > >>> >>>>>> isn't > >>> >>>>>> hermetic, precisely, but it is pretty safe to treat as a cache > of > >>> >>>>>> immutable > >>> >>>>>> data, which is no more dangerous than having a caching > incremental > >>> >>>>>> build > >>> >>>>>> system. > >>> >>>>>> > >>> >>>>>> Kenn > >>> >>>>>> > >>> >>>>>> On Wed, Nov 29, 2017 at 3:39 PM, Eugene Kirpichov > >>> >>>>>> <[email protected]> wrote: > >>> >>>>>>> > >>> >>>>>>> Our builds often hit transient Maven network issues, e.g. this > >>> >>>>>>> one > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/5331/consoleFull > >>> >>>>>>> > >>> >>>>>>> 2017-11-29T02:18:02.936 [ERROR] Failed to execute goal on > project > >>> >>>>>>> beam-sdks-java-io-hadoop-jdk1.8-tests: Could not resolve > >>> >>>>>>> dependencies for > >>> >>>>>>> project > >>> >>>>>>> > >>> >>>>>>> > org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:2.3.0-SNAPSHOT: > >>> >>>>>>> Could not transfer artifact > org.apache.derby:derby:jar:10.10.2.0 > >>> >>>>>>> from/to > >>> >>>>>>> central (https://repo.maven.apache.org/maven2): GET request > of: > >>> >>>>>>> org/apache/derby/derby/10.10.2.0/derby-10.10.2.0.jar from > central > >>> >>>>>>> failed: > >>> >>>>>>> Connection reset -> [Help 1] > >>> >>>>>>> > >>> >>>>>>> It'd be good to increase reliability of our builds. > >>> >>>>>>> repo.maven.apache.org seems quite unreliable. > >>> >>>>>>> > >>> >>>>>>> I tried finding a way to configure Maven to retry such network > >>> >>>>>>> errors > >>> >>>>>>> and it appears to be impossible [will be happy if someone > proves > >>> >>>>>>> me wrong]. > >>> >>>>>>> > >>> >>>>>>> Would this issue be resolved if we used multiple mirrors? > >>> >>>>>>> > https://maven.apache.org/guides/mini/guide-mirror-settings.html > >>> >>>>>>> Any other > >>> >>>>>>> suggestions? > >>> >>>>>> > >>> >>>>>> > >>> >>>>> > >>> >>> > >>> > > >> > >> > > >
