Le 28 avr. 2014 à 16:36, Josh Suereth <joshua.suer...@gmail.com> a écrit :
> On Mon, Apr 28, 2014 at 10:13 AM, Nicolas Lalevée < > nicolas.lale...@hibnet.org> wrote: > >> >> Le 28 avr. 2014 à 13:59, Josh Suereth <joshua.suer...@gmail.com> a écrit : >> >>> On a whim, I implemented parallel download of artifacts by hooking the >>> downloadArtifacts >>> method of Resolve engine. While this can potentially speed up download >>> performance, the resolution times (getDepedencies) still dominates (of >>> course). Parallel downloads is an often requested feature of sbt, so we >>> may still release this hook for our users, as they request it, but I >> think >>> the *really* just want faster resolution times. >>> >>> We may be doing some investigation into improving the performance of >>> getDependencies (and most importantly fetchDependencies). I was curious >> if >>> anyone else in the Ivy community has attempted this before and what sorts >>> of guidance they could offer before we dig in. >> >> This has been discussed, but AFAIR nothing has been done other than >> discussing the potential issues to tackle. >> >> Things which would require some attention would be to find a way to >> properly report the progress of the parallel download: in the console when >> it's done via Ant or via the listeners for IvyDE. >> A thing that is worrying me, is that Ivy is quite sensitive to which >> thread is actually working on the resolve, see IvyContext's thread local. >> So a parallelization may not be a very upper level than the artifact >> downloading. >> >> > Totally understand. I haven't had time to check if my resolution on > different threads has mucked up the ivy context, but I know I can propagate > context as needed, as long as downloading artifacts is idempotent (i.e. no > changes to artifacts). The first implementation just defers reporting all > progress until AFTER the entire download, then notifies, which probably > breaks IvyDE, except I'm only overriding this in sbt itself, so there is no > IvyDE there :). > > So, if this was something to contribute back to ivy (which I'm not sure it > is, given the lack of real speed-up we're seeing), I'd have to figure out > how to maintain the progress reporting notifications with the IvyContext, > possibly adapting my worker pool such that we register a thread-safe > IvyContext locally for the reporting events to get fired out of? > > Does the event model, as it exist, prevent parallelism? As far as I can tell, no special care has been taken to ensure that even inner part of Ivy can be run on multiple threads. So this is to be tested and code reviewed. > In terms of resolution improvements, mostly I was hoping that some > memoization/cache of immutable graph pieces could help avoid re-resolving > the same bits of the graph repeatedly. Again, I haven't dug fully into the > details of IvyNode, ResolveData + VisitNode, but it seems likely to me that > we should be able to avoid calling "resolve" for an artifact (the same > ModuleRevisionId) more than once during full graph resolution, which (and I > may be wrong) appears to be the case when we profile Ivy. I'm mostly > concerned with this aspect, and was wondering if anyone else has seen this > or if we're doing something very wrong in sbt. I don't think Ivy is doing a resolution twice. This part of Ivy is quite complex, so I may be wrong, but we shouldn't forget that a resolve of a module is often contextualized by the "caller", because there are some different versioning constraints, or because the root configuration is not the same. But I think that it can be parallelized. At least in theory. Nicolas --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@ant.apache.org For additional commands, e-mail: dev-h...@ant.apache.org
