Hi Maarten,
For improving resolve's performance I was thinking of caching
artifact->location data. From what I can tell the
slowness is in scanning the filesystem for matching packages. Part of the
responsability for the slowness may be in
having too many matching rules in the filesystem resolver. Once the resolver
has run and located all of the
dependencies, I was hoping to cache a simple map of the artifacts to their
locations. Then, on another call to resolve,
resolve can check if this particular dependency calculation has been resolved
and cached and if so just use those
locations with no further computation.
Another task would be needed to clear the cache (perhaps just delete the file
in which the location cache is stored).
Like I said, some of the responsability may be in having too many rules:
<!-- general resolver for pulling artifacts. -->
<filesystem name="ivycontrib">
<!-- critical -->
<artifact
pattern="${ivycontrib.dir}/m2/[organisation]/[module]/[revision]/[type]s/[artifact]-[revision].
[ext]" />
<artifact
pattern="${ivycontrib.dir}/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
/>
<!-- our custom packages -->
<artifact
pattern="${ivycontrib.dir}/[organisation]/[module]/[revision]/[artifact].[ext]"
/>
<!-- we need to obsolete these next three -->
<artifact
pattern="${ivycontrib.dir}/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact
pattern="${ivycontrib.dir}/[module]/[revision]/[artifact]-${jdkXX}-[revision].[ext]"
/>
<artifact
pattern="${ivycontrib.dir}/[module]/[revision]-${os.name}-${os.arch}/[artifact].[ext]"
/>
<artifact
pattern="${ivycontrib.dir}/[module]/[revision]/[artifact].[ext]" />
<!-- we need to obsolete these next two -->
<artifact
pattern="${ivycontrib.dir}/[organisation]-[module]/[revision]/[artifact].[ext]"
/>
<artifact
pattern="${ivycontrib.dir}/[organisation]-[module]/[revision]/[artifact]-[revision].[ext]"
/>
<ivy
pattern="${ivycontrib.dir}/m2/[organisation]/[module]/[revision]/ivy.xml" />
<ivy
pattern="${ivycontrib.dir}/[organisation]/[module]/[revision]/ivy.xml" />
<ivy pattern="${ivycontrib.dir}/[module]/[revision]/ivy.xml" />
<ivy
pattern="${ivycontrib.dir}/[organisation]-[module]/[revision]/ivy.xml" />
<ivy
pattern="${ivycontrib.dir}/[module]/[revision]-${os.name}-${os.arch}/ivy.xml" />
</filesystem>
-Ben
On Wednesday 19 November 2008 02:57:39 pm Maarten Coene wrote:
> Benjamin,
>
> could you give a bit more info about what your caching facade will do and
> how you think it will improve the resolve performance?
>
> Maarten
>
>
>
>
> ----- Original Message ----
> From: Benjamin Damm <[EMAIL PROTECTED]>
> To: [email protected]
> Cc: Colin Fleming <[EMAIL PROTECTED]>
> Sent: Wednesday, November 19, 2008 9:21:57 PM
> Subject: Re: Speeding up ivy:resolve
>
> Good stuff and good tip on the getHostName() call, could be a really simple
> win there. For the main work it'll be a few weeks before I can start on it
> but it's great to know there may be opportunities for shared effort
> especially in testing & bulletproofing any new code.
>
> Cheers!
>
> On Wednesday 19 November 2008 11:12:34 am Colin Fleming wrote:
> > Hi Ben,
> >
> > I'd be interested, for sure. We have about 190 modules in our project now
> > and resolve is a large part of our build time. I've been meaning to look
> > more closely at it for ages now, but I'm totally swamped.
> >
> > One thing I did see when I profiled our build a while back was that
> > somewhere in the depths there (I can't remember the exact spot, sorry)
> > Ivy calls getHostName() and doesn't cache the return value. This ends up
> > being called a lot of times (I don't remember the exact number but IIRC
> > in the thousands) due to recursive resolves, and would be a two-line fix.
> > I suspect this is the cause of a wide variation in our build times across
> > different machines that we're seeing (particularly that builds take much
> > longer in our integration server).
> >
> > If I get a moment I'll profile our build again (easy because it's written
> > in Java) and post any relevant info here.
> >
> > Cheers,
> > Colin
> >
> > 2008/11/19 Benjamin Damm <[EMAIL PROTECTED]>
> >
> > > Hi,
> > >
> > > Does anyone have or would anyone be interested in a caching facade to
> > > speed
> > > up ivy:resolve? I have a case where my project has about 100 modules
> > > in a few different configurations and it is taking a little longer to
> > > do the resolve than I can stand. I'm concerned about what will happen
> > > when I introduce NFS into the mix. For these reasons I'm considering
> > > what options I
> > > may have for caching the ivy:resolve (if this is possible) by perhaps
> > > storing
> > > structures to disk in a file that can be updated (perhaps by deleting
> > > it, I'm
> > > not sure yet).
> > >
> > > The difference between this cache and the ivy cache is I'm not copying
> > > the files out of their locations; where they are presently is just fine
> > > because the entire repository is on local disk (and managed by p4). So
> > > really this comes down to speeding up the filesystem resolver.
> > >
> > > -Ben
> > >
> > > --
> > > Benjamin Damm
> > > Silver Spring Networks
> > > 650-298-4200 x201
--
Benjamin Damm
Silver Spring Networks
650-298-4200 x201
