I've been thinking recently about how much copying Ivy does, and I've
filed some feature request to that effect (Thanks for getting IVY-353
done, Maarten!), but I think chipping away at the problem with feature
requests is the wrong away to address this issue, so I'm posting here
to share ideas.
My experience with Ivy has been several iterations of solving one
problem only to create another. I think my situation is kind of a
worst-case scenario for Ivy w.r.t. copying because of a combination of
factors. My company has a lot of modules with complicated
dependencies between them. At any given time, many modules are under
active development. Some of our modules produce very large artifacts,
and those tend to be some of the most frequently used modules as well
as the most frequently modified. While I think having such large
modules is generally a bad thing, it's not feasible for us to split
them up at the moment.
This situation affects our usage of Ivy in a number of ways, starting
with how we number our builds. Originally I had our continuous
integration system publish each build with a unique revision number,
but saving every build took up far too much disk space. As a result,
I switched to a system of overwriting old builds with new ones. We
use the same revision number for multiple builds because I think it's
probably easier to live with the same number referring to multiple
builds than to deal with most revision numbers referring to builds
that have been deleted to save space.
Because we use only file system repositories, I had wanted to bypass
the cache to save time and space, but since we overwrite artifacts in
our repository so frequently now, I decided it was best to use the
cache so that our developers would not have to worry about their
dependencies' artifacts changing from minute to minute.
Because we re-use revision numbers, it is necessary for us to manually
clean out the cache to guarantee that the latest version of every
module is being used. This is not a big problem for developers
because they don't clean out the cache too often, but the CI system is
not smart enough to know when it must clean out the cache, so it does
it for every build. As a result, it spends as much time copying
artifacts to its cache as it does building.
I'm very open to solutions that avoid this problem entirely, but I
think I good way to fix it might be to support creating hard links in
the cache instead of copying artifacts, preferably using a setting in
the Ivy config file rather than in ivy.xml files or build.xml files.
Because I'm proposing yet another use of links, I also think it would
be worthwhile to revisit the fix for IVY-353 that allows
<ivy:retrieve> to create symlinks. If both hard and soft links are to
be supported, and if links are to be supported in multiple places in
Ivy, I think it would be best to change the "symlink" attribute of
<ivy:retrieve> to "linkType", with values "copy", "soft", and "hard".
This same attribute could then be added to file system resolvers to
control how artifacts are copied to the cache, and the value "none"
could be used to implement IVY-360 ("Add property to control useOrigin
default").
Any thoughts?
Thanks,
jw
