On Wed, Jul 8, 2009 at 11:28 AM, Phil Hagelberg<p...@hagelb.org> wrote:
>
> I've been noodling on the problem of dependency management for a while
> now. It's definitely a pain point for projects with more than a couple
> dependencies. Currently our approach has been to use maven, but that
> involves a fair amount of arcane knowledge as well as writing a bunch of
> XML, which isn't a lot of fun. But there's been a community consensus
> that any dependency tool needs to be able to leverage all the valuable
> stuff out there that's currently in maven repositories.
>
> I've put together a proof-of-concept dependency manager called
> Corkscrew. It uses Maven under the hood as well as offering dependencies
> on source packages stored in git or subversion. A simple project.clj
> file lays out the dependencies and other project info:
>
> {:name "my-sample"
> :version "1.0"
> :dependencies [["tagsoup" "1.2" "org.ccil.cowan.tagsoup"]
> ;; group defaults to name
> ["rome" "0.9"]]
> :source-dependencies [["clojure-contrib" "r663" :svn
> "http://clojure-contrib.googlecode.com/svn/trunk";]
> ["enlive" "95b2558943f50bb9962fe7d500ede353f1b578f0"
> :git "git://github.com/cgrand/enlive.git"]]}
>
> You can install it with:
>
> $ git clone git://github.com/technomancy/corkscrew.git
> $ cd corkscrew
> $ ./install
> $ cp bin/corkscrew /somewhere/on/your/path
>
> Create a project.clj file in your project root based on the one
> above. Then you can use "corkscrew deps" to set everything up for
> you. At that point just make sure your classpath includes
> target/dependency/ and you should be good to go.
>
> It's simple, but it solves the main problems that I've been having with
> more complicated projects. I'd love to get some opinions on it.
>
> The biggest issue right now is that it runs Maven as a subprocess rather
> than using the Java API in the same VM because I can't make head or tail
> of the Maven Java API (it uses plexus.core), but shelling out works as a
> proof-of-concept even if it's tacky.
>
> -Phil
>
Thanks for bringing that up, as it's been bugging me for a some time
now (and violently ejected me out of lurk mode on the list). What
follows is a pretty long essay on the topic, I hope you stay with me,
and take part in the debate, because I think the availability of a
good buildtool/dependency manager is absolutely crucial to use Clojure
in larger projects, and as such for adoption on a wider scale.
I have basically the same problem as you in that Clojure is lacking a
proper buildtool (i.e. one that can replace Maven). I used Maven for a
long time and lived with it's strengths and weaknesses. It's far from
perfect, but Maven does still give you quite a lot:
* a dependency resolution and publishing model for generated artifacts
* a build based on conventions (can be overridden, but Maven pushes
you hard to do things in a de-facto standard way)
* a defined execution model (the phases thing. Not always nice, and
not always sufficient, but at least something)
* a defined API to hook into with your plugins (API design is so-so,
and documentation is IMHO abysmal, but ...)
* a ton of reports and plugins to extend the model (not always
perfect, and configuration is not nice, but they usually do what they
advertise)
* more?
On the other hand, some things Maven are always a pain in the back:
figuring out how to configure plugins, writing plugins because it's
the only way to do something non-standard in Maven, and
debugging/finding out what the heck is going on, if some plugin is not
doing what it should do... That's the arcane knowledge part.
But recently I had the need for a more flexible buildtool that would
allow me to build non-standard JVM based languages in a non-standard
fashion and still play nice. Case in point was that I had to marry two
different dependency resolution systems (Maven style, and
Jython/Python style), and then build a mix of code in
Scala/Java/Jython. The experience was none too pretty with Maven.
Most of the buildtools out there for JVM related stuff don't try to
compete with Maven and all it's plugins and reports, but they start by
replacing/wrapping Ant (and usually Ivy for dependency management).
That description applies to Buildr, the Groovy Ant Builder, Gant,
Grape (which is a wrapper for Ivy), SBT (Simple Build Tool - Scala),
Scalr (Scala), Lancet, and probably some more that I missed. The only
notable exception I could dig up was Gradle.
Gradle (http://gradle.org/) turns out to be very interesting. Here are
some propaganda points (not exhaustive):
* Although it currently uses mainly a Groovy DSL, the Gradle core is
written in Java, and there's an explicit goal on the roadmap to
support DSLs written in other JVM languages on top of it as to make it
an attractive choice for everybody. For Scala there's a patch in the
Bugtracker to add a Scala DSL on top. Other languages I'm unaware of.
* Convention based builds. They have taken care that they appeal to
Maven developers (all the conventions of the Java Plugin are the same
ones as in Maven), while still leaving it flexible through
configuration that every project layout can be built.
* Hierarchical, dependency based builds.
* A build is executed in two major phases (configuration and
execution), and tasks are allowed to hook into both (a task can make
it's execution dependent on whether another task is scheduled to run
*after* itself (try that with Maven))
* A build graph containing all the tasks for execution can be accessed
and manipulated programmatically.
* Gradle defines it's own task API, but provides an Ant wrapper to
give you an easy migration route or extension path (similar to Gant,
or Lancet in Clojure). Custom tasks can be easily coded up, although
you need custom things less often than with Maven and Co., because a
lot can be solved with a flexible build script (it's all code).
* The build can be extended from within a project. You don't need a
plugin project that you're only going to use in one project anyway.
There's a separate source tree in your project home that hosts
extension to your build.
* Dependency resolution is done with Ivy. But they took care to clean
the Ivy model a bit up so that it provides better support for
convention based builds.
* They have an integration layer with Maven. I don't know how far it
actually goes, but I know that you can call through to Maven for some
things (MavenPlugin).
Non-technical
* The mailinglist is very active, and receptive to input that comes
from the outside. They have github mirrors, pull requests can be
issued easily.
* The project is well setup: toplevel domain, liberal license (Apache
v2), plethora of SCM can be used (master is on subversion, mirrors on
github (git), and launchpad (bzr)) issue tracker (JIRA), wiki
(Confluence), public roadmap, continuous integration (bamboo &
teamcity, platform specific builds), selfhosting (gradle is built by
gradle), documentation is autogenerated during the build (including
the userguide HTML and PDFs).
* The guys working on Gradle have been around the build tool circus
for a long time apparently (five committers, among them: the founder
of JBoss IDE, and the author of Gant).
* The team is dedicated enough to evangelize their product largescale,
including talks on conferences around the globe, and at least one of
the participants runs a consultancy providing advice on buildsystems.
* Reasonably complete documentation (there's arguably room for
improvement, especially in structure, but you can find almost anything
in the userguide).
* They are willing to change stuff in how it works internally if
there's technical merit.
* It's developing fast, and I have the feeling that it's getting
increasingly powerful and flexible.
* It's not very complicated IMHO. It's probably slightly more
complicated than ant + sugar coating, but they Gradle tagline is:
"make the impossible possible, the possible easy, and the easy
elegant." (Moshé Feldenkrais), and I think they are definitely within
scope.
Ok, now the question why am I pushing this so hard? Because have been
annoyed at the exact same things in every language that I tried for
the JVM. Maven is not very good but it's usually the best thing you
can get. And although there's a lot of competition for build tools,
nobody seems to try to tackle what Maven solved, and only goes for a
simpler subset of the problems. Usually it goes something like: "Our
language is more productive than Java and we don't like Maven, because
it's a pain to use. Ant is nice but uses XML so we can't use it out of
the box. We can't really wrap Maven, so let's wrap Ant, and since we
are lacking dependency management in that case, let's also wrap Ivy,
add a little bit of sugar on top, and voila, new shiny build system."
Except that everybody now has to learn a gazillion new (and different)
tools if he wants to build stuff easily in a certain language, and
beware if you try to do something cross language (a problem that will
become more common with alternative languages for the JVM on the
rise).
I would very much like to see that people don't suffer the NIH
syndrome everywhere. Usually people forget that every alternative
technology implemented has an associated cost that everyone looking
for a solution has to pay, because I have to consciously discard the
tool as insufficient, and that means I have to acquire enough
knowledge about each tool to make an informed decision (the paradox of
choice). I cannot even imagine how much time has been wasted by
halfbaked (and partially or fully abandoned) projects.
If the miracle would happen and the different, rapidly growing
language communities on the JVM finally could commit themselves to
support and push a shared buildtool, then that tool would have enough
steam to grow into a decent alternative for Maven. It might even
become the preferred way of build at least non-Java projects, who
knows. If everybody is trying to get their own share of followers and
only solves what is immediately visible for the projects used by
themselves and their followers, then we will never get anywhere.
And please note that I'm not asking to start from scratch. Gradle is
there and has momentum already. I'm just hoping that I'm not the only
one that sees a benefit in joining forces with stuff that already
exists. If people in the Clojure community start to throw your weight
behind something that seems to be on the right track, can still be
influenced and made better.
To get started with Gradle, you would have to:
* write a DSL that can talk to Gradle Java API.
* write a ClojurePlugin (probably based on the JavaPlugin) for Gradle,
if you have specialized wishes (such as that src/main/clojure should
be the default convention for Clojure projects, or that you can
specify which version of Clojure should be used for compilation etc.).
</propaganda>
Sorry for the long mail/rant, but it's been too painful too many
times. And a sincere apology to Phil, who just scratched an itch for
himself, and was interested in sharing it with the larger Clojure
community. This mail is not an attack on your work, or approach. It
just triggered something that I have stumbled over too many times and
I would love to see it finally resolved. I think that Gradle could be
used to give you all you envisioned for Corkscrew, but in the process,
you will gain so much more.
Disclaimer: I'm (currently) not actively developing Gradle itself, but
i was impressed with what's there and where it's going. I also took
the liberty to crosspost this mail to the Gradle dev list (CC'd) to
invite a broader participation.
I'm really hoping that we could find some consensus on the topic.
These are my probably 20$ or so...
Daniel
(probably by now unofficial Gradle Evangelist)
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
