clojure dependency management and build
When you are building something real with Clojure and Emacs/Slime/Swank - things can get a bit hairy with dependency management. I have scoured the inter-tubes in the past couple days to see what I could find. I found Lancet, for builds: http://github.com/stuarthalloway/lancet/tree/master I haven't tried it yet. I found some people doing things with Maven: Creating a clojurue app with maven: http://pupeno.com/blog/how-to-create-a-clojure-application?set_language=eo http://pupeno.com/blog/iterative-coding-with-a-clojure-application clojure-pom: http://github.com/dysinger/clojure-pom/tree/master I heard some chatter yesterday on #clojure about using Ivy with Clojure. So there is a flurry of activity. Please let me know if there are other things that I am missing. What I am doing now from my emacs / slime-repl is hacking things in manually to my swank-clojure-extra-classpaths. This doesn't scale for working with multiple clojure projects in emacs. I will probably create a script to make things a bit nicer. But I'd like something fundamentally better. Here are the issues: -I download lots of little projects things from github and i want to munge them all together for my app. This means I need to build jars (some with ant, otehrs with maven, etc.) and in other cases I want to depend directly on the .clj files using clojures namespace-to-dir-structure conventions. So there are a couple different ways to build of the classpath - one for .clj and one for .jar. -Many projects also have their own lib foler - with both jars and cljs, so I need to pick those deps up transatively. -The work in the Clojure community is proceeding very fast, so I'd like updating all the projects from git to be automated as well. So what is a good solution to these problems? Perhaps it would be cool to build some git/maven/lancet aware infrastructure to do this refreshing of deps, building the deps, and building up the classpath. It may also be good to configure .emacs to be able to load projects and rebuild the classpath dynamically based on lancet build files - much in the way that intelliJ or eclipse load projects from ant .builds or maven poms. Is all this too much, am I missing something that already exists? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
Lancet is more at proof-of-concept stage than ready for heavy lifting -- I am using a mix of Ant and Rake on my own Clojure stuff. :-) I am happy to contribute to solving some of these issues but don't have time to lead the effort. Stuart > When you are building something real with Clojure and Emacs/Slime/ > Swank - things can get a bit hairy with dependency management. > > I have scoured the inter-tubes in the past couple days to see what I > could find. > > I found Lancet, for builds: > http://github.com/stuarthalloway/lancet/tree/master > > I haven't tried it yet. > > I found some people doing things with Maven: > > Creating a clojurue app with maven: > http://pupeno.com/blog/how-to-create-a-clojure-application?set_language=eo > http://pupeno.com/blog/iterative-coding-with-a-clojure-application > > clojure-pom: http://github.com/dysinger/clojure-pom/tree/master > > I heard some chatter yesterday on #clojure about using Ivy with > Clojure. > > So there is a flurry of activity. Please let me know if there are > other things that I am missing. > > What I am doing now from my emacs / slime-repl is hacking things in > manually to my swank-clojure-extra-classpaths. This doesn't scale > for working with multiple clojure projects in emacs. > > I will probably create a script to make things a bit nicer. But I'd > like something fundamentally better. > > Here are the issues: > > -I download lots of little projects things from github and i want to > munge them all together for my app. This means I need to build jars > (some with ant, otehrs with maven, etc.) and in other cases I want > to depend directly on the .clj files using clojures namespace-to-dir- > structure conventions. So there are a couple different ways to > build of the classpath - one for .clj and one for .jar. > -Many projects also have their own lib foler - with both jars and > cljs, so I need to pick those deps up transatively. > -The work in the Clojure community is proceeding very fast, so I'd > like updating all the projects from git to be automated as well. > > So what is a good solution to these problems? Perhaps it would be > cool to build some git/maven/lancet aware infrastructure to do this > refreshing of deps, building the deps, and building up the > classpath. It may also be good to configure .emacs to be able to > load projects and rebuild the classpath dynamically based on lancet > build files - much in the way that intelliJ or eclipse load projects > from ant .builds or maven poms. > > Is all this too much, am I missing something that already exists? > > > > > > > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
Stuart Halloway writes: > Lancet is more at proof-of-concept stage than ready for heavy lifting > -- I am using a mix of Ant and Rake on my own Clojure stuff. :-) > > I am happy to contribute to solving some of these issues but don't > have time to lead the effort. I'm also quite willing to help with such an effort, but I don't feel qualified to lead it myself as I haven't had any exposure to other dependency-management tools on the JVM, though I have hacked a lot on RubyGems. But the needs for Clojure are quite different. -Phil --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
I keep a "Clojure stuff" dir with the latest revisions of all the projects I track -- Clojure, contrib, swank, slime, etc. -- and update them all with a shell script that runs the various "svn update" and "git pull" commands. So I always have access to the latest version of everything. But I don't automatically use the latest releases in my projects! Each project that I work on has a "lib" dir for JARs, including Clojure. I manage the JARs manually, but they're part of my source control repository. I update the the JARs only when some new functionality or bug fix that I depend on comes out. That way I know that a project won't suddenly break just because I've updated to the latest Clojure SVN. I use Ant for build management, with separate build dirs for compiled Java code and AOT-compiled Clojure code. Here's my build.xml: http://tinyurl.com/c5vkfm Multi-version dependency tracking is HARD. I've never seen a system that does it perfectly -- Rubygems, CPAN, Maven, you name it. -Stuart S On Apr 8, 3:31 pm, Bradford Cross wrote: > When you are building something real with Clojure and Emacs/Slime/Swank - > things can get a bit hairy with dependency management. > > I have scoured the inter-tubes in the past couple days to see what I could > find. > > I found Lancet, for builds:http://github.com/stuarthalloway/lancet/tree/master > > I haven't tried it yet. > > I found some people doing things with Maven: > > Creating a clojurue app with > maven:http://pupeno.com/blog/how-to-create-a-clojure-application?set_langua...http://pupeno.com/blog/iterative-coding-with-a-clojure-application > > clojure-pom:http://github.com/dysinger/clojure-pom/tree/master > > I heard some chatter yesterday on #clojure about using Ivy with Clojure. > > So there is a flurry of activity. Please let me know if there are other > things that I am missing. > > What I am doing now from my emacs / slime-repl is hacking things in manually > to my swank-clojure-extra-classpaths. This doesn't scale for working with > multiple clojure projects in emacs. > > I will probably create a script to make things a bit nicer. But I'd like > something fundamentally better. > > Here are the issues: > > -I download lots of little projects things from github and i want to munge > them all together for my app. This means I need to build jars (some with > ant, otehrs with maven, etc.) and in other cases I want to depend directly > on the .clj files using clojures namespace-to-dir-structure conventions. So > there are a couple different ways to build of the classpath - one for .clj > and one for .jar. > -Many projects also have their own lib foler - with both jars and cljs, so I > need to pick those deps up transatively. > -The work in the Clojure community is proceeding very fast, so I'd like > updating all the projects from git to be automated as well. > > So what is a good solution to these problems? Perhaps it would be cool to > build some git/maven/lancet aware infrastructure to do this refreshing of > deps, building the deps, and building up the classpath. It may also be good > to configure .emacs to be able to load projects and rebuild the classpath > dynamically based on lancet build files - much in the way that intelliJ or > eclipse load projects from ant .builds or maven poms. > > Is all this too much, am I missing something that already exists? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
On Thu, Apr 9, 2009 at 11:34 AM, Stuart Sierra wrote: > > I keep a "Clojure stuff" dir with the latest revisions of all the > projects I track -- Clojure, contrib, swank, slime, etc. -- and update > them all with a shell script that runs the various "svn update" and > "git pull" commands. So I always have access to the latest version of > everything. But I don't automatically use the latest releases in my > projects! > > Each project that I work on has a "lib" dir for JARs, including > Clojure. I manage the JARs manually, but they're part of my source > control repository. I update the the JARs only when some new > functionality or bug fix that I depend on comes out. That way I know > that a project won't suddenly break just because I've updated to the > latest Clojure SVN. > > I use Ant for build management, with separate build dirs for compiled > Java code and AOT-compiled Clojure code. Here's my build.xml: > http://tinyurl.com/c5vkfm Cool, thanks, I will look over this stuff today. Sounds like more or less what I have been doing. What are you doing about building up the classpath for your projects in slime/swan? Where does the rake come into play? > > > Multi-version dependency tracking is HARD. I've never seen a system > that does it perfectly -- Rubygems, CPAN, Maven, you name it. Yea, I don't imagine will solve things beautifully at first, but we should be able to at least streamline things a bit. > > > -Stuart S > > > On Apr 8, 3:31 pm, Bradford Cross wrote: > > When you are building something real with Clojure and Emacs/Slime/Swank > - > > things can get a bit hairy with dependency management. > > > > I have scoured the inter-tubes in the past couple days to see what I > could > > find. > > > > I found Lancet, for builds: > http://github.com/stuarthalloway/lancet/tree/master > > > > I haven't tried it yet. > > > > I found some people doing things with Maven: > > > > Creating a clojurue app with maven: > http://pupeno.com/blog/how-to-create-a-clojure-application?set_langua...http://pupeno.com/blog/iterative-coding-with-a-clojure-application > > > > clojure-pom:http://github.com/dysinger/clojure-pom/tree/master > > > > I heard some chatter yesterday on #clojure about using Ivy with Clojure. > > > > So there is a flurry of activity. Please let me know if there are other > > things that I am missing. > > > > What I am doing now from my emacs / slime-repl is hacking things in > manually > > to my swank-clojure-extra-classpaths. This doesn't scale for working > with > > multiple clojure projects in emacs. > > > > I will probably create a script to make things a bit nicer. But I'd like > > something fundamentally better. > > > > Here are the issues: > > > > -I download lots of little projects things from github and i want to > munge > > them all together for my app. This means I need to build jars (some with > > ant, otehrs with maven, etc.) and in other cases I want to depend > directly > > on the .clj files using clojures namespace-to-dir-structure conventions. > So > > there are a couple different ways to build of the classpath - one for > .clj > > and one for .jar. > > -Many projects also have their own lib foler - with both jars and cljs, > so I > > need to pick those deps up transatively. > > -The work in the Clojure community is proceeding very fast, so I'd like > > updating all the projects from git to be automated as well. > > > > So what is a good solution to these problems? Perhaps it would be cool > to > > build some git/maven/lancet aware infrastructure to do this refreshing of > > deps, building the deps, and building up the classpath. It may also be > good > > to configure .emacs to be able to load projects and rebuild the classpath > > dynamically based on lancet build files - much in the way that intelliJ > or > > eclipse load projects from ant .builds or maven poms. > > > > Is all this too much, am I missing something that already exists? > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
Iv been using gant which has built in support for Ivy & Ant (for example http://tiny.cc/tRGKB), since the build scripts are code they make it easier to keep build scripts DRY. On Apr 9, 9:40 pm, Bradford Cross wrote: > On Thu, Apr 9, 2009 at 11:34 AM, Stuart Sierra > wrote: > > > > > > > I keep a "Clojure stuff" dir with the latest revisions of all the > > projects I track -- Clojure, contrib, swank, slime, etc. -- and update > > them all with a shell script that runs the various "svn update" and > > "git pull" commands. So I always have access to the latest version of > > everything. But I don't automatically use the latest releases in my > > projects! > > > Each project that I work on has a "lib" dir for JARs, including > > Clojure. I manage the JARs manually, but they're part of my source > > control repository. I update the the JARs only when some new > > functionality or bug fix that I depend on comes out. That way I know > > that a project won't suddenly break just because I've updated to the > > latest Clojure SVN. > > > I use Ant for build management, with separate build dirs for compiled > > Java code and AOT-compiled Clojure code. Here's my build.xml: > >http://tinyurl.com/c5vkfm > > Cool, thanks, I will look over this stuff today. Sounds like more or less > what I have been doing. What are you doing about building up the classpath > for your projects in slime/swan? Where does the rake come into play? > > > > > Multi-version dependency tracking is HARD. I've never seen a system > > that does it perfectly -- Rubygems, CPAN, Maven, you name it. > > Yea, I don't imagine will solve things beautifully at first, but we should > be able to at least streamline things a bit. > > > > > -Stuart S > > > On Apr 8, 3:31 pm, Bradford Cross wrote: > > > When you are building something real with Clojure and Emacs/Slime/Swank > > - > > > things can get a bit hairy with dependency management. > > > > I have scoured the inter-tubes in the past couple days to see what I > > could > > > find. > > > > I found Lancet, for builds: > >http://github.com/stuarthalloway/lancet/tree/master > > > > I haven't tried it yet. > > > > I found some people doing things with Maven: > > > > Creating a clojurue app with maven: > >http://pupeno.com/blog/how-to-create-a-clojure-application?set_langua... > > > > clojure-pom:http://github.com/dysinger/clojure-pom/tree/master > > > > I heard some chatter yesterday on #clojure about using Ivy with Clojure. > > > > So there is a flurry of activity. Please let me know if there are other > > > things that I am missing. > > > > What I am doing now from my emacs / slime-repl is hacking things in > > manually > > > to my swank-clojure-extra-classpaths. This doesn't scale for working > > with > > > multiple clojure projects in emacs. > > > > I will probably create a script to make things a bit nicer. But I'd like > > > something fundamentally better. > > > > Here are the issues: > > > > -I download lots of little projects things from github and i want to > > munge > > > them all together for my app. This means I need to build jars (some with > > > ant, otehrs with maven, etc.) and in other cases I want to depend > > directly > > > on the .clj files using clojures namespace-to-dir-structure conventions. > > So > > > there are a couple different ways to build of the classpath - one for > > .clj > > > and one for .jar. > > > -Many projects also have their own lib foler - with both jars and cljs, > > so I > > > need to pick those deps up transatively. > > > -The work in the Clojure community is proceeding very fast, so I'd like > > > updating all the projects from git to be automated as well. > > > > So what is a good solution to these problems? Perhaps it would be cool > > to > > > build some git/maven/lancet aware infrastructure to do this refreshing of > > > deps, building the deps, and building up the classpath. It may also be > > good > > > to configure .emacs to be able to load projects and rebuild the classpath > > > dynamically based on lancet build files - much in the way that intelliJ > > or > > > eclipse load projects from ant .builds or maven poms. > > > > Is all this too much, am I missing something that already exists? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
Hi Bradford, Am 08.04.2009 um 21:31 schrieb Bradford Cross: I heard some chatter yesterday on #clojure about using Ivy with Clojure. Yes. I'm working on marrying Clojure and Ivy. So far with, IMHO, great success. I set up a little ivy repo with clojure and contrib. The clojure module has the following configurations: - default => AOT compiled clojure jar - dev => the previous + a source jar - slim => Java source compiled, clojure sources as is The contrib module has (roughly) one configuration per unit, ie. def, pprint, stacktrace, with the following exceptions: - I grouped together the scattered math stuff into a math configuration. - generic and types depend on each other so their combined into the generic configuration. Similar to clojure a source configuration is included, which adds a source jar to the dependencies This allows very fine-grained control of what is needed. So if you want only def, you only download a 13k not 2.5M. To use this setup, you only have to specify the dependencies in your ivy.xml and ivysettings.xml files. Here an example: ivysettings.xml: http://kotka.de/ivy/ivysettings.xml"/> The first line sucks in the default configuration, the second tells Ivy how to resolve the clojure and contrib dependencies. ivy.xml: ... conf="default"/> rev="1.0.0" conf="default->def"/> Now you only have to integrate Ivy with your build system. It can be done very easily with ant, but may also be used standalone, eg. with make. Here you can see an example how I integrated Ivy with my lazymap project. http://bitbucket.org/kotarak/lazymap/src/ It is avalaible via the kotka.de ivysettings as conf="default"/> I can't speak for SLIME, but VimClojure can be simply started with the resolved jars from lib directory in the classpath. -I download lots of little projects things from github and i want to munge them all together for my app. This means I need to build jars (some with ant, otehrs with maven, etc.) and in other cases I want to depend directly on the .clj files using clojures namespace-to-dir- structure conventions. So there are a couple different ways to build of the classpath - one for .clj and one for .jar. I'm not sure, that Ivy can help in the general case, where you have to build the dependencies. I think it only downloads the files and makes them available. In particular it doesn't seem to handle subdirectories. So vanilla clj might not work. -Many projects also have their own lib foler - with both jars and cljs, so I need to pick those deps up transatively. This is a breeze with Ivy. It must say, that this just works beautifully. The dependencies either work with Ivy or with Maven and are resolved transitively. You can also mirror dependencies locally, to make sure they don't go away. -The work in the Clojure community is proceeding very fast, so I'd like updating all the projects from git to be automated as well. This could be easily done with Ivy. Check out the dependencies you need. Add an ivy.xml and a custom build script for ant. Then you can create a master build.xml, which iterates through all dependencies in the right order to pull changes, rebuild and publish into your local repository. I actually haven't set up production version of such a structure but I tried around and it worked without problems. The Ivy documentation has some tutorial on this. So what is a good solution to these problems? Perhaps it would be cool to build some git/maven/lancet aware infrastructure to do this refreshing of deps, building the deps, and building up the classpath. It may also be good to configure .emacs to be able to load projects and rebuild the classpath dynamically based on lancet build files - much in the way that intelliJ or eclipse load projects from ant .builds or maven poms. I'm not sure for bigger setups, but for my current needs Ivy Just Works. So I think it's certainly worth a look. Sincerely Meikel PS: May changes to clojure and contrib are attached as diffs. They add the Ivy information and a new build+ivy.xml for ant. clojure-ivy2.diff Description: Binary data contrib-ivy2.diff Description: Binary data smime.p7s Description: S/MIME cryptographic signature
Re: clojure dependency management and build
I have written up my journeys so far: http://bradfordcross.blogspot.com/2009/04/clojure-build-and-dependency-management.html I re-used Meikel's ant + ivy successfully. Next step is to get something working for the transative deps , i.e. pulling other clojure projects i depend on and building them and getting the resulting jars and transitive deps on the classpath. Rigth now I jsut pack it all up into jars at the top level lib dir and add that to my swank-extra-classpaths. I have had some good discussions with Meikel Brandmeyer and Phil Hagelberg and things seem to be coming along nicely. I'd like to get things working with ant + ivy xml and so I understand all the issues and then start building out Lancet or something similar so we can script in clojure rather than xml. :-) On Sat, Apr 11, 2009 at 7:06 AM, Meikel Brandmeyer wrote: > Hi Bradford, > > Am 08.04.2009 um 21:31 schrieb Bradford Cross: > > I heard some chatter yesterday on #clojure about using Ivy with Clojure. >> > > Yes. I'm working on marrying Clojure and Ivy. So far with, IMHO, > great success. I set up a little ivy repo with clojure and contrib. > The clojure module has the following configurations: > > - default => AOT compiled clojure jar > - dev => the previous + a source jar > - slim => Java source compiled, clojure sources as is > > The contrib module has (roughly) one configuration per unit, ie. > def, pprint, stacktrace, with the following exceptions: > > - I grouped together the scattered math stuff into a math configuration. > - generic and types depend on each other so their combined into > the generic configuration. > > Similar to clojure a source configuration is included, which adds > a source jar to the dependencies > > This allows very fine-grained control of what is needed. So if you want > only def, you only download a 13k not 2.5M. > > To use this setup, you only have to specify the dependencies in > your ivy.xml and ivysettings.xml files. Here an example: > > ivysettings.xml: > > >http://kotka.de/ivy/ivysettings.xml"/> > > > The first line sucks in the default configuration, the second tells > Ivy how to resolve the clojure and contrib dependencies. > > ivy.xml: > >... > > > conf="default"/> > conf="default->def"/> > > > > Now you only have to integrate Ivy with your build system. It can > be done very easily with ant, but may also be used standalone, > eg. with make. > > Here you can see an example how I integrated Ivy with my lazymap project. > >http://bitbucket.org/kotarak/lazymap/src/ > > It is avalaible via the kotka.de ivysettings as > > > > I can't speak for SLIME, but VimClojure can be simply started with the > resolved jars from lib directory in the classpath. > > -I download lots of little projects things from github and i want to munge >> them all together for my app. This means I need to build jars (some with >> ant, otehrs with maven, etc.) and in other cases I want to depend directly >> on the .clj files using clojures namespace-to-dir-structure conventions. So >> there are a couple different ways to build of the classpath - one for .clj >> and one for .jar. >> > > I'm not sure, that Ivy can help in the general case, where you have to > build > the dependencies. I think it only downloads the files and makes them > available. > In particular it doesn't seem to handle subdirectories. So vanilla clj > might not > work. > > -Many projects also have their own lib foler - with both jars and cljs, so >> I need to pick those deps up transatively. >> > > This is a breeze with Ivy. It must say, that this just works beautifully. > The > dependencies either work with Ivy or with Maven and are resolved > transitively. > You can also mirror dependencies locally, to make sure they don't go away. > > -The work in the Clojure community is proceeding very fast, so I'd like >> updating all the projects from git to be automated as well. >> > > This could be easily done with Ivy. Check out the dependencies you need. > Add an ivy.xml and a custom build script for ant. Then you can create a > master build.xml, which iterates through all dependencies in the right > order > to pull changes, rebuild and publish into your local repository. > > I actually haven't set up production version of such a structure but I > tried > around and it worked without problems. The Ivy documentation has some > tutorial on this. > > So what is a good solution to these problems? Perhaps it would be cool to >> build some git/maven/lancet aware infrastructure to do this refreshing of >> deps, building the deps, and building up the classpath. It may also be good >> to configure .emacs to be able to load projects and rebuild the classpath >> dynamically based on lancet build files - much in the way that intelliJ or >> eclipse load projects from ant .builds or maven poms. >> > > I'm not sure for bigger setups, but for my current needs Ivy Just Works. So > I think it's certainly worth a lo
Re: clojure dependency management and build
On Apr 13, 2009, at 4:58 PM, bradford cross wrote: I have written up my journeys so far: http://bradfordcross.blogspot.com/2009/04/clojure-build-and-dependency-management.html Thanks very much for the write-up! Given the recent discussions and interest in dependency management I thought I'd mention an idea in that area that I think is a good one. At ILC '09, Francois-Rene Rideau made a presentation about "xcvb": http://common-lisp.net/project/xcvb/ which is intended to be a successor to Common Lisp's ASDF with some nice features. One of the issues it addresses is the possibly problematic behavior of a "same world" compiler as we have in Clojure AOT: build dependencies which are intended to be explicit and correct can be incorrect, but accidentally work. Here's an example: A depends on B and C E depends on A, B, and D, but only declares A and D Compile A Compile E works Now change A so it no longer depends on B. Now compiling E "breaks". (And in the general case, the guy who made the change to A knows nothing about E.) In xcvb, the strategy they use to avoid that and make sure all dependencies are correctly specified is to make the compilation of each file start with the compiler's "image" in a known, minimal state. They break the build into a dependency discovery phase and a make-like minimal rebuild phase (which may actually use "make" itself) to accomplish this. The tradeoff in overall "system" compilation time is the reloading of compiled code every time it's needed rather than just once. For us, the corresponding strategy would be that the compilation of each namespace should start with only clojure.core loaded. We could adopt this strategy in a given Clojure build system without any changes to how Clojure works (by launching a new instance of Clojure for each compilation) which makes it particularly easy to experiment with. --Steve smime.p7s Description: S/MIME cryptographic signature
Re: clojure dependency management and build
On Mon, Apr 13, 2009 at 2:31 PM, Stephen C. Gilardi wrote: > On Apr 13, 2009, at 4:58 PM, bradford cross wrote: >> >> I have written up my journeys so far: >> http://bradfordcross.blogspot.com/2009/04/clojure-build-and-dependency-management.html >> > > Thanks very much for the write-up! > > Given the recent discussions and interest in dependency management I > thought I'd mention an idea in that area that I think is a good one. > > At ILC '09, Francois-Rene Rideau made a presentation about "xcvb": > > http://common-lisp.net/project/xcvb/ > > which is intended to be a successor to Common Lisp's ASDF with some nice > features. > > One of the issues it addresses is the possibly problematic behavior of a > "same world" compiler as we have in Clojure AOT: build dependencies which > are intended to be explicit and correct can be incorrect, but accidentally > work. > > Here's an example: > >A depends on B and C >E depends on A, B, and D, but only declares A and D > >Compile A >Compile E > >works > >Now change A so it no longer depends on B. > >Now compiling E "breaks". (And in the general case, the guy who made > the change to A knows nothing about E.) Yes, I have had nasty issues with this in the past. IMO, explicitly declaring deps and not relying on transitive resolution is the only way to go. > > > In xcvb, the strategy they use to avoid that and make sure all dependencies > are correctly specified is to make the compilation of each file start with > the compiler's "image" in a known, minimal state. They break the build into > a dependency discovery phase and a make-like minimal rebuild phase (which > may actually use "make" itself) to accomplish this. The tradeoff in overall > "system" compilation time is the reloading of compiled code every time it's > needed rather than just once. > > For us, the corresponding strategy would be that the compilation of each > namespace should start with only clojure.core loaded. > > We could adopt this strategy in a given Clojure build system without any > changes to how Clojure works (by launching a new instance of Clojure for > each compilation) which makes it particularly easy to experiment with. I believe we can achieve this via ant and ivy but i am not sure if they can force explicit deps. I'm not sure if a separate closure process is required, I have seen static analysis tools that can do the graph traversal and warn you about transitive dependency resolutions (or unused dependencies.) > > > --Steve > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
Hi, 2009/4/14 bradford cross > > > On Mon, Apr 13, 2009 at 2:31 PM, Stephen C. Gilardi wrote: > >> On Apr 13, 2009, at 4:58 PM, bradford cross wrote: >>> >>> I have written up my journeys so far: >>> http://bradfordcross.blogspot.com/2009/04/clojure-build-and-dependency-management.html >>> >> >> Thanks very much for the write-up! >> >> Given the recent discussions and interest in dependency management I >> thought I'd mention an idea in that area that I think is a good one. >> >> At ILC '09, Francois-Rene Rideau made a presentation about "xcvb": >> >> http://common-lisp.net/project/xcvb/ >> >> which is intended to be a successor to Common Lisp's ASDF with some nice >> features. >> >> One of the issues it addresses is the possibly problematic behavior of a >> "same world" compiler as we have in Clojure AOT: build dependencies which >> are intended to be explicit and correct can be incorrect, but accidentally >> work. >> >> Here's an example: >> >>A depends on B and C >>E depends on A, B, and D, but only declares A and D >> >>Compile A >>Compile E >> >>works >> >>Now change A so it no longer depends on B. >> >>Now compiling E "breaks". (And in the general case, the guy who >> made the change to A knows nothing about E.) > > > Yes, I have had nasty issues with this in the past. IMO, explicitly > declaring deps and not relying on transitive resolution is the only way to > go. > I don't understand, the example given by Stuart does not rely on any transitive resolution mechanism, but rather plain explicit dependency declaration. So I think that transitive resolution is not the problem here. The problem is that the compilation of A had a side effect left that the compilation of E benefited from. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
On Mon, Apr 13, 2009 at 3:22 PM, Laurent PETIT wrote: > Hi, > > 2009/4/14 bradford cross > >> >> >> On Mon, Apr 13, 2009 at 2:31 PM, Stephen C. Gilardi wrote: >> >>> On Apr 13, 2009, at 4:58 PM, bradford cross wrote: I have written up my journeys so far: http://bradfordcross.blogspot.com/2009/04/clojure-build-and-dependency-management.html >>> >>> Thanks very much for the write-up! >>> >>> Given the recent discussions and interest in dependency management I >>> thought I'd mention an idea in that area that I think is a good one. >>> >>> At ILC '09, Francois-Rene Rideau made a presentation about "xcvb": >>> >>> http://common-lisp.net/project/xcvb/ >>> >>> which is intended to be a successor to Common Lisp's ASDF with some nice >>> features. >>> >>> One of the issues it addresses is the possibly problematic behavior of a >>> "same world" compiler as we have in Clojure AOT: build dependencies which >>> are intended to be explicit and correct can be incorrect, but accidentally >>> work. >>> >>> Here's an example: >>> >>>A depends on B and C >>>E depends on A, B, and D, but only declares A and D >>> >>>Compile A >>>Compile E >>> >>>works >>> >>>Now change A so it no longer depends on B. >>> >>>Now compiling E "breaks". (And in the general case, the guy who >>> made the change to A knows nothing about E.) >> >> >> Yes, I have had nasty issues with this in the past. IMO, explicitly >> declaring deps and not relying on transitive resolution is the only way to >> go. >> > > I don't understand, the example given by Stuart does not rely on any > transitive resolution mechanism, but rather plain explicit dependency > declaration. So I think that transitive resolution is not the problem here. > The problem is that the compilation of A had a side effect left that the > compilation of E benefited from. > Right, let me restate it a different way. The goal is to enforce explicit deps so that you cannot rely on side effects that transitively resolve a dependency. My point is there are a couple ways to deal with this: 1) each unit (project or whatever you want to call it) builds in a different process 2) static analysis or dynamic analysis is injected into the build pipeline and causes failure (or gives warnings) if you are relying on a transitive resolution. > > > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: clojure dependency management and build
On Apr 13, 10:58 am, bradford cross wrote: > I have written up my journeys so > far:http://bradfordcross.blogspot.com/2009/04/clojure-build-and-dependenc... Nice article. > I re-used Meikel's ant + ivy successfully. Next step is to get something > working for the transative deps , i.e. pulling other clojure projects i > depend on and building them and getting the resulting jars and transitive > deps on the classpath. Rigth now I jsut pack it all up into jars at the top > level lib dir and add that to my swank-extra-classpaths. Maven already has this above solved plus multi-module projects. I currently build a half dozen interdependent clojure libs / apps with one top level "mvn install" command. The emacs classpath is solved on a per-module basis too with a custom repl generated for each module. > I have had some good discussions with Meikel Brandmeyer and Phil Hagelberg > and things seem to be coming along nicely. > > I'd like to get things working with ant + ivy xml and so I understand all > the issues and then start building out Lancet or something similar so we can > script in clojure rather than xml. :-) Me too - but for now I chose not to shave yaks and accept a little XML (one file) in my modules. I agree that it'll be awesome to have ivy/ maven power in simple DRY clojure files. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
