I previously worked with pants [1] in the context of an extremely large (~50GiB) repository which saw a huge amount of daily churn. In that context build tools like pants and Bazel become absolutely essential because the only way to have a reasonable CI/CD system and build/test times is to do change analysis to predict what tests and other products are impacted. While I think that Pants, Buck [2], Bazel and other Google Blaze derivatives have made fundamental and perhaps undesirable complexity sacrifices in order to be polygot, they do add significant value to a large organization. Presently, I work on a much smaller Clojure project encompassing several microservices which communicate over Kafka. Each service is its own sub-project in a single repository. We coordinate modules using lein- modules [3]. For a variety of reasons, the majority of our interesting tests involve Kafka and potentially integrating against other services continually built out of the same repo. These tests are wall clock expensive due to a variety of system initialization overheads which are difficult to amortize. It has already proven very profitable for us to deploy a limited amount of test invalidation by using the inter-module dependency information lein-modules exposes to restrict test runs to those modules which have seen source changes or which according to topological sort order may depend on impacted modules. This is a correct but non-minimal heuristic which leaves time on the table. Our deployment artifacts are also fairly expensive to produce, and so I'm very interested in the possibility of a polygot tool which can capture dependency information both for test invalidation and build product invalidation. It is in theory possible to write a "macro" (function) for either Pants or Buck which defines a clojure_library or clojure_app target in terms of simply generating a java_library which happens to have your clj(c) files for resources and depend on Clojure. I got this working for Buck at one point, but Buck doesn't support Maven dependencies and requires that you hand-manage all your jars which made the ergonomics unfortunate. I haven't taken a serious crack at getting Pants wired up that way. I worked with the pants developers at one point to understand what they thought it would take to get a clojure_library target integrated "the right way" and it turns out to be a mess because of how pants' JVM model is coupled to Ivy, Scala's incremental compiler. I've been sketching at Katamari [4] on and off, the thought being that the fundamental dependency and invalidation concerns are well enough understood [7] that maybe by trying to build a framework or fire customers it's possible to achieve a meaningful complexity reduction. I haven't managed that yet. Of particular frustration is notions of when to build classpaths and resolve dependencies - although forcing dependency pinning fixes this which is Buck's answer.There's also Juxt's mach [5] (not to be confused with Mozilla's build tool of the same name), krei.alpha [6] which has some of the same goals as katamari and I've seen some sketches at multi-module boot. In short, I do think that more sophisticated build tools make sense in the Clojure space. Sure Clojure itself doesn't have traditional compilation needs in the sense of say C++ or Scala, but notions of products and static dependency information is more generally valuable and can help with whole applications or systems. After spending a fair bit of time evaluating the options, I concur with Colin. If lein-modules or lein-monolith doesn't solve your problem Gradle is probably the most mature "off the shelf" solution. Reid
[1] https://pantsbuild.org/[1] [2] https://buckbuild.com/ [3] https://github.com/FundingCircle/lein-modules [4] https://github.com/arrdem/katamari [5] https://github.com/juxt/mach [6] https://github.com/SevereOverfl0w/krei.alpha [7] https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf On Mon, Jun 4, 2018, at 10:04 PM, Nathan Fisher wrote: > It's not something that as a user you'd be exposed to. The easiest way > to demonstrate the impact is to write a library and test target in two > different ways for a Java project.> > 1. using java_library target for the library. > 2. using genrule for the library that calls out to javac. > > Run the test target, introduce whitespace into the library, and run > the test target again. It should only do a full execution in the 2nd > example.> > On Sunday, 15 April 2018 17:09:12 UTC-3, Gregg Reynolds wrote: >> >> >> On Mon, Jan 29, 2018, 2:07 PM Nathan Fisher >> <nfi...@junctionbox.ca> wrote:>>> >> ... >>> In order to get the speed that the build tool promises you need the >>> ability to efficiently parse the ABI of each file and only build >>> its dependents when the ABI has changed.>> >> Can you please elaborate on this a bit? I've been using Bazel for a >> project that involve C and Java, with cross-compiles, and ABI parsing >> has never come up. It just works.>>> >>> >>> I’m not certain you can do that with Clojures compiler but someone >>> else might be able to chime in with a way that you could.>> >> I like boot for clojure stuff, not sure Bazel offers any advantage. >> Clojurescript may be a different story, tho.>> >> So far I cannot see a good reason to switch from boot to Bazel for >> 100% clojure projects. Mixed language - clojure, clojurescript, js, >> Java, kotlin, JNI,etc - may be a different story, still not sure.>> >> Then there's remote caching: >> https://docs.bazel.build/versions/master/remote-caching.html. a >> hyuuuuge win for many scenarios, not so sure about clojure.>> >> G >> G > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient > with your first post.> To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google > Groups "Clojure" group.> To unsubscribe from this group and stop receiving > emails from it, send > an email to clojure+unsubscr...@googlegroups.com.> For more options, visit > https://groups.google.com/d/optout. Links: 1. https://www.pantsbuild.org/ -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
