+1 to the initiative. It would great to have better support for Go and Docker container images. The current Go maven integration in particular is clunky [1], but I'll have to look into the details of the alternatives to see if they are better.
Henning [1] https://github.com/apache/beam/blob/master/sdks/go/BUILD.md On Mon, Oct 30, 2017 at 10:27 AM, Kenneth Knowles <k...@google.com.invalid> wrote: > I also support exploring a move away from Apache Maven for orchestrating > our build. > > For a single-module project, I still think it can be a good build tool, and > we could still use it for this sort of thing, but I think we are reaching a > multi-module scale where it does not work well. Almost all of our jobs > build things that are not needed and run tests that are redundant, and it > is not easy to do better, even with a sequence of maven commands. > > I'd like to lay out what we hope for from a tool. Here's a start: > > General: > > - Dependency-driven build so devs working on one thing build & test only > what is needed > - Supports orchestration across Protobuf, Java, Python, Go, Docker images > - Meets devs where they are, letting folks in one language use familiar > tools > - Caching across builds as much as possible > - Easily extensible for when it doesn't have the feature we need (writing > a maven plugin is too much, using maven-exec-plugin is too crufty) > - Preferably a declarative configuration language > > Java needs beyond the basics, which could be executed by the orchestrator > or my module-local mvn builds, etc. > > - Pulling deps from maven central and alternate repos > - Findbugs > - RAT > - Dependency rule enforcement > - IWYU (Include What You Use) > - Multiple Java versions in same project > - ASF release workflow > > I probably missed some must-haves or nice-to-haves. I'd love to compile > thoughts on other languages' needs. > > Based on these, another project I would consider is Bazel. We could very > easily use it to orchestrate, but use Maven (or Gradle!) on the leaves. I > also think that Gradle is also more focused on the JVM ecosystem, so it is > not quite as neutral as Bazel, and uses Groovy which is a bit more esoteric > than Python for writing Bazel rules. > > Kenn > > On Mon, Oct 30, 2017 at 9:37 AM, Lukasz Cwik <lc...@google.com.invalid> > wrote: > > > I wanted to make this thread more visible. This discussion stems from > Ken's > > thread about Jenkins pre/post commit issues[1]. > > > > I did some investigation as for ways to improve the quality of the signal > > from Jenkins by trying to modify the Jenkins jobs spawned from Groovy. I > > had limited success but everything I felt like I was doing was just > > patching symptoms of the problem which is that our build is just too > slow. > > For example, we keep adding all these profiles to Maven or tuning how a > > plugin runs to eek out a small decrease in build time. I believe swapping > > away from Apache Maven to a build tool which only builds the things which > > have changed in a PR would be the best approach. > > > > I would suggest that we migrate to Gradle as our build tool. I am > > suggesting Gradle because: > > * It is used in lots of open source projects and has a very large > community > > behind it. > > * It has better support for building languages other then Java > > (PyGradle[2], GoGradle[3], ...) > > * Its incremental build support works and only builds things that changed > > through the use of a build cache. Even without the build cache (or for > > clean builds), it is much faster. > > * Apache Infra already has Gradle v4.x installed on the Jenkins machines. > > > > Any alternatives that should be considered or should we stick with Apache > > Maven? > > > > 1: > > https://lists.apache.org/thread.html/25311e0e95be5c49afb168d9b4b4d3 > > 57984c10c39c7b01da8ff3baaf@%3Cdev.beam.apache.org%3E > > 2: https://github.com/linkedin/pygradle > > 3: https://github.com/gogradle/gogradle > > >
