Adding to Travis list these would be hcatalog's requirements for a build system: * ability to easily specify dependencies and dependency repositories (I hate ivy :) ). * good support for easily writing multi-module projects with minimal configuration (ties in with Travis thought on separating modules) * plugins for junit, clover, checkstyle, findbugs and forrest * easy way to run e2e tests.
All of these are easily satisfied by maven (e2e will require ant plugin). Gradle also satisfies them though some things are more difficult considering it is still in its infancy compared to maven. For eg: clover requires a lot of code (http://docs.codehaus.org/display/GRADLE/Cookbook), but there are examples out there already and we can use them. The positive thing with Gradle is it provides lot of flexibility unlike maven and it can make doing a lot of things easier. For eg: If you have to copy file to some other location in maven during your build you will have to use the ant plugin. It also has good integration with ant, and you can easily write ant tasks natively in gradle if a plugin is not available. The only con I see with gradle is that if we don't write the build script well modularized and cleanly, it might turn out to be unwieldy and difficult to maneuver like a build.xml file. With maven the chances of that are less because of the conventions and xml configuration. But if we have someone (Jakob) who has good experience with it then it might not be a problem. I am fine with going with either. In fact Gradle sounds more interesting just for the reason it is something new and is gaining popularity and we would be an early adopter in the hadoop family. But that's just the eager beaver in me talking :). We don't have much complexity in our builds yet. So this would be the time to decide maven or gradle and go with one of it. Regards, Rohini On Fri, Jun 22, 2012 at 5:48 PM, Travis Crawford <[email protected]>wrote: > On Fri, Jun 22, 2012 at 5:28 PM, Jakob Homan <[email protected]> wrote: > > I've been wanting to take a crack at this with Gradle, which we've > > used extensively with great results. I was hoping to show up with > > something in hand rather than suggest something new, igniting a whole > > big war. Any resistance to such an effort? > > Things I care about: > > * artifacts in the central maven repo > * per-framework adapters packaged separate from the core functionality > so dependencies are sane > > I don't have a strong preference for how that happens. Before having a > build system throwdown - do those sound like the goals we're > interested in working towards? > > --travis > > > > On Fri, Jun 22, 2012 at 5:10 PM, Travis Crawford > > <[email protected]> wrote: > >> On Fri, Jun 22, 2012 at 11:48 AM, Rohini Palaniswamy < > >> [email protected]> wrote: > >> > >>> Replying to the dev list instead of the jira as this requires > everybody's > >>> attention/thoughts. > >>> > >>> I would prefer all at once for mavenization. All the ivy stuff in ant > >>> will be throwaway in maven. The directory structure would require good > >>> amount of reorganizing (properly modularizing, moving code to standard > >>> directory structure src/main/java, src/test/java, src/test/resources). > >>> Trying to do it incrementally, moving out code and creating > sub-projects is > >>> ok, but the effort spent on changing ant scripts for that might be a > waste > >>> of effort as it is going to be thrown away. It would be better to have > the > >>> build in broken state for few days and complete mavenization faster > instead > >>> of trying to keep ant working when we move to maven. Since hcatalog is > not > >>> that complex, mavenizing should be easy and quick except for the e2e > tests. > >>> Even that can be quickly done with maven ant plugin before completely > >>> moving to maven. > >>> > >> > >> I'm down with this plan. Maybe have a hack day and crank this out? > Anyone > >> around SF who wants to work on this? I could host at the new Twitter > diggs. > >> > >> If there's interest I can see how hosting people works and we can figure > >> out a date. > >> > >> --travis > >> > >> > >> > >> > >>> > >>> Regards, > >>> Rohini > >>> > >>> > >>> > >>> On Fri, Jun 22, 2012 at 11:14 AM, Travis Crawford < > >>> [email protected]> wrote: > >>> > >>>> This is an automatically generated e-mail. To reply, visit: > >>>> https://reviews.apache.org/r/5496/ > >>>> > >>>> On June 22nd, 2012, 5:21 p.m., *Rohini Palaniswamy* wrote: > >>>> > >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build.xml< > https://reviews.apache.org/r/5496/diff/2/?file=115880#file115880line329> > (Diff > >>>> revision 2) > >>>> > >>>> 329 > >>>> > >>>> <antcall target="ivy-publish"/> > >>>> > >>>> Do we want to do ivy-publish inside the jar target. Would prefer it > to be called separately in only apache hudson build. ivy-publish can > probably depend on jar. All of us run jar in our hosts. Putting ivy-publish > here might lead to issues. > >>>> > >>>> On June 22nd, 2012, 5:27 p.m., *Travis Crawford* wrote: > >>>> > >>>> So this publishes to the local ivy cache, which is needed because > hcatalog-pig-adapter depends on hcatalog.jar. Otherwise the subproject > dependency does not work. > >>>> > >>>> On June 22nd, 2012, 6 p.m., *Rohini Palaniswamy* wrote: > >>>> > >>>> Good then. Thought that it publishes a snapshot jar to the maven > repo. Should have paid more attention to the "local" keyword there. > >>>> > >>>> What are your thoughts on this general approach of reorganizing the > repo to match the maven layout, then we can mavenize? Are you okay with > this incremental approach, or do you think it would be better all at once? > >>>> > >>>> Looking at the current build, I think we also want subprojects for: > >>>> > >>>> * hcatalog-server-extensions.jar > >>>> * hcatalog-core.jar (today just hcatalog.jar; we could keep this just > hcatalog.jar if people like it) > >>>> > >>>> I don't fully understand the storage-handler stuff, since doesn't > that stuff belong in Hive? For now I was planning to leave as-is. > >>>> > >>>> Let me know if you want to do this all at once, or incremental. > >>>> > >>>> > >>>> - Travis > >>>> > >>>> On June 22nd, 2012, 3:45 p.m., Travis Crawford wrote: > >>>> Review request for hcatalog. > >>>> By Travis Crawford. > >>>> > >>>> *Updated June 22, 2012, 3:45 p.m.* > >>>> Description > >>>> > >>>> Update HCatalog build to package pig classes as a separate jar. I did > not update ivy yet, but if the general approach looks good I will update. > This will let the core hcatalog.jar depends only on stuff needed by all > processing frameworks; then people that want to use pig can use the pig > adapter which has the pig dependency. > >>>> > >>>> I believe we'll have more adapters in the future, so I'm trying to > make this reusable. > >>>> > >>>> For example: > >>>> > >>>> Traviss-iMac:hcatalog travis$ jar -tvf > hcatalog-pig-adapter/build/hcatalog-pig-adapter-0.5.0-dev.jar > >>>> 0 Thu Jun 21 10:34:18 PDT 2012 META-INF/ > >>>> 107 Thu Jun 21 10:34:16 PDT 2012 META-INF/MANIFEST.MF > >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/ > >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/ > >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/ > >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/pig/ > >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/pig/drivers/ > >>>> 4352 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/HCatBaseLoader.class > >>>> 1261 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/HCatBaseStorer$1.class > >>>> 12413 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/HCatBaseStorer.class > >>>> 632 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/HCatLoader$1.class > >>>> 8518 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/HCatLoader.class > >>>> 6801 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/HCatStorer.class > >>>> 1019 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/PigHCatUtil$1.class > >>>> 13117 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/PigHCatUtil.class > >>>> 3711 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat$LoadFuncBasedRecordReader.class > >>>> 2383 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat.class > >>>> 2189 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat$StoreFuncBasedOutputCommitter.class > >>>> 1775 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat$StoreFuncBasedRecordWriter.class > >>>> 2647 Thu Jun 21 10:34:16 PDT 2012 > org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat.class > >>>> Traviss-iMac:hcatalog travis$ > >>>> > >>>> Diffs > >>>> > >>>> - > >>>> > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build-common-new.xml > >>>> (PRE-CREATION) > >>>> - > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build.xml > >>>> (1352540) > >>>> - > >>>> > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/hcatalog-pig-adapter/build.xml > >>>> (PRE-CREATION) > >>>> - > >>>> > http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/hcatalog-pig-adapter/ivy.xml > >>>> (PRE-CREATION) > >>>> - http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/ivy.xml > >>>> (1352540) > >>>> > >>>> View Diff <https://reviews.apache.org/r/5496/diff/> > >>>> > >>> > >>> >
