On Fri, Jun 22, 2012 at 5:28 PM, Jakob Homan <[email protected]> wrote: > I've been wanting to take a crack at this with Gradle, which we've > used extensively with great results. I was hoping to show up with > something in hand rather than suggest something new, igniting a whole > big war. Any resistance to such an effort?
Things I care about: * artifacts in the central maven repo * per-framework adapters packaged separate from the core functionality so dependencies are sane I don't have a strong preference for how that happens. Before having a build system throwdown - do those sound like the goals we're interested in working towards? --travis > On Fri, Jun 22, 2012 at 5:10 PM, Travis Crawford > <[email protected]> wrote: >> On Fri, Jun 22, 2012 at 11:48 AM, Rohini Palaniswamy < >> [email protected]> wrote: >> >>> Replying to the dev list instead of the jira as this requires everybody's >>> attention/thoughts. >>> >>> I would prefer all at once for mavenization. All the ivy stuff in ant >>> will be throwaway in maven. The directory structure would require good >>> amount of reorganizing (properly modularizing, moving code to standard >>> directory structure src/main/java, src/test/java, src/test/resources). >>> Trying to do it incrementally, moving out code and creating sub-projects is >>> ok, but the effort spent on changing ant scripts for that might be a waste >>> of effort as it is going to be thrown away. It would be better to have the >>> build in broken state for few days and complete mavenization faster instead >>> of trying to keep ant working when we move to maven. Since hcatalog is not >>> that complex, mavenizing should be easy and quick except for the e2e tests. >>> Even that can be quickly done with maven ant plugin before completely >>> moving to maven. >>> >> >> I'm down with this plan. Maybe have a hack day and crank this out? Anyone >> around SF who wants to work on this? I could host at the new Twitter diggs. >> >> If there's interest I can see how hosting people works and we can figure >> out a date. >> >> --travis >> >> >> >> >>> >>> Regards, >>> Rohini >>> >>> >>> >>> On Fri, Jun 22, 2012 at 11:14 AM, Travis Crawford < >>> [email protected]> wrote: >>> >>>> This is an automatically generated e-mail. To reply, visit: >>>> https://reviews.apache.org/r/5496/ >>>> >>>> On June 22nd, 2012, 5:21 p.m., *Rohini Palaniswamy* wrote: >>>> >>>> >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build.xml<https://reviews.apache.org/r/5496/diff/2/?file=115880#file115880line329> >>>> (Diff >>>> revision 2) >>>> >>>> 329 >>>> >>>> <antcall target="ivy-publish"/> >>>> >>>> Do we want to do ivy-publish inside the jar target. Would prefer it to >>>> be called separately in only apache hudson build. ivy-publish can probably >>>> depend on jar. All of us run jar in our hosts. Putting ivy-publish here >>>> might lead to issues. >>>> >>>> On June 22nd, 2012, 5:27 p.m., *Travis Crawford* wrote: >>>> >>>> So this publishes to the local ivy cache, which is needed because >>>> hcatalog-pig-adapter depends on hcatalog.jar. Otherwise the subproject >>>> dependency does not work. >>>> >>>> On June 22nd, 2012, 6 p.m., *Rohini Palaniswamy* wrote: >>>> >>>> Good then. Thought that it publishes a snapshot jar to the maven repo. >>>> Should have paid more attention to the "local" keyword there. >>>> >>>> What are your thoughts on this general approach of reorganizing the repo >>>> to match the maven layout, then we can mavenize? Are you okay with this >>>> incremental approach, or do you think it would be better all at once? >>>> >>>> Looking at the current build, I think we also want subprojects for: >>>> >>>> * hcatalog-server-extensions.jar >>>> * hcatalog-core.jar (today just hcatalog.jar; we could keep this just >>>> hcatalog.jar if people like it) >>>> >>>> I don't fully understand the storage-handler stuff, since doesn't that >>>> stuff belong in Hive? For now I was planning to leave as-is. >>>> >>>> Let me know if you want to do this all at once, or incremental. >>>> >>>> >>>> - Travis >>>> >>>> On June 22nd, 2012, 3:45 p.m., Travis Crawford wrote: >>>> Review request for hcatalog. >>>> By Travis Crawford. >>>> >>>> *Updated June 22, 2012, 3:45 p.m.* >>>> Description >>>> >>>> Update HCatalog build to package pig classes as a separate jar. I did not >>>> update ivy yet, but if the general approach looks good I will update. This >>>> will let the core hcatalog.jar depends only on stuff needed by all >>>> processing frameworks; then people that want to use pig can use the pig >>>> adapter which has the pig dependency. >>>> >>>> I believe we'll have more adapters in the future, so I'm trying to make >>>> this reusable. >>>> >>>> For example: >>>> >>>> Traviss-iMac:hcatalog travis$ jar -tvf >>>> hcatalog-pig-adapter/build/hcatalog-pig-adapter-0.5.0-dev.jar >>>> 0 Thu Jun 21 10:34:18 PDT 2012 META-INF/ >>>> 107 Thu Jun 21 10:34:16 PDT 2012 META-INF/MANIFEST.MF >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/ >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/ >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/ >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/pig/ >>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/pig/drivers/ >>>> 4352 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatBaseLoader.class >>>> 1261 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatBaseStorer$1.class >>>> 12413 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatBaseStorer.class >>>> 632 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatLoader$1.class >>>> 8518 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatLoader.class >>>> 6801 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatStorer.class >>>> 1019 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/PigHCatUtil$1.class >>>> 13117 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/PigHCatUtil.class >>>> 3711 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat$LoadFuncBasedRecordReader.class >>>> 2383 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat.class >>>> 2189 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat$StoreFuncBasedOutputCommitter.class >>>> 1775 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat$StoreFuncBasedRecordWriter.class >>>> 2647 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat.class >>>> Traviss-iMac:hcatalog travis$ >>>> >>>> Diffs >>>> >>>> - >>>> >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build-common-new.xml >>>> (PRE-CREATION) >>>> - http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build.xml >>>> (1352540) >>>> - >>>> >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/hcatalog-pig-adapter/build.xml >>>> (PRE-CREATION) >>>> - >>>> >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/hcatalog-pig-adapter/ivy.xml >>>> (PRE-CREATION) >>>> - http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/ivy.xml >>>> (1352540) >>>> >>>> View Diff <https://reviews.apache.org/r/5496/diff/> >>>> >>> >>>
