Hi all, I have made changes for 'Option 2' so people can take a look, its available on this branch: https://svn.apache.org/repos/asf/qpid/branches/QPID-5610-maven3-build-qmf-tools
The commits are all logged on https://issues.apache.org/jira/browse/QPID-5610 For the manual tests I elected to make another module with those as the main code. Rather than update the test shell scripts to do the equivalent (i.e point at the target dir) to what they were doing or add another assembly to package them all up so they could work anywhere, I took a different approach and added a helper profile to the module pom which can run a particular class from the test package on demand whether it was previously build or not. While I didn't like the idea of pointing the test scipts at the target dir, I did this as much to try it out as anything because I think we should be doing something similar with any client examples etc in the main tree, so we can always put the scripts back if required. See the README in qpid-qmf2-test for details. Version wise it is marked as 0.30-SNAPSHOT to reflect its 'coming in future' status, but I have made it use the 0.28-SNAPSHOT/trunk dependencies for now so it works with the current codebase. I have updated the README files somewhat to reflect the update, add details to how to build javadoc etc, but to summarise in case you want to play with it: Ensure you have the latest qpid-parent pom installed in your local repo (a temporary step until we start deploying it soon, so I didn't document this bit): svn co http://svn.apache.org/repos/asf/qpid/qpid-parent-pom/trunk <your chosen dir name> cd <your chosen dir name> mvn clean install Optionally, build the Java broker/client/etc and install them in your local repo. If you don't, the QMF2 tools build will just use the artefacts from the snapshots repo: cd <qpid trunk checkout>/qpid/java/ mvn clean install -DskipTests=true -Denable=true Build the QMF2 broker plugin and tools: svn co http://svn.apache.org/repos/asf/qpid/branches/QPID-5610-maven3-build-qmf-tools/<your chosen dir name> cd <your chosen dir name>/qpid/tools/src/java mvn clean package You will now have the jars, the tools assembly which you can extract and use the tools/gui, and a broker plugin assembly which you can extract and install in a broker release (that assembly was optionally created earlier in qpid/java/broker/target) as per the instructions. Robbie. On 13 March 2014 21:36, Robbie Gemmell <[email protected]> wrote: > > On 13 March 2014 18:31, Fraser Adams <[email protected]>wrote: > >> Thanks for your thoughts Andrew. >> >> Just to be clear I wasn't intending to come across as negative about this >> in my previous response. TBH build systems just don't "float my boat" and I >> find it hard to get excited about them, so I'm only moaning about having to >> figure out another blinking one (having had to figure out CMake on the C++ >> side too recently). >> >> If I were to put my "objective" hat on I would have to say that we should >> strive to make things as uniform as possible and so if that means messing >> with the structure so that it follows the same patterns as the Java Broker >> etc. then so be it, even if I personally dislike the layout of that. >> >> >> I've got a degree of sympathy with the argument "the use of a standard, >> convention driven build system will allow us to concentrate more on Qpid >> itself " but I have concerns about things that get done "magically" I quite >> like knowing what's going on with my system, but then I'm an old Luddite :-) >> >> A particular concern (which is I guess primarily caused by me knowing SFA >> about Maven) relates to dependencies. Now I know that Maven automagically >> handles dependencies and it seems to do things like magically pull things >> down from Maven Central, but say I always build off trunk and say the Java >> Broker keeps changing - and breaking the plugin :-) what I currently do is >> build the Java Broker and client and the main Jars are on my CLASSPATH I >> then build the QMF stuff - which sees the just built Broker. I'm not at all >> clear how one makes things behave in essentially the same way. > > > As I alluded to in my earlier mail, I think this will actually be simpler > than it is currently for the QMF bits (even more so for people who have > used maven before, who should just be able to build it without hesitation) > > Maven pulls things down from repositories (central, pus any others > configured explicitly in settings or poms) and installs them in the 'local > repo', which defaults to ~/.m2/repository. When doing new builds the local > repo is checked to see if it already has the version of the dependency. > Maven places a timestamp against each repo checked, after a configurable > time has elapsed it will check again for updates (can also be requested on > demand) rather than relying on the local file. When doing builds you have > the option to install the output in your local repo to make them available > to other subsequent builds, rather than relying on grabbing the dependency > from a remote repo. This effectively gives you two choices - let it > fetch and build against the latest [snapshot] releases published to remote > repositories (e.g. we publish a nightly snapshot currently) or just roll > your own and build against those. > > The equivalent of what you have done previously will thus become something > along the lines of: > cd qpid/java/ > mvn clean install [-DskipTests=true] > cd ../tools/qmf/src/java > mvn clean <lifecycle stage of your choice> > > With the first execution for the main java build being optional if you > don't need to have any changes since the available local/remote bits > were last published. > > If you were to try this now, youd find you need to add -Denable=true to > the command in the main tree or it will fail immediately, as I wanted > people to realise its still a work in progress. You will also currently > first need to install the qpid-parent pom (see the dir at the root of the > qpid svn repo) because we have yet to publish it while getting the initial > version in shape. Neither of these steps will be necessary in future. > > >> This wasn't helped by me looking at the AMQP 1.0 JMS client and seeing >> nowt but a solitary pom.xml and no instructions whatsoever about how on >> earth I'd even *begin* to build it, > > > There aren't any instructions for a couple of reasons - it isn't in a > particularly useful state for people to use yet and so it isn't high on my > list, and there are actually no noteworthy qpid specific things to know > about the jms client build so any basic maven instructions should suffice. > I'll consider adding somehing when it is in a state people would be > expected to pick up, but they are likely to be very minimal with perhaps > some links to the maven site for more info. I don't expect anyone who has > previously used maven to need any instructions whatsoever. > > >> I *think* it might be "mvn package" to build the jars locally, but I've >> really no clue > > > mvn package would indeed create a jar in this case (more generally, that > produces a file of whatever packaging type a particular module is declared > as using within its pom, with jar being the default packaging if not > specified) > > >> - and no great desire to learn more than I have to about Maven (see my >> earlier point about build systems not exactly floating my boat) >> > > There really isn't much you need to know, between your mail and this reply > we have already covered a large chunk of the stuff youd need to know. > > >> So the bottom line is that despite my moaning I'm actually fairly easy on >> this and will go with consistency over personal taste. >> >> HTH, >> Frase >> >> >> >> On 13/03/14 14:22, Andrew MacBean wrote: >> >>> Robbie/Fraser, >>> >>> I have been contributing to QPID-5048 for a while now, so am obviously >>> bias, but would agree with the sentiment that it would be beneficial and >>> makes most sense long term to also have the QMF stuff "mavenised". I >>> also >>> think we should "harmonise" the maven build of the 1.0 stuff with the >>> rest >>> as soon as it makes sense (common versions, plugins and overall parent >>> etc). >>> >>> I agree with Robbie wrt to the Ant build being a pain in the ass but can >>> also understand Frasers initial concerns. Maven is not perfect by any >>> means >>> and is prone to bloat, but the use of a standard, convention driven build >>> system will allow us to concentrate more on Qpid itself. It will also >>> easily allow us to split up the tree and move to a component based >>> release >>> mechanism in the near future, which I also think would be a good idea. >>> One >>> good thing about Maven is that it is very easy to pick up... a bit like a >>> virus Fraser ;) >>> >>> Based on the options you suggest Robbie, I would vote for option 2 and if >>> that is successful, further work to move to option 3 could be picked up >>> later if desired. >>> >>> Andrew >>> >>> >>> On 11 March 2014 20:53, Robbie Gemmell <[email protected]> wrote: >>> >>> On 11 March 2014 19:10, Fraser Adams <[email protected]> >>>> wrote: >>>> >>>> On 10/03/14 22:56, Robbie Gemmell wrote: >>>>> >>>>> Hi all, but especially Fraser. >>>>>> >>>>>> Now that the Maven build work progressing in the main Java tree under >>>>>> QPID-5048 is sufficiently progressed, it is time we look at providing >>>>>> a >>>>>> Maven build for the QMF2 tools so that they can integrate with the >>>>>> other >>>>>> components, just be more easily built generally, and be published to >>>>>> >>>>> maven >>>> >>>>> central for people to pick up in their builds as has been requested. I >>>>>> have >>>>>> raised https://issues.apache.org/jira/browse/QPID-5610 to track this. >>>>>> >>>>>> Sounds reasonable, though I'm slightly concerned 'cause I've got no >>>>> experience of Maven >>>>> >>>> >>>> Since I've work mainly on the Qpid Java bits, other than to complete the >>>> pieces that generate the poms in the Ant build I'd barely ever used >>>> Maven >>>> until the last year when other bits have popped up using it. Its >>>> certainly >>>> not difficult to pick up, and for the many people who do use it >>>> elsewhere >>>> it should make things far easier than picking up our Ant build is :) >>>> >>>> >>>> and TBH I've been generally avoiding it 'cause it seems like bloatware >>>>> to >>>>> me, but it does appear to have taken hold like a virus so I guess >>>>> that's >>>>> "progress" for you. >>>>> >>>>> It's all fair enough though and as you point out people have begun to >>>>> request this stuff in Maven central. >>>>> >>>>> >>>>> >>>>> As per the JIRA itself, the current layout of the qmf2 tools in the >>>>>> repo >>>>>> doesn't lend itself to a Maven build, so we will need to look at >>>>>> reorganising the content as part of the process >>>>>> >>>>>> I've go no real issues with code reorganisation, but I wouldn't mind >>>>> knowing what you mean. I'm guessing it's because Maven expects a >>>>> specific >>>>> layout so it can automagically do stuff, but as I say I know pretty >>>>> much >>>>> zero about Maven. >>>>> >>>>> >>>>> I simply meant the layout proposals described further down the mail, >>>> moving >>>> bits so there is a clearer heirarchy for each of the individual >>>> components, >>>> in part to facilate the automagic you refer to...though even the Ant >>>> build >>>> in the main Java tree couldnt cope with the existing semi nested layout. >>>> >>>> >>>> (likely at the expense of >>>>> >>>>>> the Ant build going away, it would need rewritten). >>>>>> >>>>>> I'm kind of worried about that, I use ant all the time, but moreover >>>>> I've >>>>> only got Maven2 - I'm currently running a relatively old Ubuntu and I >>>>> >>>> can't >>>> >>>>> see Maven3 in the repo. I know that I should look to upgrade, but it's >>>>> blinking disruptive and I can never find the time. If there's an easy >>>>> Maven3 install I might be less nervous but not being able to build it >>>>> >>>> would >>>> >>>>> upset me a bit. >>>>> >>>>> I still use ant to build the main Java stuff, >>>>> >>>> >>>> I dont use Ubuntu currently (but I did use Maven 3 on an old Ubuntu >>>> 8.04 VM >>>> not that long ago hehe) so I dont know if it is packaged or not, but >>>> its a >>>> simple java app with a startup script just like Ant, I dont personally >>>> tend >>>> to install either one of them using the distro package manager. The core >>>> maven install is a 5-6MB download from http://maven.apache.orgdepending >>>> on >>>> version. I'm mainly using 3.0.5 to make sure stuff works on the old >>>> releases, whereas 3.2.1 is the current version. Maven 2 has been >>>> declared >>>> EOL by the Maven project for what its worth, there hasnt been a release >>>> of >>>> it in over 4 years, with Maven 3 now being more than 3 years old. >>>> >>>> One of the reasons that we are working on a Maven build (beyond just >>>> being >>>> able to integrate easily with all the stuff that expects Maven these >>>> days) >>>> is so that we can later split the Java tree up and move to more a >>>> component-related heirarchy for the project as discussed on the list >>>> previously. We have done the work in-situ so far in order to avoid >>>> disruption and minimise the modifications required to the Ant build >>>> because >>>> touching it is generally a time hole (I say that as the person who has >>>> easily spent by far the most time tweaking it in recent years; its a >>>> complete pain in the ass), but once we do start breaking things up it >>>> will >>>> need to go. In the case of the QMF stuff, layout changes are required up >>>> front to facilitate the initial work and these changes would require a >>>> complete rewrite of the Ant build, doing which doesnt seem an ideal use >>>> of >>>> time to me given the end goal and that the Maven build should be a lot >>>> nicer to use in this case (for example, not requiring setting system >>>> properties and adding things to the classpath in order to build the QMF >>>> components as currently required). >>>> >>>> though it looks like the new AMQP 1.0 JMS client is Maven only? (TBH >>>> that's >>>> >>>>> what has put me of trying it out) >>>>> >>>>> >>>>> It is Maven only (but has no dependencies on the main tree), though >>>> that >>>> wouldnt really be what prevents you trying it out hehe, there isnt >>>> really >>>> enough there to support proper application usage yet as we have front >>>> loaded the work on the messages specifically (rather than the covering >>>> client as a whole) in tandem with the work on the JMS Mapping at OASIS >>>> in >>>> the AMQP Bindings & Mappings TC, as thats basically the most complex >>>> part >>>> of the JMS<->AMQP mapping when considering things comprehensively. >>>> >>>> >>>> >>>> >>>>> We will want to end up with a few (or more) modules such that we are >>>>>> >>>>> able >>>> >>>>> to produce the main jar, rest api jar, broker plugin jar, and >>>>>> >>>>> additionally >>>> >>>>> archives containing all the bits people would need to use either the >>>>>> broker >>>>>> plugin or the tools+GUI. >>>>>> >>>>>> Seems fair enough. >>>>> >>>>> >>>>> I have a few suggestions below as to the new >>>>> >>>>>> modules we would created in the tools/src/java directory and what the >>>>>> contents of them would be based on the current layout, with Option 1 >>>>>> >>>>> being >>>> >>>>> the closest to what would be produced now and Option 3 splitting it up >>>>>> >>>>> the >>>> >>>>> most. >>>>>> >>>>>> Thoughts? >>>>>> >>>>>> Robbie >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Option 1: >>>>>> ======= >>>>>> >>>>>> qpid-qmf2: >>>>>> -src/main/java >>>>>> #contains most of (see further down) or all the code currently in >>>>>> src/main/java >>>>>> >>>>>> qpid-qmf2-rest: >>>>>> -src/main/java >>>>>> #contains all the code currently in src/restapi/java >>>>>> >>>>>> qpid-qmf2-tools: >>>>>> -bin >>>>>> #contains everything currently in bin >>>>>> -src/main/assembly >>>>>> #contains an assembly descriptor for a release binary which would >>>>>> >>>>> result >>>> >>>>> in a tar containing the bin dir with all the scripts etc, any docs >>>>>> >>>>> wanted, >>>> >>>>> and a lib dir with the qpid-qmf2 and qpid-qmf2-rest modules and their >>>>>> dependencies (e.g the client). >>>>>> >>>>>> qpid-broker-plugins-management-qmf2: >>>>>> -src/main/java >>>>>> #contains all the code currently in >>>>>> src/qpid-broker-plugins-management-qmf2/java >>>>>> -src/main/assembly >>>>>> #contains an assembly descriptor for a release binary which would >>>>>> >>>>> result >>>> >>>>> in a tar containing the qpid-qmf2 and >>>>>> >>>>> qpid-broker-plugins-management-qmf2 >>>> >>>>> modules and their dependencies (e.g the client). >>>>>> >>>>>> Plus appropriate src/main/resources folders to handle additional >>>>>> >>>>> non-java >>>> >>>>> files as necessary. >>>>>> >>>>>> Something would also need done with the current 'tests' folder, though >>>>>> >>>>> as >>>> >>>>> they don't appear to be automated JUnit tests I haven't yet decided on >>>>>> a >>>>>> final suggestion there. >>>>>> >>>>>> >>>>>> Option 2: >>>>>> ======= >>>>>> >>>>>> As above, but modified such that the qpid-qmf2-tools module isn't just >>>>>> >>>>> an >>>> >>>>> assembly of the others, but actually contains the >>>>>> org/apache/qpid/qmf2/tools package rather than it remaining in the >>>>>> qpid-qmf2 module. >>>>>> The assembly produced by the qpid-qmf2-tools module would be updated >>>>>> to >>>>>> also include the resulting new qpid-qmf2-tools jar as well. >>>>>> >>>>>> >>>>>> Option 3: >>>>>> ======= >>>>>> >>>>>> Taking Option 2 further still, split qpid-qmf2 up in line with its >>>>>> >>>>> current >>>> >>>>> main package composition, e.g. creating the below module jars and >>>>>> incorporating them into the broker plugin and tools assemblies as >>>>>> appropriate. >>>>>> >>>>>> qpid-qmf2-agent >>>>>> qpid-qmf2-common >>>>>> qpid-qmf2-console >>>>>> qpid-qmf2-tools >>>>>> >>>>>> I guess that I'd go with the consensus on this. >>>>>> >>>>> >>>> I have my own preference of course, but I'd like to see some other >>>> thoughts >>>> before saying :) >>>> >>>> >>>> Option 3 has a certain appeal though it brings out bits I hate about >>>>> >>>> Java. >>>> >>>>> Having lots of separate source trees is a right royal pain in the >>>>> >>>> backside >>>> >>>>> when trying to find things. >>>>> >>>>> There's a lot of that in the main Java code and navigating around >>>>> broker-plugins/management-amqp/src/main/java/org/apache/ >>>>> qpid/server/management/amqp >>>>> broker-core/src/main/java/org/apache/qpid/server/model >>>>> >>>>> etc. >>>>> just gets tedious when all the "main/java/org/apache/qpid" is the same. >>>>> >>>>> >>>>> I can't say I really share the same frustration. We have a lot more >>>> directories in the main tree, and I find that that often makes it more >>>> obvious where to find things (so long as they are named appropriately). >>>> I >>>> actually had a lot more trouble than expected figuring out what/where >>>> the >>>> various parts of the QMF stuff are in order to consider how to package >>>> it >>>> and write my earlier email hehe :D >>>> >>>> Either way, editors usually [can be made to] hide or handle much of this >>>> for you. What are you using? >>>> >>>> >>>> But whatever's most useful I'll go with the flow on, I just want this >>>>> stuff to be useful and used. >>>>> >>>>> Frase >>>>> >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >
