On 6 April 2014 10:29, Fraser Adams <[email protected]> wrote:
> Hi Robbie, > Firstly thanks so much for taking the time to give such a comprehensive > response. I'll summarise my current thoughts below. > > 1. The directory structure in qpid/tools/src/java seems fine, I've got no > strong aversion to it. There might be some tweakage, and possibly as you've > suggested before there might be mileage in separating into console, agent, > common+util as most applications using it would only tend to need the > console, common and JMS stuff, but it's probably not worth doing too much > there at the moment 'cause I've no idea what carnage I'll cause when I > start to (eventually) get round to doing AMQP 1.0 Management - quite how I > handle any migration is very much TBD :-/ > > 2. I can see how Maven can (sort of) simplify build scripts, though that > only seems to apply to large projects like the Java Broker, for the QMF > stuff, which is ultimately just a JMS client it seems to have gone from one > fairly simple build.xml which was pretty straightforward to something like > six pom.xml files - there's progress for you :-D > > You only have to deal with the top level pom.xml generally when building stuff, sure there may be more files 'behind' that but you basically dont need to look at them unless you want to change them, at which point I think it actually becomes easier to find what you want to change because things are nicely split up into modules that deal with themselves, and a parent that everything inherits. We could have left the non-broker-plugin stuff as one big aggregate jar, but the reality is that the existing 'qmf.jar' contains a lot more than a single thing and so splitting it up to at least some extent seemed like the correct thing to do. The python/original QMF implementation and command line tools built around it are separated in a similar way, presumably for similar reasons. > 3. In the qpid-qmf2-parent pom I don't really understand the dependencies, > that pom seems to be responsible for the top-level building of the modules, > but why does it have dependencies of its own (log4j, slf4j, jms spec)? I > would get it if this had the common dependencies of all the sub-modules, > but the qpid-qmf2 pom has its own dependencies on slf4j and jms spec - > exposing more holes in my lack of understanding of Maven I guess. > First up, as with qpid-parent yesterday, qpid-qmf2-parent actually has no dependencies, but it does have a 'dependency management' section, which is used to ensure consistent alignment of dependency versions in the child modules and make their poms slightly simpler. See http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management Next up, inheritance. The parent pom uses 'pom' packaging and doesnt produce any output except itself, while the individual modules then use it as their parent and then inherit much of its contents - such as the dependencyManagement section that in turn ensures they all use the same versions for those things going forward. I could have left out the dependencyManagement section and just repeated the SLF4J and JMS spec dependency etries in their entirity in every module requiring them, but that becomes harder to maintain going forward and is more prone to becoming inconsistent. Some of this information is overriding bits from qpid-parent pom, which currently contains a much larger amount of dependency management, but we are considering moving some of that back to its immediate children as it doesnt all apply universally to them. I also now realise that a (highly unusual going forward) change I made yesterday to qpid-parent might not sit nice with the dependencyManagement section in qpid-qmf2-parent so I'll go fix that if required - which yes it was, update commmited :) > > 4. The main benefit that I can see of Maven, aside from (sort of) > simplifying build scripts (which only seems to apply for large builds) is > that it does seem to provide useful artefacts from the perspective of > *users*. That is to say it's biggest (or only as far as I can see ;->) > benefit is from the perspective of release packaging - you end up with some > nice tar.gz files that contain what's needed, so I sort of get why it's > nice from a user perspective - maybe that's worth the price of admission > alone. > We always had the ability to make nice tar.gz files, its just much more of a pain in the ass to do so in our Ant build, often requiring significant hoop jumping and typically coming out during the release process that they were wrong because we dont actually use them the rest of the time (due to the convenient but horrible global build dir that doesnt represent what users actually get). For me the main benefits are the consistency (this obviously doesnt apply to you yet, but the many people who know maven from use on other projects will now largely be able to pick up the Qpid build and use most of it without assistance, while the same wasn't really true of the Ant build because so much of it is bespoke) and extending from that the ease of integration with a lot of tooling that for the most part works simply with or is actually built into maven builds but usually requires pulling teeth (often mine) to get working with highly bespoke Ant builds such as ours. It will also mean we (i.e me) dont need to make a special effort to publish maven artifacts for components to maven central after our users request it (which they wont need to because we just will), as we (i.e me) always have in the past, or waste considerable amounts of [my] time verifying those generated artifacts actually contain what they should every time we release (which they often dont - I have yet to check them this time round) because we dont actually use them the rest of the time. > 5. From a *dev* perspective though, at the moment the bit of the Maven > rework that a dislike the most is more or less exactly the same as point 4. > above. In order to actually use the Java Broker I have to do mvn clean > install, You just need to run mvn package currently. You only need to use install if you want to build something entirely separate using maven which declares it depends on the artifacts for the broker you just built. Strictly speaking you only need actually do that if you want to modify the broker first, as it will otherwise download the release or snapshot version (based on your specified dependency) from the remote repositories if you dont have it in your local repo already. > I then need to take the tar.gz from <qpid>/java/broke/target copy it > somewhere and unzip it - inconvenient! As mentioned yesterday, its simply not finished. Removing this need is a 'nice to have' thing in my mind, which has ranked it below some other 'needs to be done before this is correct / suitable for use at all' things. I have basically viewed this personally along the lines of....until we do put something into the poms to start or unpack the broker, we actually have to use/do something a user e.g. looking at the website would do? Oh the horror ;) > But that's not the worse bit, when I need to figure out what's broken next > time the plugin API changes there's a whole world of copying and unzipping > to be had, which as well as being inconvenient is prone to finger trouble. Build and install the broker artifacts (or dont and it will just use the latest snapshot). Build the QMF bits, which will fail if we broke something. Dont touch the tar file, or any jar files, or set the classpath. This stage is far easier than it was before with the Ant build for the QMF bits. When it comes to running it all up to make sure it works, yes you will need to copy some things *currently*. That could certainly be done via config in the poms if needed to 'automate' it. Its probably worth noting that having some/any real automated tests for these bits would mean that this step wouldnt even be necessary much of the time. A key thing I think is being overlooked is that we will simply try harder not to break it in future. Because it wont be a complete pain to build as it is now, we will far more easily be able to ensure it is kept up to date. We would also now be able to very easily stick it in CI to make sure that we know we broke it if we hadnt noticed in advance (which I expect we will more, because it wont be an ordeal to do so anymore). > It's unfortunate that Maven install only "installs" things to the > repository and that there doesn't seem to be a way to install assembly > artefacts onto my system. I guess I could write my own private script that > does all that - but it is as I say inconvenient from a dev perspective. > Thats the primary purpose of mvn install. We can however bind anything we like to any of the build phases, perhaps adding a profile that unpacks to a specified directory as a 'system installation' type command etc etc. As I said yesterday, there are a multitude of options here, it just isnt something we have got to *yet* because it really is only a convenience whereas other things are not. > > 6. Hmmm so I say item 5 was what I dislike the most, actually one thing I > dislike just a bit more is what it does for WebApp development. What I > really like about programming in JavaScript is that I can just make a > change, hit refresh and bingo, none of this build/compile malarkey. Now > from a user perspective it's great that the QMF WebApp is neatly packaged > in qpid-qmf2-tools.tar.gz but from a dev perspective it's a car crash > waiting to happen :-( In order to run QpidRestAPI I have to unpack the > tools, but that will then use the packaged WebApp by default, now there is > a command line option so I can point webroot to > <qpid>/tools/src/java/qpid-qmf2-tools/bin/qpid-web > on trunk, so it's not like I can't work around it, but how long do you give > me before I start accidentally making changes in the "installed" version - > then accidentally lose them all next time I update the QpidRestAPI. I might > be lucky, but I have a very simple brain :-) > You could probably make a profile to it run the GUI straight out of the src tree without ever building the package. Thats basically how the 'test' classes work for example since it doesnt even need to build the jar files to run them. > > 7. I'm a bit worried about the Maven version stuff. I can see it being a > good thing wrt. being able to do reproducible builds, but does it really > mean that every time a new Qpid release comes out there will be a need to > try and remember to go through dozens of pom.xml files to update various > artefact versions? Given that there are six poms for the QMF stuff alone > that doesn't feel brilliant. > There are plugins that can help with this. > > 8. It feels like I'm pushed into Mavenising every other little Qpid test > app I've got on my system. Before I had things (albeit perhaps messily) on > my CLASSPATH, but now not. As you say, this new way is cleaner, but you cancertainly do basically what you did before if you like, you will just have to specify more locations in your classpath. For production things that's fine and the Maven packaging does seem quite > good, but for all the other hacky little things .... sigh, I really do > think Maven is a virus, once one thing catches it you have to "update" > everything else. > You arent really forced in any way in my mind. Sure, it makes something you are doing slightly uglier, but you are entirely free to keep doing that hacky thing if you like. I feel a related consideration also needs to be applied in the opposite direction, i.e the folks (lots of them) that have long had maven builds and might come along and want to use the output of our Ant build. Poor sods. > > 9. All that said, there seems to be an inexorable trend towards Maven, so > from my perspective you might as well move forward on the great Qpid > Mavenisation. I'm not going to stand in the way even if I might moan a bit > until I get used to it :-D. Feel free to moan away :) It'd be nice though if there were some solutions to the productivity issues I realise that this transition might involve a temporary productivity dip for some who arent as used to it already, but also consider the productivity increase others will get from it (I'll be honest: whilst I think that does apply generally, I do primarily mean me, who wont have to spend so much time rewriting chunks the Ant build to do really simple things, or waste as much time verifying the output we rarely generate is actually correct, or <insert time sinks here>..) and that a key driver is allowing us to progress towards making the project as a whole more component focused, which will itself improve productivity and wouldnt be possible without rewriting what we had anyway. We will get to the little things like starting the broker one its built. There is obviously a learning curve, but its not that big, once you are over it I actually think you would actually come to see increased productivity from using it generally instead of writing Ant scripts (or dealing directly with javac) and tinkering with the CLASSPATH for example. > - I already struggle with productivity given limited coding time between > work and family stuff, some weekends I feel like I've wasted it all chasing > my tail rather than doing anything useful. > > Regards, > Frase > > > > > On 05/04/14 18:57, Robbie Gemmell wrote: > >> On 4 April 2014 14:41, Fraser Adams <[email protected]> >> wrote: >> >> On 04/04/14 13:30, Rob Godfrey wrote: >>> >>> No... "install in the local repo" is what the mvn clean install will >>>> do... So, you literally just have to follow Robbie's instructions: check >>>> out the qpid-parent-pom/trunk directory, run mvn clean install from >>>> wherever you checked it out to... then go back to the broker and the >>>> maven >>>> build should complete ok. Hope this helps, Rob >>>> >>>> Cheers Rob, >>> I ended up figuring out that myself after staring at the instructions for >>> a while and deciding that I was reading way too much into things :-) >>> >>> I've now got the Java Broker etc. built using Maven but before I play >>> with >>> the QMF things I wouldn't mind answers to a few Maven questions - I'm >>> still >>> very much at the "Burn the Witch" stage wrt. my trust of Maven ;-) >>> >>> I've now ended up with a directory >>> /home/fadams/.m2/repository >>> >>> Filled with stuff. >>> >>> I'm not clear by what witchcraft Maven decides what to shove there - >>> >> >> Maven has the concept of a local repo and a remote repo, analogous to e.g. >> your installed packages in a linux distro, and the repository the package >> manager got them from. The default local repo localtion is >> ~/.m2/repository, and the default remote repo is maven central. >> >> Any dependencies it needs to download will go in the local repo, so it >> doenst necessarily need to download them again next time they are used >> (unles e.g they are snapshots and new versions came out). Additionally, >> anything you run "mvn install" on will go in the local repo configured at >> the time (can be overriden per-command for example) to allow it to be used >> by other things you build later. >> >> 'Any dependencies' above includes the dependencies of a particular >> artifact >> you are using, the [plugin] dependencies of anything needed by your build >> process to do what you are asking, and any [plugin] dependencies needed by >> maven itself. >> >> (Additional reading, there are different dependency scopes that can >> influence if/when they are needed/used etc: >> http://maven.apache.org/guides/introduction/introduction-to-dependency- >> mechanism.html#Dependency_Scope >> ) >> >> >> for example I eyeballed the parent pom.xml and saw the dependencies, >>> >> >> The parent pom actually has no real dependencies (except on its parent, >> the >> apache pom). >> >> What you likely saw was the pluginManagement and dependencyManagement >> sections, which help fix down and/or consistently specify the versions of >> particular dependencies and plugins that might be used by the child >> modules >> and their build steps. These dont actually mean things have a dependency >> on >> those artifacts, just which version etc they would get by default if they >> should happen to specify such a dependency or plugin to use. This helps >> keep things using consistent versions of artifacts, and in the case of >> plugins helps ensure reproducable builds by locking down the versions in >> use. >> >> >> but when I looked at the repository after running mvn clean install on >>> that I was kind of expecting the directories in the repository to pretty >>> much match what was in the dependencies, but it definitely didn't after >>> running mvn on the parent pom.xml, though to be fair after building the >>> main things it does all appear to match (though with a ton of other stuff >>> too). >>> >>> What you would have got after doing 'mvn install' on the qpid-parent pom >> would be mostly only the plugins needed by maven itself to carry out those >> actions. What they ship in the core maven release is augmented by >> downloading/installing things to the local repo as they are used both by >> itself and by people declaring they want to use a specific plugin in their >> build. >> >> >> When I build the Java Broker etc. it seemed to take an age, when I was >>> using ant it took under a minute on my system, but with Maven it reports >>> a >>> total time of 5:18 min and it looks like it was downloading half the >>> Internet :-D I'm *guessing* that this is a one-off cost as it fills up my >>> local repo with stuff? >>> >>> >>> Yes, the first time you would have incurred the cost of grabbing any >> plugins the build needed that you didnt already have and also the >> dependencies of the things being built (the later of which happens on your >> first Ant build too - go look in ~/.ivy2/cache, then try deleting it and >> see what happens to the Ant build time). Run it again and it will be a lot >> faster going forward. >> >> On my machine using the head of trunk (I just committed a bunch of tweaks >> earlier), 'ant clean build' vs 'mvn clean package -DskipTests=true' has >> the >> maven build being 1 or 2sec slower (32 vs 33/34), which is actually >> surprisingly close since using those commands we currenty have the maven >> build doing a bunch of extra packaging and enforcement that the ant build >> doesnt. It may also be quicker if told to build 'offline' to ensure it >> didnt check any repos for updates. >> >> (It takes a further 2mins 21sec on top of that for me to use the Ant build >> to generate the maven artifacts we currently publish to central...a >> generation step we obviously wont need to do for the maven build as they >> are already there inherantly.) >> >> >> Where do you set CLASSPATH when you build using Maven? With the ant build >>> I used to have: >>> <qpid>qpid/java/build/lib/qpid-all.jar >>> >>> >>> You dont set CLASSPATH, basically. Supplying dependency information is >> all >> handled through the poms, which maven uses to build the classpath for each >> module as it builds. If you want to build something else that uses e.g the >> qpid client, the thing your are building should have a dependency for >> qpid-client in its pom. You can make that dependency be either a published >> release [or snapshot] version available in central [or the apache >> snapshots >> repo], or you can 'mvn install' your own modified copy if you want to use >> an unpublished client release. If you dont want to use maven / want to use >> basic javac commands for the other thing you are compiling against the >> client, then you would need to make the client jars available somewhere >> and >> add the individual jars to your classpath currently. >> >> We aren't recreating qpid-all at this time because while it may be >> convenient in cases, it is actually a horrible horrible thing, which I'll >> talk a bit more to later when covering the build directory :) >> >> We have a qpid-all file in every release artifact made by the ant build, >> each having different content, and all different from the one that gets >> created in the build dir. Even if we wanted to recreate it in the maven >> build, we would need to give it a unique name for every component to ever >> publish it anyway. >> >> We could instead do something like add the classpath manifest entries >> previously contained in qpid-all.jar into the main qpid-client, >> qpid-broker >> etc jars so you could do 'java -jar qpid-foo' etc on them instead. >> >> >> Which was nice and convenient, if possibly a bit sloppy, from what I can >>> see each qpid/java subdirectory seems to have a target directory e.g. >>> qpid/java/client/target though in there there is >>> qpid-client-0.28-SNAPSHOT.jar which seems less convenient that >>> qpid-client.jar if I want to set up CLASSPATH in my .bashrc. Am I missing >>> something? >>> >>> Another thing that I'm not clear on is that in the Maven repository there >>> seems to be a bunch of jars installed - as an example >>> org/eclipse/jetty/jetty-websocket/8.1.14.v20131031/ >>> jetty-websocket-8.1.14.v20131031.jar. >>> In the "olden days" Qpid pulled in jetty and the Jar was available in >>> qpid/java/build/lib so when I was messing around with Jetty on another >>> project I had it handily on my CLASSPATH - I can't seem to see jetty >>> anywhere now except in the Maven repository? So how does the Broker see >>> it? >>> >>> The broker sees it because it says it has a runtime dependency on the >> management-http module in its pom, and the management-http module says it >> has a compile time dependency on Jetty in its pom, and so maven then knows >> the broker transitively needs jetty. The actual jars never leave the local >> repo unless we instruct the build to do something that requires that (e.g. >> package a binary assembly containing it, or copying it somewhere) in the >> pom or you execute a command manually that does (e.g. a quick Google says >> 'mvn dependency:copy-dependencies' requests the dependency plugin to run >> its copy-dependencies goal, which create a dir in target/dependency with >> everything in it). >> >> The only reason the Ant build still puts things in the lib dir (by using >> Ivy to grab thigns from the maven central repo, i.e roughly the same thing >> Maven is doing to download things) is so that I didnt need to rewrite even >> more of the Ant build than I already was when I modified it to allow >> removing the jars from the lib dir, where they were actually committed >> into >> the svn repo historically (I refer you to my previous mails comment about >> the Ant build being a pain in the ass and me being the one best placed to >> speak best to this point :P). >> >> >> >> In the olden days when I just did "ant" at the end of that I ended up >>> with >>> something that would run 'cause I had >>> <qpid-trunk>qpid/java/build/bin >>> >>> on my path and could simply do "qpid-server" but now there's no nice >>> convenient build directory. >>> >>> The existing build directory is horrible and should burn :P >> >> I'll admit, it can be convenient, but its also a complete jumble with a >> little bit of extra mess on top, mixing up all the different components >> together in lib so you cant tell which bits need what (qpid-all.jar in the >> build directory being worst of all, as you have everything on the >> classpath >> manifest in there) and doing ugly copies of stuff into subdirs of lib to >> seperate them for later use precisely because you cant tell which bits >> need >> what. I would have tried to kill it a long time ago if it didnt involve >> yet >> more 'rewriting the ant build' fun, which I just couldnt bring myself to >> subject me to :) >> >> Maven deals with modules, which each have their own target dir for their >> intermediate work and final output, and if you want something that >> aggregates multiple modules then there are varous ways to do that, >> avoiding >> the global build dir mess while still letting you compose a larger unit >> from the individual bits. It is certainly different than what our Ant >> build >> does, but in this case I think its a good thing. We can always do things >> to >> make life easier, more below. >> >> >> I noticed in >>> qpid/java/broker/target/qpid-broker-0.28-SNAPSHOT-bin.tar.gz that >>> archive >>> seems to contain qpid-server, do I *really* have to now have to faff >>> around >>> copying and unpacking archives everytime I want to update a build? That >>> seems an awful lot less convenient than simply doing "ant clean" then >>> "ant". >>> >>> Have I missed something? I hope so 'cause if not it seems a step >>> backwards >>> from a "just works" POV >>> >>> What you say is true *currently*, to run the broker built by the maven >> build you would need to unpack that tar file. We can certainly add >> functionality to remove that need, we just havent got to those final few >> 'nice to have' things yet. There are a multitude of ways we could do it, >> for example we could use a plugin and have the maven build itself start >> the >> broker (e.g see what i did with the 'tests' in the QMF tree: those >> actually >> run without even creating a jar file), or add some build config to allow >> output of an unpackaged version of it during the main build, or..<insert >> idea>. >> >> >> This might all be second nature to folks familiar with Maven, but I'm a >>> bit "old skool" and I quite like knowing what's installed on my system, >>> where it is installed, and why it's there so I'd quite like a bit of >>> reassurance :-) >>> >>> Frase >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
