Oh, I see your point now. For hadoop 1 over hadoop 2 we would just use the same profiles that we have in place. We could look into using a classifier when deploying these artifacts so users can pull down a version of minicluster that is compatible with hadoop2 without forcing them to build it themselves.
Given that we already *have* hadoop-1.x listed as the default dependency, I don't really see that as being an issue. On Tue, Sep 24, 2013 at 12:58 PM, Keith Turner <ke...@deenlo.com> wrote: > On Tue, Sep 24, 2013 at 12:48 PM, Josh Elser <josh.el...@gmail.com> wrote: > >> On Tue, Sep 24, 2013 at 12:31 PM, Keith Turner <ke...@deenlo.com> wrote: >> > On Tue, Sep 24, 2013 at 11:57 AM, Josh Elser <josh.el...@gmail.com> >> wrote: >> > >> >> I'm curious to hear what people think on this. >> >> >> >> I'm a really big fan of spinning up a minicluster instance to do some >> >> "more real" testing of software as I write it. >> >> >> >> With 1.5.0, it's a bit more painful because I have to add a bunch more >> >> dependencies to my project (which previously would only have to depend >> >> on the accumulo-minicluster artifact). The list includes, but is >> >> likely not limited to, commons-io, commons-configuration, >> >> hadoop-client, zookeeper, log4j, slf4j-api, slf4j-log4j12. >> >> >> >> Best as I understand it, the intent of this was that Hadoop will >> >> typically provide these artifacts at runtime, and therefore Accumulo >> >> doesn't need to re-bundle them itself which I'd agree with (not >> >> getting into that whole issue about the Hadoop "ecosystem"). However, >> >> I would think that the minicluster should have non-provided scope >> >> dependencies declared on these, as there is no Hadoop installation -- >> >> >> > >> > Would this require declaring dependencies on a particular version of >> hadoop >> > in the minicluster pom? Or could the minicluster pom have profiles for >> > different hadoop versions? I do not know enough about maven to know if >> you >> > can use profiles declared in a dependency (e.g. if a user depends on >> > minicluster, can they activate profiles in it?) >> >> The actual dependency in minicluster is against Apache Hadoop but >> that's besides the point. >> >> By marking the hadoop-client dependency as provided that means that >> Hadoop's dependencies are *not* included at runtime (because hadoop is >> provided, and, as such, so are its dependencies). In other words, this >> is completely beside the point of what's actually included in a >> distribution of Hadoop when you download and install it. >> >> Apache Hadoop has dependencies we need to run minicluster. By marking >> the hadoop-client artifact as 'provided', we do not get its >> dependencies and the minicluster fails to run. I think this is easy >> enough to work around by overriding the dependencies we need to run >> the minicluster in the minicluster module (e.g. make the hadoop-client >> not 'provided' in the minicluster module). Thus, as we add more things >> > > So if we mark hadoop-client as not provided, then we have to choose a > version? How easy will it be for a user to choose a different version of > hadoop for their testing? I am trying to undertand what impact this would > have a users pom that depends on Hadoop 2 if minicluster depends on Hadoop > 1.2. > > > >> to the minicluster that require other libraries, we control the >> dependency mgmt instead of forcing that onto the user. >> >> > >> > >> >> there's just the minicluster. As such, this would alleviate users from >> >> having to dig into our dependency management or trial&error to figure >> >> out what "extra" dependencies they have to include in their project to >> >> actually make it work >> >> >> >> Thoughts? >> >> >> >> - Josh >> >> >>
