Interestingly, mllib isn't even getting built by maven... I will address it as a separate patch then.
On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote: > Basically the way to think of the assembly is that it should have libraries > that users' client programs need to run. These are core, repl (needed if > they use the shell), and likely bagel and streaming and mllib, though > originally we'd opted to leave those out. We are still deciding on that -- > could go either way. > > Matei > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <[email protected]> wrote: > > > Yeah, that is true. But the assembly shouldn't include the examples project > > at all IMO -- if it does now, we should remove it. > > > > Matei > > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <[email protected]> wrote: > > > >> Matei, > >> > >> Hbase dependencies aren't actually included into the Maven assembly as of > >> this > >> moment, because scope of hbase dependency in examples' module is > >> "compile"; but > >> the assembly is only includes those with "runtime". Hence it is > >> automatically > >> excluded. > >> > >> I believe, hbase is needed for examples during the execution time, and if > >> so - > >> it would have to be fixed in the module. This will lead to need to exclude > >> it > >> from the assembly, in turn. > >> > >> And of course... :) > >> > >> s/putt/pull/ > >> > >> Cos > >> > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote: > >>> Yeah, and maybe we will want to change to Maven as the recommended tool > >>> for > >>> assembly building. I want to look into this more for the 0.8 release. > >>> > >>> Matei > >>> > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <[email protected]> wrote: > >>> > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote: > >>>>> Let's at the very least make it configurable, but an even better thing > >>>>> will > >>>>> be to make sbt assembly not include it. I think the only thing that > >>>>> depends > >>>>> on HBase is the examples project, but unfortunately SBT puts all its > >>>>> JARs in > >>>>> the lib_managed folder and just stupidly creates an assembly by grouping > >>>>> those. The Maven build, for example, should not do that. > >>>> > >>>> It is very easy to exclude dependencies in Maven assembly, like it is > >>>> done for > >>>> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, > >>>> thank > >>>> you! > >>>> > >>>> Cos > >>>> > >>>>> Matei > >>>>> > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <[email protected]> wrote: > >>>>> > >>>>>> Hello, > >>>>>> > >>>>>> after couple of days(!) of trying to understand where i get the > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now > >>>>>> includes > >>>>>> hbase. > >>>>>> > >>>>>> While it is assumed that hadoop version is specified, hbase version is > >>>>>> fixed. This seem to create problem if hbase is used with a particular > >>>>>> version of CDH hadoop client in the backend. (there's a known > >>>>>> compatibility > >>>>>> bug). > >>>>>> > >>>>>> wouldn't it make sense in this case to allow to declare hbase version > >>>>>> as > >>>>>> well, perhaps even tie it to the CDH version? > >>>>>> > >>>>>> At the very least i think it deserves a specific mention in the header > >>>>>> section to provide opportunity to override, just like hadoop version > >>>>>> does? > >>>>>> > >>>>>> Thanks. > >>>>>> -D > >>>>> > >>> > > >
