Re: Using spark MLlib without installing Spark

2015-11-21 Thread Reynold Xin
You can use MLlib and Spark directly without "installing anything". Just run Spark in local mode. On Sat, Nov 21, 2015 at 4:05 PM, Rad Gruchalski wrote: > Bowen, > > What Andy is doing in the notebook is a slightly different thing. He’s > using sbt to bring all spark jars (core, mllib, repl, wh

Re: Using spark MLlib without installing Spark

2015-11-21 Thread Rad Gruchalski
Bowen, What Andy is doing in the notebook is a slightly different thing. He’s using sbt to bring all spark jars (core, mllib, repl, what have you). You could use maven for that. He then creates a repl and submits all the spark code into it. Pretty sure spark unit tests cover similar uses cases

Re: Using spark MLlib without installing Spark

2015-11-21 Thread bowen zhang
Thanks Rad for info. I looked into the repo and see some .snb file using spark mllib. Can you give me a more specific place to look for when invoking the mllib functions? What if I just want to invoke some of the ML functions in my HelloWorld.java? From: Rad Gruchalski To: bowen zhang

Re: Using spark MLlib without installing Spark

2015-11-21 Thread Rad Gruchalski
Bowen, One project to look at could be spark-notebook: https://github.com/andypetrella/spark-notebook It uses Spark you in the way you intend to use it. Kind regards,
 Radek Gruchalski 
ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 (mailto:ra...@gruchalski.com) de.linkedin.com/

Using spark MLlib without installing Spark

2015-11-21 Thread bowen zhang
Hi folks,I am a big fan of Spark's Mllib package. I have a java web app where I want to run some ml jobs inside the web app. My question is: is there a way to just import spark-core and spark-mllib jars to invoke my ML jobs without installing the entire Spark package? All the tutorials related S

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

2015-11-21 Thread Steve Loughran
> On 20 Nov 2015, at 21:39, Reynold Xin wrote: > > OK I'm not exactly asking for a vote here :) > > I don't think we should look at it from only maintenance point of view -- > because in that case the answer is clearly supporting as few versions as > possible (or just rm -rf spark source code

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

2015-11-21 Thread Sean Owen
On Fri, Nov 20, 2015 at 10:39 PM, Reynold Xin wrote: > I don't think we should look at it from only maintenance point of view -- > because in that case the answer is clearly supporting as few versions as > possible (or just rm -rf spark source code and call it a day). It is a > tradeoff between th

Re: Unhandled case in VectorAssembler

2015-11-21 Thread Benjamin Fradet
Will do, thanks for your input. On 21 Nov 2015 2:42 a.m., "Joseph Bradley" wrote: > Yes, please, could you send a JIRA (and PR)? A custom error message would > be better. > Thank you! > Joseph > > On Fri, Nov 20, 2015 at 2:39 PM, BenFradet > wrote: > >> Hey there, >> >> I noticed that there is