You can even use it without spark as well (besides local). For example i have used the following algo in some web app: https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala
Essentially some algorithms (i havent checked them all) they will have to run the same steps in each partition so if you overlook the distributed oriented parts (spark specific) of the code there is a lot of resuable stuff. You have just to use the api where that is public and conform to the input/output contract of it. There used to be some dependencies like Breeze for example in the api hidden now (eg. https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala), but of course this is a hint not a list of what is available for your use case. Mind though that this may not be the cleanest way to implement your use case or might sound a hack ;) As an alternative choice besides spark local you could use a job server ( https://github.com/spark-jobserver/spark-jobserver) to integrate with your app as a proxy for spark or have a spark service there to respond back with results. From a design point of you its best to separate the concerns for several reasons: scaling , utilization etc. On Sun, Nov 22, 2015 at 5:03 AM, Reynold Xin <r...@databricks.com> wrote: > You can use MLlib and Spark directly without "installing anything". Just > run Spark in local mode. > > > On Sat, Nov 21, 2015 at 4:05 PM, Rad Gruchalski <ra...@gruchalski.com> > wrote: > >> Bowen, >> >> What Andy is doing in the notebook is a slightly different thing. He’s >> using sbt to bring all spark jars (core, mllib, repl, what have you). You >> could use maven for that. He then creates a repl and submits all the spark >> code into it. >> Pretty sure spark unit tests cover similar uses cases. Maybe not mllib >> per se but this kind of submission. >> >> Kind regards, >> Radek Gruchalski >> ra...@gruchalski.com <ra...@gruchalski.com> >> de.linkedin.com/in/radgruchalski/ >> >> >> *Confidentiality:*This communication is intended for the above-named >> person and may be confidential and/or legally privileged. >> If it has come to you in error you must take no action based on it, nor >> must you copy or show it to anyone; please delete/destroy and inform the >> sender immediately. >> >> On Sunday, 22 November 2015 at 01:01, bowen zhang wrote: >> >> Thanks Rad for info. I looked into the repo and see some .snb file using >> spark mllib. Can you give me a more specific place to look for when >> invoking the mllib functions? What if I just want to invoke some of the ML >> functions in my HelloWorld.java? >> >> ------------------------------ >> *From:* Rad Gruchalski <ra...@gruchalski.com> >> *To:* bowen zhang <bowenzhang...@yahoo.com> >> *Cc:* "dev@spark.apache.org" <dev@spark.apache.org> >> *Sent:* Saturday, November 21, 2015 3:43 PM >> *Subject:* Re: Using spark MLlib without installing Spark >> >> Bowen, >> >> One project to look at could be spark-notebook: >> https://github.com/andypetrella/spark-notebook >> It uses Spark you in the way you intend to use it. >> Kind regards, >> Radek Gruchalski >> ra...@gruchalski.com <ra...@gruchalski.com> >> de.linkedin.com/in/radgruchalski/ >> >> >> *Confidentiality:*This communication is intended for the above-named >> person and may be confidential and/or legally privileged. >> If it has come to you in error you must take no action based on it, nor >> must you copy or show it to anyone; please delete/destroy and inform the >> sender immediately. >> >> >> On Sunday, 22 November 2015 at 00:38, bowen zhang wrote: >> >> Hi folks, >> I am a big fan of Spark's Mllib package. I have a java web app where I >> want to run some ml jobs inside the web app. My question is: is there a way >> to just import spark-core and spark-mllib jars to invoke my ML jobs without >> installing the entire Spark package? All the tutorials related Spark seems >> to indicate installing Spark is a pre-condition for this. >> >> Thanks, >> Bowen >> >> >> >> >> >> > -- Stavros Kontopoulos <http://www.typesafe.com> <http://www.typesafe.com>
