Re: Play framework
().setMaster(local[*]).setAppName(sMy Awesome
App)
// call conf.set for any other configuration you want
val sc = new SparkContext(sparkConf)
It is important to keep in mind you cannot have multiple local contexts
(you can create them but you'll get odd errors), so if you are running
things in parallel within your app (even unit tests) you'd need to share a
context in this case. If you are running sequentially you can create a new
local context each time, but you must make sure to call SparkContext.stop()
when you're done.
Running against a cluster is a bit more complicated because you need to
add all your dependency jars. I'm not sure how to get this to work with play
run. I stick to building the app with play dist and then running against
the packaged application, because it very conveniently provides all the
dependencies in a lib folder. Here is some code to load all the paths you
need from the dist:
def libs : Seq[String] = {
val libDir = play.api.Play.application.getFile(lib)
logger.info(sSparkContext will be initialized with libraries
from directory $libDir)
return if ( libDir.exists ) {
libDir.listFiles().map(_.getCanonicalFile().getAbsolutePath()).filter(_.endsWith(.jar))
} else {
throw new IllegalStateException(slib dir is missing: $libDir)
}
}
Creating the context is similar to above, but with this extra line:
conf.setJars(libs)
I hope this helps. I should note that I don't use play run very much, at
least not for when I'm actually executing Spark jobs. So I'm not sure if
this integrates properly with that. I have unit tests which execute on
Spark and have executed the dist package both locally and on a cluster. To
make working with the dist locally easier, I wrote myself a little shell
script to unzip and run the dist.
On Wed, Oct 15, 2014 at 10:51 PM, Mohammed Guller moham...@glassbeam.com
wrote:
Hi –
Has anybody figured out how to integrate a Play application with Spark and
run it on a Spark cluster using spark-submit script? I have seen some blogs
about creating a simple Play app and running it locally on a dev machine
with sbt run command. However, those steps don’t work for Spark-submit.
If you have figured out how to build and run a Play app with Spark-submit,
I would appreciate if you could share the steps and the sbt settings for
your Play app.
Thanks,
Mohammed
--
Daniel Siegmann, Software Developer
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
E: daniel.siegm...@velos.io W: www.velos.io
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v suren.hira...@sociocast.comelos.io
W: www.velos.io
--
--
Ramaraju Indukuri
Re: Nested Case Classes (Found and Required Same)
Upgraded to 1.1 and the issue is resolved. Thanks. I still wonder if there is a better way to approach a large attribute dataset. On Fri, Sep 12, 2014 at 12:20 PM, Prashant Sharma scrapco...@gmail.com wrote: What is your spark version ? This was fixed I suppose. Can you try it with latest release ? Prashant Sharma On Fri, Sep 12, 2014 at 9:47 PM, Ramaraju Indukuri iramar...@gmail.com wrote: This is only a problem in shell, but works fine in batch mode though. I am also interested in how others are solving the problem of case class limitation on number of variables. Regards Ram On Fri, Sep 12, 2014 at 12:12 PM, iramaraju iramar...@gmail.com wrote: I think this is a popular issue, but need help figuring a way around if this issue is unresolved. I have a dataset that has more than 70 columns. To have all the columns fit into my RDD, I am experimenting the following. (I intend to use the InputData to parse the file and have 3 or 4 columnsets to accommodate the full list of variables) case class ColumnSet(C1: Double , C2: Double , C3: Double) case class InputData(EQN: String, ts: String,Set1 :ColumnSet,Set2 :ColumnSet) val set1 = ColumnSet(1,2,3) val a = InputData(a,a,set1,set1) returns the following console:16: error: type mismatch; found : ColumnSet required: ColumnSet val a = InputData(a,a,set1,set1) Where as the same code works fine in my scala console. Is there a work around for my problem ? Regards Ram -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Nested-Case-Classes-Found-and-Required-Same-tp14096.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- -- Ramaraju Indukuri -- -- Ramaraju Indukuri
Re: Nested Case Classes (Found and Required Same)
This is only a problem in shell, but works fine in batch mode though. I am also interested in how others are solving the problem of case class limitation on number of variables. Regards Ram On Fri, Sep 12, 2014 at 12:12 PM, iramaraju iramar...@gmail.com wrote: I think this is a popular issue, but need help figuring a way around if this issue is unresolved. I have a dataset that has more than 70 columns. To have all the columns fit into my RDD, I am experimenting the following. (I intend to use the InputData to parse the file and have 3 or 4 columnsets to accommodate the full list of variables) case class ColumnSet(C1: Double , C2: Double , C3: Double) case class InputData(EQN: String, ts: String,Set1 :ColumnSet,Set2 :ColumnSet) val set1 = ColumnSet(1,2,3) val a = InputData(a,a,set1,set1) returns the following console:16: error: type mismatch; found : ColumnSet required: ColumnSet val a = InputData(a,a,set1,set1) Where as the same code works fine in my scala console. Is there a work around for my problem ? Regards Ram -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Nested-Case-Classes-Found-and-Required-Same-tp14096.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- -- Ramaraju Indukuri
