Re: Scala examples for Spark do not work as written in documentation
Hey, sorry to reanimate this thread, but just a quick question: why do the
examples (on http://spark.apache.org/examples.html) use spark for the
SparkContext reference? This is minor, but it seems like it could be a little
confusing for people who want to run them in the shell and need to change
spark to sc. (I noticed because this was a speedbump for a colleague who
is trying out Spark.)
thanks,
wb
- Original Message -
From: Andy Konwinski andykonwin...@gmail.com
To: dev@spark.apache.org
Sent: Tuesday, May 20, 2014 4:06:33 PM
Subject: Re: Scala examples for Spark do not work as written in documentation
I fixed the bug, but I kept the parameter i instead of _ since that (1)
keeps it more parallel to the python and java versions which also use
functions with a named variable and (2) doesn't require readers to know
this particular use of the _ syntax in Scala.
Thanks for catching this Glenn.
Andy
On Fri, May 16, 2014 at 12:38 PM, Mark Hamstra
m...@clearstorydata.comwrote:
Sorry, looks like an extra line got inserted in there. One more try:
val count = spark.parallelize(1 to NUM_SAMPLES).map { _ =
val x = Math.random()
val y = Math.random()
if (x*x + y*y 1) 1 else 0
}.reduce(_ + _)
On Fri, May 16, 2014 at 12:36 PM, Mark Hamstra m...@clearstorydata.com
wrote:
Actually, the better way to write the multi-line closure would be:
val count = spark.parallelize(1 to NUM_SAMPLES).map { _ =
val x = Math.random()
val y = Math.random()
if (x*x + y*y 1) 1 else 0
}.reduce(_ + _)
On Fri, May 16, 2014 at 9:41 AM, GlennStrycker glenn.stryc...@gmail.com
wrote:
On the webpage http://spark.apache.org/examples.html, there is an
example
written as
val count = spark.parallelize(1 to NUM_SAMPLES).map(i =
val x = Math.random()
val y = Math.random()
if (x*x + y*y 1) 1 else 0
).reduce(_ + _)
println(Pi is roughly + 4.0 * count / NUM_SAMPLES)
This does not execute in Spark, which gives me an error:
console:2: error: illegal start of simple expression
val x = Math.random()
^
If I rewrite the query slightly, adding in {}, it works:
val count = spark.parallelize(1 to 1).map(i =
{
val x = Math.random()
val y = Math.random()
if (x*x + y*y 1) 1 else 0
}
).reduce(_ + _)
println(Pi is roughly + 4.0 * count / 1.0)
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Scala-examples-for-Spark-do-not-work-as-written-in-documentation-tp6593.html
Sent from the Apache Spark Developers List mailing list archive at
Nabble.com.
Re: Scala examples for Spark do not work as written in documentation
Those are pretty old - but I think the reason Matei did that was to
make it less confusing for brand new users. `spark` is actually a
valid identifier because it's just a variable name (val spark = new
SparkContext()) but I agree this could be confusing for users who want
to drop into the shell.
On Fri, Jun 20, 2014 at 12:04 PM, Will Benton wi...@redhat.com wrote:
Hey, sorry to reanimate this thread, but just a quick question: why do the
examples (on http://spark.apache.org/examples.html) use spark for the
SparkContext reference? This is minor, but it seems like it could be a
little confusing for people who want to run them in the shell and need to
change spark to sc. (I noticed because this was a speedbump for a
colleague who is trying out Spark.)
thanks,
wb
- Original Message -
From: Andy Konwinski andykonwin...@gmail.com
To: dev@spark.apache.org
Sent: Tuesday, May 20, 2014 4:06:33 PM
Subject: Re: Scala examples for Spark do not work as written in documentation
I fixed the bug, but I kept the parameter i instead of _ since that (1)
keeps it more parallel to the python and java versions which also use
functions with a named variable and (2) doesn't require readers to know
this particular use of the _ syntax in Scala.
Thanks for catching this Glenn.
Andy
On Fri, May 16, 2014 at 12:38 PM, Mark Hamstra
m...@clearstorydata.comwrote:
Sorry, looks like an extra line got inserted in there. One more try:
val count = spark.parallelize(1 to NUM_SAMPLES).map { _ =
val x = Math.random()
val y = Math.random()
if (x*x + y*y 1) 1 else 0
}.reduce(_ + _)
On Fri, May 16, 2014 at 12:36 PM, Mark Hamstra m...@clearstorydata.com
wrote:
Actually, the better way to write the multi-line closure would be:
val count = spark.parallelize(1 to NUM_SAMPLES).map { _ =
val x = Math.random()
val y = Math.random()
if (x*x + y*y 1) 1 else 0
}.reduce(_ + _)
On Fri, May 16, 2014 at 9:41 AM, GlennStrycker glenn.stryc...@gmail.com
wrote:
On the webpage http://spark.apache.org/examples.html, there is an
example
written as
val count = spark.parallelize(1 to NUM_SAMPLES).map(i =
val x = Math.random()
val y = Math.random()
if (x*x + y*y 1) 1 else 0
).reduce(_ + _)
println(Pi is roughly + 4.0 * count / NUM_SAMPLES)
This does not execute in Spark, which gives me an error:
console:2: error: illegal start of simple expression
val x = Math.random()
^
If I rewrite the query slightly, adding in {}, it works:
val count = spark.parallelize(1 to 1).map(i =
{
val x = Math.random()
val y = Math.random()
if (x*x + y*y 1) 1 else 0
}
).reduce(_ + _)
println(Pi is roughly + 4.0 * count / 1.0)
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Scala-examples-for-Spark-do-not-work-as-written-in-documentation-tp6593.html
Sent from the Apache Spark Developers List mailing list archive at
Nabble.com.
Current status of Sparrow
Hi What is the current status of Sparrow integration with Spark? I would like to integrate Sparrow with Spark 1.0 on a 100 node cluster. Any suggestions? Thanks a lot for your help! Liquan
