Those are pretty old - but I think the reason Matei did that was to
make it less confusing for brand new users. `spark` is actually a
valid identifier because it's just a variable name (val spark = new
SparkContext()) but I agree this could be confusing for users who want
to drop into the shell.
On Fri, Jun 20, 2014 at 12:04 PM, Will Benton wrote:
> Hey, sorry to reanimate this thread, but just a quick question: why do the
> examples (on http://spark.apache.org/examples.html) use "spark" for the
> SparkContext reference? This is minor, but it seems like it could be a
> little confusing for people who want to run them in the shell and need to
> change "spark" to "sc". (I noticed because this was a speedbump for a
> colleague who is trying out Spark.)
>
>
> thanks,
> wb
>
> - Original Message -
>> From: "Andy Konwinski"
>> To: dev@spark.apache.org
>> Sent: Tuesday, May 20, 2014 4:06:33 PM
>> Subject: Re: Scala examples for Spark do not work as written in documentation
>>
>> I fixed the bug, but I kept the parameter "i" instead of "_" since that (1)
>> keeps it more parallel to the python and java versions which also use
>> functions with a named variable and (2) doesn't require readers to know
>> this particular use of the "_" syntax in Scala.
>>
>> Thanks for catching this Glenn.
>>
>> Andy
>>
>>
>> On Fri, May 16, 2014 at 12:38 PM, Mark Hamstra
>> wrote:
>>
>> > Sorry, looks like an extra line got inserted in there. One more try:
>> >
>> > val count = spark.parallelize(1 to NUM_SAMPLES).map { _ =>
>> > val x = Math.random()
>> > val y = Math.random()
>> > if (x*x + y*y < 1) 1 else 0
>> > }.reduce(_ + _)
>> >
>> >
>> >
>> > On Fri, May 16, 2014 at 12:36 PM, Mark Hamstra > > >wrote:
>> >
>> > > Actually, the better way to write the multi-line closure would be:
>> > >
>> > > val count = spark.parallelize(1 to NUM_SAMPLES).map { _ =>
>> > >
>> > > val x = Math.random()
>> > > val y = Math.random()
>> > > if (x*x + y*y < 1) 1 else 0
>> > > }.reduce(_ + _)
>> > >
>> > >
>> > > On Fri, May 16, 2014 at 9:41 AM, GlennStrycker > > >wrote:
>> > >
>> > >> On the webpage http://spark.apache.org/examples.html, there is an
>> > example
>> > >> written as
>> > >>
>> > >> val count = spark.parallelize(1 to NUM_SAMPLES).map(i =>
>> > >> val x = Math.random()
>> > >> val y = Math.random()
>> > >> if (x*x + y*y < 1) 1 else 0
>> > >> ).reduce(_ + _)
>> > >> println("Pi is roughly " + 4.0 * count / NUM_SAMPLES)
>> > >>
>> > >> This does not execute in Spark, which gives me an error:
>> > >> :2: error: illegal start of simple expression
>> > >> val x = Math.random()
>> > >> ^
>> > >>
>> > >> If I rewrite the query slightly, adding in {}, it works:
>> > >>
>> > >> val count = spark.parallelize(1 to 1).map(i =>
>> > >>{
>> > >>val x = Math.random()
>> > >>val y = Math.random()
>> > >>if (x*x + y*y < 1) 1 else 0
>> > >>}
>> > >> ).reduce(_ + _)
>> > >> println("Pi is roughly " + 4.0 * count / 1.0)
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> View this message in context:
>> > >>
>> > http://apache-spark-developers-list.1001551.n3.nabble.com/Scala-examples-for-Spark-do-not-work-as-written-in-documentation-tp6593.html
>> > >> Sent from the Apache Spark Developers List mailing list archive at
>> > >> Nabble.com.
>> > >>
>> > >
>> > >
>> >
>>
