Those are pretty old - but I think the reason Matei did that was to make it less confusing for brand new users. `spark` is actually a valid identifier because it's just a variable name (val spark = new SparkContext()) but I agree this could be confusing for users who want to drop into the shell.
On Fri, Jun 20, 2014 at 12:04 PM, Will Benton <wi...@redhat.com> wrote: > Hey, sorry to reanimate this thread, but just a quick question: why do the > examples (on http://spark.apache.org/examples.html) use "spark" for the > SparkContext reference? This is minor, but it seems like it could be a > little confusing for people who want to run them in the shell and need to > change "spark" to "sc". (I noticed because this was a speedbump for a > colleague who is trying out Spark.) > > > thanks, > wb > > ----- Original Message ----- >> From: "Andy Konwinski" <andykonwin...@gmail.com> >> To: dev@spark.apache.org >> Sent: Tuesday, May 20, 2014 4:06:33 PM >> Subject: Re: Scala examples for Spark do not work as written in documentation >> >> I fixed the bug, but I kept the parameter "i" instead of "_" since that (1) >> keeps it more parallel to the python and java versions which also use >> functions with a named variable and (2) doesn't require readers to know >> this particular use of the "_" syntax in Scala. >> >> Thanks for catching this Glenn. >> >> Andy >> >> >> On Fri, May 16, 2014 at 12:38 PM, Mark Hamstra >> <m...@clearstorydata.com>wrote: >> >> > Sorry, looks like an extra line got inserted in there. One more try: >> > >> > val count = spark.parallelize(1 to NUM_SAMPLES).map { _ => >> > val x = Math.random() >> > val y = Math.random() >> > if (x*x + y*y < 1) 1 else 0 >> > }.reduce(_ + _) >> > >> > >> > >> > On Fri, May 16, 2014 at 12:36 PM, Mark Hamstra <m...@clearstorydata.com >> > >wrote: >> > >> > > Actually, the better way to write the multi-line closure would be: >> > > >> > > val count = spark.parallelize(1 to NUM_SAMPLES).map { _ => >> > > >> > > val x = Math.random() >> > > val y = Math.random() >> > > if (x*x + y*y < 1) 1 else 0 >> > > }.reduce(_ + _) >> > > >> > > >> > > On Fri, May 16, 2014 at 9:41 AM, GlennStrycker <glenn.stryc...@gmail.com >> > >wrote: >> > > >> > >> On the webpage http://spark.apache.org/examples.html, there is an >> > example >> > >> written as >> > >> >> > >> val count = spark.parallelize(1 to NUM_SAMPLES).map(i => >> > >> val x = Math.random() >> > >> val y = Math.random() >> > >> if (x*x + y*y < 1) 1 else 0 >> > >> ).reduce(_ + _) >> > >> println("Pi is roughly " + 4.0 * count / NUM_SAMPLES) >> > >> >> > >> This does not execute in Spark, which gives me an error: >> > >> <console>:2: error: illegal start of simple expression >> > >> val x = Math.random() >> > >> ^ >> > >> >> > >> If I rewrite the query slightly, adding in {}, it works: >> > >> >> > >> val count = spark.parallelize(1 to 10000).map(i => >> > >> { >> > >> val x = Math.random() >> > >> val y = Math.random() >> > >> if (x*x + y*y < 1) 1 else 0 >> > >> } >> > >> ).reduce(_ + _) >> > >> println("Pi is roughly " + 4.0 * count / 10000.0) >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> -- >> > >> View this message in context: >> > >> >> > http://apache-spark-developers-list.1001551.n3.nabble.com/Scala-examples-for-Spark-do-not-work-as-written-in-documentation-tp6593.html >> > >> Sent from the Apache Spark Developers List mailing list archive at >> > >> Nabble.com. >> > >> >> > > >> > > >> > >>
