The code is the example given on Spark site:
/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object SimpleApp {
def main(args: Array[String]) {
val logFile =
"C:/Users/D062844/Desktop/HandsOnSpark/Install/spark-1.0.2-bin-hadoop2/README.md"
// Should be some file on your system
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile, 2).cache()
val numAs = logData.filter(line => line.contains("a")).count()
val numBs = logData.filter(line => line.contains("b")).count()
println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
}
}
It goes on like this and doesn’t show me the result of count in the file. I had
installed pre-built version of Spark1.0.2 named Hadoop2 version Spark on the
site. The problem what I think is because I am running it on local machine and
it is not able to find some dependencies of Hadoop. Please tell me what file
should I download to work on my local machine (pre-built, so that I don’t have
to build it again).
From: Akhil Das [mailto:[email protected]]
Sent: Mittwoch, 27. August 2014 13:35
To: Hingorani, Vineet
Cc: [email protected]
Subject: Re: Example File not running
It should point to your hadoop installation directory. (like C:\hadoop\)
Since you don't have hadoop installed, What is the code that you are running?
Thanks
Best Regards
On Wed, Aug 27, 2014 at 4:50 PM, Hingorani, Vineet
<[email protected]<mailto:[email protected]>> wrote:
What should I put the value of that environment variable? I want to run the
scripts locally on my machine and do not have any Hadoop installed.
Thank you
From: Akhil Das
[mailto:[email protected]<mailto:[email protected]>]
Sent: Mittwoch, 27. August 2014 12:54
To: Hingorani, Vineet
Cc: [email protected]<mailto:[email protected]>
Subject: Re: Example File not running
The statement java.io.IOException: Could not locate executable
null\bin\winutils.exe
explains that the null is received when expanding or replacing an Environment
Variable.
I'm guessing that you are missing HADOOP_HOME in the environment variables.
Thanks
Best Regards
On Wed, Aug 27, 2014 at 3:52 PM, Hingorani, Vineet
<[email protected]<mailto:[email protected]>> wrote:
Hello all,
I am able to use Spark in the shell but I am not able to run a spark file. I am
using sbt and the jar is created but even the SimpleApp class example given on
the site http://spark.apache.org/docs/latest/quick-start.html is not running. I
installed a prebuilt version of spark and >sbt package is compiling the scala
file to jar. I am running it locally on my machine. The error log is huge but
it starts with something like this:
14/08/27 12:14:21 INFO SecurityManager: Using Spark's default log4j profile:
org/apache/sp
ark/log4j-defaults.properties
14/08/27 12:14:21 INFO SecurityManager: Changing view acls to: D062844
14/08/27 12:14:21 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls
disabled; users with view permissions: Set(D062844)
14/08/27 12:14:22 INFO Slf4jLogger: Slf4jLogger started
14/08/27 12:14:22 INFO Remoting: Starting remoting
14/08/27 12:14:22 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://[email protected]:51157<http://[email protected]:51157>]
14/08/27 12:14:22 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://[email protected]
4.74.159:51157]
14/08/27 12:14:22 INFO SparkEnv: Registering MapOutputTracker
14/08/27 12:14:22 INFO SparkEnv: Registering BlockManagerMaster
14/08/27 12:14:22 INFO DiskBlockManager: Created local directory at
C:\Users\D062844\AppDa
ta\Local\Temp\spark-local-20140827121422-dec8
14/08/27 12:14:22 INFO MemoryStore: MemoryStore started with capacity 294.9 MB.
14/08/27 12:14:22 INFO ConnectionManager: Bound socket to port 51160 with id =
ConnectionM
anagerId(10.94.74.159,51160)
14/08/27 12:14:22 INFO BlockManagerMaster: Trying to register BlockManager
14/08/27 12:14:22 INFO BlockManagerInfo: Registering block manager
10.94.74.159:51160<http://10.94.74.159:51160> with
294.9 MB RAM
14/08/27 12:14:22 INFO BlockManagerMaster: Registered BlockManager
14/08/27 12:14:22 INFO HttpServer: Starting HTTP Server
14/08/27 12:14:22 INFO HttpBroadcast: Broadcast server started at
http://10.94.74.159:5116
1
14/08/27 12:14:22 INFO HttpFileServer: HTTP File server directory is
C:\Users\D062844\AppD
ata\Local\Temp\spark-d79d2857-3d85-4b16-8d76-ade83d465f10
14/08/27 12:14:22 INFO HttpServer: Starting HTTP Server
14/08/27 12:14:22 INFO SparkUI: Started SparkUI at http://10.94.74.159:4040
14/08/27 12:14:23 WARN NativeCodeLoader: Unable to load native-hadoop library
for your pla
tform... using builtin-java classes where applicable
14/08/27 12:14:23 INFO SparkContext: Added JAR
file:/C:/Users/D062844/Desktop/HandsOnSpark
/Install/spark-1.0.2-bin-hadoop2/target/scala-2.10/simple-project_2.10-1.0.jar
at
http://10.94.74.159:51162/jars/simple-project_2.10-1.0.jar<http://0.94.74.159:51162/jars/simple-project_2.10-1.0.jar>
with timestamp 1409134463198
14/08/27 12:14:23 INFO MemoryStore: ensureFreeSpace(138763) called with
curMem=0, maxMem=3
09225062
14/08/27 12:14:23 INFO MemoryStore: Block broadcast_0 stored as values to
memory (estimate
d size 135.5 KB, free 294.8 MB)
14/08/27 12:14:23 ERROR Shell: Failed to locate the winutils binary in the
hadoop binary p
ath
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop binar
ies.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:362
)
at
org.apache.spark.SparkContext$$anonfun$22.apply(SparkContext.scala:546)
at
org.apache.spark.SparkContext$$anonfun$22.apply(SparkContext.scala:546)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:145)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:145)
at scala.Option.map(Option.scala:145)
at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:145)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:168)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.FilteredRDD.getPartitions(FilteredRDD.scala:29)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
………………
……………..
It goes on like this and doesn’t show me the result of count in the file. I had
installed pre0built version named Hadoop2. The problem what I think is because
I am running it on local machine and it is not able to find some dependencies
of Hadoop. Please give me what file should I download to work on my local
machine (pre-built, so that I don’t have to build it again).
Thank you
Regards,
Vineet Hingorani
