Hi, What kind of error do you receive?
Best regards, Jörn Le 24 août 2014 08:29, "Deep Pradhan" <pradhandeep1...@gmail.com> a écrit : > Hi, > I was going through the SparkPageRank code and want to see the > intermediate steps, like the RDDs formed in the intermediate steps. > Here is a part of the code along with the lines that I added in order to > print the RDDs. > I want to print the "*parts*" in the code (denoted by the comment in Bold > letters). But, when I try to do the same thing there, it gives an error. > Can someone suggest what I should be doing? > Thank You > > CODE: > > object SparkPageRank { > def main(args: Array[String]) { > val sparkConf = new SparkConf().setAppName("PageRank") > var iters = args(1).toInt > val ctx = new SparkContext(sparkConf) > val lines = ctx.textFile(args(0), 1) > println("The lines RDD is") > lines.collect().foreach(println) > val links = lines.map{ s => > val parts = s.split("\\s+") > (parts(0), parts(1)) */*I want to print this "parts"*/* > }.distinct().groupByKey().cache() > println("The links RDD is") > links.collect().foreach(println) > var ranks = links.mapValues(v => 1.0) > println("The ranks RDD is") > ranks.collect().foreach(println) > for (i <- 1 to iters) { > val contribs = links.join(ranks).values.flatMap{ case (urls, rank) => > val size = urls.size > urls.map(url => (url, rank / size)) > } > println("The contribs RDD is") > contribs.collect().foreach(println) > ranks = contribs.reduceByKey(_ + _).mapValues(0.15 + 0.85 * _) > } > println("The second ranks RDD is") > ranks.collect().foreach(println) > > val output = ranks.collect() > output.foreach(tup => println(tup._1 + " has rank: " + tup._2 + ".")) > > ctx.stop() > } > } > > > >