Hi,
What kind of error do you receive?
Best regards,
Jörn
Le 24 août 2014 08:29, "Deep Pradhan" <pradhandeep1...@gmail.com> a écrit :
> Hi,
> I was going through the SparkPageRank code and want to see the
> intermediate steps, like the RDDs formed in the intermediate steps.
> Here is a part of the code along with the lines that I added in order to
> print the RDDs.
> I want to print the "*parts*" in the code (denoted by the comment in Bold
> letters). But, when I try to do the same thing there, it gives an error.
> Can someone suggest what I should be doing?
> Thank You
>
> CODE:
>
> object SparkPageRank {
> def main(args: Array[String]) {
> val sparkConf = new SparkConf().setAppName("PageRank")
> var iters = args(1).toInt
> val ctx = new SparkContext(sparkConf)
> val lines = ctx.textFile(args(0), 1)
> println("The lines RDD is")
> lines.collect().foreach(println)
> val links = lines.map{ s =>
> val parts = s.split("\\s+")
> (parts(0), parts(1)) */*I want to print this "parts"*/*
> }.distinct().groupByKey().cache()
> println("The links RDD is")
> links.collect().foreach(println)
> var ranks = links.mapValues(v => 1.0)
> println("The ranks RDD is")
> ranks.collect().foreach(println)
> for (i <- 1 to iters) {
> val contribs = links.join(ranks).values.flatMap{ case (urls, rank) =>
> val size = urls.size
> urls.map(url => (url, rank / size))
> }
> println("The contribs RDD is")
> contribs.collect().foreach(println)
> ranks = contribs.reduceByKey(_ + _).mapValues(0.15 + 0.85 * _)
> }
> println("The second ranks RDD is")
> ranks.collect().foreach(println)
>
> val output = ranks.collect()
> output.foreach(tup => println(tup._1 + " has rank: " + tup._2 + "."))
>
> ctx.stop()
> }
> }
>
>
>
>
