How to debug Spark source using IntelliJ/ Eclipse

2015-12-05 Thread jatinganhotra
Hi, I am trying to understand Spark internal code and wanted to debug Spark source, to add a new feature. I have tried the steps lined out here on the Spark Wiki page IDE setup , but they

How can I access data on RDDs?

2015-10-06 Thread jatinganhotra
Consider the following 2 scenarios: *Scenario #1* val pagecounts = sc.textFile("data/pagecounts") pagecounts.checkpoint pagecounts.count *Scenario #2* val pagecounts = sc.textFile("data/pagecounts") pagecounts.count The total time show in the Spark shell Application UI was different for both