What about a simple call to nanotime? long startTime = System.nanoTime();
//Spark work here long endTime = System.nanoTime(); long duration = (endTime - startTime) println(duration) Count recomputes the df so it makes sense it takes longer for you. ---- On Tue, 02 Apr 2019 07:06:30 -0700 koloka...@ics.forth.gr wrote ---- Hello, I want to ask if there any way to measure HDFS data loading time at the start of my program. I tried to add an action e.g count() after val data = sc.textFile() call. But I notice that my program takes more time to finish than before adding count call. Is there any other way to do it ? Thanks, --Iacovos --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
