Hi all, I only have one stage which is "mapToPair" and inside the function, I have a for loop which will do about 133433 times.
But then it becomes slow, when I replace 133433 with just 133, it works very fast. But I think this is just a simple operation even in normal Java. You can look at the code in my github. (https://github.com/allanj/myspark/blob/master/sparktest.java) When I replace the for loop size 133433 with 133, it runs fast. Otherwise, very very slow, I can see it from the web console. The input records increase very slowly. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/The-stage-slow-when-I-have-for-loop-inside-Java-tp23010.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
