I recently started using Spark version 1.3.0 in standalone mode (with Scala 2.10.3), and I'm running into an odd problem. I'm loading data from a file using sc.textFile, doing some conversion of the data, and then clustering it. When I do this with a small file (10 lines, 9 KB), it works fine, and the program terminates.
However, when I load my full data file (400,000 lines, 167 MB), the process hangs sometime after the last line of my main() method. (The last line is a print statement, and it gets printed.) I am creating quite a few objects, so I don't know if perhaps the garbage collection is just taking a really long time? (5+ minutes; I haven't had the patience to let it go longer than that.) Is there anything I should try to fix this, or to help diagnose the issue? Thanks, Michael -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-hanging-after-main-method-completes-tp22544.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
