I have a task to remap the index to actual uuid in ALS prediction results. But it consistently fail due to lost executors. I noticed there's large shuffle spill memory but I don't know how to improve it.
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n26683/24.png> I've tried to reduce the number of executors while assigning each to have bigger memory. <http://apache-spark-user-list.1001560.n3.nabble.com/file/n26683/31.png> But it still doesn't seem big enough. I don't know what to do. Below is my code: user = load_user() product = load_product() user.cache() product.cache() model = load_model(model_path) all_pairs = user.map(lambda x: x[1]).cartesian(product.map(lambda x: x[1])) all_prediction = model.predictAll(all_pairs) user_reverse = user.map(lambda r: (r[1], r[0])) product_reverse = product.map(lambda r: (r[1], r[0])) user_reversed = all_prediction.map(lambda u: (u[0], (u[1], u[2]))).join(user_reverse).map(lambda r: (r[1][0][0], (r[1][1], r[1][0][1]))) both_reversed = user_reversed.join(product_reverse).map(lambda r: (r[1][0][0], r[1][1], r[1][0][1])) both_reversed.map(lambda x: '{}|{}|{}'.format(x[0], x[1], x[2])).saveAsTextFile(recommendation_path) Both user and products are (uuid, index) tuples. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/lost-executor-due-to-large-shuffle-spill-memory-tp26683.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
