Re: lost executor due to large shuffle spill memory

2016-04-06 Thread Michael Slavitch
ambda r: (r[1][0][0], (r[1][1], > > r[1][0][1]))) > > both_reversed = user_reversed.join(product_reverse).map(lambda r: > > (r[1][0][0], r[1][1], r[1][0][1])) > > both_reversed.map(lambda x: '{}|{}|{}'.format(x[0], x[1], > > x[2])).saveAsTextFile(recommendation_path) &g

Re: lost executor due to large shuffle spill memory

2016-04-06 Thread Lishu Liu
mbda u: (u[0], (u[1], > > u[2]))).join(user_reverse).map(lambda r: (r[1][0][0], (r[1][1], > > r[1][0][1]))) > > both_reversed = user_reversed.join(product_reverse).map(lambda r: > > (r[1][0][0], r[1][1], r[1][0][1])) > > both_reversed.map(lambda x: '{}|{}|{}'.format(x[0], x[1], > >

Re: lost executor due to large shuffle spill memory

2016-04-05 Thread Michael Slavitch
: > (r[1][0][0], r[1][1], r[1][0][1])) > both_reversed.map(lambda x: '{}|{}|{}'.format(x[0], x[1], > x[2])).saveAsTextFile(recommendation_path) > > Both user and products are (uuid, index) tuples. > > > > -- > View this message in context: > http://apache-sp

lost executor due to large shuffle spill memory

2016-04-05 Thread lllll
ndation_path) Both user and products are (uuid, index) tuples. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/lost-executor-due-to-large-shuffle-spill-memory-tp26683.html Sent from the Apache Spark User List mailing list archive a