Executor memory : 45g X 4 executors , 1 Driver with 45g memory Data Source is from S3 and I've logs that tells me the Rating objects are loaded fine.
On Fri, Mar 11, 2016 at 2:13 PM, Nick Pentreath <nick.pentre...@gmail.com> wrote: > Hmmm, something else is going on there. What data source are you reading > from? How much driver and executor memory have you provided to Spark? > > > > On Fri, 11 Mar 2016 at 09:21 Deepak Gopalakrishnan <dgk...@gmail.com> > wrote: > >> 1. I'm using about 1 million users against few thousand products. I >> basically have around a million ratings >> 2. Spark 1.6 on Amazon EMR >> >> On Fri, Mar 11, 2016 at 12:46 PM, Nick Pentreath < >> nick.pentre...@gmail.com> wrote: >> >>> Could you provide more details about: >>> 1. Data set size (# ratings, # users and # products) >>> 2. Spark cluster set up and version >>> >>> Thanks >>> >>> On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dgk...@gmail.com> >>> wrote: >>> >>>> Hello All, >>>> >>>> I've been running Spark's ALS on a dataset of users and rated items. I >>>> first encode my users to integers by using an auto increment function ( >>>> just like zipWithIndex), I do the same for my items. I then create an RDD >>>> of the ratings and feed it to ALS. >>>> >>>> My issue is that the ALS algorithm never completes. Attached is a >>>> screenshot of the stages window. >>>> >>>> Any help will be greatly appreciated >>>> >>>> -- >>>> Regards, >>>> *Deepak Gopalakrishnan* >>>> *Mobile*:+918891509774 >>>> *Skype* : deepakgk87 >>>> http://myexps.blogspot.com >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >> >> -- >> Regards, >> *Deepak Gopalakrishnan* >> *Mobile*:+918891509774 >> *Skype* : deepakgk87 >> http://myexps.blogspot.com >> >> -- Regards, *Deepak Gopalakrishnan* *Mobile*:+918891509774 *Skype* : deepakgk87 http://myexps.blogspot.com
