Just a clarification, I am using Spark ALS explicit feedback on standalone
cluster without deploying zookeeper master HA option yet...
When in the standalone spark cluster, worker fails due to GC error, the
worker dies as well and I have to restart the workerUnderstanding this
issue will be us
Thanks Sean. Looking into executor memory options now...
I am at incubator_spark head. Does that has all the fixes or I need spark
head ? I can deploy the spark head as well...
I am not running implicit feedback yet...I remember memory enhancements
were mainly for implicit right ?
For ulimit let
Much of this sounds related to the memory issue mentioned earlier in this
thread. Are you using a build that has fixed that? That would be by far
most important here.
If the raw memory requirement is 8GB, the actual heap size necessary could
be a lot larger -- object overhead, all the other stuff
Hi,
For our usecases we are looking into 20 x 1M matrices which comes in the
similar ranges as outlined by the paper over here:
http://sandeeptata.blogspot.com/2012/12/sparkler-large-scale-matrix.html
Is the exponential runtime growth in spark ALS as outlined by the blog
still exists in recommen
