Re: Need for advice - performance improvement and out of memory resolution

2016-05-12 Thread AlexModestov
Hello. I'm sorry but did you find the answer? I have the similar error and I can not solve it... No one answered me... Spark driver dies and I get the error "Answer from Java side is empty". I thought that it is so because I made a mistake this conf-file I use Sparkling Water 1.6.3, Spark

Need for advice - performance improvement and out of memory resolution

2015-09-30 Thread Camelia Elena Ciolac
Hello, I am working on a machine learning project, currently using spark-1.4.1-bin-hadoop2.6 in local mode on a laptop (Ubuntu 14.04 OS running on a Dell laptop with i7-5600@2.6 GHz * 4 cores, 15.6 GB RAM). I also mention working in Python from an IPython notebook. I face the following

Need for advice - performance improvement and out of memory resolution

2015-09-30 Thread Camelia Elena Ciolac
Hello, I am working on a machine learning project, currently using spark-1.4.1-bin-hadoop2.6 in local mode on a laptop (Ubuntu 14.04 OS running on a Dell laptop with i7-5600@2.6 GHz * 4 cores, 15.6 GB RAM). I also mention working in Python from an IPython notebook. I face the following

Need for advice - performance improvement and out of memory resolution

2015-09-30 Thread camelia
Hello, I am working on a machine learning project, currently using spark-1.4.1-bin-hadoop2.6 in local mode on a laptop (Ubuntu 14.04 OS running on a Dell laptop with i7-5600@2.6 GHz * 4 cores, 15.6 GB RAM). I also mention working in Python from an IPython notebook. I face the following

RE: Need for advice - performance improvement and out of memory resolution

2015-09-30 Thread Ewan Leith
Try reducing the number of workers to 2, and increasing their memory up to 6GB. However I've seen mention of a bug in the pyspark API for when calling head() on a dataframe in spark 1.5.0 and 1.4, it's got a big performance hit. https://issues.apache.org/jira/browse/SPARK-10731 It's fixed in