Re: DataFrame Sort gives Cannot allocate a page with more than 17179869176 bytes

You can get some more insights by using the Spark history server (http://spark.apache.org/docs/latest/monitoring.html), it can show you which task is failing and some other information that might help you debugging the issue. On 05/10/2016 19:00, Babak Alipour wrote: > The issue seems to lie in

Spark ML Interaction

Hi, Did anyone here manage to write an example of the following ML feature transformer http://spark.apache.org/docs/latest/api/java/org/apache/spark/ml/feature/Interaction.html ? It's not documented on the official Spark ML features pages but it can be found in the package API javadocs. Thanks,

Dynamic jar loading

Hello guys, Do you know if the method SparkContext.addJar("file:///...") can be used on a running context (an already started spark-shell) ? And if so, does it add the jar to the class-path of the Spark workers (Yarn containers in case of yarn-client) ? Thanks,

Re: Database does not exist: (Spark-SQL ===> Hive)

Can you test with latest version of spark ? I had the same issue with 1.3 and it was resolved 1.5. On 15/12/2015 04:31, Jeff Zhang wrote: > Do you put hive-site.xml on the classpath ? > > On Tue, Dec 15, 2015 at 11:14 AM, Gokula Krishnan D > >

Re: Save RandomForest Model from ML package

It's an open issue : https://issues.apache.org/jira/browse/SPARK-4587 That's being said, you can workaround the issue by serializing the Model (simple java serialization) and then restoring it before calling the predicition job. Best Regards, On 22/10/2015 14:33, Sebastian Kuepers wrote: >