Is There Any Benchmarks Comparing C++ MPI with Spark

2014-06-16 Thread Wei Da
Hi guys, We are making choices between C++ MPI and Spark. Is there any official comparation between them? Thanks a lot! Wei

Is there a step-by-step instruction on how to build Spark App with IntelliJ IDEA?

2014-06-01 Thread Wei Da
Hi guys, I'm using IntelliJ IDEA 13.1.2 Community Edition, and I have installed Scala plugin and Maven 3.2.1. I want to develop Spark applications with IntelliJ IDEA through Maven. In IntelliJ, I create a Maven project with the archetype ID spark-core_2.10, but got the following messages in the

Re: How to stop system info output in spark shell

2014-04-15 Thread Wei Da
The solution: Edit /opt/spark-0.9.0-incubating-bin-hadoop2/conf/log4j.properties, changing Spark's output to WARN. Done! Refer to: https://github.com/amplab-extras/SparkR-pkg/blob/master/pkg/src/src/main/resources/log4j.properties#L8 eduardocalfaia wrote Have you already tried in