RE: Error: could not find function "includePackage"

2015-10-05 Thread Koen Vantomme
Verzonden vanaf mijn Sony Xperia™-smartphone jayendra.par...@yahoo.in schreef >As mentioned on the website that “includePackage” command can be used to >include existing R packages, but when I am using this command R is giving >this error :- > >Error: could not find function

RE: distributing large matrices

2015-08-07 Thread Koen Vantomme
Verzonden vanaf mijn Sony Xperia™-smartphone iceback schreef Is this the sort of problem spark can accommodate? I need to compare 10,000 matrices with each other (10^10 comparison). The matrices are 100x10 (10^7 int values). I have 10 machines with 2 to 8 cores (8-32

RE: Spark master driver UI: How to keep it after process finished?

2015-08-07 Thread Koen Vantomme
Verzonden vanaf mijn Sony Xperia™-smartphone saif.a.ell...@wellsfargo.com schreef !-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */

Re: miniBatchFraction for LinearRegressionWithSGD

2015-08-07 Thread Koen Vantomme
Verzonden vanaf mijn Sony Xperia™-smartphone Meihua Wu schreef Feynman, thanks for clarifying. If we default miniBatchFraction = (1 / numInstances), then we will only hit one row for every iteration of SGD regardless the number of partitions and executors. In other words the

Question - writing data to Cassandra to Spark gives a strange error message

2015-06-24 Thread Koen Vantomme
Hello, Trying to write data from Spark to Cassandra. Reading data from Cassandra is ok, but writing seems to give a strange error. Exception in thread main scala.ScalaReflectionException: none is not a term at scala.reflect.api.Symbols$SymbolApi$class.asTerm(Symbols.scala:259) The

Spark 1.3 - Connect to to Cassandra - cassandraTable is not recognised by sc

2015-06-22 Thread Koen Vantomme
Hello, I'm writing an application in Scala to connect to Cassandra to read the data. My setup is Intellij with maven. When I try to compile the application I get the following *error: object datastax is not a member of package com* *error: value cassandraTable is not a member of

Re: Spark 1.3 - Connect to to Cassandra - cassandraTable is not recognised by sc

2015-06-22 Thread Koen Vantomme
its fixed now, adding dependecies in pom.xml fixed it dependency groupIdcom.datastax.spark/groupId artifactIdspark-cassandra-connector-embedded_2.10/artifactId version1.4.0-M1/version /dependency On Mon, Jun 22, 2015 at 10:46 AM, Koen Vantomme koen.vanto...@gmail.com wrote: Hello

Cassandra - Spark 1.3 - reading data from cassandra table with PYSpark

2015-06-19 Thread Koen Vantomme
Hello, I'm trying to read data from a table stored in cassandra with pyspark. I found the scala code to loop through the table : cassandra_rdd.toArray.foreach(println) How can this be translated into PySpark ? code snipplet : sc_cass = CassandraSparkContext(conf=conf) cassandra_rdd =

Re: Spark code development practice

2015-03-05 Thread Koen Vantomme
use the spark-shell command and the shell will open type :paste abd then paste your code, after control-d open spark-shell: sparks/bin ./spark-shell Verstuurd vanaf mijn iPhone Op 6-mrt.-2015 om 02:28 heeft fightf...@163.com fightf...@163.com het volgende geschreven: Hi, You can first