Re: Disable all spark logging

. -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80

Re: Execution blocked when collect()ing some relatively big blocks on spark 0.9

-- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 eXenSa S.A.S. 41, rue Prier - 92120 Montrouge - FRANCE Tel +33(0)1

Re: Execution blocked when collect()ing some relatively big blocks on spark 0.9

? It should be logged -- Guillaume PITEL, Prsident +33(0)6

Re: How to efficiently join this two complicated rdds

;,("L1",11)) ("2",("L1",22)) ("3",("L1",33)) ("1",("L2",11)) ("3",("L2",33)) ("5",("L2",55)) --

Execution blocked when collect()ing some relatively big blocks on spark 0.9

Guillaume -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80

Re: Execution blocked when collect()ing some relatively big blocks on spark 0.9

-- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 eXenSa

Re: How to efficiently join this two complicated rdds

the (key, value) if value is zero. And maybe the type one RDD has a lot key numbers of 1 but a few of 15877. We want to fine a fast way to solve this problem. Sincerely thanks Bo Han . -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-efficiently-join-this-two-complicated-rdds-tp1665.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark Cluster Size

for 'Production' usecase? -Bharath -- Guillaume PITEL, Prsident

Re: Performance and serialization: use case

Pierre -- Guillaume

Re: Interleaving stages

ry pattern? -- Guillaume PITEL

Shuffle file not found Exception

related to the # of file descriptors Guillaume -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 / +33(0)9

Re: Shuffle file not found Exception

which is causing a file to not be written -- I think I've seen latent OOMEs induce this behavior, for instance. On Sun, Feb 9, 2014 at 8:28 AM, Guillaume Pitel guillaume.pi...@exensa.com wrote

Re: How to create RDD over hashmap?

= sc.parallelize(cr)                                    ^ -- Guillaume PITEL, Président

Advices if your worker die often

cancelling a job : sys.addShutdownHook( { sc.stop() } ) Hope this can be useful to someone Guillaume -- Guillaume

Re: Forcing RDD computation with something else than count() ?

Thanks so much ! Guillaume sc.runJob(rdd, (iter: Iterator[_]) = {}) -- Guillaume

Forcing RDD computation with something else than count() ?

sically call newAPIHadoopFile and saveAsNewAPIHadoopFile -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80

Re: Forcing RDD computation with something else than count() ?

. -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80

Re: Dying workers since migration to 0.8.1

is often unpredictable. -- Guillaume PITEL

Re: Dying workers since migration to 0.8.1

. -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80

Troubles with the Spark-EC2 stuff

-- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 / +33(0)9 70 44 67 53 eXenSa S.A.S. 41, rue Prier - 92120 Montrouge

Re: Troubles with the Spark-EC2 stuff

-- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 / +33(0)9 70 44 67 53

Re: Turning kryo on does not decrease binary output

yo support. Any idea why my simple tuple type should not get kryo benefits? -- Guillaume PITEL, Prsident

Re: Turning kryo on does not decrease binary output

is the right place to set these properties? On Fri, Jan 3, 2014 at 4:56 PM, Guillaume Pitel guillaume.pi...@exensa.com wrote: Hi, I believe Kryo is only use during RDD

Re: Turning kryo on does not decrease binary output

RDD[T]#saveAsObjectFile()? -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 / +33(0)9 7

Re: Turning kryo on does not decrease binary output

saveAsObjectFile? -- Guillaume PITEL, Prsident

Re: Turning kryo on does not decrease binary output

(according to Guillaume) for saveAsHadoopFile()? -- Guillaume PITEL

Re: How to map each line to (line number, line)?

. -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 / +33(0)9 70 44 67 53 eXenSa S.A.S

Re: How to map each line to (line number, line)?

. -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 eXenSa S.A.S. 41, rue Prier - 92120 Montrouge - FRANCE Tel +33(0)1 84

Re: How to map each line to (line number, line)?

) } } -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 eXenSa S.A.S. 41, rue Prier - 92120 Montrouge - FRANCE Tel +33(0)1

Re: How to map each line to (line number, line)?

} } -- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 / +33(0)9 70 44 67 53 eXenSa S.A.S. 41, rue Prier - 92120

DoubleMatrix vs Array[Array[Double]] : Question about debugging serialization performance issues

-- Guillaume PITEL, Prsident +33(0)6 25 48 86 80 eXenSa S.A.S. 41, rue Prier

Re: DoubleMatrix vs Array[Array[Double]] : Question about debugging serialization performance issues

Guillaume -- Guillaume PITEL, Président +33(0)6 25 48 86 80 / +33(0)9 70 44 67 53 eXenSa S.A.S. 41,

Random StreamCorruptedException during task execution

pen, but the process takes significantly longer to finish, especially for the second stage (after the reduceByKey) Any help would be very appreciated Cheers, Guillaume --