Re: error when running spark from oozie launcher

It sounds like a mismatch in the spark version ship in oozie and the runtime one. Regards JB On Aug 18, 2016, 07:36, at 07:36, tkg_cangkul wrote: >hi olivier, thx for your reply. > >this is the full stacktrace : > >Failing Oozie Launcher, Main class

Re: Aggregations with scala pairs

Agreed. Regards JB On Aug 18, 2016, 07:32, at 07:32, Olivier Girardot wrote: >CC'ing dev list, you should open a Jira and a PR related to it to >discuss it c.f.

Re: Spark Streaming UI duration numbers mismatch

-- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Apache Beam Spark runner

! The Apache Beam (http://beam.incubator.apache.org/) project is a unified model for building data pipelines using Google’s Dataflow programming model, and now it supports Spark as well! Take it for a ride on your Spark cluster! Thanks, Amit -- Jean-Baptiste Onofré jbono...@apache.org http

Re: Double Counting When Using Accumulators with Spark Streaming

IS "+ *accum.value(*)); *return**null*;} }); jssc.start(); If I comment rdd.saveAsTextFile I get correct count, but with rdd.saveAsTextFile for each input I am getting multiple accumulator count. Thanks, Rachana -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talen

Re: Is Spark 1.6 released?

-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Is Spark 1.6 released?

as a stable release? Thanks Jung -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail

Re: Download Problem with Spark 1.5.2 pre-built for Hadoop 1.X

-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user

Re: Compiling ERROR for Spark MetricsSystem

d.java:606) at com.martiansoftware.nailgun.NGSession.run(NGSession.java:319) The IDE I'm using is Jetbrains IntelliJIDea 15 CE. I'm suspecting it's classpath issue but not sure how to further debug. Thanks Haijia -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend

Re: Spark Submit - java.lang.IllegalArgumentException: requirement failed

.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For add

Re: Spark streaming with Kinesis broken?

1.5.1. Can someone please attempt to reproduce before I open a JIRA issue for it? -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: u

Re: How to test https://issues.apache.org/jira/browse/SPARK-10648 fix

-- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: How and where to update release notes for spark rel 1.6?

1.6? pls help. There are a few methods with changed params, and a few deprecated ones that need to be documented. Thanks Ravi -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Standalone cluster not using multiple workers for single application

. If you are not a designated recipient, you may not review, use, copy or distribute this message. If you received this in error, please notify the sender by reply e-mail and delete this message. -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Error : - No filesystem for scheme: spark

> # spark.driver.memory 5g > # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three" > > > regards > Bala > > On 2 November 2015 at 12:21, Je

Re: Error : - No filesystem for scheme: spark

-2015 3:15 pm, "Balachandar R.A." <balachandar...@gmail.com <mailto:balachandar...@gmail.com>> wrote: No.. I am not using yarn. Yarn is not running in my cluster. So, it is standalone one. Regards Bala On 02-Nov-2015 3:11 pm, "Jean-Baptiste

Re: Error : - No filesystem for scheme: spark

eSystemClass(FileSystem.java:2584)at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)| Any help here to move towards fixing this will be of great help Thanks Bala -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.n

Re: spark-1.5.1 application detail ui url

the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http

RE: Spark on Yarn

Hi The compiled version (master side) and client version diverge on spark network JavaUtils. You should use the same/aligned version. RegardsJB Sent from my Samsung device Original message From: Raghuveer Chanda Date: 21/10/2015 12:33

Re: Spark 1.5 Streaming and Kinesis

instead of the runtime type. Cheers, Eugen 2015-10-15 13:41 GMT+07:00 Jean-Baptiste Onofré <j...@nanthrax.net <mailto:j...@nanthrax.net> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>>

Re: Spark 1.5 Streaming and Kinesis

hil On Thu, Oct 15, 2015 at 10:35 PM, Jean-Baptiste Onofré <j...@nanthrax.net <mailto:j...@nanthrax.net>> wrote: Hi Phil, sorry I didn't have time to investigate yesterday (I was on a couple of other Apache projects ;)). I will try to do it today. I keep you posted.

Re: Spark 1.5 Streaming and Kinesis

Hi Phil, as you can see in the Jira, I tried with both Spark 1.5.1 and 1.6.0-SNAPSHOT, and I'm not able to reproduce your issue. What's your Java version (and I guess you use scala 2.10 for compilation, not -Pscala-2.11) ? Regards JB On 10/20/2015 03:59 PM, Jean-Baptiste Onofré wrote: Hi

Re: Can not subscript to mailing list

Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://b

Re: Spark 1.5 Streaming and Kinesis

ning the streaming job locally, and also in EMR with release 4.1.0 that includes Spark 1.5 Very strange! -- Forwarded message ------ From: "Jean-Baptiste Onofré" <j...@nanthrax.net <mailto:j...@nanthrax.net>> To: user@spark.apache.org <mailto:user@sp

Re: Spark 1.5 Streaming and Kinesis

initely happen depending on the lib used and its configuration that it just doesn't preserve the concrete type. So it would deserialize using the compile type instead of the runtime type. Cheers, Eugen 2015-10-15 13:41 GMT+07:00 Jean-Baptiste O

Re: Spark 1.5 Streaming and Kinesis

://spark.apache.org/docs/latest/streaming-kinesis-integration.html with bin/run-example streaming.KinesisWordCountASL Am I doing something incorrect? -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com

Re: Spark 1.5 Streaming and Kinesis

By correct, I mean: the map declaration looks good to me, so the ClassCastException is weird ;) I'm trying to reproduce the issue in order to investigate. Regards JB On 10/15/2015 08:03 AM, Jean-Baptiste Onofré wrote: Hi Phil, KinesisReceiver is part of extra. Just a dumb question: did you

Re: Why is my spark executor is terminated?

, Wang, Ningjun (LNG-NPV) wrote: I checked master log before and did not find anything wrong. Unfortunately I have lost the master log now. So you think master log will tell you why executor is down? Regards, Ningjun Wang -Original Message- From: Jean-Baptiste Onofré [mailto:j..

Re: Why is the Spark Web GUI failing with JavaScript "Uncaught SyntaxError"?

that my browser proxy is not the cause of this problem. Joshua -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org F

Re: Why is the Spark Web GUI failing with JavaScript "Uncaught SyntaxError"?

Thanks for the update Joshua. Let me try with Spark 1.4.1. I keep you posted. Regards JB On 10/13/2015 04:17 PM, Joshua Fox wrote: * Spark 1.4.1, part of EMR emr-4.0.0 * Chrome Version 41.0.2272.118 (64-bit) on Ubuntu On Tue, Oct 13, 2015 at 3:27 PM, Jean-Baptiste Onofré &l

Re: Spark shuffle service does not work in stand alone

eader.load(DataFrameReader.scala:120) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104) at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:219) at org.apache.saif.loaders.DataLoader$.load_json(DataLoader.scala:86) -- Jean-Baptiste Onofré jbono...@apach

Re: Why is my spark executor is terminated?

- Thanks Ningjun Wang -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h

Re: Does Spark use more memory than MapReduce?

at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http

Re: Build Failure

r-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Running Spark in Yarn-client mode

but are unable to connect back to the driver. How do I resolve this? Also, I need help in locating the driver and executor node logs. Thanks. Regards, Sushrut Ikhar https://about.me/sushrutikhar <https://about.me/sushrutikhar?promo=email_sig> -- Jean-Baptiste Onofré jbono...@apache.or

Re: Help needed to reproduce bug

5.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré

Re: OutOfMemoryError

2 - 32 GB memory ). my input data of size 128 GB. How to solve this exception ? is it depends on driver.memory and execuitor.memory setting ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://ww

Re: DStream Transformation to save JSON in Cassandra 2.1

nformation transmitted by this email including damage from virus." --------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Jean-Baptiste Onofré jbono...@