Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions

> On 5 Oct 2015, at 16:48, Alex Rovner wrote: > > Hey Steve, > > Are you referring to the 1.5 version of the history server? > Yes. I should warn, however, that there's no guarantee that a history server running the 1.4 code will handle the histories of a 1.5+

Re: Spark on YARN using Java 1.8 fails

YARN 2.7.1 (running on the cluster) was built with Java 1.8, I assume. Have you used the following command to retrieve / inspect logs ? yarn logs -applicationId Cheers On Mon, Oct 5, 2015 at 8:41 AM, mvle wrote: > Hi, > > I have successfully run pyspark on Spark 1.5.1 on YARN

Re: Spark on YARN / aws - executor lost on node restart

lto:user@spark.apache.org>" Subject: Re: Spark on YARN / aws - executor lost on node restart Hi guys, Digging up this question after spending some more time trying to replicate it. It seems to be an issue with the YARN – spark integration, wondering if there is a bug already tracking this?

Re: Spark on Yarn vs Standalone

I think you need to increase the memory size of executor through command arguments "--executor-memory", or configuration "spark.executor.memory". Also yarn.scheduler.maximum-allocation-mb in Yarn side if necessary. Thanks Saisai On Mon, Sep 21, 2015 at 5:13 PM, Alexander Pivovarov

Re: Spark on Yarn vs Standalone

The warning your seeing in Spark is no issue. The scratch space lives inside the heap, so it'll never result in YARN killing the container by itself. The issue is that Spark is using some off-heap space on top of that. You'll need to bump the spark.yarn.executor.memoryOverhead property to give

Re: Spark on Yarn vs Standalone

I repartitioned input RDD from 4,800 to 24,000 partitions After that the stage (24000 tasks) was done in 22 min on 100 boxes Shuffle read/write: 905 GB / 710 GB Task Metrics (Dur/GC/Read/Write) Min: 7s/1s/38MB/30MB Med: 22s/9s/38MB/30MB Max:1.8min/1.6min/38MB/30MB On Mon, Sep 21, 2015 at 5:55

Re: Spark on Yarn vs Standalone

I noticed that some executors have issue with scratch space. I see the following in yarn app container stderr around the time when yarn killed the executor because it uses too much memory. -- App container stderr -- 15/09/21 21:43:22 WARN storage.MemoryStore: Not enough space to cache rdd_6_346

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

Hi Nick/Igor, ​​ Any solution for this ? Even I am having the same issue and copying jar to each executor is not feasible if we use lot of jars. Thanks, Vipul

Re: Spark on YARN / aws - executor lost on node restart

Hi guys, Digging up this question after spending some more time trying to replicate it. It seems to be an issue with the YARN – spark integration, wondering if there is a bug already tracking this? If I just kill the process on the machine, YARN detects the container is dead and the spark

Re: Spark w/YARN Scheduling Questions...

Task set is a set of tasks within one stage. Executor will be killed when it is idle for a period of time (default is 60s). The problem you mentioned is bug, scheduler should not allocate tasks on this to-be killed executors. I think it is fixed in 1.5. Thanks Saisai On Thu, Sep 17, 2015 at

Re: Spark on Yarn vs Standalone

YARN will never kill processes for being unresponsive. It may kill processes for occupying more memory than it allows. To get around this, you can either bump spark.yarn.executor.memoryOverhead or turn off the memory checks entirely with yarn.nodemanager.pmem-check-enabled. -Sandy On Tue, Sep

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

as a starting point, attach your stacktrace... ps: look for duplicates in your classpath, maybe you include another jar with same class On 8 September 2015 at 06:38, Nicholas R. Peterson wrote: > I'm trying to run a Spark 1.4.1 job on my CDH5.4 cluster, through Yarn. >

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

Thans, Igor; I've got it running again right now, and can attach the stack trace when it finishes. In the mean time, I've noticed something interesting: in the Spark UI, the application jar that I submit is not being included on the classpath. It has been successfully uploaded to the nodes -- in

Re: Spark on Yarn vs Standalone

Those settings seem reasonable to me. Are you observing performance that's worse than you would expect? -Sandy On Mon, Sep 7, 2015 at 11:22 AM, Alexander Pivovarov wrote: > Hi Sandy > > Thank you for your reply > Currently we use r3.2xlarge boxes (vCPU: 8, Mem: 61 GiB) >

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

Yes, the jar contains the class: $ jar -tf lumiata-evaluation-assembly-1.0.jar | grep 2028/Document/Document com/i2028/Document/Document$1.class com/i2028/Document/Document.class What else can I do? Is there any way to get more information about the classes available to the particular

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

Here is the stack trace: (Sorry for the duplicate, Igor -- I forgot to include the list.) 15/09/08 05:56:43 WARN scheduler.TaskSetManager: Lost task 183.0 in stage 41.0 (TID 193386, ds-compute2.lumiata.com): java.io.IOException: com.esotericsoftware.kryo.KryoException: Error constructing

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

java.lang.ClassNotFoundException: com.i2028.Document.Document 1. so have you checked that jar that you create(fat jar) contains this class? 2. might be there is some stale cache issue...not sure though On 8 September 2015 at 16:12, Nicholas R. Peterson wrote: > Here is

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

hmm...out of ideas. can you check in spark ui environment tab that this jar is not somehow appears 2 times or more...? or more generally - any 2 jars that can contain this class by any chance regarding your question about classloader - no idea, probably there is, I remember stackoverflow has some

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

another idea - you can add this fat jar explicitly to the classpath of executors...it's not a solution, but might be it work... I mean place it somewhere locally on executors and add it to cp with spark.executor.extraClassPath On 8 September 2015 at 18:30, Nick Peterson

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

Yeah... none of the jars listed on the classpath contain this class. The only jar that does is the fat jar that I'm submitting with spark-submit, which as mentioned isn't showing up on the classpath anywhere. -- Nick On Tue, Sep 8, 2015 at 8:26 AM Igor Berman wrote: >

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

Yes, putting the jar on each node and adding it manually to the executor classpath does it. So, it seems that's where the issue lies. I'll do some experimenting and see if I can narrow down the problem; but, for now, at least I can run my job! Thanks for your help. On Tue, Sep 8, 2015 at 8:40

Re: Spark on Yarn vs Standalone

The problem which we have now is skew data (2360 tasks done in 5 min, 3 tasks in 40 min and 1 task in 2 hours) Some people from the team worry that the executor which runs the longest task can be killed by YARN (because executor might be unresponsive because of GC or it might occupy more memory

Re: Spark on Yarn vs Standalone

Hi Alex, If they're both configured correctly, there's no reason that Spark Standalone should provide performance or memory improvement over Spark on YARN. -Sandy On Fri, Sep 4, 2015 at 1:24 PM, Alexander Pivovarov wrote: > Hi Everyone > > We are trying the latest aws

Re: Spark on Yarn vs Standalone

Hi Sandy Thank you for your reply Currently we use r3.2xlarge boxes (vCPU: 8, Mem: 61 GiB) with emr setting for Spark "maximizeResourceAllocation": "true" It is automatically converted to Spark settings spark.executor.memory47924M spark.yarn.executor.memoryOverhead 5324 we also set

Re: Spark-on-YARN LOCAL_DIRS location

Yes, you can set the SPARK_LOCAL_DIR in the spark-env.sh or spark.local.dir in the spark-defaults.conf file, then it would use this location for the shuffle writes etc. Thanks Best Regards On Wed, Aug 26, 2015 at 6:56 PM, michael.engl...@nomura.com wrote: Hi, I am having issues with /tmp

Re: Spark on YARN

Hi, I have looked at the UI scheduler tab and it appears my new user was allocated less cores than my other user, is there any way i can avoid this happening? Thanks, Jem On Sat, Aug 8, 2015 at 8:32 PM Shushant Arora shushantaror...@gmail.com wrote: which is the scheduler on your cluster.

Re: Spark on YARN

Hi Jem, Do they fail with any particular exception? Does YARN just never end up giving them resources? Does an application master start? If so, what are in its logs? If not, anything suspicious in the YARN ResourceManager logs? -Sandy On Fri, Aug 7, 2015 at 1:48 AM, Jem Tucker

Re: Spark on YARN

Hi Sandy, The application doesn't fail, it gets accepted by yarn but the application master never starts and the application state never changes to running. I have checked in the resource manager and node manager logs and nothing jumps out. Thanks Jem On Sat, 8 Aug 2015 at 09:20, Sandy Ryza

Re: Spark on YARN

Hi dustin, Yes there are enough resources available, the same application run with a different user works fine so I think it is something to do with permissions but I can't work out where. Thanks , Jem On Sat, 8 Aug 2015 at 17:35, Dustin Cote dc...@cloudera.com wrote: Hi Jem, In the top of

Re: Spark on YARN

which is the scheduler on your cluster. Just check on RM UI scheduler tab and see your user and max limit of vcores for that user , is currently other applications of that user have occupies till max vcores of this user then that could be the reason of not allocating vcores to this user but for

Re: Spark on YARN

it. If you met similar problem, you could increase this configuration “yarn.nodemanager.vmem-pmem-ratio”. Thanks Jerry *From:* Jeff Zhang [mailto:zjf...@gmail.com] *Sent:* Thursday, July 30, 2015 4:36 PM *To:* Jeetendra Gangele *Cc:* user *Subject:* Re: Spark on YARN 15/07/30 12:13:35

Re: Spark on YARN

I can't see the application logs here. All the logs are going into stderr. can anybody help here? On 30 July 2015 at 12:21, Jeetendra Gangele gangele...@gmail.com wrote: I am running below command this is default spark PI program but this is not running all the log are going in stderr but at

Re: Spark on YARN

15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM AM is killed somehow, may due to preemption. Does it always happen ? Resource manager log would be helpful. On Thu, Jul 30, 2015 at 4:17 PM, Jeetendra Gangele gangele...@gmail.com wrote: I can't see the application

RE: Spark on YARN

Gangele Cc: user Subject: Re: Spark on YARN 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM AM is killed somehow, may due to preemption. Does it always happen ? Resource manager log would be helpful. On Thu, Jul 30, 2015 at 4:17 PM, Jeetendra Gangele gangele

Re: spark on yarn

On Tue, Jul 14, 2015 at 9:57 AM, Shushant Arora shushantaror...@gmail.com wrote: When I specify --executor-cores 4 it fails to start the application. When I give --executor-cores as 4 , it works fine. Do you have any NM that advertises more than 4 available cores? Also, it's always worth it

Re: spark on yarn

Ok thanks a lot! few more doubts : What happens in a streaming application say with spark-submit --class classname --num-executors 10 --executor-cores 4 --master masteradd jarname Will it allocate 10 containers throughout the life of streaming application on same nodes until any node failure

Re: spark on yarn

On Tue, Jul 14, 2015 at 11:13 AM, Shushant Arora shushantaror...@gmail.com wrote: spark-submit --class classname --num-executors 10 --executor-cores 4 --master masteradd jarname Will it allocate 10 containers throughout the life of streaming application on same nodes until any node failure

Re: spark on yarn

On Tue, Jul 14, 2015 at 12:03 PM, Shushant Arora shushantaror...@gmail.com wrote: Can a container have multiple JVMs running in YARN? Yes and no. A container runs a single command, but that process can start other processes, and those also count towards the resource usage of the container

Re: spark on yarn

Can a container have multiple JVMs running in YARN? I am comparing Hadoop Mapreduce running on yarn vs spark running on yarn here : 1.Is the difference is in Hadoop Mapreduce job - say I specify 20 reducers and my job uses 10 map tasks then, it need total 30 containers or 30 vcores ? I guess 30

Re: spark on yarn

On Tue, Jul 14, 2015 at 10:40 AM, Shushant Arora shushantaror...@gmail.com wrote: My understanding was --executor-cores(5 here) are maximum concurrent tasks possible in an executor and --num-executors (10 here)are no of executors or containers demanded by Application master/Spark driver

Re: spark on yarn

Is yarn.scheduler.maximum-allocation-vcores the setting for max vcores per container? Whats the setting for max limit of --num-executors ? On Tue, Jul 14, 2015 at 11:18 PM, Marcelo Vanzin van...@cloudera.com wrote: On Tue, Jul 14, 2015 at 10:40 AM, Shushant Arora shushantaror...@gmail.com

Re: spark on yarn

Shushant : Please also see 'Debugging your Application' section of https://spark.apache.org/docs/latest/running-on-yarn.html On Tue, Jul 14, 2015 at 10:48 AM, Marcelo Vanzin van...@cloudera.com wrote: On Tue, Jul 14, 2015 at 10:40 AM, Shushant Arora shushantaror...@gmail.com wrote: My

Re: spark on yarn

On Tue, Jul 14, 2015 at 10:55 AM, Shushant Arora shushantaror...@gmail.com wrote: Is yarn.scheduler.maximum-allocation-vcores the setting for max vcores per container? I don't remember YARN config names by heart, but that sounds promising. I'd look at the YARN documentation for details.

Re: spark on yarn

got the below exception in logs: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requested virtual cores 0, or requested virtual cores max configured, requestedVirtualCores=5, maxVirtualCores=4 at

Re: Spark on Yarn - How to configure

Hi Ashish, For Spark on YARN, you actually only need the Spark files on one machine - the submission client. This machine could even live outside of the cluster. Then all you need to do is point YARN_CONF_DIR to the directory containing your hadoop configuration files (e.g. yarn-site.xml) on that

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi, We switched from ParallelGC to CMS, and the symptom is gone. On Thu, Jun 4, 2015 at 3:37 PM, Ji ZHANG zhangj...@gmail.com wrote: Hi, I set spark.shuffle.io.preferDirectBufs to false in SparkConf and this setting can be seen in web ui's environment tab. But, it still eats memory, i.e.

Re: Spark-sql(yarn-client) java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/ExecutorLauncher

btw, user listt will be a better place for this thread. On Thu, Jun 18, 2015 at 8:19 AM, Yin Huai yh...@databricks.com wrote: Is it the full stack trace? On Thu, Jun 18, 2015 at 6:39 AM, Sea 261810...@qq.com wrote: Hi, all: I want to run spark sql on yarn(yarn-client), but ... I already

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Glad to hear that. :) On Thu, Jun 18, 2015 at 6:25 AM, Ji ZHANG zhangj...@gmail.com wrote: Hi, We switched from ParallelGC to CMS, and the symptom is gone. On Thu, Jun 4, 2015 at 3:37 PM, Ji ZHANG zhangj...@gmail.com wrote: Hi, I set spark.shuffle.io.preferDirectBufs to false in

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi, I set spark.shuffle.io.preferDirectBufs to false in SparkConf and this setting can be seen in web ui's environment tab. But, it still eats memory, i.e. -Xmx set to 512M but RES grows to 1.5G in half a day. On Wed, Jun 3, 2015 at 12:02 PM, Shixiong Zhu zsxw...@gmail.com wrote: Could you

Re: Spark 1.4 YARN Application Master fails with 500 connect refused

Just testing with Spark 1.3, it looks like it sets the proxy correctly to be the YARN RM host (0101) 15/06/03 10:34:19 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT] 15/06/03 10:34:20 INFO yarn.ApplicationMaster: ApplicationAttemptId:

Re: Spark 1.4 YARN Application Master fails with 500 connect refused

That code hasn't changed at all between 1.3 and 1.4; it also has been working fine for me. Are you sure you're using exactly the same Hadoop libraries (since you're building with -Phadoop-provided) and Hadoop configuration in both cases? On Tue, Jun 2, 2015 at 5:29 PM, Night Wolf

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi, Thanks for you information. I'll give spark1.4 a try when it's released. On Wed, Jun 3, 2015 at 11:31 AM, Tathagata Das t...@databricks.com wrote: Could you try it out with Spark 1.4 RC3? Also pinging, Cloudera folks, they may be aware of something. BTW, the way I have debugged memory

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Could you try it out with Spark 1.4 RC3? Also pinging, Cloudera folks, they may be aware of something. BTW, the way I have debugged memory leaks in the past is as follows. Run with a small driver memory, say 1 GB. Periodically (maybe a script), take snapshots of histogram and also do memory

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi, Thanks for you reply. Here's the top 30 entries of jmap -histo:live result: num #instances #bytes class name -- 1: 40802 145083848 [B 2: 99264 12716112 methodKlass 3: 99264 12291480

Re: Spark 1.4 YARN Application Master fails with 500 connect refused

Thanks Marcelo - looks like it was my fault. Seems when we deployed the new version of spark it was picking up the wrong yarn site and setting the wrong proxy host. All good now! On Wed, Jun 3, 2015 at 11:01 AM, Marcelo Vanzin van...@cloudera.com wrote: That code hasn't changed at all between

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi Zhang, Could you paste your code in a gist? Not sure what you are doing inside the code to fill up memory. Thanks Best Regards On Thu, May 28, 2015 at 10:08 AM, Ji ZHANG zhangj...@gmail.com wrote: Hi, Yes, I'm using createStream, but the storageLevel param is by default

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi, I wrote a simple test job, it only does very basic operations. for example: val lines = KafkaUtils.createStream(ssc, zkQuorum, group, Map(topic - 1)).map(_._2) val logs = lines.flatMap { line = try { Some(parse(line).extract[Impression]) } catch { case _:

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Can you replace your counting part with this? logs.filter(_.s_id 0).foreachRDD(rdd = logger.info(rdd.count())) Thanks Best Regards On Thu, May 28, 2015 at 1:02 PM, Ji ZHANG zhangj...@gmail.com wrote: Hi, I wrote a simple test job, it only does very basic operations. for example:

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi, Unfortunately, they're still growing, both driver and executors. I run the same job with local mode, everything is fine. On Thu, May 28, 2015 at 5:26 PM, Akhil Das ak...@sigmoidanalytics.com wrote: Can you replace your counting part with this? logs.filter(_.s_id 0).foreachRDD(rdd =

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

After submitting the job, if you do a ps aux | grep spark-submit then you can see all JVM params. Are you using the highlevel consumer (receiver based) for receiving data from Kafka? In that case if your throughput is high and the processing delay exceeds batch interval then you will hit this

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi, Yes, I'm using createStream, but the storageLevel param is by default MEMORY_AND_DISK_SER_2. Besides, the driver's memory is also growing. I don't think Kafka messages will be cached in driver. On Thu, May 28, 2015 at 12:24 AM, Akhil Das ak...@sigmoidanalytics.com wrote: Are you using the

Re: Spark Streming yarn-cluster Mode Off-heap Memory Is Constantly Growing

Hi Akhil, Thanks for your reply. Accoding to the Streaming tab of Web UI, the Processing Time is around 400ms, and there's no Scheduling Delay, so I suppose it's not the Kafka messages that eat up the off-heap memory. Or maybe it is, but how to tell? I googled about how to check the off-heap

Re: Spark on Yarn : Map outputs lifetime ?

Neither of those two. Instead, the shuffle data is cleaned up when the stage they are from get GC'ed by the jvm. that is, when you are no longer holding any references to anything which points to the old stages, and there is an appropriate gc event. The data is not cleaned up right after the

Re: Spark-on-YARN architecture

Thanks for the quick reply. I am running the application in YARN client mode. And I want to run the AM on the same node as RM inorder use the node which otherwise would run AM. How can I get AM run on the same node as RM? On Tue, Mar 10, 2015 at 3:49 PM, Sean Owen so...@cloudera.com wrote:

Re: Spark-on-YARN architecture

I suppose you just provision enough resource to run both on that node... but it really shouldn't matter. The RM and your AM aren't communicating heavily. On Tue, Mar 10, 2015 at 10:23 AM, Harika Matha matha.har...@gmail.com wrote: Thanks for the quick reply. I am running the application in

Re: Spark-on-YARN architecture

In YARN cluster mode, there is no Spark master, since YARN is your resource manager. Yes you could force your AM somehow to run on the same node as the RM, but why -- what do think is faster about that? On Tue, Mar 10, 2015 at 10:06 AM, Harika matha.har...@gmail.com wrote: Hi all, I have Spark

Re: Spark on Yarn: java.lang.IllegalArgumentException: Invalid rule

The version I'm using was already pre-built for Hadoop 2.3. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-java-lang-IllegalArgumentException-Invalid-rule-tp21382p21485.html Sent from the Apache Spark User List mailing list archive at

Re: spark on yarn succeeds but exit code 1 in logs

Can you look inside RM log to see if you can find some clue there ? You can pastebin part of the RM log around the time your job ran ? What hadoop version are you using ? Thanks On Sat, Jan 31, 2015 at 11:24 AM, Koert Kuipers ko...@tresata.com wrote: i have a simple spark app that i run with

Re: spark on yarn succeeds but exit code 1 in logs

it is CDH 5.3 with the spark that ships with it. i went through the RM logs line by line, and i found the exit code in there: container_1422728945460_0001_01_29 Container Transitioned from NEW to RESERVED 2015-01-31 18:30:49,633 INFO

Re: Spark on YARN: java.lang.ClassCastException SerializedLambda to org.apache.spark.api.java.function.Function in instance of org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1

Here is the same issues: [1] http://stackoverflow.com/questions/28186607/java-lang-classcastexception-using-lambda-expressions-in-spark-job-on-remote-ser [2]

Re: Spark on Yarn: java.lang.IllegalArgumentException: Invalid rule

Then your spark is not built for yarn. Try to build with sbt/sbt -Dhadoop.version=2.3.0 -Pyarn assembly -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-java-lang-IllegalArgumentException-Invalid-rule-tp21382p21404.html Sent from the Apache

Re: Spark on Yarn: java.lang.IllegalArgumentException: Invalid rule

Thanks, Ted. Kerberos is enabled on the cluster. I'm new to the world of kerberos, so pease excuse my ignorance here. Do you know if there are any additional steps I need to take in addition to setting HADOOP_CONF_DIR? For instance, does hadoop.security.auth_to_local require any specific setting

Re: Spark on Yarn: java.lang.IllegalArgumentException: Invalid rule

Thanks, Siddardha. I did but got the same error. Kerberos is enabled on my cluster and I may be missing a configuration step somewhere. -- View this message in context:

Re: Spark on Yarn: java.lang.IllegalArgumentException: Invalid rule

Caused by: java.lang.IllegalArgumentException: Invalid rule: L RULE:[2:$1@$0](.*@XXXCOMPANY.COM http://xxxcompany.com/)s/(.*)@ XXXCOMPANY.COM/$1/L http://xxxcompany.com/$1/L DEFAULT Can you put the rule on a single line (not sure whether there is newline or space between L and DEFAULT) ? Looks

Re: Spark on YARN: java.lang.ClassCastException SerializedLambda to org.apache.spark.api.java.function.Function in instance of org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1

Update: I deployed a stand-alone spark in localhost then set Master as spark://localhost:7077 and it met the same issue Don't know how to solve it. -- View this message in context:

Re: spark-network-yarn 2.11 depends on spark-network-shuffle 2.10

Actually it does causes builds with SBT 0.13.7 to fail with the error Conflicting cross-version suffixes. I have raised a defect SPARK-5143 for this. On Wed Jan 07 2015 at 23:44:21 Marcelo Vanzin van...@cloudera.com wrote: This particular case shouldn't cause problems since both of those

Re: spark-network-yarn 2.11 depends on spark-network-shuffle 2.10

This particular case shouldn't cause problems since both of those libraries are java-only (the scala version appended there is just for helping the build scripts). But it does look weird, so it would be nice to fix it. On Wed, Jan 7, 2015 at 12:25 AM, Aniket Bhatnagar aniket.bhatna...@gmail.com

Re: Spark 1.2.0 Yarn not published

/JW1q5vd61V1/Spark-yarn+1.2.0subj=Re+spark+yarn_2+10+1+2+0+artifacts Cheers On Dec 28, 2014, at 11:13 PM, Aniket Bhatnagar aniket.bhatna...@gmail.com wrote: Hi all I just realized that spark-yarn artifact hasn't been published for 1.2.0 release. Any particular reason for that? I was using

Re: Spark 1.2.0 Yarn not published

See this thread: http://search-hadoop.com/m/JW1q5vd61V1/Spark-yarn+1.2.0subj=Re+spark+yarn_2+10+1+2+0+artifacts Cheers On Dec 28, 2014, at 11:13 PM, Aniket Bhatnagar aniket.bhatna...@gmail.com wrote: Hi all I just realized that spark-yarn artifact hasn't been published for 1.2.0 release

Re: Spark on YARN memory utilization

Thanks Sandy! On Mon, Dec 8, 2014 at 23:15 Sandy Ryza sandy.r...@cloudera.com wrote: Another thing to be aware of is that YARN will round up containers to the nearest increment of yarn.scheduler.minimum-allocation-mb, which defaults to 1024. -Sandy On Sat, Dec 6, 2014 at 3:48 PM, Denny Lee

Re: Spark on YARN memory utilization

Another thing to be aware of is that YARN will round up containers to the nearest increment of yarn.scheduler.minimum-allocation-mb, which defaults to 1024. -Sandy On Sat, Dec 6, 2014 at 3:48 PM, Denny Lee denny.g@gmail.com wrote: Got it - thanks! On Sat, Dec 6, 2014 at 14:56 Arun Ahuja

Re: Spark on YARN memory utilization

Hi Denny, This is due the spark.yarn.memoryOverhead parameter, depending on what version of Spark you are on the default of this may differ, but it should be the larger of 1024mb per executor or .07 * executorMemory. When you set executor memory, the yarn resource request is executorMemory +

Re: Spark on YARN memory utilization

Got it - thanks! On Sat, Dec 6, 2014 at 14:56 Arun Ahuja aahuj...@gmail.com wrote: Hi Denny, This is due the spark.yarn.memoryOverhead parameter, depending on what version of Spark you are on the default of this may differ, but it should be the larger of 1024mb per executor or .07 *

Re: Spark on YARN

Hi all! Thanks for answering! @Sean, I tried to run with 30 executor-cores , and 1 machine still without processing. @Vanzin, I checked RM's web UI, and all nodes were detecteds and RUNNING. The interesting fact is that available memory and available core of 1 node was different of other 2, with

Re: Spark on YARN

I think your config may be the issue then. It sounds like 1 server is configured in a different YARN group that states it has way less resource than it does. On Wed, Nov 19, 2014 at 5:27 PM, Alan Prando a...@scanboo.com.br wrote: Hi all! Thanks for answering! @Sean, I tried to run with 30

Re: Spark On Yarn Issue: Initial job has not accepted any resources

Not sure how to solve this, but spotted these lines in the logs: 14/11/18 14:28:23 INFO YarnAllocationHandler: Container marked as *failed*: container_1415961020140_0325_01_02 14/11/18 14:28:38 INFO YarnAllocationHandler: Container marked as *failed*: container_1415961020140_0325_01_03

Re: Spark on YARN

I run my Spark on YARN jobs as: HADOOP_CONF_DIR=/etc/hadoop/conf/ /app/data/v606014/dist/bin/spark-submit --master yarn --jars test-job.jar --executor-cores 4 --num-executors 10 --executor-memory 16g --driver-memory 4g --class TestClass test.jar It uses HADOOP_CONF_DIR to schedule executors and

Re: Spark on YARN

Can you check in your RM's web UI how much of each resource does Yarn think you have available? You can also check that in the Yarn configuration directly. Perhaps it's not configured to use all of the available resources. (If it was set up with Cloudera Manager, CM will reserve some room for

Re: Spark on YARN

Hey Alan, Spark's application master will take up 1 core on one of the nodes on the cluster. This means that that node will only have 31 cores remaining, not enough to fit your third executor. -Sandy On Tue, Nov 18, 2014 at 10:03 AM, Alan Prando a...@scanboo.com.br wrote: Hi Folks! I'm

Re: Spark on YARN

My guess is you're asking for all cores of all machines but the driver needs at least one core, so one executor is unable to find a machine to fit on. On Nov 18, 2014 7:04 PM, Alan Prando a...@scanboo.com.br wrote: Hi Folks! I'm running Spark on YARN cluster installed with Cloudera Manager

Re: Spark on YARN, ExecutorLostFailure for long running computations in map

So it seems that this problem was related to  http://apache-spark-developers-list.1001551.n3.nabble.com/Lost-executor-on-YARN-ALS-iterations-td7916.html and increasing the executor memory worked for me. __ Hi, I am getting 

Re: Spark on Yarn probably trying to load all the data to RAM

I have tried it out to merge the file to one, Spark is now working with RAM as I've expected. Unfortunately after doing this there appears another problem. Now Spark running on YARN is scheduling all the work only to one worker node as a one big job. Is there some way, how to force Spark and

Re: Spark on Yarn probably trying to load all the data to RAM

Ok so the problem was solved, it that the file was gziped and it looks that Spark does not support direct .gz file distribution to workers.  Thank you very much fro the suggestion to merge the files. Best regards, Jan  __ I have

Re: Spark on Yarn probably trying to load all the data to RAM

Could you please give me an example or send me a link of how to use Hadoop CombinedFileInputFormat? It sound very interesting to me and it would probably save me several hours of my pipeline computation. Merging of the files is currently the bottleneck in my system.

Re: Spark on Yarn probably trying to load all the data to RAM

On Sun, Nov 2, 2014 at 1:35 AM, jan.zi...@centrum.cz wrote: Hi, I am using Spark on Yarn, particularly Spark in Python. I am trying to run: myrdd = sc.textFile(s3n://mybucket/files/*/*/*.json) How many files do you have? and the average size of each file? myrdd.getNumPartitions()

Re: Spark on Yarn probably trying to load all the data to RAM

I have 3 datasets in all the datasets the average file size is 10-12Kb.  I am able to run my code on the dataset with 70K files, but I am not able to run it on datasets with 1.1M and 3.8M files.  __ On Sun, Nov 2, 2014 at 1:35 AM,  

Re: spark 1.1.0/yarn hang

We have narrowed this hanging issue down to the calliope package that we used to create RDD from reading cassandra table. The calliope native RDD interface seems hanging and I have decided to switch to the calliope cql3 RDD interface. -- View this message in context:

Re: Spark on YARN driver memory allocation bug?

It may also cause a problem when running in the yarn-client mode. If --driver-memory is large, Yarn has to allocate a lot of memory to the AM container, but AM doesn't really need the memory. Boduo -- View this message in context:

Re: Spark on YARN driver memory allocation bug?

: Andrew Or and...@databricks.commailto:and...@databricks.com Date: Wednesday, October 8, 2014 3:25 PM To: Greg greg.h...@rackspace.commailto:greg.h...@rackspace.com Cc: user@spark.apache.orgmailto:user@spark.apache.org user@spark.apache.orgmailto:user@spark.apache.org Subject: Re: Spark on YARN driver

Re: Spark on YARN driver memory allocation bug?

@spark.apache.org user@spark.apache.org Subject: Re: Spark on YARN driver memory allocation bug? Hi Greg, It does seem like a bug. What is the particular exception message that you see? Andrew 2014-10-08 12:12 GMT-07:00 Greg Hill greg.h...@rackspace.com: So, I think this is a bug, but I wanted

<    1   2   3   >