Re: Running Spark und YARN on AWS EMR

Hi Josh, As you say, I also recognize the problem. I feel I got a warning when specifying a huge data set. We also adjust the partition size but we are doing command options instead of default settings, or in code. Regards, Takashi 2017-07-18 6:48 GMT+09:00 Josh Holbrook

Re: Running Spark und YARN on AWS EMR

I just ran into this issue! Small world. As far as I can tell, by default spark on EMR is completely untuned, but it comes with a flag that you can set to tell EMR to autotune spark. In your configuration.json file, you can add something like: { "Classification": "spark", "Properties":

Re: Running Spark und YARN on AWS EMR

Hi Takashi, thanks for your help. After a further investigation, I figure out that the killed container was the driver process. After setting spark.yarn.driver.memoryOverhead instead of spark.yarn.executor.memoryOverhead the error was gone and application is executed without error. Maybe it

Re: Running Spark und YARN on AWS EMR

Hi Pascal, The error also occurred frequently in our project. As a solution, it was effective to specify the memory size directly with spark-submit command. eg. spark-submit executor-memory 2g Regards, Takashi > 2017-07-18 5:18 GMT+09:00 Pascal Stammer : >> Hi, >> >>

Running Spark und YARN on AWS EMR

Hi, I am running a Spark 2.1.x Application on AWS EMR with YARN and get following error that kill my application: AM Container for appattempt_1500320286695_0001_01 exited with exitCode: -104 For more detailed output, check application tracking

Re: problem running spark with yarn-client not using spark-submit

cc| > |"user @spark" <user@spark.apache.org> > | > | >

Re: problem running spark with yarn-client not using spark-submit

ject| | [Spam][SMG] Re: problem running spark with yarn

Re: problem running spark with yarn-client not using spark-submit

Hi, Trying to run spark with yarn-client not using spark-submit here what are you using to submit the job? spark-shell, spark-sql or anything else Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

problem running spark with yarn-client not using spark-submit

Hello guys, Trying to run spark with yarn-client not using spark-submit here but the jobs kept failed while AM launching executor. The error collected by yarn like below. Looks like some environment setting is missing? Could someone help me out with this. Thanks in advance! HY Chung Java

Re: SLF4J binding error while running Spark using YARN as Cluster Manager

> I am having log4j trouble while running Spark using YARN as cluster manager > in CDH 5.3.3. > I get the following error:- > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/data/12/yarn/nm/filecache/34/spark-assemb

SLF4J binding error while running Spark using YARN as Cluster Manager

Hi, I am having log4j trouble while running Spark using YARN as cluster manager in CDH 5.3.3. I get the following error:- SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/12/yarn/nm/filecache/34/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl

Long(20+ seconds) startup delay for jobs when running Spark on YARN

Hi, I'm running Spark(1.6.1) on YARN(2.5.1), cluster mode. It's taking 20+ seconds for application to move from ACCEPTED to RUNNING state, here's logs 16/04/21 09:06:56 INFO impl.YarnClientImpl: Submitted application application_1461229289298_0001 16/04/21 09:06:57 INFO yarn.Client: Application

Re: Running Spark on Yarn-Client/Cluster mode

unch Spark-Shell in yarn-client mode > > Any suggestion ? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26752.html > Sent from the Apache Spark U

Re: Running Spark on Yarn-Client/Cluster mode

I have updated all my nodes in the Cluster to have 4GB RAM memory , but still face the same error when trying to launch Spark-Shell in yarn-client mode Any suggestion ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster

Re: Running Spark on Yarn-Client/Cluster mode

-Spark-on-Yarn-Client-Cluster-mode-tp26691p26739.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h

Re: Running Spark on Yarn-Client/Cluster mode

0.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26717.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional command

Re: Running Spark on Yarn-Client/Cluster mode

Few more added information with Nodes Memory and Core ptfhadoop01v - 4GB ntpcam01v - 1GB ntpcam03v - 2GB Each of the VM has only 1 core CPU -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26714.html Sent

Re: Running Spark on Yarn-Client/Cluster mode

his message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26713.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe,

Re: Running Spark on Yarn-Client/Cluster mode

560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26710.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands,

Re: Running Spark on Yarn-Client/Cluster mode

or your quick thoughts on this issue. Regards Ashesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26709.html Sent from the Apache Spark User List mailing list

Re: Running Spark on Yarn-Client/Cluster mode

sembly.jar -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26704.html Sent from the Apache Spark User List mailing list archive at

Running Spark on Yarn-Client/Cluster mode

n each node in cluster ? How do i start the spark-shell in yarn-client mode. Thanks in advance. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691.html Sent from the Apache Spark User List mailin

Re: Running Spark on Yarn

RM NM logs traced below, RM --> 2016-03-30 14:59:15,498 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1459326455972_0004_01_01, NodeId: myhost:60653, NodeHttpAddress: myhost:8042, Resource:

Re: Running Spark on Yarn

ok, start EMR-4.3.0 or 4.2.0 cluster and look at how to configure spark on yarn properly

Re: Running Spark on Yarn

:~/Downloads/package/spark-1.6.1-bin-hadoop2.6$ bin/spark-shell --master yarn-client 16/03/30 03:24:43 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost sending #138 16/03/30 03:24:43 DEBUG ipc.Client: IPC Client (111576772) connection to

Re: Running Spark on Yarn

Looks like still the same while the other MR application is working fine, On Wed, Mar 30, 2016 at 3:15 AM, Alexander Pivovarov wrote: > for small cluster set the following settings > > yarn-site.xml > > > yarn.scheduler.minimum-allocation-mb > 32 > > > >

Re: Running Spark on Yarn

for small cluster set the following settings yarn-site.xml yarn.scheduler.minimum-allocation-mb 32 capacity-scheduler.xml yarn.scheduler.capacity.maximum-am-resource-percent 0.5 Maximum percent of resources in the cluster which can be used to run application

Re: Running Spark on Yarn

Yarn seems to be running fine, I have successful MR jobs completed on the same, *Cluster Metrics* *Apps Submitted Apps Pending Apps Running Apps Completed Containers Running Memory Used Memory Total Memory Reserved VCores Used VCores Total VCores Reserved Active Nodes Decommissioned Nodes Lost

Re: Running Spark on Yarn

check resource manager and node manager logs. Maybe you find smth explaining why 1 app is pending do you have any app run successfully? *Apps Completed is 0 on the UI* On Tue, Mar 29, 2016 at 2:13 PM, Vineet Mishra wrote: > Hi Alex/Surendra, > > Hadoop is up and

Re: Running Spark on Yarn

Hi Alex/Surendra, Hadoop is up and running fine and I am able to run example on the same. *Cluster Metrics* *Apps Submitted Apps Pending Apps Running Apps Completed Containers Running Memory Used Memory Total Memory Reserved VCores Used VCores Total VCores Reserved Active Nodes Decommissioned

Re: Running Spark on Yarn

check 8088 ui - how many cores and memory available - how many slaves are active run teragen or pi from hadoop examples to make sure that yarn works On Tue, Mar 29, 2016 at 1:25 PM, Surendra , Manchikanti < surendra.manchika...@gmail.com> wrote: > Hi Vineeth, > > Can you please check

Re: Running Spark on Yarn

Hi Vineeth, Can you please check resource(RAM,Cores) availability in your local cluster, And change accordingly. Regards, Surendra M -- Surendra Manchikanti On Tue, Mar 29, 2016 at 1:15 PM, Vineet Mishra wrote: > Hi All, > > While starting Spark on Yarn on local

Running Spark on Yarn

Hi All, While starting Spark on Yarn on local cluster(Single Node Hadoop 2.6 yarn) I am facing some issues. As I try to start the Spark Shell it keeps on iterating in a endless loop while initiating, *6/03/30 01:32:38 DEBUG ipc.Client: IPC Client (1782965120) connection to

Re: How to display the web ui when running Spark on YARN?

t; > Hi all, > > I am running Spark in yarn-client mode, but every time I access the web > ui, the browser redirect me to one of the worker nodes and shows nothing. > The url looks like > http://hadoop-node31.company.com:8088/proxy/application_1453797301246_120264 > . > > >

Re: How to display the web ui when running Spark on YARN?

On 3 Mar 2016, at 09:17, Shady Xu <shad...@gmail.com<mailto:shad...@gmail.com>> wrote: Hi all, I am running Spark in yarn-client mode, but every time I access the web ui, the browser redirect me to one of the worker nodes and shows nothing. The url looks like http://h

How to display the web ui when running Spark on YARN?

Hi all, I am running Spark in yarn-client mode, but every time I access the web ui, the browser redirect me to one of the worker nodes and shows nothing. The url looks like http://hadoop-node31.company.com:8088/proxy/application_1453797301246_120264 . I googled a lot and found some possible

Re: Running Spark in Yarn-client mode

Hey Jean, Thanks for the quick response. I am using spark 1.4.1 pre-built with hadoop 2.6. Yes the Yarn cluster has multiple running worker nodes. It would a great help if you can tell how to look for the executors logs. Regards, Sushrut Ikhar [image: https://]about.me/sushrutikhar

Running Spark in Yarn-client mode

Hi, I am new to Spark and I have been trying to run Spark in yarn-client mode. I get this error in yarn logs : Error: Could not find or load main class org.apache.spark.executor.CoarseGrainedExecutorBackend Also, I keep getting these warnings: WARN YarnScheduler: Initial job has not accepted

Re: Running Spark in Yarn-client mode

Hi Sushrut, which packaging of Spark do you use ? Do you have a working Yarn cluster (with at least one worker) ? spark-hadoop-x ? Regards JB On 10/08/2015 07:23 AM, Sushrut Ikhar wrote: Hi, I am new to Spark and I have been trying to run Spark in yarn-client mode. I get this error in yarn

Re: Is it possible to see Spark jobs on MapReduce job history ? (running Spark on YARN cluster)

...@gmail.com wrote: Hi all, I wonder if anyone has used use MapReduce Job History to show Spark jobs. I can see my Spark jobs (Spark running on Yarn cluster) on Resource manager (RM). I start Spark History server, and then through Spark's web-based user interface I can monitor

Is it possible to see Spark jobs on MapReduce job history ? (running Spark on YARN cluster)

Hi all, I wonder if anyone has used use MapReduce Job History to show Spark jobs. I can see my Spark jobs (Spark running on Yarn cluster) on Resource manager (RM). I start Spark History server, and then through Spark's web-based user interface I can monitor the cluster (and track cluster

running spark on yarn

Hello, folks. We just recently switched to using Yarn on our cluster (when upgrading to cloudera 5.4.1) I'm trying to run a spark job from within a broader application (a web service running on Jetty), so I can't just start it using spark-submit. Does anyone know of an instructions page on how

Re: Who manage the log4j appender while running spark on yarn?

.1001560.n3.nabble.com/Who-manage-the-log4j-appender-while-running-spark-on-yarn-tp20778p20818.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org

Re: Who manage the log4j appender while running spark on yarn?

then. Any one could offer some guide? Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Who-manage-the-log4j-appender-while-running-spark-on-yarn-tp20778.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Who manage the log4j appender while running spark on yarn?

.n3.nabble.com/Who-manage-the-log4j-appender-while-running-spark-on-yarn-tp20778.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org

Re: vcores used in cluster metrics(yarn resource manager ui) when running spark on yarn

Hi yuemeng, Are you possibly running the Capacity Scheduler with the default resource calculator? -Sandy On Sat, Dec 6, 2014 at 7:29 PM, yuemeng1 yueme...@huawei.com wrote: Hi, all When i running an app with this cmd: ./bin/spark-sql --master yarn-client --num-executors 2

vcores used in cluster metrics(yarn resource manager ui) when running spark on yarn

Hi, all When i running an app with this cmd: ./bin/spark-sql --master yarn-client --num-executors 2 --executor-cores 3, i noticed that yarn resource manager ui shows the `vcores used` in cluster metrics is 3. It seems `vcores used` show wrong num (should be 7?)? Or i miss something?

Re: hadoop_conf_dir when running spark on yarn

Hi, On Mon, Nov 3, 2014 at 1:29 PM, Amey Chaugule ambr...@gmail.com wrote: I thought that only applied when you're trying to run a job using spark-submit or in the shell... And how are you starting your Yarn job, if not via spark-submit? Tobias

Re: hadoop_conf_dir when running spark on yarn

by the configuration that I pull from sc.hadoopConfiguration() is incorrect. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: hadoop_conf_dir when running spark on yarn

-spark-user-list.1001560.n3.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org

hadoop_conf_dir when running spark on yarn

.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail

Running Spark On Yarn without Spark-Submit

Hi, My requirement is to run Spark on Yarn without using the script spark-submit. I have a servlet and a tomcat server. As and when request comes, it creates a new SC and keeps it alive for the further requests, I ma setting my master in sparkConf as sparkConf.setMaster(yarn-cluster) but the

Re: Running Spark On Yarn without Spark-Submit

including user@spark.apache.org. On Fri, Aug 29, 2014 at 2:03 PM, Archit Thakur archit279tha...@gmail.com wrote: Hi, My requirement is to run Spark on Yarn without using the script spark-submit. I have a servlet and a tomcat server. As and when request comes, it creates a new SC and

Re: Running Spark On Yarn without Spark-Submit

Archit We are using yarn-cluster mode , and calling spark via Client class directly from servlet server. It works fine. To establish a communication channel to give further requests, It should be possible with yarn client, but not with yarn server. Yarn client mode, spark driver

Running Spark on Yarn vs Mesos

.nabble.com/Running-Spark-on-Yarn-vs-Mesos-tp9320.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Question about running spark on yarn

In this page http://spark.apache.org/docs/0.9.0/running-on-yarn.html We have to use spark assembly to submit spark apps to yarn cluster. And I checked the assembly jars of spark. It contains some yarn classes which are added during compile time. The yarn classes are not what I want. My question

Re: Question about running spark on yarn

Hi Gordon, We recently handled this in SPARK-1064. As of 1.0.0, you'll be able to pass -Phadoop-provided to Maven and avoid including Hadoop and its dependencies in the assembly jar. -Sandy On Tue, Apr 22, 2014 at 2:43 AM, Gordon Wang gw...@gopivotal.com wrote: In this page

Re: Question about running spark on yarn

Hi Sandy, Thanks for your reply ! Does this work for sbt ? I checked the commit, looks like only maven build has such option. On Wed, Apr 23, 2014 at 12:38 AM, Sandy Ryza sandy.r...@cloudera.comwrote: Hi Gordon, We recently handled this in SPARK-1064. As of 1.0.0, you'll be able to

Re: Question about running spark on yarn

I currently don't have plans to work on that. -Sandy On Apr 22, 2014, at 8:06 PM, Gordon Wang gw...@gopivotal.com wrote: Thanks I see. Do you guys have plan to port this to sbt? On Wed, Apr 23, 2014 at 10:24 AM, Sandy Ryza sandy.r...@cloudera.com wrote: Right, it only works for Maven