Re: Networking issues with Spark on EC2

Hi, Nopes. I was trying to use EC2(due to a few constraints) where I faced the problem. With EMR, it works flawlessly. But, I would like to go back and use EC2 if I can fix this issue. Has anybody set up a spark cluster using plain EC2 machines. What steps did you follow? Thanks and Regards, Suraj

Re: Networking issues with Spark on EC2

Hi, Are you using EMR ? Natu On Sat, Sep 26, 2015 at 6:55 AM, SURAJ SHETH wrote: > Hi Ankur, > Thanks for the reply. > This is already done. > If I wait for a long amount of time(10 minutes), a few tasks get > successful even on slave nodes. Sometime, a fraction of the tasks(20%) are > complet

Re: Networking issues with Spark on EC2

Hi Ankur, Thanks for the reply. This is already done. If I wait for a long amount of time(10 minutes), a few tasks get successful even on slave nodes. Sometime, a fraction of the tasks(20%) are completed on all the machines in the initial 5 seconds and then, it slows down drastically. Thanks and R

Re: Networking issues with Spark on EC2

Hi Suraj, Spark uses a lot of ports to communicate between nodes. Probably your security group is restrictive and does not allow instances to communicate on all networks. The easiest way to resolve it is to add a Rule to allow all Inbound traffic on all ports (0-65535) to instances in same securit

Networking issues with Spark on EC2

Hi, I am using Spark 1.2 and facing network related issues while performing simple computations. This is a custom cluster set up using ec2 machines and spark prebuilt binary from apache site. The problem is only when we have workers on other machines(networking involved). Having a single node for

Re: Required settings for permanent HDFS Spark on EC2

rsistent > HDFS - it always looks for 9000 port regardless of options I set for 9010 > persistent HDFS. Have you figured out a solution? Thanks > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Required-settings-for-permanent-HDFS

Re: Required settings for permanent HDFS Spark on EC2

e.com/Required-settings-for-permanent-HDFS-Spark-on-EC2-tp22860p23157.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e

Required settings for permanent HDFS Spark on EC2

ll / scala shell to use persistent instead of ephemeral? Best, Tony -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Required-settings-for-permanent-HDFS-Spark-on-EC2-tp22860.html Sent from the Apache Spark

Re: How to run customized Spark on EC2?

code and built it > using: > > mvn package -DskipTests > > I checked the new version in my own computer and it works. However, when I > ran spark on EC2, the spark code EC2 machines ran is the original version. > > Anyone knows how to deploy the changed spark source

Re: How to run self-build spark on EC2?

ackage -DskipTests > > I checked the new version in my own computer and it works. However, when I > ran spark on EC2, the spark code EC2 machines ran is the original version. > > Anyone knows how to deploy the changed spark source code

How to run customized Spark on EC2?

Hi experts, I have an issue. I added some timestamps in Spark source code and built it using: mvn package -DskipTests I checked the new version in my own computer and it works. However, when I ran spark on EC2, the spark code EC2 machines ran is the original version. Anyone knows how to

How to run self-build spark on EC2?

Hi all, I have an issue. I added some timestamps in Spark source code and built it using: mvn package -DskipTests I checked the new version in my own computer and it works. However, when I ran spark on EC2, the spark code EC2 machines ran is the original version. Anyone knows how to deploy

Re: Spark on EC2

You're probably requesting more instances than allowed by your account, so the error gets generated for the extra instances. Try launching a smaller cluster. On Wed, Apr 1, 2015 at 12:41 PM, Vadim Bichutskiy < vadim.bichuts...@gmail.com> wrote: > Hi all, > > I just tried launching a Spark cluster

Spark on EC2

Hi all, I just tried launching a Spark cluster on EC2 as described in http://spark.apache.org/docs/1.3.0/ec2-scripts.html I got the following response: *"PendingVerificationYour account is currently being verified. Verification normally takes less than 2 hours. Until your account is verified,

Re: Spark on EC2

ng purposes. > :) > > Thanks > Best Regards > > On Tue, Feb 24, 2015 at 8:25 PM, Deep Pradhan > wrote: > >> Hi, >> I have just signed up for Amazon AWS because I learnt that it provides >> service for free for the first 12 months. >> I want to run Spark on EC2 cluster. Will they charge me for this? >> >> Thank You >> > >

Re: Spark on EC2

;> I have just signed up for Amazon AWS because I learnt that it provides >>> service for free for the first 12 months. >>> I want to run Spark on EC2 cluster. Will they charge me for this? >>> >>> Thank You >>> >> >> >

Re: Spark on EC2

t that it provides > service for free for the first 12 months. > I want to run Spark on EC2 cluster. Will they charge me for this? > > Thank You >

Re: Spark on EC2

The charge of AWS depends on how much and the types of machine that you >>> launched, but not on the utilisation of machine. >>> >>> Hope it would help. >>> >>> Cheers >>> Gen >>> >>> >>> On Tue, Feb 24, 2015 at 3:55 PM

Re: Spark on EC2

adhan >> wrote: >> >>> Hi, >>> I have just signed up for Amazon AWS because I learnt that it provides >>> service for free for the first 12 months. >>> I want to run Spark on EC2 cluster. Will they charge me for this? >>> >>> Thank You >>> >> >> >

Re: Spark on EC2

t; >> run into as you don't get much provisioning with these free instances. > >> > >> So, no you really can't use this for anything that is at all CPU > >> intensive. It's for, say, running a low-traffic web service. > >> > >> On Tue, Feb 24, 2015 at 2:55 PM, Deep Pradhan < > pradhandeep1...@gmail.com> > >> wrote: > >> > Hi, > >> > I have just signed up for Amazon AWS because I learnt that it provides > >> > service for free for the first 12 months. > >> > I want to run Spark on EC2 cluster. Will they charge me for this? > >> > > >> > Thank You > > > > >

Re: Spark on EC2

running a low-traffic web service. >> >> On Tue, Feb 24, 2015 at 2:55 PM, Deep Pradhan >> wrote: >> > Hi, >> > I have just signed up for Amazon AWS because I learnt that it provides >> > service for free for the first 12 months. >> > I want to

Re: Spark on EC2

all CPU >> intensive. It's for, say, running a low-traffic web service. >> >> On Tue, Feb 24, 2015 at 2:55 PM, Deep Pradhan >> wrote: >> > Hi, >> > I have just signed up for Amazon AWS because I learnt that it provides >> > service for free for the first 12 months. >> > I want to run Spark on EC2 cluster. Will they charge me for this? >> > >> > Thank You >> > >

Re: Spark on EC2

that is at all CPU > intensive. It's for, say, running a low-traffic web service. > > On Tue, Feb 24, 2015 at 2:55 PM, Deep Pradhan > wrote: > > Hi, > > I have just signed up for Amazon AWS because I learnt that it provides > > service for free for the first 12 months. > > I want to run Spark on EC2 cluster. Will they charge me for this? > > > > Thank You >

Re: Spark on EC2

Pradhan wrote: > Hi, > I have just signed up for Amazon AWS because I learnt that it provides > service for free for the first 12 months. > I want to run Spark on EC2 cluster. Will they charge me for this? > > Thank You --

Re: Spark on EC2

ides >> service for free for the first 12 months. >> I want to run Spark on EC2 cluster. Will they charge me for this? >> >> Thank You >> > >

Re: Spark on EC2

, but not on the utilisation of machine. Hope it would help. Cheers Gen On Tue, Feb 24, 2015 at 3:55 PM, Deep Pradhan wrote: > Hi, > I have just signed up for Amazon AWS because I learnt that it provides > service for free for the first 12 months. > I want to run Spark on EC2 cluste

Spark on EC2

Hi, I have just signed up for Amazon AWS because I learnt that it provides service for free for the first 12 months. I want to run Spark on EC2 cluster. Will they charge me for this? Thank You

Re: spark on ec2

Oh yeah, they picked up changes after restart, thanks! On Thu, Feb 5, 2015 at 8:13 PM, Charles Feduke wrote: > I don't see anything that says you must explicitly restart them to load > the new settings, but usually there is some sort of signal trapped [or > brute force full restart] to get a con

Re: spark on ec2

I don't see anything that says you must explicitly restart them to load the new settings, but usually there is some sort of signal trapped [or brute force full restart] to get a configuration reload for most daemons. I'd take a guess and use the $SPARK_HOME/sbin/{stop,start}-slaves.sh scripts on yo

spark on ec2

Hi, I'm trying to change setting as described here: http://spark.apache.org/docs/1.2.0/ec2-scripts.html export SPARK_WORKER_CORES=6 Then I ran ~/spark-ec2/copy-dir /root/spark/conf to distribute to slaves, but without any effect. Do I have to restart workers? How to do that with spark-ec2? Than

Re: Spark on EC2

quot; To: user@spark.apache.org Sent: Thursday, September 18, 2014 11:48:03 AM Subject: Spark on EC2 Hello, I am trying to run a python script that makes use of the kmeans MLIB and I'm not getting anywhere. I'm using an c3.xlarge instance as master, and 10 c3.large instances as slaves. In the cod

Spark on EC2

Hello, I am trying to run a python script that makes use of the kmeans MLIB and I'm not getting anywhere. I'm using an c3.xlarge instance as master, and 10 c3.large instances as slaves. In the code I make a map of a 600MB csv file in S3, where each row has 128 integer columns. The problem is that a

Re: Issue with Spark on EC2 using spark-ec2 script

I'm also getting into same issue and is blocked here. Did any of you were able to go past this issue? I tried using both ephimeral and persistent-hdfs. I'm getting the same issue. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-on

Re: Down-scaling Spark on EC2 cluster

What about down-scaling when I use Mesos, does that really deteriorate the performance ? Otherwise we would probably go for spark on mesos on ec2 :) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494p12109.html Sent

Re: Issue with Spark on EC2 using spark-ec2 script

k-ec2 script? >> >> The environment I am running on is a 4 data node 1 master spark cluster >> generated by the spark-ec2 script. I haven't modified anything in the >> environment except for adding data to the ephemeral hdfs. >> >> >> >> -- >>

Re: Issue with Spark on EC2 using spark-ec2 script

master spark cluster > generated by the spark-ec2 script. I haven't modified anything in the > environment except for adding data to the ephemeral hdfs. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-on

Re: Issue with Spark on EC2 using spark-ec2 script

o the ephemeral hdfs. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-on-EC2-using-spark-ec2-script-tp11088p7.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Issue with Spark on EC2 using spark-ec2 script

Forgot to add that I tried your program with the same input file path. It worked fine. (I used local[2], however...) Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition (O'Reilly) Typesafe @deanwampler

Re: Issue with Spark on EC2 using spark-ec2 script

The stack trace suggests it was trying to create a temporary file, not read your file. Of course, it doesn't say what file it couldn't create. Could there be a configuration file, like a Hadoop config file, that was read with a temp dir setting that's invalid for your machine? dean Dean Wampler,

Issue with Spark on EC2 using spark-ec2 script

Hey all, I was able to spawn up a cluster, but when I'm trying to submit a simple jar via spark-submit it fails to run. I am trying to run the simple "Standalone Application" from the quickstart. Oddly enough, I could get another application running through the spark-shell. What am I doing wrong

Re: Down-scaling Spark on EC2 cluster

t; community. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494p10639.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: Down-scaling Spark on EC2 cluster

Any idea about the probable dates for this implementation. I believe it would be a wonderful (and essential) functionality to gain more acceptance in the community. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster

Re: Down-scaling Spark on EC2 cluster

ter > performance. > > Thanks > Best Regards > > > On Wed, Jul 23, 2014 at 6:36 PM, Shubhabrata > wrote: > >> Hello, >> >> We plan to use Spark on EC2 for our data science pipeline. We successfully >> manage to set up cluster as-well-as launch and

Re: Down-scaling Spark on EC2 cluster

etter performance. Thanks Best Regards On Wed, Jul 23, 2014 at 6:36 PM, Shubhabrata wrote: > Hello, > > We plan to use Spark on EC2 for our data science pipeline. We successfully > manage to set up cluster as-well-as launch and run applications on > remote-clusters. However, to enhance

Down-scaling Spark on EC2 cluster

Hello, We plan to use Spark on EC2 for our data science pipeline. We successfully manage to set up cluster as-well-as launch and run applications on remote-clusters. However, to enhance scalability we would like to implement auto-scaling in EC2 for Spark applications. However, I did not find any

Re: Spark on EC2

ontext: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-EC2-tp6638p6681.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >

Re: Spark on EC2

I haven't set up AMI yet. I am just trying to run a simple job on the EC2 cluster. So, is setting up AMI a prerequisite for running simple Spark example like org.apache.spark.examples.GroupByTest? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spa

Re: Spark on EC2

nt.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-EC2-tp6638.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > -- Jeremy Lee BCompSci(Hons) The Unorthodox Engineers

Spark on EC2

View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-EC2-tp6638.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: CDH5 Spark on EC2

Thanks Mayur - I thought I had done those configurations but perhaps I'm pointing to the wrong master IP. > On Apr 2, 2014, at 9:34 AM, Mayur Rustagi wrote: > > The cluster is not running. You need to add MASTER environment variable & > point to your master IP to connect with it. > Also if

Re: CDH5 Spark on EC2

The cluster is not running. You need to add MASTER environment variable & point to your master IP to connect with it. Also if you are running in distributed mode the workers should be registered. Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi

CDH5 Spark on EC2

I’ve been able to get CDH5 up and running on EC2 and according to Cloudera Manager, Spark is running healthy. But when I try to run spark-shell, I eventually get the error: 14/04/02 07:18:18 INFO client.AppClient$ClientActor: Connecting to master  spark://ip-172-xxx-xxx-xxx:7077... 14/04/02 07:1