Hi Hassan
Typically I log on to my master to submit my app.
[ec2-user@ip-172-31-11-222 bin]$ echo $SPARK_ROOT
/root/spark
[ec2-user@ip-172-31-11-222 bin]$echo $MASTER_URL
spark://ec2-54-215-11-222.us-west-1.compute.amazonaws.com:7077
[ec2-user@ip-172-31-11-222 bin]$ $SPARK_ROOT/bin/spark
Hi
I have launched my cluster and I am trying to submit my application to run
on cluster but its not allowing me to connect . It prompts the following
error "*Master endpoint
spark://**ec2-54-187-59-117.us-west-2.compute.amazonaws.com:7077
<http://ec2-54-187-59-117.u
Thanks for the info and the tip! I'll look into writing our own script
based on the spark-ec2 scripts.
Best,
Augustus
On Thu, Nov 12, 2015 at 10:01 AM, Jason Rubenstein <
jasondrubenst...@gmail.com> wrote:
> Hi,
>
> With some minor changes to spark-ec2/spark/init.sh and w
spark-ec2 does not offer a way to upgrade an existing cluster, and from
what I gather, it wasn't intended to be used to manage long-lasting
infrastructure. The recommended approach really is to just destroy your
existing cluster and launch a new one with the desired configuration.
If you want
Hi,
With some minor changes to spark-ec2/spark/init.sh and writing your own
"upgrade-spark.sh" script, you can upgrade spark in place.
(Make sure to call not only spark/init.sh but also spark/setup.sh, because
the latter uses copy-dir to get your ner version of spark to the slaves)
I
Hey All,
I have a Spark cluster(running version 1.5.0) on EC2 launched with the
provided spark-ec2 scripts. If I want to upgrade Spark to 1.5.2 in the same
cluster, what's the safest / recommended way to do that?
I know I can spin up a new cluster running 1.5.2, but it doesn't seem
efficient
t;>>
>>> All TCP
>>> TCP
>>> 0 - 65535
>>> your security group
>>>
>>> Hope this helps!!
>>>
>>> Thanks
>>> Ankur
>>>
>>> On Thu, Sep 24, 2015 at 7:09 AM SURAJ SHETH <shet...@gmail.com> wrote:
&g
<shet...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am using Spark 1.2 and facing network related issues while performing
>>> simple computations.
>>>
>>> This is a custom cluster set up using ec2 machines and spark prebuilt
>&g
kur
>
> On Thu, Sep 24, 2015 at 7:09 AM SURAJ SHETH <shet...@gmail.com> wrote:
>
>> Hi,
>>
>> I am using Spark 1.2 and facing network related issues while performing
>> simple computations.
>>
>> This is a custom cluster set up using ec2 machines and spark p
Hi,
I am using Spark 1.2 and facing network related issues while performing
simple computations.
This is a custom cluster set up using ec2 machines and spark prebuilt
binary from apache site. The problem is only when we have workers on other
machines(networking involved). Having a single node
t;
> This is a custom cluster set up using ec2 machines and spark prebuilt
> binary from apache site. The problem is only when we have workers on other
> machines(networking involved). Having a single node for the master and the
> slave works correctly.
>
> The error log from slav
You refer to `aws_security_token`, but I'm not sure where you're specifying
it. Can you elaborate? Is it an environment variable?
On Mon, Jul 27, 2015 at 4:21 AM Jan Zikeš jan.zi...@centrum.cz wrote:
Hi,
I would like to ask if it is currently possible to use spark-ec2 script
together
Hi,
I would like to ask if it is currently possible to use spark-ec2 script
together with credentials that are consisting not only from:
aws_access_key_id and aws_secret_access_key, but it also contains
aws_security_token.
When I try to run the script I am getting following error message
Hi,
I would like to ask if it is currently possible to use spark-ec2 script
together with credentials that are consisting not only from: aws_access_key_id
and aws_secret_access_key, but it also contains aws_security_token.
When I try to run the script I am getting following error message
to persistent
HDFS - it always looks for 9000 port regardless of options I set for 9010
persistent HDFS. Have you figured out a solution? Thanks
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Required-settings-for-permanent-HDFS-Spark-on-EC2-tp22860p23157
/Required-settings-for-permanent-HDFS-Spark-on-EC2-tp22860p23157.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail
code and built it
using:
mvn package -DskipTests
I checked the new version in my own computer and it works. However, when I
ran spark on EC2, the spark code EC2 machines ran is the original version.
Anyone knows how to deploy the changed spark source code into EC2?
Thx a lot
Bo Fu
-DskipTests
I checked the new version in my own computer and it works. However, when I
ran spark on EC2, the spark code EC2 machines ran is the original version.
Anyone knows how to deploy the changed spark source code into EC2?
Thx a lot
Bo Fu
Hi all,
I have an issue. I added some timestamps in Spark source code and built it
using:
mvn package -DskipTests
I checked the new version in my own computer and it works. However, when I ran
spark on EC2, the spark code EC2 machines ran is the original version.
Anyone knows how to deploy
Hi experts,
I have an issue. I added some timestamps in Spark source code and built it
using:
mvn package -DskipTests
I checked the new version in my own computer and it works. However, when I ran
spark on EC2, the spark code EC2 machines ran is the original version.
Anyone knows how
Hi all,
I just tried launching a Spark cluster on EC2 as described in
http://spark.apache.org/docs/1.3.0/ec2-scripts.html
I got the following response:
*ResponseErrorsErrorCodePendingVerification/CodeMessageYour
account is currently being verified. Verification normally takes less than
2
You're probably requesting more instances than allowed by your account, so
the error gets generated for the extra instances. Try launching a smaller
cluster.
On Wed, Apr 1, 2015 at 12:41 PM, Vadim Bichutskiy
vadim.bichuts...@gmail.com wrote:
Hi all,
I just tried launching a Spark cluster
Hi,
I have just signed up for Amazon AWS because I learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me for this?
Thank You
wrote:
Hi,
I have just signed up for Amazon AWS because I learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me for this?
Thank You
-
To unsubscribe, e-mail
.
On Tue, Feb 24, 2015 at 2:55 PM, Deep Pradhan pradhandeep1...@gmail.com
wrote:
Hi,
I have just signed up for Amazon AWS because I learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me for this?
Thank You
,
I have just signed up for Amazon AWS because I learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me for this?
Thank You
familiar with spark. You can do this on your laptop as well as on ec2. In
fact, running ./ec2/spark-ec2 means launching spark standalone mode on a
cluster, you can find more details here:
https://spark.apache.org/docs/latest/spark-standalone.html
Cheers
Gen
On Tue, Feb 24, 2015 at 4:07 PM, Deep
testing purposes.
:)
Thanks
Best Regards
On Tue, Feb 24, 2015 at 8:25 PM, Deep Pradhan pradhandeep1...@gmail.com
wrote:
Hi,
I have just signed up for Amazon AWS because I learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me
for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me for this?
Thank You
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h
for Amazon AWS because I learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me for this?
Thank You
because I learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me for this?
Thank You
learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2 cluster. Will they charge me for this?
Thank You
Kindly bear with my questions as I am new to this.
If you run spark on local mode on a ec2 machine
What does this mean? Is it that I launch Spark cluster from my local
machine,i.e., by running the shell script that is there in /spark/ec2?
On Tue, Feb 24, 2015 at 8:32 PM, gen tang gen.tan
, but not on the utilisation of machine.
Hope it would help.
Cheers
Gen
On Tue, Feb 24, 2015 at 3:55 PM, Deep Pradhan pradhandeep1...@gmail.com
wrote:
Hi,
I have just signed up for Amazon AWS because I learnt that it provides
service for free for the first 12 months.
I want to run Spark on EC2
.
For your question: it means that you just download spark and unzip it.
Then run spark shell by ./bin/spark-shell or ./bin/pyspark. It is useful to
get familiar with spark. You can do this on your laptop as well as on ec2.
In fact, running ./ec2/spark-ec2 means launching spark standalone mode
SPARK_WORKER_CORES=6
Then I ran ~/spark-ec2/copy-dir /root/spark/conf to distribute to
slaves, but without any effect. Do I have to restart workers?
How to do that with spark-ec2?
Thanks.
kane.ist...@gmail.com wrote:
Hi,
I'm trying to change setting as described here:
http://spark.apache.org/docs/1.2.0/ec2-scripts.html
export SPARK_WORKER_CORES=6
Then I ran ~/spark-ec2/copy-dir /root/spark/conf to distribute to
slaves, but without any effect. Do I have to restart workers?
How
Hi,
I'm trying to change setting as described here:
http://spark.apache.org/docs/1.2.0/ec2-scripts.html
export SPARK_WORKER_CORES=6
Then I ran ~/spark-ec2/copy-dir /root/spark/conf to distribute to slaves,
but without any effect. Do I have to restart workers?
How to do that with spark-ec2
We found the problem and already fixed it. Basically, spark-ec2 requires ec2
instances to have external ip addresses. You need to specify this in the ASW
console.
From: nicholas.cham...@gmail.com
Date: Tue, 27 Jan 2015 17:19:21 +
Subject: Re: spark 1.2 ec2 launch script hang
Hmm, I can’t see why using ~ would be problematic, especially if you
confirm that echo ~/path/to/pem expands to the correct path to your
identity file.
If you have a simple reproduction of the problem, please send it over. I’d
love to look into this. When I pass paths with ~ to spark-ec2 on my
that echo ~/path/to/pem expands to the correct path to your
identity file.
If you have a simple reproduction of the problem, please send it over. I’d
love to look into this. When I pass paths with ~ to spark-ec2 on my system,
it works fine. I’m using bash, but zsh handles tilde expansion the same
that will be fixed as part of
SPARK-5242 https://issues.apache.org/jira/browse/SPARK-5242.
Charles,
Thanks for the info. In your case, when does spark-ec2 hang? Only when the
specified path to the identity file doesn't exist? Or also when you specify
the path as a relative path or with ~?
Nick
Thanks for sending this over, Peter.
What if you try this? (i.e. Remove the = after --identity-file.)
ec2/spark-ec2 --key-pair=spark-streaming-kp --identity-file
~/.pzkeys/spark-streaming-kp.pem --region=us-east-1 login
pz-spark-cluster
If that works, then I think the problem in this case
if you try this? (i.e. Remove the = after --identity-file.)
ec2/spark-ec2 --key-pair=spark-streaming-kp --identity-file
~/.pzkeys/spark-streaming-kp.pem --region=us-east-1 login pz-spark-cluster
If that works, then I think the problem in this case is simply that Bash
cannot expand the tilde
Below is trace from trying to access with ~/path. I also did the echo as
per Nick (see the last line), looks ok to me. This is my development box
with Spark 1.2.0 running CentOS 6.5, Python 2.6.6
[pete.zybrick@pz-lt2-ipc spark-1.2.0]$ ec2/spark-ec2
--key-pair=spark-streaming-kp
--identity-file
an absolute path to the pem file
On Jan 26, 2015, at 8:57 PM, ey-chih chow eyc...@hotmail.com wrote:
Hi,
I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
modified the script according to
https://github.com/grzegorz-dubicki/spark/commit
For those who found that absolute vs. relative path for the pem file
mattered, what OS and shell are you using? What version of Spark are you
using?
~/ vs. absolute path shouldn’t matter. Your shell will expand the ~/ to the
absolute path before sending it to spark-ec2. (i.e. tilde expansion
Try using an absolute path to the pem file
On Jan 26, 2015, at 8:57 PM, ey-chih chow eyc...@hotmail.com wrote:
Hi,
I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
modified the script according to
https://github.com/grzegorz-dubicki/spark/commit
Hi,
I used the spark-ec2 script of spark 1.2 to launch a cluster. I have
modified the script according to
https://github.com/grzegorz-dubicki/spark/commit/5dd8458d2ab9753aae939b3bb33be953e2c13a70
But the script was still hung at the following message:
Waiting for cluster to enter 'ssh-ready
will
finish the launch of cluster.
Cheers
Gen
On Sat, Jan 17, 2015 at 7:00 PM, Nathan Murthy nathan.mur...@gmail.com
wrote:
Originally posted here:
http://stackoverflow.com/questions/28002443/cluster-hangs-in-ssh-ready-state-using-spark-1-2-ec2-launch-script
I'm trying to launch a standalone
Originally posted here:
http://stackoverflow.com/questions/28002443/cluster-hangs-in-ssh-ready-state-using-spark-1-2-ec2-launch-script
I'm trying to launch a standalone Spark cluster using its pre-packaged EC2
scripts, but it just indefinitely hangs in an 'ssh-ready' state:
ubuntu@machine
will
finish the launch of cluster.
Cheers
Gen
On Sat, Jan 17, 2015 at 7:00 PM, Nathan Murthy nathan.mur...@gmail.com
wrote:
Originally posted here:
http://stackoverflow.com/questions/28002443/cluster-hangs-in-ssh-ready-state-using-spark-1-2-ec2-launch-script
I'm trying to launch a standalone Spark
Hi,
Im facing this error on spark ec2 cluster when a job is submitted its says
that native hadoop libraries are not found I have checked spark-env.sh and
all the folders in the path but unable to find the problem even though the
folder are containing. are there any performance drawbacks if we use
KMeans
from numpy import array
from math import sqrt
conf = (SparkConf()
.setMaster(spark://ec2-54-207-84-167.sa-east-1.compute.amazonaws.com:7077)
.setAppName(Kmeans App)
.set(spark.akka.frameSize, 20)
.set(spark.executor.memory, 2048m))
sc = SparkContext(conf = conf)
# Load
To: user@spark.apache.org
Sent: Thursday, September 18, 2014 11:48:03 AM
Subject: Spark on EC2
Hello, I am trying to run a python script that makes use of the kmeans MLIB and
I'm not getting anywhere. I'm using an c3.xlarge instance as master, and 10
c3.large instances as slaves. In the code I make
I'm also getting into same issue and is blocked here. Did any of you were
able to go past this issue? I tried using both ephimeral and
persistent-hdfs. I'm getting the same issue.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-on-EC2-using
What about down-scaling when I use Mesos, does that really deteriorate the
performance ? Otherwise we would probably go for spark on mesos on ec2 :)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494p12109.html
Sent
://polyglotprogramming.com
On Thu, Jul 31, 2014 at 8:37 PM, ratabora ratab...@gmail.com wrote:
Hey Dean! Thanks!
Did you try running this on a local environment or one generated by the
spark-ec2 script?
The environment I am running on is a 4 data node 1 master spark cluster
generated
Hey all,
I was able to spawn up a cluster, but when I'm trying to submit a simple
jar via spark-submit it fails to run. I am trying to run the simple
Standalone Application from the quickstart.
Oddly enough, I could get another application running through the
spark-shell. What am I doing wrong
Hey Dean! Thanks!
Did you try running this on a local environment or one generated by the
spark-ec2 script?
The environment I am running on is a 4 data node 1 master spark cluster
generated by the spark-ec2 script. I haven't modified anything in the
environment except for adding data
Any idea about the probable dates for this implementation. I believe it would
be a wonderful (and essential) functionality to gain more acceptance in the
community.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster
in the
community.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Down-scaling-Spark-on-EC2-cluster-tp10494p10639.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Hello,
We plan to use Spark on EC2 for our data science pipeline. We successfully
manage to set up cluster as-well-as launch and run applications on
remote-clusters. However, to enhance scalability we would like to implement
auto-scaling in EC2 for Spark applications. However, I did not find any
performance.
Thanks
Best Regards
On Wed, Jul 23, 2014 at 6:36 PM, Shubhabrata mail2shu...@gmail.com wrote:
Hello,
We plan to use Spark on EC2 for our data science pipeline. We successfully
manage to set up cluster as-well-as launch and run applications on
remote-clusters. However, to enhance
There is a JIRA issue to track adding such functionality to spark-ec2:
SPARK-2008 https://issues.apache.org/jira/browse/SPARK-2008 - Enhance
spark-ec2 to be able to add and remove slaves to an existing cluster
On Wed, Jul 23, 2014 at 10:12 AM, Akhil Das ak...@sigmoidanalytics.com
wrote:
Hi
Hi All,
I've used the spark-ec2 scripts to build a simple 1.0.1 Standalone cluster on
EC2. It appears that the spark-submit script is not bundled with a spark-ec2
install. Given that: What is the recommended way to execute spark jobs on a
standalone EC2 cluster? Spark-submit provides
The script should be there, in the spark/bin directory. What command did you
use to launch the cluster?
Matei
On Jul 14, 2014, at 1:12 PM, Josh Happoldt josh.happo...@trueffect.com wrote:
Hi All,
I've used the spark-ec2 scripts to build a simple 1.0.1 Standalone cluster on
EC2
:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-EC2-tp6638
I haven't set up AMI yet. I am just trying to run a simple job on the EC2
cluster. So, is setting up AMI a prerequisite for running simple Spark
example like org.apache.spark.examples.GroupByTest?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark
No, you don't have to set up your own AMI. Actually it's probably simpler
and less error prone if you let spark-ec2 manage that for you as you first
start to get comfortable with Spark. Just spin up a cluster without any
explicit mention of AMI and it will do the right thing.
2014년 6월 1일 일요일
this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-EC2-tp6638.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
I’ve been able to get CDH5 up and running on EC2 and according to Cloudera
Manager, Spark is running healthy.
But when I try to run spark-shell, I eventually get the error:
14/04/02 07:18:18 INFO client.AppClient$ClientActor: Connecting to master
spark://ip-172-xxx-xxx-xxx:7077...
14/04/02
72 matches
Mail list logo
