Hello spark-users,
I am testing the behavior of remote job submission with ec2/spark_ec2.py in
spark distribution 1.5.2. I submit SparkPi to a remote ec2 instance using
spark-submit using the "standalone mode" (spark://) protocol. Connecting to
the master via ssh works, but submission fails. The server logs report:
Association with remote system [akka.tcp://sparkDriver@192.168.0.4:58498] has
failed
Use case: run Zeppelin to develop test, and save code on local machine with
spark local, but intermittently connect to an EC2 cluster to scale out. Thus
ssh to the master first for job submission is not acceptable.
Please find below a complete reproduction.
My questions:
1) Is this kind of remote submission over standalone mode port 7077 supported?
2) What is the root cause of the protocol failure?
3) Is there a spark-env.sh or other server-side setting which will make the
remote submission work?
Regards,
Jeff Henrikson
# reproduction shown with:
- "jq" json query
- pyenv, virtualenv
- awscli
# set configuration
export VPC=. . . # your VPC
export SPARK_HOME=. . ./spark-1.5.2-bin-hadoop2.6 # Just the binary
spark distribution 1.5.2
export IP4_SOURCE=. . . # the IP of the
gateway for my internet access
export KP=. . . # the name of a
keypair
# throughout, cluster is named "cluster2"
# region is us-west-2
# keypair given is ~/.ssh/$KP and registered in us-west-2 as $KP
# setup python/virtualenv
pushd $SPARK_HOME
pyenv local 2.7.6
cd $SPARK_HOME/ec2
virtualenv ../venv
../venv/bin/pip install awscli
# launch cluster
../venv/bin/python spark_ec2.py --vpc-id=$VPC --region=us-west-2
--instance-type=t2.medium --key-pair=KP -i ~/.ssh/KP launch cluster2
# authorize firewall port 7077
SG_MASTER=$(../venv/bin/aws ec2 describe-security-groups | jq -r
'.SecurityGroups[] | select(.["GroupName"] == "cluster2-master") | .GroupId')
../venv/bin/aws ec2 authorize-security-group-ingress --group-id
$SG_MASTER --protocol tcp --port 7077 --cidr $IP4_SOURCE/32
# verify connectivity to master port 7077
nc -v $DNS_MASTER 7077
ec2-. . . 7077 open
# locate ec2 public dns name
export DNS_MASTER=$(../venv/bin/aws ec2 describe-instances | jq -r
'.Reservations[].Instances[] | select(.SecurityGroups[].GroupName ==
"cluster2-master") | .PublicDnsName')
# submit job
SPARK_HOME/bin/spark-submit --master spark://$DNS_MASTER:7077
--driver-memory 1g --executor-memory 1g --executor-cores 1 --class
org.apache.spark.examples.SparkPi
$SPARK_HOME/lib/spark-examples-1.5.2-hadoop2.6.0.jar
# destroy cluster
../venv/bin/python spark_ec2.py --vpc-id=vpc-d35651b6
--region=us-west-2 --instance-type=t2.medium --key-pair=KP -i ~/.ssh/KP destroy
cluster2
# actual result:
16/02/23 12:28:36 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20160223201742-0000/21 on hostPort 172.31.13.146:40392 with 1 cores, 1024.0
MB RAM
16/02/23 12:28:36 INFO AppClient$ClientEndpoint: Executor updated:
app-20160223201742-0000/20 is now LOADING
16/02/23 12:28:36 INFO AppClient$ClientEndpoint: Executor updated:
app-20160223201742-0000/21 is now LOADING
16/02/23 12:28:36 INFO AppClient$ClientEndpoint: Executor updated:
app-20160223201742-0000/20 is now RUNNING
16/02/23 12:28:36 INFO AppClient$ClientEndpoint: Executor updated:
app-20160223201742-0000/21 is now RUNNING
16/02/23 12:28:43 WARN TaskSchedulerImpl: Initial job has not accepted
any resources; check your cluster UI to ensure that workers are registered and
have sufficient resources
# expected result:
Pi is approximately . . .
# actual result:
# tail logs
ssh -i ~/.ssh/KP root@$DNS_MASTER
sudo tail -f -n0 /root/spark/logs/*
16/02/23 20:42:42 INFO Master: 192.168.0.4:58498 got disassociated,
removing it.
16/02/23 20:42:42 WARN ReliableDeliverySupervisor: Association with
remote system [akka.tcp://sparkDriver@192.168.0.4:58498] has failed, address is
now gated for [5000] ms. Reason is: [Disassociated].
