Hi, I'm working with an infrastructure that already has its own web server set up on EC2. I would like to set up a separate spark cluster on EC2 with the scripts and have the web server submit jobs to this spark cluster.
Is it possible to do this? I'm getting some errors running the spark shell from the spark shell on the web server: "Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory". I have heard that it's not possible for any local computer to connect to the spark cluster, but I was wondering if other EC2 nodes could have their firewalls configured to allow this. We don't want to deploy the web server on the master node of the spark cluster. Thanks, -Matt Cheah
