Re: FW: Submitting jobs to Spark EC2 cluster remotely

2015-02-23 Thread Franc Carter
verPropsFetcher' on port 34609. > /***/ > > > So it seems that there is a problem with starting executors... > > > Hopefully this clarifies the environment and workflow. I'd be happy to > provide any additional information. > > Again, thanks a lot

RE: FW: Submitting jobs to Spark EC2 cluster remotely

2015-02-23 Thread Oleg Shirokikh
in the cluster), it's extremely interesting to understand how remoting works with Spark. And in general it would be very useful to have the ability to submit jobs remotely. Thanks, Oleg -Original Message- From: Patrick Wendell [mailto:pwend...@gmail.com] Sent: Monday, Februa

Re: FW: Submitting jobs to Spark EC2 cluster remotely

2015-02-23 Thread Patrick Wendell
What happens if you submit from the master node itself on ec2 (in client mode), does that work? What about in cluster mode? It would be helpful if you could print the full command that the executor is failing. That might show that spark.driver.host is being set strangely. IIRC we print the launch

FW: Submitting jobs to Spark EC2 cluster remotely

2015-02-23 Thread Oleg Shirokikh
Patrick, I haven't changed the configs much. I just executed ec2-script to create 1 master, 2 slaves cluster. Then I try to submit the jobs from remote machine leaving all defaults configured by Spark scripts as default. I've tried to change configs as suggested in other mailing-list and stack