RE: FW: Submitting jobs to Spark EC2 cluster remotely

2015-02-23 Thread Oleg Shirokikh
be very useful to have the ability to submit jobs remotely. Thanks, Oleg -Original Message- From: Patrick Wendell [mailto:pwend...@gmail.com] Sent: Monday, February 23, 2015 1:22 AM To: Oleg Shirokikh Cc: user@spark.apache.org Subject: Re: FW: Submitting jobs to Spark EC2 cluster remotely

FW: Submitting jobs to Spark EC2 cluster remotely

2015-02-23 Thread Oleg Shirokikh
Patrick, I haven't changed the configs much. I just executed ec2-script to create 1 master, 2 slaves cluster. Then I try to submit the jobs from remote machine leaving all defaults configured by Spark scripts as default. I've tried to change configs as suggested in other mailing-list and stack

Re: FW: Submitting jobs to Spark EC2 cluster remotely

2015-02-23 Thread Patrick Wendell
What happens if you submit from the master node itself on ec2 (in client mode), does that work? What about in cluster mode? It would be helpful if you could print the full command that the executor is failing. That might show that spark.driver.host is being set strangely. IIRC we print the launch

Re: FW: Submitting jobs to Spark EC2 cluster remotely

2015-02-23 Thread Franc Carter
Shirokikh Cc: user@spark.apache.org Subject: Re: FW: Submitting jobs to Spark EC2 cluster remotely What happens if you submit from the master node itself on ec2 (in client mode), does that work? What about in cluster mode? It would be helpful if you could print the full command