Re: Networking issues with Spark on EC2

Hi, Nopes. I was trying to use EC2(due to a few constraints) where I faced the problem. With EMR, it works flawlessly. But, I would like to go back and use EC2 if I can fix this issue. Has anybody set up a spark cluster using plain EC2 machines. What steps did you follow? Thanks and Regards,

Re: Networking issues with Spark on EC2

Hi, Are you using EMR ? Natu On Sat, Sep 26, 2015 at 6:55 AM, SURAJ SHETH wrote: > Hi Ankur, > Thanks for the reply. > This is already done. > If I wait for a long amount of time(10 minutes), a few tasks get > successful even on slave nodes. Sometime, a fraction of the

Re: Networking issues with Spark on EC2

Hi Ankur, Thanks for the reply. This is already done. If I wait for a long amount of time(10 minutes), a few tasks get successful even on slave nodes. Sometime, a fraction of the tasks(20%) are completed on all the machines in the initial 5 seconds and then, it slows down drastically. Thanks and

Networking issues with Spark on EC2

Hi, I am using Spark 1.2 and facing network related issues while performing simple computations. This is a custom cluster set up using ec2 machines and spark prebuilt binary from apache site. The problem is only when we have workers on other machines(networking involved). Having a single node

Re: Networking issues with Spark on EC2

Hi Suraj, Spark uses a lot of ports to communicate between nodes. Probably your security group is restrictive and does not allow instances to communicate on all networks. The easiest way to resolve it is to add a Rule to allow all Inbound traffic on all ports (0-65535) to instances in same