Hi Andrew,
for this test I only have one machine which provides the master and only
worker.
So all I'd need is communication to the Internet to access the twitter API.
I've tried assigning a specific port to the driver and creating iptables rules
for this port, but that didn't work.
Best regards,
Simon
On Aug 6, 2014 11:37 AM, "Andrew Or-2 [via Apache Spark User List]"
<[email protected]> wrote:
Hi Simon, The drivers and executors currently choose random ports to
talk to each other, so the Spark nodes will have to have full TCP access to
each other. This is changed in a very recent commit, where all of these random
ports will become configurable:
https://github.com/apache/spark/commit/09f7e4587bbdf74207d2629e8c1314f93d865999
. This will be available in Spark 1.1, but for now you will have to open all
ports among the nodes in your cluster.
-Andrew 2014-08-06 10:23 GMT-07:00 durin < [hidden email] >:
<blockquote style='border-left:2px solid #CCCCCC;padding:0 1em'
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">Update: I can get it to work by disabling
iptables temporarily. I can,
however, not figure out on which port I have to accept traffic. 4040 and any
of the Master or Worker ports mentioned in the previous post don't work.
Can it be one of the randomly assigned ones in the 30k to 60k range? Those
appear to change every time, making it difficult to apply any sensible
rules.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11556.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
If you reply to this email, your message will be added to the
discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11561.html
To unsubscribe from Spark Streaming fails - where is the
problem?, click here .
NAML
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11566.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
