Spark 1.2 – How to change Default (Random) port ….
Hello,
Recently, I have upgraded my setup to Spark 1.2 from Spark 1.1.
I have 4 node Ubuntu Spark Cluster.
With Spark 1.1, I used to write Spark Scala program in Eclipse on my Windows
development host and submit the job on Ubuntu Cluster, from Eclipse (Windows
machine).
As on my network not all ports between Spark cluster and development machine
are open, I set spark process ports to valid ports.
On Spark 1.1 this works perfectly.
When I try to run the same program with same user defined ports on Spark 1.2
cluster it gives me connection time out for port *56117*.
I referred the Spark 1.2 configuration page
(http://spark.apache.org/docs/1.2.0/configuration.html) but there are no new
ports mentioned.
*Here is my code for reference:*
val conf = new SparkConf()
.setMaster(sparkMaster)
.setAppName("Spark SVD")
.setSparkHome("/usr/local/spark")
.setJars(jars)
.set("spark.driver.host",
"consb2a") //Windows host
(Development machine)
.set("spark.driver.port",
"51810")
.set("spark.fileserver.port",
"51811")
.set("spark.broadcast.port",
"51812")
.set("spark.replClassServer.port", "51813")
.set("spark.blockManager.port",
"51814")
.set("spark.executor.port",
"51815")
.set("spark.executor.memory",
"2g")
.set("spark.driver.memory",
"4g")
val sc = new SparkContext(conf)
*Here is Exception:*
15/01/21 15:44:08 INFO BlockManagerMasterActor: Registering block manager
wynchcs217.wyn.cnw.co.nz:37173 with 1059.9 MB RAM, BlockManagerId(2,
wynchcs217.wyn.cnw.co.nz, 37173)
15/01/21 15:44:08 INFO BlockManagerMasterActor: Registering block manager
wynchcs219.wyn.cnw.co.nz:53850 with 1059.9 MB RAM, BlockManagerId(1,
wynchcs219.wyn.cnw.co.nz, 53850)
15/01/21 15:44:08 INFO BlockManagerMasterActor: Registering block manager
wynchcs220.wyn.cnw.co.nz:35670 with 1060.3 MB RAM, BlockManagerId(0,
wynchcs220.wyn.cnw.co.nz, 35670)
15/01/21 15:44:08 INFO BlockManagerMasterActor: Registering block manager
wynchcs218.wyn.cnw.co.nz:46890 with 1059.9 MB RAM, BlockManagerId(3,
wynchcs218.wyn.cnw.co.nz, 46890)
15/01/21 15:52:23 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
wynchcs217.wyn.cnw.co.nz): java.io.IOException: Connecting to
CONSB2A.cnw.co.nz/143.96.130.27:56117 timed out (12 ms)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:188)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156)
at
org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:78)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
15/01/21 15:52:23 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID
2, wynchcs220.wyn.cnw.co.nz, NODE_LOCAL, 1366 bytes)
15/01/21 15:55:35 INFO TaskSchedulerImpl: Cancelling stage 0
15/01/21 15:55:35 INFO TaskSchedulerImpl: Stage 0 was cancelled
15/01/21 15:55:35 INFO DAGScheduler: Job 0 failed: count at
RowMatrix.scala:76, took 689.331309 s
Exception in thread "main" org.apache.spark.SparkException: Job 0 cancelled
because Stage 0 was cancelled
Can you please let me know how can I define the port 56117 to some other
port ?
Thanks,
Shailesh
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-How-to-change-Default-Random-port-tp21306.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark 1.2 – How to change Default (Random) port ….
Can anyone please let me know ? I don't want to open all ports on n/w. So, am interested in the property by which this new port I can configure. Shailesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-How-to-change-Default-Random-port-tp21306p21360.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark 1.2 – How to change Default (Random) port ….
This was a regression caused by Netty Block Transfer Service. The fix for this just barely missed the 1.2 release, and you can see the associated JIRA here: https://issues.apache.org/jira/browse/SPARK-4837 Current master has the fix, and the Spark 1.2.1 release will have it included. If you don't want to rebuild from master or wait, then you can turn it off by setting "spark.shuffle.blockTransferService" to "nio". On Sun, Jan 25, 2015 at 6:28 PM, Shailesh Birari wrote: > Can anyone please let me know ? > I don't want to open all ports on n/w. So, am interested in the property by > which this new port I can configure. > > Shailesh > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-How-to-change-Default-Random-port-tp21306p21360.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Spark 1.2 – How to change Default (Random) port ….
Thanks. But after setting "spark.shuffle.blockTransferService" to "nio" application fails with Akka Client disassociation. 15/01/27 13:38:11 ERROR TaskSchedulerImpl: Lost executor 3 on wynchcs218.wyn.cnw.co.nz: remote Akka client disassociated 15/01/27 13:38:11 INFO TaskSetManager: Re-queueing tasks for 3 from TaskSet 0.0 15/01/27 13:38:11 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 7, wynchcs218.wyn.cnw.co.nz): ExecutorLostFailure (executor lost) 15/01/27 13:38:11 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job 15/01/27 13:38:11 WARN TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6, wynchcs218.wyn.cnw.co.nz): ExecutorLostFailure (executor lost) 15/01/27 13:38:11 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 15/01/27 13:38:11 INFO TaskSchedulerImpl: Cancelling stage 0 15/01/27 13:38:11 INFO DAGScheduler: Failed to run count at RowMatrix.scala:71 Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, wynchcs218.wyn.cnw.co.nz): ExecutorLostFailure (executor lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 15/01/27 13:38:11 INFO DAGScheduler: Executor lost: 3 (epoch 3) 15/01/27 13:38:11 INFO BlockManagerMasterActor: Trying to remove executor 3 from BlockManagerMaster. 15/01/27 13:38:11 INFO BlockManagerMaster: Removed 3 successfully in removeExecutor On Mon, Jan 26, 2015 at 6:34 PM, Aaron Davidson wrote: > This was a regression caused by Netty Block Transfer Service. The fix for > this just barely missed the 1.2 release, and you can see the associated > JIRA here: https://issues.apache.org/jira/browse/SPARK-4837 > > Current master has the fix, and the Spark 1.2.1 release will have it > included. If you don't want to rebuild from master or wait, then you can > turn it off by setting "spark.shuffle.blockTransferService" to "nio". > > On Sun, Jan 25, 2015 at 6:28 PM, Shailesh Birari > wrote: > >> Can anyone please let me know ? >> I don't want to open all ports on n/w. So, am interested in the property >> by >> which this new port I can configure. >> >> Shailesh >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-How-to-change-Default-Random-port-tp21306p21360.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
Re: Spark 1.2 – How to change Default (Random) port ….
Hi SM, Apologize for delayed response. No, the issue is with Spark 1.2.0. There is a bug in Spark 1.2.0. Recently Spark have latest 1.3.0 release so it might have fixed in it. I am not planning to test it soon, may be after some time. You can try for it. Regards, Shailesh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-How-to-change-Default-Random-port-tp21306p22063.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
