Re: Use Spark as multi-threading library and deprecate web UI
Do you mean disable the web UI? spark.ui.enabled=false Sure, it's useful with master = local[*] too. On Thu, Feb 5, 2015 at 9:30 AM, Shuai Zheng szheng.c...@gmail.com wrote: Hi All, It might sounds weird, but I think spark is perfect to be used as a multi-threading library in some cases. The local mode will naturally boost multiple thread when required. Because it is more restrict and less chance to have potential bug in the code (because it is more data oriental, not thread oriental). Of course, it cannot be used for all cases, but in most of my applications, it is enough (90%). I want to hear other people’s idea about this. BTW: if I run spark in local mode, how to deprecate the web UI (default listen on 4040), because I don’t want to start the UI every time if I use spark as a local library. Regards, Shuai - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Use Spark as multi-threading library and deprecate web UI
You can use akka, that is the underlying Multithreading library Spark uses. On Thu, Feb 5, 2015 at 9:56 PM, Shuai Zheng szheng.c...@gmail.com wrote: Nice. I just try and it works. Thanks very much! And I notice there is below in the log: 15/02/05 11:19:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@NY02913D.global.local:8162] 15/02/05 11:19:10 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@NY02913D.global.local:8162/user/HeartbeatReceiver As I understand. The local mode will have driver and executors in the same java process. So is there any way for me to also disable above two listeners? Or they are not optional even in local mode? Regards, Shuai -Original Message- From: Sean Owen [mailto:so...@cloudera.com] Sent: Thursday, February 05, 2015 10:53 AM To: Shuai Zheng Cc: user@spark.apache.org Subject: Re: Use Spark as multi-threading library and deprecate web UI Do you mean disable the web UI? spark.ui.enabled=false Sure, it's useful with master = local[*] too. On Thu, Feb 5, 2015 at 9:30 AM, Shuai Zheng szheng.c...@gmail.com wrote: Hi All, It might sounds weird, but I think spark is perfect to be used as a multi-threading library in some cases. The local mode will naturally boost multiple thread when required. Because it is more restrict and less chance to have potential bug in the code (because it is more data oriental, not thread oriental). Of course, it cannot be used for all cases, but in most of my applications, it is enough (90%). I want to hear other people’s idea about this. BTW: if I run spark in local mode, how to deprecate the web UI (default listen on 4040), because I don’t want to start the UI every time if I use spark as a local library. Regards, Shuai - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- [image: Sigmoid Analytics] http://htmlsig.com/www.sigmoidanalytics.com *Arush Kharbanda* || Technical Teamlead ar...@sigmoidanalytics.com || www.sigmoidanalytics.com
RE: Use Spark as multi-threading library and deprecate web UI
Nice. I just try and it works. Thanks very much! And I notice there is below in the log: 15/02/05 11:19:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@NY02913D.global.local:8162] 15/02/05 11:19:10 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@NY02913D.global.local:8162/user/HeartbeatReceiver As I understand. The local mode will have driver and executors in the same java process. So is there any way for me to also disable above two listeners? Or they are not optional even in local mode? Regards, Shuai -Original Message- From: Sean Owen [mailto:so...@cloudera.com] Sent: Thursday, February 05, 2015 10:53 AM To: Shuai Zheng Cc: user@spark.apache.org Subject: Re: Use Spark as multi-threading library and deprecate web UI Do you mean disable the web UI? spark.ui.enabled=false Sure, it's useful with master = local[*] too. On Thu, Feb 5, 2015 at 9:30 AM, Shuai Zheng szheng.c...@gmail.com wrote: Hi All, It might sounds weird, but I think spark is perfect to be used as a multi-threading library in some cases. The local mode will naturally boost multiple thread when required. Because it is more restrict and less chance to have potential bug in the code (because it is more data oriental, not thread oriental). Of course, it cannot be used for all cases, but in most of my applications, it is enough (90%). I want to hear other people’s idea about this. BTW: if I run spark in local mode, how to deprecate the web UI (default listen on 4040), because I don’t want to start the UI every time if I use spark as a local library. Regards, Shuai - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
