Re: Interactive interface tool for spark
Hi, Please check Zeppelin, too. http://zeppelin-project.org https://github.com/nflabs/zeppelin Which is similar to scala notebook. Best, moon 2014년 10월 9일 목요일, andy petrellaandy.petre...@gmail.com님이 작성한 메시지: Sure! I'll post updates as well in the ML :-) I'm doing it on twitter for now (until doc is ready). The repo is there (branch spark) : https://github.com/andypetrella/scala-notebook/tree/spark Some tweets: * very first working stuff: https://twitter.com/noootsab/status/508758335982927872/photo/1 * using graphx: https://twitter.com/noootsab/status/517073481104908289/photo/1 * using sql (it has already evolved in order to declare variable names): https://twitter.com/noootsab/status/518917295226515456/photo/1 * using ADAM+mllib: https://twitter.com/noootsab/status/511270449054220288/photo/1 There are plenty of others stuffs but will need some time for the write-up (soon) cheers, andy aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] http://about.me/noootsab On Wed, Oct 8, 2014 at 4:57 PM, Michael Allman mich...@videoamp.com javascript:_e(%7B%7D,'cvml','mich...@videoamp.com'); wrote: Hi Andy, This sounds awesome. Please keep us posted. Meanwhile, can you share a link to your project? I wasn't able to find it. Cheers, Michael On Oct 8, 2014, at 3:38 AM, andy petrella andy.petre...@gmail.com javascript:_e(%7B%7D,'cvml','andy.petre...@gmail.com'); wrote: Heya You can check Zeppellin or my fork of the Scala notebook. I'm going this week end to push some efforts on the doc, because it supports for realtime graphing, Scala, SQL, dynamic loading of dependencies and I started this morning a widget to track the progress of the jobs. I'm quite happy with it so far, I used it with graphx, mllib, ADAM and the Cassandra connector so far. However, its major drawback is that it is a one man (best) effort ftm! :-S Le 8 oct. 2014 11:16, Dai, Kevin yun...@ebay.com javascript:_e(%7B%7D,'cvml','yun...@ebay.com'); a écrit : Hi, All We need an interactive interface tool for spark in which we can run spark job and plot graph to explorer the data interactively. Ipython notebook is good, but it only support python (we want one supporting scala)… BR, Kevin.
Re: REPL like interface for Spark
Hi, There is project called Zeppelin. You can checkout here https://github.com/NFLabs/zeppelin Homepage is here. http://zeppelin-project.org/ It's notebook style tool (like databrics demo, scala-notebook) with nice UI, with built-in Spark integration. It's in active development, so don't hesitate ask questions, request features to the mailing list. Thanks. - moon On Mon, Sep 29, 2014 at 5:27 PM, andy petrella andy.petre...@gmail.com wrote: Heya, I started to port the scala-notebook to Spark some weeks ago (but doing it in my sparse time and for my Spark talks ^^). It's a WIP but works quite fine ftm, you can check my fork and branch over here: https://github.com/andypetrella/scala-notebook/tree/spark Feel free to ask any questions, I'll happy to help of course (PRs are more than welcome :-P) Cheers, aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] http://about.me/noootsab On Mon, Sep 29, 2014 at 10:19 AM, IT CTO goi@gmail.com wrote: Hi, Has anyone know of a REPL interface for spark on GIT which support similar user experience as presented by Databricks in there cloud demo? We are looking for something similar but one that can be deployed on premise and not on the cloud. -- Eran | CTO
Re: REPL like interface for Spark
There're little histories about using Java. Any feedback is welcomed warmly. On Mon, Sep 29, 2014 at 5:57 PM, andy petrella andy.petre...@gmail.com wrote: However (I must say ^^) that it's funny that it has been build using usual plain old Java stuffs :-D. aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] http://about.me/noootsab On Mon, Sep 29, 2014 at 10:51 AM, andy petrella andy.petre...@gmail.com wrote: Cool!!! I'll give it a try ASAP! aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] http://about.me/noootsab On Mon, Sep 29, 2014 at 10:48 AM, moon soo Lee leemoon...@gmail.com wrote: Hi, There is project called Zeppelin. You can checkout here https://github.com/NFLabs/zeppelin Homepage is here. http://zeppelin-project.org/ It's notebook style tool (like databrics demo, scala-notebook) with nice UI, with built-in Spark integration. It's in active development, so don't hesitate ask questions, request features to the mailing list. Thanks. - moon On Mon, Sep 29, 2014 at 5:27 PM, andy petrella andy.petre...@gmail.com wrote: Heya, I started to port the scala-notebook to Spark some weeks ago (but doing it in my sparse time and for my Spark talks ^^). It's a WIP but works quite fine ftm, you can check my fork and branch over here: https://github.com/andypetrella/scala-notebook/tree/spark Feel free to ask any questions, I'll happy to help of course (PRs are more than welcome :-P) Cheers, aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] http://about.me/noootsab On Mon, Sep 29, 2014 at 10:19 AM, IT CTO goi@gmail.com wrote: Hi, Has anyone know of a REPL interface for spark on GIT which support similar user experience as presented by Databricks in there cloud demo? We are looking for something similar but one that can be deployed on premise and not on the cloud. -- Eran | CTO
SparkContext and multi threads
Hi, I'm trying to make spark work on multithreads java application. What i'm trying to do is, - Create a Single SparkContext - Create Multiple SparkILoop and SparkIMain - Inject created SparkContext into SparkIMain interpreter. Thread is created by every user request and take a SparkILoop and interpret some code. My problem is - If a thread take first SparkILoop instance, than everything works fine. - If a thread take other SparkILoop instance, Spark can not find closure / case classes that i defined inside of interpreter. I read some previous topic and I think it's related with SparkEnv and ClosureCleaner. tried SparkEnv.set(env) with the env i can get right after SparkContext created. i not still no class found exception. Can anyone give me some idea? Thanks. Best, moon
Spark driver application can not connect to Spark-Master
Hi, I'm developing an application with Spark.
My java application trying to creates spark context like
Creating spark context
public SparkContext createSparkContext(){
String execUri = System.getenv(SPARK_EXECUTOR_URI);
String[] jars = SparkILoop.getAddedJars();
SparkConf conf = new SparkConf().setMaster(getMaster())
.setAppName(App name).setJars(jars)
.set(spark.repl.class.uri, interpreter.intp().classServer().uri());
if (execUri != null) {
conf.set(spark.executor.uri, execUri);
}
if (System.getenv(SPARK_HOME) != null) {
conf.setSparkHome(System.getenv(SPARK_HOME));
}
SparkContext sparkContext = new SparkContext(conf);
return sparkContext;
}
public String getMaster() {
String envMaster = System.getenv().get(MASTER);
if(envMaster!=null) return envMaster;
String propMaster = System.getProperty(spark.master);
if(propMaster!=null) return propMaster;
return local[*];
}
But when i call createSparkContext(), in driver side, i got logs like
-- My application's log -
INFO [2014-09-01 17:28:37,092] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Changing view acls to: root
INFO [2014-09-01 17:28:37,092] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(root)
INFO [2014-09-01 17:28:37,093] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Starting HTTP Server
INFO [2014-09-01 17:28:37,096] ({pool-1-thread-2}
Server.java[doStart]:272) - jetty-8.1.14.v20131031
INFO [2014-09-01 17:28:37,099] ({pool-1-thread-2}
AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:46610
INFO [2014-09-01 17:28:40,050] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Changing view acls to: root
INFO [2014-09-01 17:28:40,050] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(root)
INFO [2014-09-01 17:28:40,589] ({spark-akka.actor.default-dispatcher-2}
Slf4jLogger.scala[applyOrElse]:80) - Slf4jLogger started
INFO [2014-09-01 17:28:40,626] ({spark-akka.actor.default-dispatcher-2}
Slf4jLogger.scala[apply$mcV$sp]:74) - Starting remoting
INFO [2014-09-01 17:28:40,833] ({spark-akka.actor.default-dispatcher-3}
Slf4jLogger.scala[apply$mcV$sp]:74) - Remoting started; listening on
addresses :[akka.tcp://spark@222.122.122.122:46833]
INFO [2014-09-01 17:28:40,835] ({spark-akka.actor.default-dispatcher-4}
Slf4jLogger.scala[apply$mcV$sp]:74) - Remoting now listens on addresses:
[akka.tcp://spark@222.122.122.122:46833]
INFO [2014-09-01 17:28:40,858] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Registering MapOutputTracker
INFO [2014-09-01 17:28:40,861] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Registering BlockManagerMaster
INFO [2014-09-01 17:28:40,877] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Created local directory at
/tmp/spark-local-20140901172840-baf4
INFO [2014-09-01 17:28:40,881] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - MemoryStore started with capacity 546.3 MB.
INFO [2014-09-01 17:28:40,912] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Bound socket to port 42671 with id =
ConnectionManagerId(222.122.122.122,42671)
INFO [2014-09-01 17:28:40,917] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Trying to register BlockManager
INFO [2014-09-01 17:28:40,920] ({spark-akka.actor.default-dispatcher-4}
Logging.scala[logInfo]:58) - Registering block manager 222.122.122.122:42671
with 546.3 MB RAM
INFO [2014-09-01 17:28:40,921] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Registered BlockManager
INFO [2014-09-01 17:28:40,932] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Starting HTTP Server
INFO [2014-09-01 17:28:40,933] ({pool-1-thread-2}
Server.java[doStart]:272) - jetty-8.1.14.v20131031
INFO [2014-09-01 17:28:40,935] ({pool-1-thread-2}
AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:52020
INFO [2014-09-01 17:28:40,936] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Broadcast server started at
http://222.122.122.122:52020
INFO [2014-09-01 17:28:40,943] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - HTTP File server directory is
/tmp/spark-fc4cc226-c740-4cec-ad0f-6f88762d365c
INFO [2014-09-01 17:28:40,943] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Starting HTTP Server
INFO [2014-09-01 17:28:40,944] ({pool-1-thread-2}
Server.java[doStart]:272) - jetty-8.1.14.v20131031
INFO [2014-09-01 17:28:40,946] ({pool-1-thread-2}
AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:59458
INFO [2014-09-01 17:28:41,167] ({pool-1-thread-2}
Server.java[doStart]:272) - jetty-8.1.14.v20131031
INFO [2014-09-01 17:28:41,177] ({pool-1-thread-2}
AbstractConnector.java[doStart]:338) - Started
SelectChannelConnector@0.0.0.0:4040
INFO [2014-09-01 17:28:41,180] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Started SparkUI at http://222.122.122.122:4040
INFO [2014-09-01 17:28:41,410]
