Spark driver application can not connect to Spark-Master

2014-09-01 Thread moon soo Lee 
Hi, I'm developing an application with Spark.

My java application trying to creates spark context like


 Creating spark context 

public SparkContext createSparkContext(){
String execUri = System.getenv("SPARK_EXECUTOR_URI");
 String[] jars = SparkILoop.getAddedJars();
SparkConf conf = new SparkConf().setMaster(getMaster())
.setAppName("App name").setJars(jars)
 .set("spark.repl.class.uri", interpreter.intp().classServer().uri());
if (execUri != null) {
 conf.set("spark.executor.uri", execUri);
}
if (System.getenv("SPARK_HOME") != null) {
 conf.setSparkHome(System.getenv("SPARK_HOME"));
}
SparkContext sparkContext = new SparkContext(conf);
 return sparkContext;
}
 public String getMaster() {
 String envMaster = System.getenv().get("MASTER");
if(envMaster!=null) return envMaster;
String propMaster = System.getProperty("spark.master");
 if(propMaster!=null) return propMaster;
return "local[*]";
}


But when i call createSparkContext(), in driver side, i got logs like


-- My application's log -
 INFO [2014-09-01 17:28:37,092] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Changing view acls to: root
 INFO [2014-09-01 17:28:37,092] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(root)
 INFO [2014-09-01 17:28:37,093] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Starting HTTP Server
 INFO [2014-09-01 17:28:37,096] ({pool-1-thread-2}
Server.java[doStart]:272) - jetty-8.1.14.v20131031
 INFO [2014-09-01 17:28:37,099] ({pool-1-thread-2}
AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:46610
 INFO [2014-09-01 17:28:40,050] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Changing view acls to: root
 INFO [2014-09-01 17:28:40,050] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(root)
 INFO [2014-09-01 17:28:40,589] ({spark-akka.actor.default-dispatcher-2}
Slf4jLogger.scala[applyOrElse]:80) - Slf4jLogger started
 INFO [2014-09-01 17:28:40,626] ({spark-akka.actor.default-dispatcher-2}
Slf4jLogger.scala[apply$mcV$sp]:74) - Starting remoting
 INFO [2014-09-01 17:28:40,833] ({spark-akka.actor.default-dispatcher-3}
Slf4jLogger.scala[apply$mcV$sp]:74) - Remoting started; listening on
addresses :[akka.tcp://spark@222.122.122.122:46833]
 INFO [2014-09-01 17:28:40,835] ({spark-akka.actor.default-dispatcher-4}
Slf4jLogger.scala[apply$mcV$sp]:74) - Remoting now listens on addresses:
[akka.tcp://spark@222.122.122.122:46833]
 INFO [2014-09-01 17:28:40,858] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Registering MapOutputTracker
 INFO [2014-09-01 17:28:40,861] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Registering BlockManagerMaster
 INFO [2014-09-01 17:28:40,877] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Created local directory at
/tmp/spark-local-20140901172840-baf4
 INFO [2014-09-01 17:28:40,881] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - MemoryStore started with capacity 546.3 MB.
 INFO [2014-09-01 17:28:40,912] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Bound socket to port 42671 with id =
ConnectionManagerId(222.122.122.122,42671)
 INFO [2014-09-01 17:28:40,917] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Trying to register BlockManager
 INFO [2014-09-01 17:28:40,920] ({spark-akka.actor.default-dispatcher-4}
Logging.scala[logInfo]:58) - Registering block manager 222.122.122.122:42671
with 546.3 MB RAM
 INFO [2014-09-01 17:28:40,921] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Registered BlockManager
 INFO [2014-09-01 17:28:40,932] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Starting HTTP Server
 INFO [2014-09-01 17:28:40,933] ({pool-1-thread-2}
Server.java[doStart]:272) - jetty-8.1.14.v20131031
 INFO [2014-09-01 17:28:40,935] ({pool-1-thread-2}
AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:52020
 INFO [2014-09-01 17:28:40,936] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Broadcast server started at
http://222.122.122.122:52020
 INFO [2014-09-01 17:28:40,943] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - HTTP File server directory is
/tmp/spark-fc4cc226-c740-4cec-ad0f-6f88762d365c
 INFO [2014-09-01 17:28:40,943] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Starting HTTP Server
 INFO [2014-09-01 17:28:40,944] ({pool-1-thread-2}
Server.java[doStart]:272) - jetty-8.1.14.v20131031
 INFO [2014-09-01 17:28:40,946] ({pool-1-thread-2}
AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:59458
 INFO [2014-09-01 17:28:41,167] ({pool-1-thread-2}
Server.java[doStart]:272) - jetty-8.1.14.v20131031
 INFO [2014-09-01 17:28:41,177] ({pool-1-thread-2}
AbstractConnector.java[doStart]:338) - Started
SelectChannelConnector@0.0.0.0:4040
 INFO [2014-09-01 17:28:41,180] ({pool-1-thread-2}
Logging.scala[logInfo]:58) - Started SparkUI at http://222.122.122.122:4040
 INFO [2014-09-01 17:28:41,410] ({spark-akka.act

SparkContext and multi threads

2014-09-11 Thread moon soo Lee 
Hi,

I'm trying to make spark work on multithreads java application.
What i'm trying to do is,

- Create a Single SparkContext
- Create Multiple SparkILoop and SparkIMain
- Inject created SparkContext into SparkIMain interpreter.

Thread is created by every user request and take a SparkILoop and interpret
some code.

My problem is
 - If a thread take first SparkILoop instance, than everything works fine.
 - If a thread take other SparkILoop instance, Spark can not find closure /
case classes that i defined inside of interpreter.

I read some previous topic and I think it's related with SparkEnv and
ClosureCleaner. tried SparkEnv.set(env) with the env i can get right after
SparkContext created. i not still no class found exception.

Can anyone give me some idea?
Thanks.


Best,
moon

Re: REPL like interface for Spark

2014-09-29 Thread moon soo Lee 
Hi,

There is project called "Zeppelin".

You can checkout here
https://github.com/NFLabs/zeppelin

Homepage is here.
http://zeppelin-project.org/

It's notebook style tool (like databrics demo, scala-notebook) with nice
UI, with built-in Spark integration.

It's in active development, so don't hesitate ask questions, request
features to the mailing list.

Thanks.

- moon

On Mon, Sep 29, 2014 at 5:27 PM, andy petrella 
wrote:

> Heya,
>
> I started to port the scala-notebook to Spark some weeks ago (but doing it
> in my sparse time and for my Spark talks ^^). It's a WIP but works quite
> fine ftm, you can check my fork and branch over here:
> https://github.com/andypetrella/scala-notebook/tree/spark
>
> Feel free to ask any questions, I'll happy to help of course (PRs are more
> than welcome :-P)
>
> Cheers,
>
> aℕdy ℙetrella
> about.me/noootsab
> [image: aℕdy ℙetrella on about.me]
>
> 
>
> On Mon, Sep 29, 2014 at 10:19 AM, IT CTO  wrote:
>
>> Hi,
>> Has anyone know of a REPL interface for spark on GIT which support
>> similar user experience as presented by Databricks in there cloud demo?
>>
>> We are looking for something similar but one that can be deployed on
>> premise and not on the cloud.
>>
>> --
>> Eran | CTO
>>
>
>

Re: REPL like interface for Spark

2014-09-29 Thread moon soo Lee 
There're little histories about using Java.
Any feedback is welcomed warmly.


On Mon, Sep 29, 2014 at 5:57 PM, andy petrella 
wrote:

> However (I must say ^^) that it's funny that it has been build using usual
> plain old Java stuffs :-D.
>
> aℕdy ℙetrella
> about.me/noootsab
> [image: aℕdy ℙetrella on about.me]
>
> <http://about.me/noootsab>
>
> On Mon, Sep 29, 2014 at 10:51 AM, andy petrella 
> wrote:
>
>> Cool!!! I'll give it a try ASAP!
>>
>> aℕdy ℙetrella
>> about.me/noootsab
>> [image: aℕdy ℙetrella on about.me]
>>
>> <http://about.me/noootsab>
>>
>> On Mon, Sep 29, 2014 at 10:48 AM, moon soo Lee 
>> wrote:
>>
>>> Hi,
>>>
>>> There is project called "Zeppelin".
>>>
>>> You can checkout here
>>> https://github.com/NFLabs/zeppelin
>>>
>>> Homepage is here.
>>> http://zeppelin-project.org/
>>>
>>> It's notebook style tool (like databrics demo, scala-notebook) with nice
>>> UI, with built-in Spark integration.
>>>
>>> It's in active development, so don't hesitate ask questions, request
>>> features to the mailing list.
>>>
>>> Thanks.
>>>
>>> - moon
>>>
>>> On Mon, Sep 29, 2014 at 5:27 PM, andy petrella 
>>> wrote:
>>>
>>>> Heya,
>>>>
>>>> I started to port the scala-notebook to Spark some weeks ago (but doing
>>>> it in my sparse time and for my Spark talks ^^). It's a WIP but works quite
>>>> fine ftm, you can check my fork and branch over here:
>>>> https://github.com/andypetrella/scala-notebook/tree/spark
>>>>
>>>> Feel free to ask any questions, I'll happy to help of course (PRs are
>>>> more than welcome :-P)
>>>>
>>>> Cheers,
>>>>
>>>> aℕdy ℙetrella
>>>> about.me/noootsab
>>>> [image: aℕdy ℙetrella on about.me]
>>>>
>>>> <http://about.me/noootsab>
>>>>
>>>> On Mon, Sep 29, 2014 at 10:19 AM, IT CTO  wrote:
>>>>
>>>>> Hi,
>>>>> Has anyone know of a REPL interface for spark on GIT which support
>>>>> similar user experience as presented by Databricks in there cloud demo?
>>>>>
>>>>> We are looking for something similar but one that can be deployed on
>>>>> premise and not on the cloud.
>>>>>
>>>>> --
>>>>> Eran | CTO
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Interactive interface tool for spark

2014-10-08 Thread moon soo Lee 
Hi,

Please check Zeppelin, too.

http://zeppelin-project.org
https://github.com/nflabs/zeppelin

Which is similar to scala notebook.

Best,
moon

2014년 10월 9일 목요일, andy petrella님이 작성한 메시지:

> Sure! I'll post updates as well in the ML :-)
> I'm doing it on twitter for now (until doc is ready).
>
> The repo is there (branch spark) :
> https://github.com/andypetrella/scala-notebook/tree/spark
>
> Some tweets:
> * very first working stuff:
> https://twitter.com/noootsab/status/508758335982927872/photo/1
> * using graphx:
> https://twitter.com/noootsab/status/517073481104908289/photo/1
> * using sql (it has already evolved in order to declare variable names):
> https://twitter.com/noootsab/status/518917295226515456/photo/1
> * using ADAM+mllib:
> https://twitter.com/noootsab/status/511270449054220288/photo/1
>
> There are plenty of others stuffs but will need some time for the write-up
> (soon)
>
>
> cheers,
> andy
>
> aℕdy ℙetrella
> about.me/noootsab
> [image: aℕdy ℙetrella on about.me]
>
> 
>
> On Wed, Oct 8, 2014 at 4:57 PM, Michael Allman  > wrote:
>
>> Hi Andy,
>>
>> This sounds awesome. Please keep us posted. Meanwhile, can you share a
>> link to your project? I wasn't able to find it.
>>
>> Cheers,
>>
>> Michael
>>
>> On Oct 8, 2014, at 3:38 AM, andy petrella > > wrote:
>>
>> Heya
>>
>> You can check Zeppellin or my fork of the Scala notebook.
>> I'm going this week end to push some efforts on the doc, because it
>> supports for realtime graphing, Scala, SQL, dynamic loading of dependencies
>> and I started this morning a widget to track the progress of the jobs.
>> I'm quite happy with it so far, I used it with graphx, mllib, ADAM and
>> the Cassandra connector so far.
>> However, its major drawback is that it is a one man (best) effort ftm! :-S
>>  Le 8 oct. 2014 11:16, "Dai, Kevin" > > a écrit :
>>
>>>  Hi, All
>>>
>>>
>>>
>>> We need an interactive interface tool for spark in which we can run
>>> spark job and plot graph to explorer the data interactively.
>>>
>>> Ipython notebook is good, but it only support python (we want one
>>> supporting scala)…
>>>
>>>
>>>
>>> BR,
>>>
>>> Kevin.
>>>
>>>
>>>
>>>
>>>
>>
>>
>