Re: Development model

2017-01-31 Thread Luciano Resende 
On Tue, Jan 31, 2017 at 9:56 AM, Marius van Niekerk <
marius.v.niek...@gmail.com> wrote:

> I think just documenting development workflow and process properly will
> help a great deal.
> Building Toree is a non-trivial exercise sadly.
>
>
Completely agree that is a non-trivial exercise, but should not be.


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

[jira] (TOREE-362) How do I define ports specifically?

2017-01-31 Thread Luciano Resende (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TOREE-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luciano Resende resolved TOREE-362.
---
Resolution: Not A Problem

As [~jodersky] mentioned, Toree already exposed a way to override all the ports 
used by Toree.

> How do I define ports specifically?
> ---
>
> Key: TOREE-362
> URL: https://issues.apache.org/jira/browse/TOREE-362
> Project: TOREE
>  Issue Type: Bug
>Reporter: Richard Quan
>
> Hello,
> I am using https://github.com/apache/incubator-toree to connect to a Spark 
> instance.  For some reason, on my company's proxy, I can sometimes connect 
> and more often cannot.  It seems to depend on the port numbers i.e. the 533xx 
> numbers here: 
> 17/01/23 14:56:55 [INFO] o.a.t.Main$$anon$1 - Connection Profile: {
>   "stdin_port" : 53327,
>   "control_port" : 53328,
>   "hb_port" : 53329,
>   "shell_port" : 53325,
>   "iopub_port" : 53326,
>   "ip" : "127.0.0.1",
>   "transport" : "tcp",
>   "signature_scheme" : "hmac-sha256",
>   "key" : "0b022f16-eff6-4cd3-8056-3a81cd5f289a"
> }
> I am wondering if it is possible to specify the port numbers as it almost 
> seems random, using the run.sh file.  Perhaps the ${TOREE_OPTS}?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Development model

2017-01-31 Thread Marius van Niekerk 
I think just documenting development workflow and process properly will
help a great deal.
Building Toree is a non-trivial exercise sadly.


On Tue, 31 Jan 2017 at 11:14 Luciano Resende  wrote:

> On Mon, Jan 30, 2017 at 5:47 PM, Jakob Odersky  wrote:
>
> > Hi everyone,
> > I was wondering how everyone usually develops toree, specifically how
> > changes to toree are tested with a jupyter notebook? I couldn't find
> > any documentation on the website so I thought I'd ask here.
> >
> > I tried running the various makefile targets, including `make dev` and
> > `make jupyter`, however those targets use a docker image that already
> > contains a published upstream version of toree, which thereby
> > overrides any changes made to the local version. Running "make
> > release", then pip-installing the archive and finally running "jupyter
> > install toree" somehow also uses an upstream version of toree.
> >
> > Is there an easy way to build and install a local source tree?
> >
> > thanks,
> > --Jakob
> >
>
>
> It seems that the project has various, and mostly undocumented, utilities
> done via make. To me this is good, but don't usually follow the patterns of
> projects using a specific build system (e.g. maven, sbt, etc).
>
> IMHO I believe we should be able to conform with the standards of a basic
> sbt project, and enable compile, unit test, package jars and distribution
> and publish locally using vanilla sbt commands. And then use and document
> the make facilities for more complex tasks such as tests that require
> docker, environment setup, build a jupyter env, etc  Which will then make
> it easy for newcomers that are used to sbt to get started while they learn
> about the make magics.
>
> Thoughts ?
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
-- 
regards
Marius van Niekerk

Toree client programming model

2017-01-31 Thread Luciano Resende 
The Toree client seems to be a client utility to enable other applications
to interact with Toree/Spark. These applications are not based or aware of
the ZeroMQ protocol, but nevertheless, we are imposing them on a very
similar programming model while evaluating code using the client.

See an example below, where the application programmer has to handle
different options, where more than one option can be called per eval
result, and the order cannot be known prior to the execution time :

val exRes: DeferredExecution = client.execute(code)
.onResult(executeResult => {
  ...
}).onError(executeReplyError =>{
  ...
}).onSuccess(executeReplyOk => {
  ...
}).onStream(streamResult => {
  ...
})


I was wondering what are the community thoughts on enhancing the client to
abstract its programming model, and having all this abstracted and handled
on the client implementation, and the eval would return one promise (e.g.
onResult which one option would be status success or error) or at most two
promises (e.g. onSuccess and onError).

The only issue I haven't fully understood yet is how streaming is handled
and what are the implication of these changes on this scenario.

Thoughts?

-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/