Re: toree does not work out of the box

2016-11-15 Thread Marius van Niekerk 
So currently for spark 2.0 support you have to built the master branch
yourself.
On Tue, Nov 15, 2016 at 14:51 Ryan Ward Kelley  wrote:

> Hi,
>
> I've used Apache-Toree in the past and its a great product; however, I
> cannot get it to work properly using Spark 2.0.  I tried to raise an issue
> on the jira page but I don't have permission.  This was the best place I
> thought to ask about my issue.  If you know a better place, can you please
> recommend one?
>
> Here is my specs:
>
> OSX 10.11.latest
> spark 2.0.2
> scala 2.11.8
>
> Here is what I'm trying to do:
>
> Install using the instruction on the website:
>
> [rwkelley@vector incubator-toree]$  pip install --pre toree
>
> Collecting toree
>
> Requirement already satisfied: jupyter-core<5.0,>=4.0 in
> /usr/local/lib/python2.7/site-packages (from toree)
>
> Requirement already satisfied: jupyter-client<5.0,>=4.0 in
> /usr/local/lib/python2.7/site-packages (from toree)
>
> Requirement already satisfied: traitlets<5.0,>=4.0 in
> /usr/local/lib/python2.7/site-packages (from toree)
>
> Requirement already satisfied: pyzmq>=13 in
> /usr/local/lib/python2.7/site-packages (from
> jupyter-client<5.0,>=4.0->toree)
>
> Requirement already satisfied: ipython-genutils in
> /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree)
>
> Requirement already satisfied: decorator in
> /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree)
>
> Requirement already satisfied: enum34; python_version == "2.7" in
> /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree)
>
> Requirement already satisfied: six in
> /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree)
>
> Installing collected packages: toree
>
> Successfully installed toree-0.1.0.dev8
>
>
> Then I tried next command to install it to jupyter:
>
> [rwkelley@vector incubator-toree]$  jupyter toree install
>
> [ToreeInstall] Installing Apache Toree version 0.1.0.dev8
>
> [ToreeInstall]
>
> Apache Toree is an effort undergoing incubation at the Apache Software
>
> Foundation (ASF), sponsored by the Apache Incubator PMC.
>
>
> Incubation is required of all newly accepted projects until a further
> review
>
> indicates that the infrastructure, communications, and decision making
> process
>
> have stabilized in a manner consistent with other successful ASF projects.
>
>
> While incubation status is not necessarily a reflection of the completeness
>
> or stability of the code, it does indicate that the project has yet to be
>
> fully endorsed by the ASF.
>
>
> Additionally, this release is not fully compliant with Apache release
> policy
>
> and includes a runtime dependency that is licensed as LGPL v3 (plus a
> static
>
> linking exception). This package is currently under an effort to re-license
>
> (https://github.com/zeromq/jeromq/issues/327).
>
> [ToreeInstall] Creating kernel Scala
>
> [ToreeInstall] Removing existing kernelspec in
> /usr/local/share/jupyter/kernels/apache_toree_scala
>
> [ToreeInstall] Installed kernelspec apache_toree_scala in
> /usr/local/share/jupyter/kernels/apache_toree_scala
>
>
> Then I tried to lauch jupyter and start a toree kernel:
>
>
> [rwkelley@vector incubator-toree]$  jupyter notebook
>
> [I 10:57:25.926 NotebookApp] The port  is already in use, trying
> another port.
>
> [I 10:57:25.937 NotebookApp] Serving notebooks from local directory:
> /Users/rwkelley/Downloads/incubator-toree
>
> [I 10:57:25.937 NotebookApp] 0 active kernels
>
> [I 10:57:25.937 NotebookApp] The Jupyter Notebook is running at:
> http://localhost:8889/
>
> [I 10:57:25.938 NotebookApp] Use Control-C to stop this server and shut
> down all kernels (twice to skip confirmation).
>
> [I 10:57:31.623 NotebookApp] Creating new notebook in
>
> [I 10:57:32.164 NotebookApp] Kernel started:
> ab31e5cc-068b-4884-ab31-bb0dc4cf8286
>
> Starting Spark Kernel with
> SPARK_HOME=/usr/local/Cellar/apache-spark/2.0.2/libexec
>
> [W 10:57:32.275 NotebookApp] 404 GET
> /nbextensions/widgets/notebook/js/extension.js?v=20161115105725 (::1)
> 11.24ms referer=
>
> http://localhost:8889/notebooks/Untitled.ipynb?kernel_name=apache_toree_scala
>
> Warning: Ignoring non-spark config property:
> hive.metastore.schema.verification=true
>
> Ivy Default Cache set to: /Users/rwkelley/.ivy2/cache
>
> The jars for the packages stored in: /Users/rwkelley/.ivy2/jars
>
> :: loading settings :: url =
>
> jar:file:/usr/local/Cellar/apache-spark/2.0.2/libexec/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
>
> com.amazonaws#aws-java-sdk-core added as a dependency
>
> com.amazonaws#aws-java-sdk-redshift added as a dependency
>
> com.amazonaws#aws-java-sdk-s3 added as a dependency
>
> com.google.guava#guava added as a dependency
>
> org.apache.commons#commons-csv added as a dependency
>
> org.apache.hadoop#hadoop-aws added as a dependency
>
> mysql#mysql-connector-java added as a dependency
>
> org.apache.spark#spark-core_2.11 added as a

toree does not work out of the box

2016-11-15 Thread Ryan Ward Kelley 
Hi,

I've used Apache-Toree in the past and its a great product; however, I
cannot get it to work properly using Spark 2.0.  I tried to raise an issue
on the jira page but I don't have permission.  This was the best place I
thought to ask about my issue.  If you know a better place, can you please
recommend one?

Here is my specs:

OSX 10.11.latest
spark 2.0.2
scala 2.11.8

Here is what I'm trying to do:

Install using the instruction on the website:

[rwkelley@vector incubator-toree]$  pip install --pre toree

Collecting toree

Requirement already satisfied: jupyter-core<5.0,>=4.0 in
/usr/local/lib/python2.7/site-packages (from toree)

Requirement already satisfied: jupyter-client<5.0,>=4.0 in
/usr/local/lib/python2.7/site-packages (from toree)

Requirement already satisfied: traitlets<5.0,>=4.0 in
/usr/local/lib/python2.7/site-packages (from toree)

Requirement already satisfied: pyzmq>=13 in
/usr/local/lib/python2.7/site-packages (from
jupyter-client<5.0,>=4.0->toree)

Requirement already satisfied: ipython-genutils in
/usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree)

Requirement already satisfied: decorator in
/usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree)

Requirement already satisfied: enum34; python_version == "2.7" in
/usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree)

Requirement already satisfied: six in
/usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree)

Installing collected packages: toree

Successfully installed toree-0.1.0.dev8


Then I tried next command to install it to jupyter:

[rwkelley@vector incubator-toree]$  jupyter toree install

[ToreeInstall] Installing Apache Toree version 0.1.0.dev8

[ToreeInstall]

Apache Toree is an effort undergoing incubation at the Apache Software

Foundation (ASF), sponsored by the Apache Incubator PMC.


Incubation is required of all newly accepted projects until a further review

indicates that the infrastructure, communications, and decision making
process

have stabilized in a manner consistent with other successful ASF projects.


While incubation status is not necessarily a reflection of the completeness

or stability of the code, it does indicate that the project has yet to be

fully endorsed by the ASF.


Additionally, this release is not fully compliant with Apache release policy

and includes a runtime dependency that is licensed as LGPL v3 (plus a static

linking exception). This package is currently under an effort to re-license

(https://github.com/zeromq/jeromq/issues/327).

[ToreeInstall] Creating kernel Scala

[ToreeInstall] Removing existing kernelspec in
/usr/local/share/jupyter/kernels/apache_toree_scala

[ToreeInstall] Installed kernelspec apache_toree_scala in
/usr/local/share/jupyter/kernels/apache_toree_scala


Then I tried to lauch jupyter and start a toree kernel:


[rwkelley@vector incubator-toree]$  jupyter notebook

[I 10:57:25.926 NotebookApp] The port  is already in use, trying
another port.

[I 10:57:25.937 NotebookApp] Serving notebooks from local directory:
/Users/rwkelley/Downloads/incubator-toree

[I 10:57:25.937 NotebookApp] 0 active kernels

[I 10:57:25.937 NotebookApp] The Jupyter Notebook is running at:
http://localhost:8889/

[I 10:57:25.938 NotebookApp] Use Control-C to stop this server and shut
down all kernels (twice to skip confirmation).

[I 10:57:31.623 NotebookApp] Creating new notebook in

[I 10:57:32.164 NotebookApp] Kernel started:
ab31e5cc-068b-4884-ab31-bb0dc4cf8286

Starting Spark Kernel with
SPARK_HOME=/usr/local/Cellar/apache-spark/2.0.2/libexec

[W 10:57:32.275 NotebookApp] 404 GET
/nbextensions/widgets/notebook/js/extension.js?v=20161115105725 (::1)
11.24ms referer=
http://localhost:8889/notebooks/Untitled.ipynb?kernel_name=apache_toree_scala

Warning: Ignoring non-spark config property:
hive.metastore.schema.verification=true

Ivy Default Cache set to: /Users/rwkelley/.ivy2/cache

The jars for the packages stored in: /Users/rwkelley/.ivy2/jars

:: loading settings :: url =
jar:file:/usr/local/Cellar/apache-spark/2.0.2/libexec/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml

com.amazonaws#aws-java-sdk-core added as a dependency

com.amazonaws#aws-java-sdk-redshift added as a dependency

com.amazonaws#aws-java-sdk-s3 added as a dependency

com.google.guava#guava added as a dependency

org.apache.commons#commons-csv added as a dependency

org.apache.hadoop#hadoop-aws added as a dependency

mysql#mysql-connector-java added as a dependency

org.apache.spark#spark-core_2.11 added as a dependency

com.databricks#spark-redshift_2.11 added as a dependency

com.databricks#spark-xml_2.11 added as a dependency

org.postgresql#postgresql added as a dependency

:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0

confs: [default]

found com.amazonaws#aws-java-sdk-core;1.10.75 in list

found commons-logging#commons-logging;1.1.3 in list

found