So currently for spark 2.0 support you have to built the master branch yourself. On Tue, Nov 15, 2016 at 14:51 Ryan Ward Kelley <kelle...@gmail.com> wrote:
> Hi, > > I've used Apache-Toree in the past and its a great product; however, I > cannot get it to work properly using Spark 2.0. I tried to raise an issue > on the jira page but I don't have permission. This was the best place I > thought to ask about my issue. If you know a better place, can you please > recommend one? > > Here is my specs: > > OSX 10.11.latest > spark 2.0.2 > scala 2.11.8 > > Here is what I'm trying to do: > > Install using the instruction on the website: > > [rwkelley@vector incubator-toree]$ pip install --pre toree > > Collecting toree > > Requirement already satisfied: jupyter-core<5.0,>=4.0 in > /usr/local/lib/python2.7/site-packages (from toree) > > Requirement already satisfied: jupyter-client<5.0,>=4.0 in > /usr/local/lib/python2.7/site-packages (from toree) > > Requirement already satisfied: traitlets<5.0,>=4.0 in > /usr/local/lib/python2.7/site-packages (from toree) > > Requirement already satisfied: pyzmq>=13 in > /usr/local/lib/python2.7/site-packages (from > jupyter-client<5.0,>=4.0->toree) > > Requirement already satisfied: ipython-genutils in > /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree) > > Requirement already satisfied: decorator in > /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree) > > Requirement already satisfied: enum34; python_version == "2.7" in > /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree) > > Requirement already satisfied: six in > /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree) > > Installing collected packages: toree > > Successfully installed toree-0.1.0.dev8 > > > Then I tried next command to install it to jupyter: > > [rwkelley@vector incubator-toree]$ jupyter toree install > > [ToreeInstall] Installing Apache Toree version 0.1.0.dev8 > > [ToreeInstall] > > Apache Toree is an effort undergoing incubation at the Apache Software > > Foundation (ASF), sponsored by the Apache Incubator PMC. > > > Incubation is required of all newly accepted projects until a further > review > > indicates that the infrastructure, communications, and decision making > process > > have stabilized in a manner consistent with other successful ASF projects. > > > While incubation status is not necessarily a reflection of the completeness > > or stability of the code, it does indicate that the project has yet to be > > fully endorsed by the ASF. > > > Additionally, this release is not fully compliant with Apache release > policy > > and includes a runtime dependency that is licensed as LGPL v3 (plus a > static > > linking exception). This package is currently under an effort to re-license > > (https://github.com/zeromq/jeromq/issues/327). > > [ToreeInstall] Creating kernel Scala > > [ToreeInstall] Removing existing kernelspec in > /usr/local/share/jupyter/kernels/apache_toree_scala > > [ToreeInstall] Installed kernelspec apache_toree_scala in > /usr/local/share/jupyter/kernels/apache_toree_scala > > > Then I tried to lauch jupyter and start a toree kernel: > > > [rwkelley@vector incubator-toree]$ jupyter notebook > > [I 10:57:25.926 NotebookApp] The port 8888 is already in use, trying > another port. > > [I 10:57:25.937 NotebookApp] Serving notebooks from local directory: > /Users/rwkelley/Downloads/incubator-toree > > [I 10:57:25.937 NotebookApp] 0 active kernels > > [I 10:57:25.937 NotebookApp] The Jupyter Notebook is running at: > http://localhost:8889/ > > [I 10:57:25.938 NotebookApp] Use Control-C to stop this server and shut > down all kernels (twice to skip confirmation). > > [I 10:57:31.623 NotebookApp] Creating new notebook in > > [I 10:57:32.164 NotebookApp] Kernel started: > ab31e5cc-068b-4884-ab31-bb0dc4cf8286 > > Starting Spark Kernel with > SPARK_HOME=/usr/local/Cellar/apache-spark/2.0.2/libexec > > [W 10:57:32.275 NotebookApp] 404 GET > /nbextensions/widgets/notebook/js/extension.js?v=20161115105725 (::1) > 11.24ms referer= > > http://localhost:8889/notebooks/Untitled.ipynb?kernel_name=apache_toree_scala > > Warning: Ignoring non-spark config property: > hive.metastore.schema.verification=true > > Ivy Default Cache set to: /Users/rwkelley/.ivy2/cache > > The jars for the packages stored in: /Users/rwkelley/.ivy2/jars > > :: loading settings :: url = > > jar:file:/usr/local/Cellar/apache-spark/2.0.2/libexec/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml > > com.amazonaws#aws-java-sdk-core added as a dependency > > com.amazonaws#aws-java-sdk-redshift added as a dependency > > com.amazonaws#aws-java-sdk-s3 added as a dependency > > com.google.guava#guava added as a dependency > > org.apache.commons#commons-csv added as a dependency > > org.apache.hadoop#hadoop-aws added as a dependency > > mysql#mysql-connector-java added as a dependency > > org.apache.spark#spark-core_2.11 added as a dependency > > com.databricks#spark-redshift_2.11 added as a dependency > > com.databricks#spark-xml_2.11 added as a dependency > > org.postgresql#postgresql added as a dependency > > :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 > > confs: [default] > > found com.amazonaws#aws-java-sdk-core;1.10.75 in list > > found commons-logging#commons-logging;1.1.3 in list > > found org.apache.httpcomponents#httpclient;4.3.6 in list > > found org.apache.httpcomponents#httpcore;4.3.3 in list > > found commons-codec#commons-codec;1.6 in central > > found com.fasterxml.jackson.core#jackson-databind;2.5.3 in central > > found com.fasterxml.jackson.core#jackson-annotations;2.5.0 in central > > found com.fasterxml.jackson.core#jackson-core;2.5.3 in central > > found com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.5.3 in > list > > found joda-time#joda-time;2.8.1 in list > > found com.amazonaws#aws-java-sdk-redshift;1.10.75 in list > > found com.amazonaws#aws-java-sdk-s3;1.10.75 in list > > found com.amazonaws#aws-java-sdk-kms;1.10.75 in list > > found com.google.guava#guava;19.0 in central > > found org.apache.commons#commons-csv;1.4 in central > > found org.apache.hadoop#hadoop-aws;2.7.3 in central > > found org.apache.hadoop#hadoop-common;2.7.3 in list > > found org.apache.hadoop#hadoop-annotations;2.7.3 in list > > found commons-cli#commons-cli;1.2 in list > > found org.apache.commons#commons-math3;3.1.1 in list > > found xmlenc#xmlenc;0.52 in list > > found commons-httpclient#commons-httpclient;3.1 in list > > found commons-io#commons-io;2.4 in list > > found commons-net#commons-net;3.1 in list > > found commons-collections#commons-collections;3.2.2 in list > > found javax.servlet#servlet-api;2.5 in central > > found org.mortbay.jetty#jetty;6.1.26 in list > > found org.mortbay.jetty#jetty-util;6.1.26 in list > > found com.sun.jersey#jersey-core;1.9 in list > > found com.sun.jersey#jersey-json;1.9 in list > > found org.codehaus.jettison#jettison;1.1 in list > > found com.sun.xml.bind#jaxb-impl;2.2.3-1 in list > > found javax.xml.bind#jaxb-api;2.2.2 in list > > found javax.xml.stream#stax-api;1.0-2 in list > > found javax.activation#activation;1.1 in list > > found org.codehaus.jackson#jackson-core-asl;1.9.13 in list > > found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in list > > found org.codehaus.jackson#jackson-jaxrs;1.9.13 in list > > found org.codehaus.jackson#jackson-xc;1.9.13 in list > > found com.sun.jersey#jersey-server;1.9 in list > > found asm#asm;3.2 in central > > found log4j#log4j;1.2.17 in list > > found net.java.dev.jets3t#jets3t;0.9.0 in central > > found com.jamesmurty.utils#java-xmlbuilder;0.4 in central > > found commons-lang#commons-lang;2.6 in list > > found commons-configuration#commons-configuration;1.6 in list > > found commons-digester#commons-digester;1.8 in list > > found commons-beanutils#commons-beanutils;1.7.0 in list > > found commons-beanutils#commons-beanutils-core;1.8.0 in list > > found org.slf4j#slf4j-api;1.7.10 in list > > found org.apache.avro#avro;1.7.4 in list > > found com.thoughtworks.paranamer#paranamer;2.3 in list > > found org.xerial.snappy#snappy-java;1.0.4.1 in list > > found org.apache.commons#commons-compress;1.4.1 in list > > found org.tukaani#xz;1.0 in list > > found com.google.protobuf#protobuf-java;2.5.0 in list > > found com.google.code.gson#gson;2.2.4 in list > > found org.apache.hadoop#hadoop-auth;2.7.3 in list > > found org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 in list > > found org.apache.directory.server#apacheds-i18n;2.0.0-M15 in list > > found org.apache.directory.api#api-asn1-api;1.0.0-M20 in list > > found org.apache.directory.api#api-util;1.0.0-M20 in list > > found org.apache.zookeeper#zookeeper;3.4.6 in list > > found org.slf4j#slf4j-log4j12;1.7.10 in list > > found io.netty#netty;3.6.2.Final in local-m2-cache > > found org.apache.curator#curator-framework;2.7.1 in list > > found org.apache.curator#curator-client;2.7.1 in list > > found com.jcraft#jsch;0.1.42 in central > > found org.apache.curator#curator-recipes;2.7.1 in list > > found com.google.code.findbugs#jsr305;3.0.0 in list > > found org.apache.htrace#htrace-core;3.1.0-incubating in list > > found javax.servlet.jsp#jsp-api;2.1 in list > > found jline#jline;0.9.94 in list > > found junit#junit;4.11 in central > > found org.hamcrest#hamcrest-core;1.3 in list > > found com.amazonaws#aws-java-sdk;1.7.4 in central > > found joda-time#joda-time;2.9.6 in central > > [2.9.6] joda-time#joda-time;[2.2,) > > found mysql#mysql-connector-java;6.0.4 in central > > found com.databricks#spark-redshift_2.11;3.0.0-preview1 in list > > found com.eclipsesource.minimal-json#minimal-json;0.9.4 in list > > found com.databricks#spark-avro_2.11;3.0.0 in list > > found org.apache.avro#avro;1.7.6 in central > > found org.xerial.snappy#snappy-java;1.0.5 in list > > found com.databricks#spark-xml_2.11;0.4.0 in list > > found org.postgresql#postgresql;9.4.1210.jre7 in central > > :: resolution report :: resolve 3816ms :: artifacts dl 30ms > > :: modules in use: > > asm#asm;3.2 from central in [default] > > com.amazonaws#aws-java-sdk;1.7.4 from central in [default] > > com.amazonaws#aws-java-sdk-core;1.10.75 from list in [default] > > com.amazonaws#aws-java-sdk-kms;1.10.75 from list in [default] > > com.amazonaws#aws-java-sdk-redshift;1.10.75 from list in [default] > > com.amazonaws#aws-java-sdk-s3;1.10.75 from list in [default] > > com.databricks#spark-avro_2.11;3.0.0 from list in [default] > > com.databricks#spark-redshift_2.11;3.0.0-preview1 from list in [default] > > com.databricks#spark-xml_2.11;0.4.0 from list in [default] > > com.eclipsesource.minimal-json#minimal-json;0.9.4 from list in [default] > > com.fasterxml.jackson.core#jackson-annotations;2.5.0 from central in > [default] > > com.fasterxml.jackson.core#jackson-core;2.5.3 from central in [default] > > com.fasterxml.jackson.core#jackson-databind;2.5.3 from central in [default] > > com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.5.3 from list in > [default] > > com.google.code.findbugs#jsr305;3.0.0 from list in [default] > > com.google.code.gson#gson;2.2.4 from list in [default] > > com.google.guava#guava;19.0 from central in [default] > > com.google.protobuf#protobuf-java;2.5.0 from list in [default] > > com.jamesmurty.utils#java-xmlbuilder;0.4 from central in [default] > > com.jcraft#jsch;0.1.42 from central in [default] > > com.sun.jersey#jersey-core;1.9 from list in [default] > > com.sun.jersey#jersey-json;1.9 from list in [default] > > com.sun.jersey#jersey-server;1.9 from list in [default] > > com.sun.xml.bind#jaxb-impl;2.2.3-1 from list in [default] > > com.thoughtworks.paranamer#paranamer;2.3 from list in [default] > > commons-beanutils#commons-beanutils;1.7.0 from list in [default] > > commons-beanutils#commons-beanutils-core;1.8.0 from list in [default] > > commons-cli#commons-cli;1.2 from list in [default] > > commons-codec#commons-codec;1.6 from central in [default] > > commons-collections#commons-collections;3.2.2 from list in [default] > > commons-configuration#commons-configuration;1.6 from list in [default] > > commons-digester#commons-digester;1.8 from list in [default] > > commons-httpclient#commons-httpclient;3.1 from list in [default] > > commons-io#commons-io;2.4 from list in [default] > > commons-lang#commons-lang;2.6 from list in [default] > > commons-logging#commons-logging;1.1.3 from list in [default] > > commons-net#commons-net;3.1 from list in [default] > > io.netty#netty;3.6.2.Final from local-m2-cache in [default] > > javax.activation#activation;1.1 from list in [default] > > javax.servlet#servlet-api;2.5 from central in [default] > > javax.servlet.jsp#jsp-api;2.1 from list in [default] > > javax.xml.bind#jaxb-api;2.2.2 from list in [default] > > javax.xml.stream#stax-api;1.0-2 from list in [default] > > jline#jline;0.9.94 from list in [default] > > joda-time#joda-time;2.9.6 from central in [default] > > junit#junit;4.11 from central in [default] > > log4j#log4j;1.2.17 from list in [default] > > mysql#mysql-connector-java;6.0.4 from central in [default] > > net.java.dev.jets3t#jets3t;0.9.0 from central in [default] > > org.apache.avro#avro;1.7.6 from central in [default] > > org.apache.commons#commons-compress;1.4.1 from list in [default] > > org.apache.commons#commons-csv;1.4 from central in [default] > > org.apache.commons#commons-math3;3.1.1 from list in [default] > > org.apache.curator#curator-client;2.7.1 from list in [default] > > org.apache.curator#curator-framework;2.7.1 from list in [default] > > org.apache.curator#curator-recipes;2.7.1 from list in [default] > > org.apache.directory.api#api-asn1-api;1.0.0-M20 from list in [default] > > org.apache.directory.api#api-util;1.0.0-M20 from list in [default] > > org.apache.directory.server#apacheds-i18n;2.0.0-M15 from list in [default] > > org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 from list in > [default] > > org.apache.hadoop#hadoop-annotations;2.7.3 from list in [default] > > org.apache.hadoop#hadoop-auth;2.7.3 from list in [default] > > org.apache.hadoop#hadoop-aws;2.7.3 from central in [default] > > org.apache.hadoop#hadoop-common;2.7.3 from list in [default] > > org.apache.htrace#htrace-core;3.1.0-incubating from list in [default] > > org.apache.httpcomponents#httpclient;4.3.6 from list in [default] > > org.apache.httpcomponents#httpcore;4.3.3 from list in [default] > > org.apache.zookeeper#zookeeper;3.4.6 from list in [default] > > org.codehaus.jackson#jackson-core-asl;1.9.13 from list in [default] > > org.codehaus.jackson#jackson-jaxrs;1.9.13 from list in [default] > > org.codehaus.jackson#jackson-mapper-asl;1.9.13 from list in [default] > > org.codehaus.jackson#jackson-xc;1.9.13 from list in [default] > > org.codehaus.jettison#jettison;1.1 from list in [default] > > org.hamcrest#hamcrest-core;1.3 from list in [default] > > org.mortbay.jetty#jetty;6.1.26 from list in [default] > > org.mortbay.jetty#jetty-util;6.1.26 from list in [default] > > org.postgresql#postgresql;9.4.1210.jre7 from central in [default] > > org.slf4j#slf4j-api;1.7.10 from list in [default] > > org.slf4j#slf4j-log4j12;1.7.10 from list in [default] > > org.tukaani#xz;1.0 from list in [default] > > org.xerial.snappy#snappy-java;1.0.5 from list in [default] > > xmlenc#xmlenc;0.52 from list in [default] > > :: evicted modules: > > joda-time#joda-time;2.8.1 by [joda-time#joda-time;2.9.6] in [default] > > com.google.guava#guava;11.0.2 by [com.google.guava#guava;19.0] in [default] > > commons-codec#commons-codec;1.4 by [commons-codec#commons-codec;1.6] in > [default] > > org.apache.httpcomponents#httpclient;4.2.5 by > [org.apache.httpcomponents#httpclient;4.3.6] in [default] > > org.apache.httpcomponents#httpcore;4.2.5 by > [org.apache.httpcomponents#httpcore;4.3.3] in [default] > > org.apache.avro#avro;1.7.4 by [org.apache.avro#avro;1.7.6] in [default] > > org.xerial.snappy#snappy-java;1.0.4.1 by > [org.xerial.snappy#snappy-java;1.0.5] in [default] > > com.fasterxml.jackson.core#jackson-databind;2.2.3 by > [com.fasterxml.jackson.core#jackson-databind;2.5.3] in [default] > > com.fasterxml.jackson.core#jackson-annotations;2.2.3 by > [com.fasterxml.jackson.core#jackson-annotations;2.5.0] in [default] > > com.fasterxml.jackson.core#jackson-core;2.2.3 by > [com.fasterxml.jackson.core#jackson-core;2.5.3] in [default] > > org.slf4j#slf4j-api;1.7.5 by [org.slf4j#slf4j-api;1.7.10] in [default] > > org.slf4j#slf4j-api;1.6.4 by [org.slf4j#slf4j-api;1.7.10] in [default] > > --------------------------------------------------------------------- > > | | modules || artifacts | > > | conf | number| search|dwnlded|evicted|| number|dwnlded| > > --------------------------------------------------------------------- > > | default | 94 | 7 | 6 | 12 || 82 | 0 | > > --------------------------------------------------------------------- > > > :: problems summary :: > > :::: ERRORS > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver null > > > unknown resolver null > > > unknown resolver null > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver null > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver null > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver sbt-chain > > > unknown resolver null > > > unknown resolver sbt-chain > > > > :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS > > :: retrieving :: org.apache.spark#spark-submit-parent > > confs: [default] > > 0 artifacts copied, 82 already retrieved (0kB/17ms) > > Exception in thread "main" java.lang.NoSuchMethodError: > > scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet; > > at akka.actor.ActorCell$.<init>(ActorCell.scala:336) > > at akka.actor.ActorCell$.<clinit>(ActorCell.scala) > > at akka.actor.RootActorPath.$div(ActorPath.scala:185) > > at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:465) > > at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:453) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at > > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78) > > at scala.util.Try$.apply(Try.scala:192) > > at > > akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73) > > at > > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) > > at > > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) > > at scala.util.Success.flatMap(Try.scala:231) > > at > > akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84) > > at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585) > > at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:578) > > at akka.actor.ActorSystem$.apply(ActorSystem.scala:142) > > at akka.actor.ActorSystem$.apply(ActorSystem.scala:109) > > at > > org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71) > > at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:35) > > at > > org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60) > > at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:35) > > at > org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:72) > > at org.apache.toree.Main$delayedInit$body.apply(Main.scala:40) > > at scala.Function0$class.apply$mcV$sp(Function0.scala:34) > > at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) > > at scala.App$$anonfun$main$1.apply(App.scala:76) > > at scala.App$$anonfun$main$1.apply(App.scala:76) > > at scala.collection.immutable.List.foreach(List.scala:381) > > at > > scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) > > at scala.App$class.main(App.scala:76) > > at org.apache.toree.Main$.main(Main.scala:24) > > at org.apache.toree.Main.main(Main.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) > > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) > > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) > > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > [W 10:57:42.178 NotebookApp] Timeout waiting for kernel_info reply from > ab31e5cc-068b-4884-ab31-bb0dc4cf8286 > > So at this point, I don't know how to use Toree. I tried to build it from > source but there are no instruction on how to install from source. > > Can you guys take a look at this? > > Thanks, > Ryan > -- regards Marius van Niekerk
