Hi, I've used Apache-Toree in the past and its a great product; however, I cannot get it to work properly using Spark 2.0. I tried to raise an issue on the jira page but I don't have permission. This was the best place I thought to ask about my issue. If you know a better place, can you please recommend one?
Here is my specs: OSX 10.11.latest spark 2.0.2 scala 2.11.8 Here is what I'm trying to do: Install using the instruction on the website: [rwkelley@vector incubator-toree]$ pip install --pre toree Collecting toree Requirement already satisfied: jupyter-core<5.0,>=4.0 in /usr/local/lib/python2.7/site-packages (from toree) Requirement already satisfied: jupyter-client<5.0,>=4.0 in /usr/local/lib/python2.7/site-packages (from toree) Requirement already satisfied: traitlets<5.0,>=4.0 in /usr/local/lib/python2.7/site-packages (from toree) Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python2.7/site-packages (from jupyter-client<5.0,>=4.0->toree) Requirement already satisfied: ipython-genutils in /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree) Requirement already satisfied: decorator in /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree) Requirement already satisfied: enum34; python_version == "2.7" in /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree) Requirement already satisfied: six in /usr/local/lib/python2.7/site-packages (from traitlets<5.0,>=4.0->toree) Installing collected packages: toree Successfully installed toree-0.1.0.dev8 Then I tried next command to install it to jupyter: [rwkelley@vector incubator-toree]$ jupyter toree install [ToreeInstall] Installing Apache Toree version 0.1.0.dev8 [ToreeInstall] Apache Toree is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. Additionally, this release is not fully compliant with Apache release policy and includes a runtime dependency that is licensed as LGPL v3 (plus a static linking exception). This package is currently under an effort to re-license (https://github.com/zeromq/jeromq/issues/327). [ToreeInstall] Creating kernel Scala [ToreeInstall] Removing existing kernelspec in /usr/local/share/jupyter/kernels/apache_toree_scala [ToreeInstall] Installed kernelspec apache_toree_scala in /usr/local/share/jupyter/kernels/apache_toree_scala Then I tried to lauch jupyter and start a toree kernel: [rwkelley@vector incubator-toree]$ jupyter notebook [I 10:57:25.926 NotebookApp] The port 8888 is already in use, trying another port. [I 10:57:25.937 NotebookApp] Serving notebooks from local directory: /Users/rwkelley/Downloads/incubator-toree [I 10:57:25.937 NotebookApp] 0 active kernels [I 10:57:25.937 NotebookApp] The Jupyter Notebook is running at: http://localhost:8889/ [I 10:57:25.938 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [I 10:57:31.623 NotebookApp] Creating new notebook in [I 10:57:32.164 NotebookApp] Kernel started: ab31e5cc-068b-4884-ab31-bb0dc4cf8286 Starting Spark Kernel with SPARK_HOME=/usr/local/Cellar/apache-spark/2.0.2/libexec [W 10:57:32.275 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20161115105725 (::1) 11.24ms referer= http://localhost:8889/notebooks/Untitled.ipynb?kernel_name=apache_toree_scala Warning: Ignoring non-spark config property: hive.metastore.schema.verification=true Ivy Default Cache set to: /Users/rwkelley/.ivy2/cache The jars for the packages stored in: /Users/rwkelley/.ivy2/jars :: loading settings :: url = jar:file:/usr/local/Cellar/apache-spark/2.0.2/libexec/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml com.amazonaws#aws-java-sdk-core added as a dependency com.amazonaws#aws-java-sdk-redshift added as a dependency com.amazonaws#aws-java-sdk-s3 added as a dependency com.google.guava#guava added as a dependency org.apache.commons#commons-csv added as a dependency org.apache.hadoop#hadoop-aws added as a dependency mysql#mysql-connector-java added as a dependency org.apache.spark#spark-core_2.11 added as a dependency com.databricks#spark-redshift_2.11 added as a dependency com.databricks#spark-xml_2.11 added as a dependency org.postgresql#postgresql added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found com.amazonaws#aws-java-sdk-core;1.10.75 in list found commons-logging#commons-logging;1.1.3 in list found org.apache.httpcomponents#httpclient;4.3.6 in list found org.apache.httpcomponents#httpcore;4.3.3 in list found commons-codec#commons-codec;1.6 in central found com.fasterxml.jackson.core#jackson-databind;2.5.3 in central found com.fasterxml.jackson.core#jackson-annotations;2.5.0 in central found com.fasterxml.jackson.core#jackson-core;2.5.3 in central found com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.5.3 in list found joda-time#joda-time;2.8.1 in list found com.amazonaws#aws-java-sdk-redshift;1.10.75 in list found com.amazonaws#aws-java-sdk-s3;1.10.75 in list found com.amazonaws#aws-java-sdk-kms;1.10.75 in list found com.google.guava#guava;19.0 in central found org.apache.commons#commons-csv;1.4 in central found org.apache.hadoop#hadoop-aws;2.7.3 in central found org.apache.hadoop#hadoop-common;2.7.3 in list found org.apache.hadoop#hadoop-annotations;2.7.3 in list found commons-cli#commons-cli;1.2 in list found org.apache.commons#commons-math3;3.1.1 in list found xmlenc#xmlenc;0.52 in list found commons-httpclient#commons-httpclient;3.1 in list found commons-io#commons-io;2.4 in list found commons-net#commons-net;3.1 in list found commons-collections#commons-collections;3.2.2 in list found javax.servlet#servlet-api;2.5 in central found org.mortbay.jetty#jetty;6.1.26 in list found org.mortbay.jetty#jetty-util;6.1.26 in list found com.sun.jersey#jersey-core;1.9 in list found com.sun.jersey#jersey-json;1.9 in list found org.codehaus.jettison#jettison;1.1 in list found com.sun.xml.bind#jaxb-impl;2.2.3-1 in list found javax.xml.bind#jaxb-api;2.2.2 in list found javax.xml.stream#stax-api;1.0-2 in list found javax.activation#activation;1.1 in list found org.codehaus.jackson#jackson-core-asl;1.9.13 in list found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in list found org.codehaus.jackson#jackson-jaxrs;1.9.13 in list found org.codehaus.jackson#jackson-xc;1.9.13 in list found com.sun.jersey#jersey-server;1.9 in list found asm#asm;3.2 in central found log4j#log4j;1.2.17 in list found net.java.dev.jets3t#jets3t;0.9.0 in central found com.jamesmurty.utils#java-xmlbuilder;0.4 in central found commons-lang#commons-lang;2.6 in list found commons-configuration#commons-configuration;1.6 in list found commons-digester#commons-digester;1.8 in list found commons-beanutils#commons-beanutils;1.7.0 in list found commons-beanutils#commons-beanutils-core;1.8.0 in list found org.slf4j#slf4j-api;1.7.10 in list found org.apache.avro#avro;1.7.4 in list found com.thoughtworks.paranamer#paranamer;2.3 in list found org.xerial.snappy#snappy-java;1.0.4.1 in list found org.apache.commons#commons-compress;1.4.1 in list found org.tukaani#xz;1.0 in list found com.google.protobuf#protobuf-java;2.5.0 in list found com.google.code.gson#gson;2.2.4 in list found org.apache.hadoop#hadoop-auth;2.7.3 in list found org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 in list found org.apache.directory.server#apacheds-i18n;2.0.0-M15 in list found org.apache.directory.api#api-asn1-api;1.0.0-M20 in list found org.apache.directory.api#api-util;1.0.0-M20 in list found org.apache.zookeeper#zookeeper;3.4.6 in list found org.slf4j#slf4j-log4j12;1.7.10 in list found io.netty#netty;3.6.2.Final in local-m2-cache found org.apache.curator#curator-framework;2.7.1 in list found org.apache.curator#curator-client;2.7.1 in list found com.jcraft#jsch;0.1.42 in central found org.apache.curator#curator-recipes;2.7.1 in list found com.google.code.findbugs#jsr305;3.0.0 in list found org.apache.htrace#htrace-core;3.1.0-incubating in list found javax.servlet.jsp#jsp-api;2.1 in list found jline#jline;0.9.94 in list found junit#junit;4.11 in central found org.hamcrest#hamcrest-core;1.3 in list found com.amazonaws#aws-java-sdk;1.7.4 in central found joda-time#joda-time;2.9.6 in central [2.9.6] joda-time#joda-time;[2.2,) found mysql#mysql-connector-java;6.0.4 in central found com.databricks#spark-redshift_2.11;3.0.0-preview1 in list found com.eclipsesource.minimal-json#minimal-json;0.9.4 in list found com.databricks#spark-avro_2.11;3.0.0 in list found org.apache.avro#avro;1.7.6 in central found org.xerial.snappy#snappy-java;1.0.5 in list found com.databricks#spark-xml_2.11;0.4.0 in list found org.postgresql#postgresql;9.4.1210.jre7 in central :: resolution report :: resolve 3816ms :: artifacts dl 30ms :: modules in use: asm#asm;3.2 from central in [default] com.amazonaws#aws-java-sdk;1.7.4 from central in [default] com.amazonaws#aws-java-sdk-core;1.10.75 from list in [default] com.amazonaws#aws-java-sdk-kms;1.10.75 from list in [default] com.amazonaws#aws-java-sdk-redshift;1.10.75 from list in [default] com.amazonaws#aws-java-sdk-s3;1.10.75 from list in [default] com.databricks#spark-avro_2.11;3.0.0 from list in [default] com.databricks#spark-redshift_2.11;3.0.0-preview1 from list in [default] com.databricks#spark-xml_2.11;0.4.0 from list in [default] com.eclipsesource.minimal-json#minimal-json;0.9.4 from list in [default] com.fasterxml.jackson.core#jackson-annotations;2.5.0 from central in [default] com.fasterxml.jackson.core#jackson-core;2.5.3 from central in [default] com.fasterxml.jackson.core#jackson-databind;2.5.3 from central in [default] com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.5.3 from list in [default] com.google.code.findbugs#jsr305;3.0.0 from list in [default] com.google.code.gson#gson;2.2.4 from list in [default] com.google.guava#guava;19.0 from central in [default] com.google.protobuf#protobuf-java;2.5.0 from list in [default] com.jamesmurty.utils#java-xmlbuilder;0.4 from central in [default] com.jcraft#jsch;0.1.42 from central in [default] com.sun.jersey#jersey-core;1.9 from list in [default] com.sun.jersey#jersey-json;1.9 from list in [default] com.sun.jersey#jersey-server;1.9 from list in [default] com.sun.xml.bind#jaxb-impl;2.2.3-1 from list in [default] com.thoughtworks.paranamer#paranamer;2.3 from list in [default] commons-beanutils#commons-beanutils;1.7.0 from list in [default] commons-beanutils#commons-beanutils-core;1.8.0 from list in [default] commons-cli#commons-cli;1.2 from list in [default] commons-codec#commons-codec;1.6 from central in [default] commons-collections#commons-collections;3.2.2 from list in [default] commons-configuration#commons-configuration;1.6 from list in [default] commons-digester#commons-digester;1.8 from list in [default] commons-httpclient#commons-httpclient;3.1 from list in [default] commons-io#commons-io;2.4 from list in [default] commons-lang#commons-lang;2.6 from list in [default] commons-logging#commons-logging;1.1.3 from list in [default] commons-net#commons-net;3.1 from list in [default] io.netty#netty;3.6.2.Final from local-m2-cache in [default] javax.activation#activation;1.1 from list in [default] javax.servlet#servlet-api;2.5 from central in [default] javax.servlet.jsp#jsp-api;2.1 from list in [default] javax.xml.bind#jaxb-api;2.2.2 from list in [default] javax.xml.stream#stax-api;1.0-2 from list in [default] jline#jline;0.9.94 from list in [default] joda-time#joda-time;2.9.6 from central in [default] junit#junit;4.11 from central in [default] log4j#log4j;1.2.17 from list in [default] mysql#mysql-connector-java;6.0.4 from central in [default] net.java.dev.jets3t#jets3t;0.9.0 from central in [default] org.apache.avro#avro;1.7.6 from central in [default] org.apache.commons#commons-compress;1.4.1 from list in [default] org.apache.commons#commons-csv;1.4 from central in [default] org.apache.commons#commons-math3;3.1.1 from list in [default] org.apache.curator#curator-client;2.7.1 from list in [default] org.apache.curator#curator-framework;2.7.1 from list in [default] org.apache.curator#curator-recipes;2.7.1 from list in [default] org.apache.directory.api#api-asn1-api;1.0.0-M20 from list in [default] org.apache.directory.api#api-util;1.0.0-M20 from list in [default] org.apache.directory.server#apacheds-i18n;2.0.0-M15 from list in [default] org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 from list in [default] org.apache.hadoop#hadoop-annotations;2.7.3 from list in [default] org.apache.hadoop#hadoop-auth;2.7.3 from list in [default] org.apache.hadoop#hadoop-aws;2.7.3 from central in [default] org.apache.hadoop#hadoop-common;2.7.3 from list in [default] org.apache.htrace#htrace-core;3.1.0-incubating from list in [default] org.apache.httpcomponents#httpclient;4.3.6 from list in [default] org.apache.httpcomponents#httpcore;4.3.3 from list in [default] org.apache.zookeeper#zookeeper;3.4.6 from list in [default] org.codehaus.jackson#jackson-core-asl;1.9.13 from list in [default] org.codehaus.jackson#jackson-jaxrs;1.9.13 from list in [default] org.codehaus.jackson#jackson-mapper-asl;1.9.13 from list in [default] org.codehaus.jackson#jackson-xc;1.9.13 from list in [default] org.codehaus.jettison#jettison;1.1 from list in [default] org.hamcrest#hamcrest-core;1.3 from list in [default] org.mortbay.jetty#jetty;6.1.26 from list in [default] org.mortbay.jetty#jetty-util;6.1.26 from list in [default] org.postgresql#postgresql;9.4.1210.jre7 from central in [default] org.slf4j#slf4j-api;1.7.10 from list in [default] org.slf4j#slf4j-log4j12;1.7.10 from list in [default] org.tukaani#xz;1.0 from list in [default] org.xerial.snappy#snappy-java;1.0.5 from list in [default] xmlenc#xmlenc;0.52 from list in [default] :: evicted modules: joda-time#joda-time;2.8.1 by [joda-time#joda-time;2.9.6] in [default] com.google.guava#guava;11.0.2 by [com.google.guava#guava;19.0] in [default] commons-codec#commons-codec;1.4 by [commons-codec#commons-codec;1.6] in [default] org.apache.httpcomponents#httpclient;4.2.5 by [org.apache.httpcomponents#httpclient;4.3.6] in [default] org.apache.httpcomponents#httpcore;4.2.5 by [org.apache.httpcomponents#httpcore;4.3.3] in [default] org.apache.avro#avro;1.7.4 by [org.apache.avro#avro;1.7.6] in [default] org.xerial.snappy#snappy-java;1.0.4.1 by [org.xerial.snappy#snappy-java;1.0.5] in [default] com.fasterxml.jackson.core#jackson-databind;2.2.3 by [com.fasterxml.jackson.core#jackson-databind;2.5.3] in [default] com.fasterxml.jackson.core#jackson-annotations;2.2.3 by [com.fasterxml.jackson.core#jackson-annotations;2.5.0] in [default] com.fasterxml.jackson.core#jackson-core;2.2.3 by [com.fasterxml.jackson.core#jackson-core;2.5.3] in [default] org.slf4j#slf4j-api;1.7.5 by [org.slf4j#slf4j-api;1.7.10] in [default] org.slf4j#slf4j-api;1.6.4 by [org.slf4j#slf4j-api;1.7.10] in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 94 | 7 | 6 | 12 || 82 | 0 | --------------------------------------------------------------------- :: problems summary :: :::: ERRORS unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver null unknown resolver null unknown resolver null unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver null unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver null unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver sbt-chain unknown resolver null unknown resolver sbt-chain :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 82 already retrieved (0kB/17ms) Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet; at akka.actor.ActorCell$.<init>(ActorCell.scala:336) at akka.actor.ActorCell$.<clinit>(ActorCell.scala) at akka.actor.RootActorPath.$div(ActorPath.scala:185) at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:465) at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:453) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78) at scala.util.Try$.apply(Try.scala:192) at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) at scala.util.Success.flatMap(Try.scala:231) at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84) at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585) at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:578) at akka.actor.ActorSystem$.apply(ActorSystem.scala:142) at akka.actor.ActorSystem$.apply(ActorSystem.scala:109) at org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71) at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:35) at org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60) at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:35) at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:72) at org.apache.toree.Main$delayedInit$body.apply(Main.scala:40) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at org.apache.toree.Main$.main(Main.scala:24) at org.apache.toree.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [W 10:57:42.178 NotebookApp] Timeout waiting for kernel_info reply from ab31e5cc-068b-4884-ab31-bb0dc4cf8286 So at this point, I don't know how to use Toree. I tried to build it from source but there are no instruction on how to install from source. Can you guys take a look at this? Thanks, Ryan
