[jira] [Resolved] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result
[ https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-508. --- Fix Version/s: 0.4.0 Assignee: Fabiano V. Santos Resolution: Fixed > Reply errors to iopub.error instead of iopub.exec_result > > > Key: TOREE-508 > URL: https://issues.apache.org/jira/browse/TOREE-508 > Project: TOREE > Issue Type: Bug >Reporter: Fabiano V. Santos >Assignee: Fabiano V. Santos >Priority: Major > Fix For: 0.4.0 > > > Errors of any kind (compilation, exceptions, etc) returns as normal results > instead of using iopub.error, so jupyter interpret the cell execution as > success which lead to problems if you use tools like papermill to schedule > notebooks, as you never know if the notebook execution was successfully or > not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (TOREE-518) File Path for toree-0.2.0 devl.tar.gz is broken
[
https://issues.apache.org/jira/browse/TOREE-518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende closed TOREE-518.
-
Fix Version/s: Not Applicable
Assignee: Luciano Resende
Resolution: Invalid
Releases are now available directly from pypi.
> File Path for toree-0.2.0 devl.tar.gz is broken
> ---
>
> Key: TOREE-518
> URL: https://issues.apache.org/jira/browse/TOREE-518
> Project: TOREE
> Issue Type: Bug
>Reporter: Ajana Chandiruthil Sathian
>Assignee: Luciano Resende
>Priority: Major
> Fix For: Not Applicable
>
> Attachments: apapche_toree_exception.png
>
>
> The file path in the README.md documentation is broken.
> Executing this command as per the documentation is giving an 404 excepiton.
>
> {{pip install
> [https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshots/dev1/toree-pip/toree-0.2.0.dev1.tar.gz]}}
> {{}}
> {{So please update the document or put the gz file on that location.}}
> {{}}
> {{Any kind of help is greatly appreciated!}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (TOREE-516) Kerberos error while working with Toree Kernel
[
https://issues.apache.org/jira/browse/TOREE-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende closed TOREE-516.
-
Fix Version/s: Not Applicable
Assignee: Luciano Resende
Resolution: Workaround
Suggested using Jupyter Enterprise Gateway to provide the end-to-end requested
flow.
> Kerberos error while working with Toree Kernel
> --
>
> Key: TOREE-516
> URL: https://issues.apache.org/jira/browse/TOREE-516
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: RHEL 7.7, HDP 3 , Spark 2.3.2
>Reporter: Bharath
>Assignee: Luciano Resende
>Priority: Critical
> Labels: security
> Fix For: Not Applicable
>
>
> We have a Kerberied HDP 3.1.0 cluster, configured Jupyter/Toree - It throws
> Keberos error, we have a valid kerberos ticket in place. Here is the
> kernel.json
> {
> "argv": [
> "/opt/jupyterhub/share/jupyter/kernels/apache_toree_scala/bin/run.sh",
> "--profile",
> "\{connection_file}"
> ],
> "env": {
> "DEFAULT_INTERPRETER": "Scala",
> "TOREE_SPARK_OPTS": "--master yarn --deploy-mode cluster",
> "TOREE_OPTS": "",
> "HADOOP_CONF_DIR": "/etc/hadoop/conf",
> "SPARK_HOME": "/usr/hdp/3.1.0.0-78/spark2",
> "PYTHONPATH":
> "/opt/jupyterhub/bin/python:/opt/anaconda3/bin/python:/usr/hdp/3.1.0.0-78/spark2/python/lib/py4j-0.10.7-src.zip:/usr/hdp/3.1.0.0-78/spark2/python:/usr/hdp/3.1.0.0-78/spark2/python/lib",
> "PYTHON_EXEC": "python",
> "HBASE_CONF_DIR": "/etc/hbase/conf",
> "HBASE_HOME": "/usr/hdp/current/hbase-client",
> "JAVA_HOME": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64"
> },
> "display_name": "Toree - Scala-kernel",
> "language": "scala",
> "interrupt_mode": "signal",
> "metadata": {}
> }
> Error Message:
> 20/03/26 18:32:28 INFO RpcRetryingCallerImpl: Call exception, tries=8,
> retries=36, started=18874 ms ago, cancelled=false, msg=Call to
> compute-4.datalake.ntt/xxx.xxx.xx.xxx:16020 failed on local exception:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)], details=row 'name_data:exchg_yearly,,99' on
> table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=compute-4.datalake.ntt
> With above configuration, kernel dies sometimes.
> Please suggest if the way we are passing configs in Kernel.Json is the right
> way ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (TOREE-499) K8: Not be able to submit Job in k8
[ https://issues.apache.org/jira/browse/TOREE-499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende closed TOREE-499. - Fix Version/s: Not Applicable Assignee: Kevin Bates Resolution: Workaround Workaround provided. Please reopen if this is still an issue. > K8: Not be able to submit Job in k8 > --- > > Key: TOREE-499 > URL: https://issues.apache.org/jira/browse/TOREE-499 > Project: TOREE > Issue Type: Bug >Reporter: Leonardo Barrientos C. >Assignee: Kevin Bates >Priority: Blocker > Fix For: Not Applicable > > > Hello, i can't submit jobs to k8, the assembly jar resides in the local > folder directory, so the Driver and executors in K8 cannot get the .jar; it > is possible? Please help. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (TOREE-426) Spark 2.2.0 and scala 2.12.1
[ https://issues.apache.org/jira/browse/TOREE-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende closed TOREE-426. - Fix Version/s: 0.4.0 Resolution: Not A Problem Works after building with the proper scala version > Spark 2.2.0 and scala 2.12.1 > > > Key: TOREE-426 > URL: https://issues.apache.org/jira/browse/TOREE-426 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Environment: MAC OS 10.12.5 >Reporter: Manish Ranjan >Priority: Major > Fix For: 0.4.0 > > > on running command `jupyter notebook` > greeted with > Starting Spark Kernel with > SPARK_HOME=/Users/manish.ranjan/Spark/Installer/spark-2.2/ > log4j:WARN No appenders could be found for logger > (org.apache.toree.Main$$anon$1). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet; > at akka.actor.ActorCell$.(ActorCell.scala:336) > at akka.actor.ActorCell$.(ActorCell.scala) > at akka.actor.RootActorPath.$div(ActorPath.scala:185) > at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:465) > at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:453) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78) > at scala.util.Try$.apply(Try.scala:192) > at > akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73) > at > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) > at > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) > at scala.util.Success.flatMap(Try.scala:231) > at > akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84) > at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585) > at akka.actor.ActorSystemImpl.(ActorSystem.scala:578) > at akka.actor.ActorSystem$.apply(ActorSystem.scala:142) > at akka.actor.ActorSystem$.apply(ActorSystem.scala:109) > at > org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71) > at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:34) > at > org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60) > at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:34) > at > org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:70) > at org.apache.toree.Main$delayedInit$body.apply(Main.scala:39) > at scala.Function0$class.apply$mcV$sp(Function0.scala:34) > at > scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) > at scala.App$$anonfun$main$1.apply(App.scala:76) > at scala.App$$anonfun$main$1.apply(App.scala:76) > at scala.collection.immutable.List.foreach(List.scala:381) > at > scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) > at scala.App$class.main(App.scala:76) > at org.apache.toree.Main$.main(Main.scala:23) > at org.apache.toree.Main.main(Main.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > [W 23:39:20.049 NotebookApp] Timeout waiting for kernel_info reply from > 282ebec0-b01a-405e-b002-f4569179a43d -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TOREE-519) Toree 0.5.0 Release (Spark 3.0 support)
[ https://issues.apache.org/jira/browse/TOREE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-519: -- Component/s: Build Fix Version/s: 0.5.0 > Toree 0.5.0 Release (Spark 3.0 support) > --- > > Key: TOREE-519 > URL: https://issues.apache.org/jira/browse/TOREE-519 > Project: TOREE > Issue Type: Bug > Components: Build >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.5.0 > > > We need to merge the Scala 2.12 change and publish a Toree release that > supports Spark 3.0 right after toree release 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TOREE-519) Toree 0.5.0 Release (Spark 3.0 support)
[ https://issues.apache.org/jira/browse/TOREE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-519: - Assignee: Luciano Resende > Toree 0.5.0 Release (Spark 3.0 support) > --- > > Key: TOREE-519 > URL: https://issues.apache.org/jira/browse/TOREE-519 > Project: TOREE > Issue Type: Bug >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > > We need to merge the Scala 2.12 change and publish a Toree release that > supports Spark 3.0 right after toree release 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-519) Toree 0.5.0 Release (Spark 3.0 support)
Luciano Resende created TOREE-519: - Summary: Toree 0.5.0 Release (Spark 3.0 support) Key: TOREE-519 URL: https://issues.apache.org/jira/browse/TOREE-519 Project: TOREE Issue Type: Bug Reporter: Luciano Resende We need to merge the Scala 2.12 change and publish a Toree release that supports Spark 3.0 right after toree release 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159418#comment-17159418 ] Luciano Resende commented on TOREE-510: --- Sorry it took a long time, the release vote is up on the mailing list https://lists.apache.org/thread.html/re7030486dd143fbbcdbb53166b222fb04f0ae536be64b2a751722697%40%3Cdev.toree.apache.org%3E > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés >Assignee: Luciano Resende >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TOREE-516) Kerberos error while working with Toree Kernel
[
https://issues.apache.org/jira/browse/TOREE-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071306#comment-17071306
]
Luciano Resende commented on TOREE-516:
---
While Toree kernel can be submitted in yarn cluster mode, vanilla Jupyter
Notebooks expect kernels as local processes and cannot discover where in the
YARN cluster the actual kernel launched. The proper way to handle this is to
have Jupyter Notebook -> Jupyter Enterprise Gateway -> Toree Kernel, and this
is described in
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/kernel-yarn-cluster-mode.html
> Kerberos error while working with Toree Kernel
> --
>
> Key: TOREE-516
> URL: https://issues.apache.org/jira/browse/TOREE-516
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: RHEL 7.7, HDP 3 , Spark 2.3.2
>Reporter: Bharath
>Priority: Critical
> Labels: security
>
> We have a Kerberied HDP 3.1.0 cluster, configured Jupyter/Toree - It throws
> Keberos error, we have a valid kerberos ticket in place. Here is the
> kernel.json
> {
> "argv": [
> "/opt/jupyterhub/share/jupyter/kernels/apache_toree_scala/bin/run.sh",
> "--profile",
> "\{connection_file}"
> ],
> "env": {
> "DEFAULT_INTERPRETER": "Scala",
> "TOREE_SPARK_OPTS": "--master yarn --deploy-mode cluster",
> "TOREE_OPTS": "",
> "HADOOP_CONF_DIR": "/etc/hadoop/conf",
> "SPARK_HOME": "/usr/hdp/3.1.0.0-78/spark2",
> "PYTHONPATH":
> "/opt/jupyterhub/bin/python:/opt/anaconda3/bin/python:/usr/hdp/3.1.0.0-78/spark2/python/lib/py4j-0.10.7-src.zip:/usr/hdp/3.1.0.0-78/spark2/python:/usr/hdp/3.1.0.0-78/spark2/python/lib",
> "PYTHON_EXEC": "python",
> "HBASE_CONF_DIR": "/etc/hbase/conf",
> "HBASE_HOME": "/usr/hdp/current/hbase-client",
> "JAVA_HOME": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64"
> },
> "display_name": "Toree - Scala-kernel",
> "language": "scala",
> "interrupt_mode": "signal",
> "metadata": {}
> }
> Error Message:
> 20/03/26 18:32:28 INFO RpcRetryingCallerImpl: Call exception, tries=8,
> retries=36, started=18874 ms ago, cancelled=false, msg=Call to
> compute-4.datalake.ntt/xxx.xxx.xx.xxx:16020 failed on local exception:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)], details=row 'name_data:exchg_yearly,,99' on
> table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=compute-4.datalake.ntt
> With above configuration, kernel dies sometimes.
> Please suggest if the way we are passing configs in Kernel.Json is the right
> way ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010134#comment-17010134 ] Luciano Resende commented on TOREE-510: --- [~andresdoncel-stratio] done a few dependency updates as part of https://github.com/apache/incubator-toree/pull/183 > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés >Assignee: Luciano Resende >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TOREE-514) Update dependencies in preparation for 0.4.0 release
[ https://issues.apache.org/jira/browse/TOREE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-514. --- Fix Version/s: 0.4.0 Assignee: Luciano Resende Resolution: Fixed > Update dependencies in preparation for 0.4.0 release > > > Key: TOREE-514 > URL: https://issues.apache.org/jira/browse/TOREE-514 > Project: TOREE > Issue Type: Task > Components: Build >Affects Versions: 0.4.0 >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TOREE-513) Add logo resource for Toree Kernel
[ https://issues.apache.org/jira/browse/TOREE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-513. --- Fix Version/s: 0.4.0 Assignee: Luciano Resende Resolution: Fixed > Add logo resource for Toree Kernel > -- > > Key: TOREE-513 > URL: https://issues.apache.org/jira/browse/TOREE-513 > Project: TOREE > Issue Type: Task >Affects Versions: 0.4.0 >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.4.0 > > Attachments: jupyterlab-launcher-with-kernel-logo.png > > > JupyterLab exposes logos for kernels in the new launcher widget, this adds a > logo for Toree installed kernels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-514) Update dependencies in preparation for 0.4.0 release
Luciano Resende created TOREE-514: - Summary: Update dependencies in preparation for 0.4.0 release Key: TOREE-514 URL: https://issues.apache.org/jira/browse/TOREE-514 Project: TOREE Issue Type: Task Components: Build Affects Versions: 0.4.0 Reporter: Luciano Resende -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TOREE-513) Add logo resource for Toree Kernel
[ https://issues.apache.org/jira/browse/TOREE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-513: -- Attachment: jupyterlab-launcher-with-kernel-logo.png > Add logo resource for Toree Kernel > -- > > Key: TOREE-513 > URL: https://issues.apache.org/jira/browse/TOREE-513 > Project: TOREE > Issue Type: Task >Affects Versions: 0.4.0 >Reporter: Luciano Resende >Priority: Major > Attachments: jupyterlab-launcher-with-kernel-logo.png > > > JupyterLab exposes logos for kernels in the new launcher widget, this adds a > logo for Toree installed kernels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-513) Add logo resource for Toree Kernel
Luciano Resende created TOREE-513: - Summary: Add logo resource for Toree Kernel Key: TOREE-513 URL: https://issues.apache.org/jira/browse/TOREE-513 Project: TOREE Issue Type: Task Affects Versions: 0.4.0 Reporter: Luciano Resende JupyterLab exposes logos for kernels in the new launcher widget, this adds a logo for Toree installed kernels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TOREE-511) Rework "make dev/make jupyter" build tasks
[ https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-511. --- Resolution: Fixed > Rework "make dev/make jupyter" build tasks > -- > > Key: TOREE-511 > URL: https://issues.apache.org/jira/browse/TOREE-511 > Project: TOREE > Issue Type: Task > Components: Build >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.4.0 > > > Toree has `make dev` and `make jupyter` commands that are using some old > dependencies and some weird way to build them. We should create a docker > file that extends from `jupyter/all-spark-notebook` to make maintenance > easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-512) Refresh support for running Toree in Jupyter Binder
Luciano Resende created TOREE-512: - Summary: Refresh support for running Toree in Jupyter Binder Key: TOREE-512 URL: https://issues.apache.org/jira/browse/TOREE-512 Project: TOREE Issue Type: Task Components: Build Reporter: Luciano Resende Assignee: Luciano Resende Fix For: 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-511) Rework "make dev/make jupyter" build tasks
Luciano Resende created TOREE-511: - Summary: Rework "make dev/make jupyter" build tasks Key: TOREE-511 URL: https://issues.apache.org/jira/browse/TOREE-511 Project: TOREE Issue Type: Task Components: Build Reporter: Luciano Resende Assignee: Luciano Resende Fix For: 0.4.0 Toree has `make dev` and `make jupyter` commands that are using some old dependencies and some weird way to build them. We should create a docker file that extends from `jupyter/all-spark-notebook` to make maintenance easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-510: - Assignee: Luciano Resende > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés >Assignee: Luciano Resende >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17001806#comment-17001806 ] Luciano Resende commented on TOREE-510: --- Let me take a look if there are any cleanups needed, and update Apache Spark default builds and start working on a release. > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés >Assignee: Luciano Resende >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-510: -- Summary: Toree 0.4.0 Release (was: New Toree Release) > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups
[
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-421.
---
Fix Version/s: 0.4.0
Assignee: Kevin Bates
Resolution: Fixed
> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.4.0
>
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some
> issues as the code in our job calls into one of our libraries which tries to
> create threads in its own ThreadGroup:
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip:
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace: at
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> at java.lang.Thread.init(Thread.java:394)
> at java.lang.Thread.init(Thread.java:349)
> at java.lang.Thread.(Thread.java:599)
> at
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
> }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (TOREE-506) Enable TravisCI to build using Open JDK
Luciano Resende created TOREE-506: - Summary: Enable TravisCI to build using Open JDK Key: TOREE-506 URL: https://issues.apache.org/jira/browse/TOREE-506 Project: TOREE Issue Type: Task Components: Build Reporter: Luciano Resende Recently TravisCI build on JDK 8 started failing, and running the tests on openJDK enables a safe net in case of JDK not being available on the user env. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-505) Travis CI builds are failing due to issues installing JDK8
Luciano Resende created TOREE-505: - Summary: Travis CI builds are failing due to issues installing JDK8 Key: TOREE-505 URL: https://issues.apache.org/jira/browse/TOREE-505 Project: TOREE Issue Type: Bug Components: Build Reporter: Luciano Resende Assignee: Luciano Resende $ ~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" --workspace "/home/travis/.cache/install-jdk" --feature "8" --license "BCL" Ignoring license option: BCL -- using GPLv2+CE by default install-jdk.sh 2019-10-16 Expected feature release number in range of 9 to 14, but got: 8 The command "~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" --workspace "/home/travis/.cache/install-jdk" --feature "8" --license "BCL"" failed and exited with 3 during . Details on the following build log: https://travis-ci.org/apache/incubator-toree?utm_medium=notification_source=email -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups
[
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955245#comment-16955245
]
Luciano Resende commented on TOREE-421:
---
Sorry for the delay, I particularly don't have strong opinions one way or
another, but it seems that there are at least a few use cases where folks would
benefit from having a little more flexibility if we could enable different
thread groups properly, so +1 for that.
> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
> Issue Type: Bug
>Reporter: Piyush Narang
>Priority: Major
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some
> issues as the code in our job calls into one of our libraries which tries to
> create threads in its own ThreadGroup:
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip:
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace: at
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> at java.lang.Thread.init(Thread.java:394)
> at java.lang.Thread.init(Thread.java:349)
> at java.lang.Thread.(Thread.java:599)
> at
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
> }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TOREE-504) Version missing from shutdown_reply header
[
https://issues.apache.org/jira/browse/TOREE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933050#comment-16933050
]
Luciano Resende commented on TOREE-504:
---
[~kbates] Do we need a release with this change?
> Version missing from shutdown_reply header
> --
>
> Key: TOREE-504
> URL: https://issues.apache.org/jira/browse/TOREE-504
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Kevin Bates
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.4.0
>
>
> The 5.0 Jupyter message protocol adds a {{version}} field to [message
> headers|https://jupyter-client.readthedocs.io/en/latest/messaging.html#general-message-format].
> However, shutdown_reply headers are not setting the protocol version. As a
> result, clients that adhere to the protocol can have issues when {{version}}
> is present in the response header as an empty string, leading to noise and
> possible incorrect behaviors following Toree's shutdown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (TOREE-504) Version missing from shutdown_reply header
[
https://issues.apache.org/jira/browse/TOREE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-504.
---
Fix Version/s: 0.4.0
Assignee: Kevin Bates
Resolution: Fixed
> Version missing from shutdown_reply header
> --
>
> Key: TOREE-504
> URL: https://issues.apache.org/jira/browse/TOREE-504
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Kevin Bates
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.4.0
>
>
> The 5.0 Jupyter message protocol adds a {{version}} field to [message
> headers|https://jupyter-client.readthedocs.io/en/latest/messaging.html#general-message-format].
> However, shutdown_reply headers are not setting the protocol version. As a
> result, clients that adhere to the protocol can have issues when {{version}}
> is present in the response header as an empty string, leading to noise and
> possible incorrect behaviors following Toree's shutdown.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (TOREE-502) Toree TravisCI fails installing OpenJDK
[ https://issues.apache.org/jira/browse/TOREE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-502. --- Resolution: Fixed Fix Version/s: 0.4.0 > Toree TravisCI fails installing OpenJDK > --- > > Key: TOREE-502 > URL: https://issues.apache.org/jira/browse/TOREE-502 > Project: TOREE > Issue Type: Bug > Components: Build >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.4.0 > > > Here is the logs from Travis CI > https://travis-ci.org/apache/incubator-toree/jobs/546650927 > Removing intermediate container e1ac74d99565 > Step 4/6 : RUN echo 'deb http://ftp.debian.org/debian jessie-backports main' > > /etc/apt/sources.list.d/jessie-backports.list && apt-get -y update && > apt-get install -y --no-install-recommends -t jessie-backports > openjdk-8-jre-headless && rm > /etc/apt/sources.list.d/jessie-backports.list && apt-get clean && rm > -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/* && > update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java > ---> Running in a548a8ea42e3 > Ign http://ftp.debian.org jessie-backports InRelease > Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB] > Get:2 http://repos.mesosphere.io wheezy InRelease [5,279 B] > Ign http://httpredir.debian.org jessie InRelease > Get:3 http://httpredir.debian.org jessie-updates InRelease [7,340 B] > Get:4 http://httpredir.debian.org jessie Release.gpg [2,420 B] > Get:5 http://httpredir.debian.org jessie Release [148 kB] > Ign http://ftp.debian.org jessie-backports Release.gpg > Get:6 http://security.debian.org jessie/updates/main amd64 Packages [851 kB] > Ign http://ftp.debian.org jessie-backports Release > Err http://ftp.debian.org jessie-backports/main amd64 Packages > > Get:7 http://repos.mesosphere.io wheezy/main amd64 Packages [3,625 B] > Get:8 http://httpredir.debian.org jessie/main amd64 Packages [9,098 kB] > Err http://ftp.debian.org jessie-backports/main amd64 Packages > > Err http://ftp.debian.org jessie-backports/main amd64 Packages > > Err http://ftp.debian.org jessie-backports/main amd64 Packages > > Err http://ftp.debian.org jessie-backports/main amd64 Packages > 404 Not Found > W: Failed to fetch > http://httpredir.debian.org/debian/dists/jessie-updates/InRelease Unable to > find expected entry 'main/binary-amd64/Packages' in Release file (Wrong > sources.list entry or malformed file) > W: Failed to fetch > http://ftp.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages > 404 Not Found > E: Some index files failed to download. They have been ignored, or old ones > used instead. > Fetched 10.2 MB in 1s (5,361 kB/s) > The command '/bin/sh -c echo 'deb http://ftp.debian.org/debian > jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list && > apt-get -y update && apt-get install -y --no-install-recommends -t > jessie-backports openjdk-8-jre-headless && rm > /etc/apt/sources.list.d/jessie-backports.list && apt-get clean && rm > -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/* && > update-alternatives --set java > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java' returned a non-zero code: 100 > make: *** [.system-test-image] Error 100 > The command "make all-travis" exited with 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TOREE-500) Failed to start Toree on Spark 2.4.2
[
https://issues.apache.org/jira/browse/TOREE-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853837#comment-16853837
]
Luciano Resende edited comment on TOREE-500 at 6/1/19 11:32 PM:
I will take a look at this, but I am going to be slow due to day job
workload... having said that, feel free to provide patches if you have some
cycles.
The ideal solution would be to produce scala 2.11 and 2.12 artifacts when
performing a build/release.
was (Author: luciano resende):
I will take a look at this, but I am going to be slow due to day job
workload... having said that, feel free to provide patches if you have some
cycles.
The ideal solution would be to produce scala 2.11 and 2.12 artifacts when
performing a release.
> Failed to start Toree on Spark 2.4.2
>
>
> Key: TOREE-500
> URL: https://issues.apache.org/jira/browse/TOREE-500
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: * Docker Container(openjdk8)
> * Python 3.5
>Reporter: Masayuki Takahashi
>Priority: Major
>
> {code}
> [I 04:07:48.726 NotebookApp] Kernel started:
> 1d46275c-71d6-4062-bc22-a98593d6c374
> Starting Spark Kernel with SPARK_HOME=/spark
> 19/05/05 04:07:53 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
> at org.apache.toree.Main$.(Main.scala:24)
> at org.apache.toree.Main$.(Main.scala)
> at org.apache.toree.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: scala.App$class
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
> Toree on Spark 2.4.1 works for me.
> See: https://github.com/jupyter/docker-stacks/issues/860
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-500) Failed to start Toree on Spark 2.4.2
[
https://issues.apache.org/jira/browse/TOREE-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853837#comment-16853837
]
Luciano Resende commented on TOREE-500:
---
I will take a look at this, but I am going to be slow due to day job
workload... having said that, feel free to provide patches if you have some
cycles.
The ideal solution would be to produce scala 2.11 and 2.12 artifacts when
performing a release.
> Failed to start Toree on Spark 2.4.2
>
>
> Key: TOREE-500
> URL: https://issues.apache.org/jira/browse/TOREE-500
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: * Docker Container(openjdk8)
> * Python 3.5
>Reporter: Masayuki Takahashi
>Priority: Major
>
> {code}
> [I 04:07:48.726 NotebookApp] Kernel started:
> 1d46275c-71d6-4062-bc22-a98593d6c374
> Starting Spark Kernel with SPARK_HOME=/spark
> 19/05/05 04:07:53 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
> at org.apache.toree.Main$.(Main.scala:24)
> at org.apache.toree.Main$.(Main.scala)
> at org.apache.toree.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: scala.App$class
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
> Toree on Spark 2.4.1 works for me.
> See: https://github.com/jupyter/docker-stacks/issues/860
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (TOREE-495) Add deps-dir option to toree configuration options
[
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789841#comment-16789841
]
Luciano Resende edited comment on TOREE-495 at 3/11/19 9:40 PM:
The current implementation takes anything added during the notebook runtime
(e.g. %addDeps and %addJar) as transient, and it is created as a temporary file
that is deleted after the process finished independently of the location where
it is created.
It seems that, what you are looking for, is the ability to make a list of
"dependencies" available in kernels. You can accomplish that by updating the
Toree run.sh to list and submit dependencies as part of --packages or --jars
from spark-submit which will take care of making those available in the worker
nodes.
was (Author: luciano resende):
The current implementation takes anything added during the notebook runtime
(e.g. %addDeps and %addJar) as transient, and independent of the location it is
created it is created using the createTmp OS API that deletes the file after
the process is deleted. It seems that, what you need is just the ability to
have the user "dependencies" available in kernels, and you can do that by doing
the map as you mentioned together with updating the kernelspecto use additional
PYTHONPATH configuration (e.g. --conf
spark.yarn.appMasterEnv.PYTHONPATH=${HOME}/.local/lib/python2.7/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip)
the only caveat is that, permanent libraries would have to be added outside of
the notebook environment.
> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
> Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-496) Toree Kernel fail to add Brunel Magic
[ https://issues.apache.org/jira/browse/TOREE-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-496. --- Resolution: Not A Bug Assignee: Luciano Resende Fix Version/s: Not Applicable The version in use is too old and incompatible with Spark 2.3. Please use version 2.6 or above > Toree Kernel fail to add Brunel Magic > - > > Key: TOREE-496 > URL: https://issues.apache.org/jira/browse/TOREE-496 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: SHOBHIT SHUKLA >Assignee: Luciano Resende >Priority: Major > Fix For: Not Applicable > > Attachments: Screenshot 2019-03-11 at 11.31.22 PM.png, Screenshot > 2019-03-11 at 11.31.35 PM.png, Spark 2.3.2 and Brunel 2.6 - Brunel Graph.png, > Spark 2.3.2 and Brunel 2.6.png > > > Toree Kernel fail to add Brunel Magic : > %AddJar -magic [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] > Starting download from > [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] Finished download > of spark-kernel-brunel-all-2.2.jar > Out[2]: Magic AddJar failed to execute with error: > org/apache/toree/magic/CellMagic > > ENV : Spark 2.3.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TOREE-496) Toree Kernel fail to add Brunel Magic
[ https://issues.apache.org/jira/browse/TOREE-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-496: -- Attachment: Spark 2.3.2 and Brunel 2.6.png Spark 2.3.2 and Brunel 2.6 - Brunel Graph.png > Toree Kernel fail to add Brunel Magic > - > > Key: TOREE-496 > URL: https://issues.apache.org/jira/browse/TOREE-496 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: SHOBHIT SHUKLA >Assignee: Luciano Resende >Priority: Major > Fix For: Not Applicable > > Attachments: Screenshot 2019-03-11 at 11.31.22 PM.png, Screenshot > 2019-03-11 at 11.31.35 PM.png, Spark 2.3.2 and Brunel 2.6 - Brunel Graph.png, > Spark 2.3.2 and Brunel 2.6.png > > > Toree Kernel fail to add Brunel Magic : > %AddJar -magic [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] > Starting download from > [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] Finished download > of spark-kernel-brunel-all-2.2.jar > Out[2]: Magic AddJar failed to execute with error: > org/apache/toree/magic/CellMagic > > ENV : Spark 2.3.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-496) Toree Kernel fail to add Brunel Magic
[ https://issues.apache.org/jira/browse/TOREE-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789912#comment-16789912 ] Luciano Resende commented on TOREE-496: --- According to Shobhit, this is using Spark 2.3.0 > Toree Kernel fail to add Brunel Magic > - > > Key: TOREE-496 > URL: https://issues.apache.org/jira/browse/TOREE-496 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: SHOBHIT SHUKLA >Priority: Major > Attachments: Screenshot 2019-03-11 at 11.31.22 PM.png, Screenshot > 2019-03-11 at 11.31.35 PM.png > > > Toree Kernel fail to add Brunel Magic : > %AddJar -magic [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] > Starting download from > [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] Finished download > of spark-kernel-brunel-all-2.2.jar > Out[2]: Magic AddJar failed to execute with error: > org/apache/toree/magic/CellMagic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-495) Add deps-dir option to toree configuration options
[
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789844#comment-16789844
]
Luciano Resende commented on TOREE-495:
---
BTW, we have been doing something similar on a layer between Jupyter and the
Kernel in Jupyter Enterprise Gateway for Kubernetes ([see this blog
post|https://blog.jupyter.org/on-demand-notebooks-with-jupyterhub-jupyter-enterprise-gateway-and-kubernetes-e8e423695cbf])
> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
> Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-495) Add deps-dir option to toree configuration options
[
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789841#comment-16789841
]
Luciano Resende commented on TOREE-495:
---
The current implementation takes anything added during the notebook runtime
(e.g. %addDeps and %addJar) as transient, and independent of the location it is
created it is created using the createTmp OS API that deletes the file after
the process is deleted. It seems that, what you need is just the ability to
have the user "dependencies" available in kernels, and you can do that by doing
the map as you mentioned together with updating the kernelspecto use additional
PYTHONPATH configuration (e.g. --conf
spark.yarn.appMasterEnv.PYTHONPATH=${HOME}/.local/lib/python2.7/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip)
the only caveat is that, permanent libraries would have to be added outside of
the notebook environment.
> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
> Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-495) Add deps-dir option to toree configuration options
[
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788099#comment-16788099
]
Luciano Resende commented on TOREE-495:
---
[~standa] I thought I was following your use case, untill the moment you
mentioned that the --deps-dir would be defined during the installation. To me,
%addJar and %addDeps are to add dependencies required to a specific notebook
and are transient in the sense that are not always required to everybody, or
different users might have different versions, and so on... If you sort of know
where these are, and these are always on the same location, why not add the
path to Python classpath in the kernelspec?
Please feel free to give more details on what issue you are trying to solve by
adding the --deps-dir outside from the magic parameters itself. Also, please
consider how would this work in a distributed environment (e.g. YARN cluster
mode).
> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
> Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-493) The location for Dev snapshots of Toree is empty
[ https://issues.apache.org/jira/browse/TOREE-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738509#comment-16738509 ] Luciano Resende commented on TOREE-493: --- Agreed, please reopen and submit a patch for the docs. Thanks for catching this... > The location for Dev snapshots of Toree is empty > > > Key: TOREE-493 > URL: https://issues.apache.org/jira/browse/TOREE-493 > Project: TOREE > Issue Type: Bug > Components: Website >Affects Versions: Not Applicable >Reporter: Romulo Goncalves >Assignee: Luciano Resende >Priority: Major > Fix For: Not Applicable > > > The documentation to install Toree: > [https://github.com/apache/incubator-toree#install] > Tells us from where to get Dev Snapshots: > "Dev snapshots of Toree are located at > [https://dist.apache.org/repos/dist/dev/incubator/toree]. " > However, the location is empty. Was toree snapshots moved elsewhere and > documentation not updated? Or something happened to the server? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-493) The location for Dev snapshots of Toree is empty
[ https://issues.apache.org/jira/browse/TOREE-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-493. --- Resolution: Won't Fix Assignee: Luciano Resende Users should use the toree release available from pip (0.3.0) Contributors should build master. Having said that, 0.3.0 is very current. > The location for Dev snapshots of Toree is empty > > > Key: TOREE-493 > URL: https://issues.apache.org/jira/browse/TOREE-493 > Project: TOREE > Issue Type: Bug > Components: Website >Affects Versions: Not Applicable >Reporter: Romulo Goncalves >Assignee: Luciano Resende >Priority: Major > Fix For: Not Applicable > > > The documentation to install Toree: > [https://github.com/apache/incubator-toree#install] > Tells us from where to get Dev Snapshots: > "Dev snapshots of Toree are located at > [https://dist.apache.org/repos/dist/dev/incubator/toree]. " > However, the location is empty. Was toree snapshots moved elsewhere and > documentation not updated? Or something happened to the server? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TOREE-459) Failure when adding a Jar.
[
https://issues.apache.org/jira/browse/TOREE-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende updated TOREE-459:
--
Fix Version/s: (was: 0.3.0)
0.4.0
> Failure when adding a Jar.
> --
>
> Key: TOREE-459
> URL: https://issues.apache.org/jira/browse/TOREE-459
> Project: TOREE
> Issue Type: Bug
>Reporter: Naveen Swamy
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> I am trying to use toree with Spark 2.2.7(latest) and Scala 2.11 however when
> I add a jar it fails with the error below
> {{
> spark-2.2.7/jars/scala-library-2.11.8.jar(scala/reflect/package.class)' is
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> error: error while loading package, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/package.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> error: error while loading Serializable, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Serializable.class)' is
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> res2:
> Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.LineMagicOutput]
> = Right(())
> error: error while loading Predef, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Predef.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> }}
> The jar I am trying to add is:
> %AddJar
> https://search.maven.org/remotecontent?filepath=ml/dmlc/mxnet/mxnet-full_2.11-osx-x86_64-cpu/0.9.3a/mxnet-full_2.11-osx-x86_64-cpu-0.9.3a.jar
> Spark version 2.2.1
> Scala version 2.11.8
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-491) Jupyter Kernel creation has bug in kernel.json and SparkR doesn't work
[ https://issues.apache.org/jira/browse/TOREE-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-491. --- Resolution: Won't Fix Assignee: Luciano Resende Fix Version/s: 0.3.0 Toree SparkR is not supported, please see details at https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html > Jupyter Kernel creation has bug in kernel.json and SparkR doesn't work > -- > > Key: TOREE-491 > URL: https://issues.apache.org/jira/browse/TOREE-491 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Pasle Choix >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > > After Toree installation for Scala and SparkR, both kernel.json files contain > an entry that is wrong: > sudo pip install --upgrade toree > sudo jupyter toree install > --spark_home=/opt/cloudera/parcels/SPARK2/lib/spark2 --interpreters=Scala > > Kernel: > "/opt/cloudera/parcels/SPARK2/lib/spark2": > "/opt/cloudera/parcels/SPARK2/lib/spark2" > > After I modified the name from "/opt/cloudera/parcels/SPARK2/lib/spark2" to > "SPARK_HOME", Scala kernel seems working now, but SparkR kernel cannot be > started still. > > Can anyone help to sort it out please? Thank you very much. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-489) Apache Toree kernel throw error when executing code in jupyter lab
[
https://issues.apache.org/jira/browse/TOREE-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642528#comment-16642528
]
Luciano Resende commented on TOREE-489:
---
Please use Toree master, as the fix has already been merged
> Apache Toree kernel throw error when executing code in jupyter lab
> --
>
> Key: TOREE-489
> URL: https://issues.apache.org/jira/browse/TOREE-489
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: Not Applicable
> Environment: OS: Ubuntu 18:10
> Jupyter lab = 0.35.0
> Python = 3.7.0
> Jupyter notebook = 5.7.0
> Spark = 2.3.2
> Java = 1.8.0_181
> Scala = 2.11.8
>Reporter: Felipe Kenji Yoshida
>Priority: Minor
> Fix For: 0.3.0
>
>
> When I try to execute any code (Ex: "_5+5_") in the Apache Toree Kernel
> inside jupyter lab, no output is visible in the notebook and the jupyter lab
> log contains the following message:
> {code:java}
> 2018-10-08 21:26:50 ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-10-08 21:26:50 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748)
> 2018-10-08 21:26:50 ERROR
> LocalActorRefProvider(akka://spark-kernel-actor-system):159 - guardian
> failed, shutting down system
> java.lang.AssertionError: assertion failed: Socket is not alive to be able to
> send messages!
> at scala.Predef$.assert(Predef.scala:170)
> at
> org.apache.toree.communication.socket.JeroMQSocket.send(JeroMQSocket.scala:39)
> at
> org.apache.toree.communication.actors.RouterSocketActor$$anonfun$receive$1.applyOrElse(RouterSocketActor.scala:47)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.communication.actors.RouterSocketActor.aroundReceive(RouterSocketActor.scala:31)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
[jira] [Closed] (TOREE-489) Apache Toree kernel throw error when executing code in jupyter lab
[
https://issues.apache.org/jira/browse/TOREE-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende closed TOREE-489.
-
Resolution: Duplicate
Fix Version/s: 0.3.0
> Apache Toree kernel throw error when executing code in jupyter lab
> --
>
> Key: TOREE-489
> URL: https://issues.apache.org/jira/browse/TOREE-489
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: Not Applicable
> Environment: OS: Ubuntu 18:10
> Jupyter lab = 0.35.0
> Python = 3.7.0
> Jupyter notebook = 5.7.0
> Spark = 2.3.2
> Java = 1.8.0_181
> Scala = 2.11.8
>Reporter: Felipe Kenji Yoshida
>Priority: Minor
> Fix For: 0.3.0
>
>
> When I try to execute any code (Ex: "_5+5_") in the Apache Toree Kernel
> inside jupyter lab, no output is visible in the notebook and the jupyter lab
> log contains the following message:
> {code:java}
> 2018-10-08 21:26:50 ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-10-08 21:26:50 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748)
> 2018-10-08 21:26:50 ERROR
> LocalActorRefProvider(akka://spark-kernel-actor-system):159 - guardian
> failed, shutting down system
> java.lang.AssertionError: assertion failed: Socket is not alive to be able to
> send messages!
> at scala.Predef$.assert(Predef.scala:170)
> at
> org.apache.toree.communication.socket.JeroMQSocket.send(JeroMQSocket.scala:39)
> at
> org.apache.toree.communication.actors.RouterSocketActor$$anonfun$receive$1.applyOrElse(RouterSocketActor.scala:47)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.communication.actors.RouterSocketActor.aroundReceive(RouterSocketActor.scala:31)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
[jira] [Resolved] (TOREE-488) Remove Spark R intepreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-488. --- Resolution: Fixed Fix Version/s: 0.3.0 > Remove Spark R intepreter support in Toree > -- > > Key: TOREE-488 > URL: https://issues.apache.org/jira/browse/TOREE-488 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-487) Remove PySpark Interpreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-487. --- Resolution: Fixed Fix Version/s: 0.3.0 > Remove PySpark Interpreter support in Toree > --- > > Key: TOREE-487 > URL: https://issues.apache.org/jira/browse/TOREE-487 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-488) Remove Spark R intepreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-488: - Assignee: Luciano Resende > Remove Spark R intepreter support in Toree > -- > > Key: TOREE-488 > URL: https://issues.apache.org/jira/browse/TOREE-488 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-487) Remove PySpark Interpreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-487: - Assignee: Luciano Resende > Remove PySpark Interpreter support in Toree > --- > > Key: TOREE-487 > URL: https://issues.apache.org/jira/browse/TOREE-487 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-488) Remove Spark R intepreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637694#comment-16637694 ] Luciano Resende commented on TOREE-488: --- GitHub user lresende opened a pull request: [https://github.com/apache/incubator-toree/pull/166] TOREE-487TOREE-488 Remove PySpark and SparkR interpreters Instead, please use a supported kernel such IPython or IRKernel Closes #166 You can merge this pull request into a Git repository by running: $ git pull [https://github.com/lresende/incubator-toree] toree-487-488-clean-python-r Alternatively you can review and apply these changes as the patch at: [https://github.com/apache/incubator-toree/pull/166.patch] To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #166 commit 1459403fc8ad5b4bb1e956513ef4190a0fc5b6f4 Author: Luciano Resende Date: 2018-10-03T19:26:14Z TOREE-487TOREE-488 Remove PySpark and SparkR interpreters Instead, please use a supported kernel such IPython or IRKernel Closes #166 > Remove Spark R intepreter support in Toree > -- > > Key: TOREE-488 > URL: https://issues.apache.org/jira/browse/TOREE-488 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Reporter: Luciano Resende >Priority: Major > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-482) Truncation magic does not work
[ https://issues.apache.org/jira/browse/TOREE-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-482. --- Resolution: Fixed Fix Version/s: 0.3.0 > Truncation magic does not work > -- > > Key: TOREE-482 > URL: https://issues.apache.org/jira/browse/TOREE-482 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TOREE-487) Remove PySpark Interpreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-487: -- Component/s: Kernel > Remove PySpark Interpreter support in Toree > --- > > Key: TOREE-487 > URL: https://issues.apache.org/jira/browse/TOREE-487 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Reporter: Luciano Resende >Priority: Major > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TOREE-488) Remove Spark R intepreter support in Toree
Luciano Resende created TOREE-488: - Summary: Remove Spark R intepreter support in Toree Key: TOREE-488 URL: https://issues.apache.org/jira/browse/TOREE-488 Project: TOREE Issue Type: Improvement Components: Kernel Reporter: Luciano Resende Detailed discussion on the mailing-list https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TOREE-487) Remove PySpark Interpreter support in Toree
Luciano Resende created TOREE-487: - Summary: Remove PySpark Interpreter support in Toree Key: TOREE-487 URL: https://issues.apache.org/jira/browse/TOREE-487 Project: TOREE Issue Type: Improvement Reporter: Luciano Resende Detailed discussion on the mailing-list https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-486) Typo in %AddJar magic in Scala notebook ==> Strange classloader errors
[ https://issues.apache.org/jira/browse/TOREE-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-486: - Assignee: Luciano Resende > Typo in %AddJar magic in Scala notebook ==> Strange classloader errors > --- > > Key: TOREE-486 > URL: https://issues.apache.org/jira/browse/TOREE-486 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.2.0 >Reporter: SHOBHIT SHUKLA >Assignee: Luciano Resende >Priority: Major > Attachments: 5cd4964a-9b1d-11e8-95fe-ec9b604cd332.png > > > If the user makes a typo while entering an %AddJar magic with an HTTP URL, > the Spark classloader ends up in a strange state that produces errors. > Attachment have the error logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-485) Not able to run Toree kernel in JupyterLab 0.34
[
https://issues.apache.org/jira/browse/TOREE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-485.
---
Resolution: Fixed
Fix Version/s: 0.3.0
> Not able to run Toree kernel in JupyterLab 0.34
> ---
>
> Key: TOREE-485
> URL: https://issues.apache.org/jira/browse/TOREE-485
> Project: TOREE
> Issue Type: Bug
> Environment: Ubuntu 18.04, Python 3.6.6, JupyterLab v0.34.12, Apache
> Toree 0.2.0-rc6
>Reporter: Sergey Kuznetsov
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.3.0
>
>
> Hello! I've been using Toree kernel in JupyterLab (Python 2.7) versions 0.32
> and 0.33 for some time and it worked great. Recently I updated JupyterLab to
> version 0.34 that supports only Python 3 and now I'm not ably to run Scala
> code in Toree. The kernel starts fine, without any warnings, but once I
> executed the cell with simple import I get the following error:
> {noformat}
> ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-09-26 14:45:29 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748){noformat}
> I tried to update Toree: install with pip3 the latest rc6 release, but the
> issue remains. Is there any way to fix this?
>
> Thanks,
> Sergey
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-475) Sql statement can't resolve predefined variables in scala cells
[
https://issues.apache.org/jira/browse/TOREE-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-475.
---
Resolution: Fixed
Fix Version/s: 0.3.0
> Sql statement can't resolve predefined variables in scala cells
> ---
>
> Key: TOREE-475
> URL: https://issues.apache.org/jira/browse/TOREE-475
> Project: TOREE
> Issue Type: Improvement
> Components: Kernel
>Reporter: Huafeng Wang
>Assignee: Huafeng Wang
>Priority: Major
> Fix For: 0.3.0
>
>
> Current the SQL cell does not support code like
> "
> {{%%sql}}
> {{select * from ${tableName} limit ${limitNum}}}
> "
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (TOREE-475) Sql statement can't resolve predefined variables in scala cells
[
https://issues.apache.org/jira/browse/TOREE-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende reassigned TOREE-475:
-
Assignee: Huafeng Wang
> Sql statement can't resolve predefined variables in scala cells
> ---
>
> Key: TOREE-475
> URL: https://issues.apache.org/jira/browse/TOREE-475
> Project: TOREE
> Issue Type: Improvement
> Components: Kernel
>Reporter: Huafeng Wang
>Assignee: Huafeng Wang
>Priority: Major
>
> Current the SQL cell does not support code like
> "
> {{%%sql}}
> {{select * from ${tableName} limit ${limitNum}}}
> "
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-485) Not able to run Toree kernel in JupyterLab 0.34
[
https://issues.apache.org/jira/browse/TOREE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628981#comment-16628981
]
Luciano Resende commented on TOREE-485:
---
Ok, this seems like a recent change in Lab that is passing additional metadata
on the requests and Scala type-safe objects are not accommodating these changes:
The IDs of all deleted cells since the last run cell are now passed as part of
the cell metadata on execution. The IDs of deleted cells since the last run
cell are stored as ``deletedCells`` in ``NotebookModel``. This helps in
developing reactive kernels. (`#5037
<[https://github.com/jupyterlab/jupyterlab/pull/5037]>`__)
Contents looks like:
{"deletedCells":[],"cellId":"d01fdb74-4760-4bd2-91d7-11f85977cb43"}
The issue is that v5.Metadata does not seem to accommodate these new values.
> Not able to run Toree kernel in JupyterLab 0.34
> ---
>
> Key: TOREE-485
> URL: https://issues.apache.org/jira/browse/TOREE-485
> Project: TOREE
> Issue Type: Bug
> Environment: Ubuntu 18.04, Python 3.6.6, JupyterLab v0.34.12, Apache
> Toree 0.2.0-rc6
>Reporter: Sergey Kuznetsov
>Priority: Major
>
> Hello! I've been using Toree kernel in JupyterLab (Python 2.7) versions 0.32
> and 0.33 for some time and it worked great. Recently I updated JupyterLab to
> version 0.34 that supports only Python 3 and now I'm not ably to run Scala
> code in Toree. The kernel starts fine, without any warnings, but once I
> executed the cell with simple import I get the following error:
> {noformat}
> ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-09-26 14:45:29 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748){noformat}
> I tried to update Toree: install with pip3 the latest rc6 release, but the
> issue remains. Is there any way to fix this?
>
> Thanks,
> Sergey
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-485) Not able to run Toree kernel in JupyterLab 0.34
[
https://issues.apache.org/jira/browse/TOREE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628839#comment-16628839
]
Luciano Resende commented on TOREE-485:
---
Just FYI, we have already released Toree 0.2.0 and that is already available on
[PyPi|[https://pypi.org/project/toree].] I would recommend moving to the
official release instead of using the RC.
As for JupyterLab, I have updated to version 0.34.12 and I don't seem to have
an issue during kernel start, but when I try to execute a cell then I DO get
the port already in use exception. I will need to further investigate the
issue, as I don't seem to have a similar issue using the IPython kernel.
> Not able to run Toree kernel in JupyterLab 0.34
> ---
>
> Key: TOREE-485
> URL: https://issues.apache.org/jira/browse/TOREE-485
> Project: TOREE
> Issue Type: Bug
> Environment: Ubuntu 18.04, Python 3.6.6, JupyterLab v0.34.12, Apache
> Toree 0.2.0-rc6
>Reporter: Sergey Kuznetsov
>Priority: Major
>
> Hello! I've been using Toree kernel in JupyterLab (Python 2.7) versions 0.32
> and 0.33 for some time and it worked great. Recently I updated JupyterLab to
> version 0.34 that supports only Python 3 and now I'm not ably to run Scala
> code in Toree. The kernel starts fine, without any warnings, but once I
> executed the cell with simple import I get the following error:
> {noformat}
> ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-09-26 14:45:29 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748){noformat}
> I tried to update Toree: install with pip3 the latest rc6 release, but the
> issue remains. Is there any way to fix this?
>
> Thanks,
> Sergey
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-474) support for custom resolvers (for %AddDeps)
[ https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-474. --- Resolution: Fixed Fix Version/s: 0.3.0 > support for custom resolvers (for %AddDeps) > --- > > Key: TOREE-474 > URL: https://issues.apache.org/jira/browse/TOREE-474 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Evgeniy Tokarev >Assignee: Russell McEntyre >Priority: Major > Fix For: 0.3.0 > > > Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 > which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality > anymore. > Lot of libs using custom repositories like Bintray therefore seem like is a > big limitation of project > which doesn't allow to use custom repositories (Zeppelin and spark-notebooks > has that functionality) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-474) support for custom resolvers (for %AddDeps)
[ https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-474: - Assignee: Russell McEntyre > support for custom resolvers (for %AddDeps) > --- > > Key: TOREE-474 > URL: https://issues.apache.org/jira/browse/TOREE-474 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Evgeniy Tokarev >Assignee: Russell McEntyre >Priority: Major > Fix For: 0.3.0 > > > Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 > which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality > anymore. > Lot of libs using custom repositories like Bintray therefore seem like is a > big limitation of project > which doesn't allow to use custom repositories (Zeppelin and spark-notebooks > has that functionality) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-483) Add %showOutput magic to disable console output
[ https://issues.apache.org/jira/browse/TOREE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-483. --- Resolution: Fixed Fix Version/s: 0.3.0 > Add %showOutput magic to disable console output > --- > > Key: TOREE-483 > URL: https://issues.apache.org/jira/browse/TOREE-483 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-484) Toree kernels keeps crashing on Spark 1.6
[ https://issues.apache.org/jira/browse/TOREE-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-484. --- Resolution: Won't Fix Fix Version/s: 0.2.0 Apache Toree 0.1.0 supports Spark 1.x while Apache Toree 0.2.0 supports Spark 2.x. > Toree kernels keeps crashing on Spark 1.6 > - > > Key: TOREE-484 > URL: https://issues.apache.org/jira/browse/TOREE-484 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Environment: RHEL 6.10(Santiago) > Anaconda2-5.2.0 > Python 2.7 > Jupyter 4.4.0 > HDP 2.5.3 >Reporter: Amit Jha >Priority: Major > Fix For: 0.2.0 > > > Have installed Jupyter using Anaconda. Installed Toree using: > pip install toree > jupyter toree uninstall --spark_home=$SPARK_HOME > --interpreters=Scala,PySpark,SparkR,SQL > kernelspec lists all the toree kernels and other jupyter kernels are > running(Python2/R) but all Toree kernels keeps restarting with below > error message: > Starting Spark Kernel with SPARK_HOME=/usr/hdp/current/spark-client > [I 07:47:20.211 NotebookApp] Kernel started: > 81f1ee05-9486-4c44-9c40-0c8557c75675 > Exception in thread "main" java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at > org.apache.toree.boot.CommandLineOptions.toConfig(CommandLineOptions.scala:161) > at org.apache.toree.Main$$anon$1.(Main.scala:35) > at > org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:35) > at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24) > at scala.Function0$class.apply$mcV$sp(Function0.scala:40) > at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) > at scala.App$$anonfun$main$1.apply(App.scala:71) > at scala.App$$anonfun$main$1.apply(App.scala:71) > at scala.collection.immutable.List.foreach(List.scala:318) > at > scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32) > at scala.App$class.main(App.scala:71) > at org.apache.toree.Main$.main(Main.scala:24) > at org.apache.toree.Main.main(Main.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > [I 07:47:23.209 NotebookApp] KernelRestarter: restarting kernel (1/5), new > random ports > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-245) Publish Toree jars
[ https://issues.apache.org/jira/browse/TOREE-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609673#comment-16609673 ] Luciano Resende commented on TOREE-245: --- Ok, but we must wait for the next release to publish official versions of the other jars. Also, if you want to provide a pr to enable publish them when building a release it would be appreciated: https://github.com/apache/incubator-toree/blob/master/etc/tools/release-build.sh > Publish Toree jars > -- > > Key: TOREE-245 > URL: https://issues.apache.org/jira/browse/TOREE-245 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Ron Jadin >Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > Hello There- > I'm just offering this as an enhancement request. Would it possible to have > an option available for easy consumption/utilization that does not require > building from source? This is an excellent project. I just think this would > make it more accessible to a broader audience. > Thank you, > John -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-245) Publish Toree jars
[ https://issues.apache.org/jira/browse/TOREE-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609647#comment-16609647 ] Luciano Resende commented on TOREE-245: --- Also note that, because the assembly jar is shaded, it will enable deploying the kernel in different spark distributions/mode that might bring conflicting dependencies into the classpath. > Publish Toree jars > -- > > Key: TOREE-245 > URL: https://issues.apache.org/jira/browse/TOREE-245 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Ron Jadin >Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > Hello There- > I'm just offering this as an enhancement request. Would it possible to have > an option available for easy consumption/utilization that does not require > building from source? This is an excellent project. I just think this would > make it more accessible to a broader audience. > Thank you, > John -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-245) Publish Toree jars
[ https://issues.apache.org/jira/browse/TOREE-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609642#comment-16609642 ] Luciano Resende commented on TOREE-245: --- The kernel or standalone applications only need the assembly jar to work. Maybe you could give us more details about what you are trying to accomplish, etc But otherwise, we were not planning to publish any other jars. > Publish Toree jars > -- > > Key: TOREE-245 > URL: https://issues.apache.org/jira/browse/TOREE-245 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Ron Jadin >Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > Hello There- > I'm just offering this as an enhancement request. Would it possible to have > an option available for easy consumption/utilization that does not require > building from source? This is an excellent project. I just think this would > make it more accessible to a broader audience. > Thank you, > John -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-245) Publish Toree jars
[ https://issues.apache.org/jira/browse/TOREE-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-245. --- Resolution: Fixed Assignee: Luciano Resende (was: Gino Bustelo) Fix Version/s: 0.2.0 This is available as of the last release: https://search.maven.org/artifact/org.apache.toree/toree-assembly/0.2.0-incubating/jar > Publish Toree jars > -- > > Key: TOREE-245 > URL: https://issues.apache.org/jira/browse/TOREE-245 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Ron Jadin >Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > Hello There- > I'm just offering this as an enhancement request. Would it possible to have > an option available for easy consumption/utilization that does not require > building from source? This is an excellent project. I just think this would > make it more accessible to a broader audience. > Thank you, > John -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-482) Truncation magic does not work
[ https://issues.apache.org/jira/browse/TOREE-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-482: - Assignee: Luciano Resende > Truncation magic does not work > -- > > Key: TOREE-482 > URL: https://issues.apache.org/jira/browse/TOREE-482 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-467) Showtypes magic doesn't work
[ https://issues.apache.org/jira/browse/TOREE-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-467. --- Resolution: Fixed Assignee: Luciano Resende Fix Version/s: 0.3.0 > Showtypes magic doesn't work > > > Key: TOREE-467 > URL: https://issues.apache.org/jira/browse/TOREE-467 > Project: TOREE > Issue Type: Bug >Reporter: Maciej Bryński >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > Attachments: image-2018-02-21-12-02-56-348.png > > > !image-2018-02-21-12-02-56-348.png! > > I'm using Toree version 0.2.0-incubating-rc3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-480) Deprecate nosparkcontext in favor of spark-context-initialization-mode none
[ https://issues.apache.org/jira/browse/TOREE-480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-480. --- Resolution: Fixed Fix Version/s: 0.3.0 > Deprecate nosparkcontext in favor of spark-context-initialization-mode none > --- > > Key: TOREE-480 > URL: https://issues.apache.org/jira/browse/TOREE-480 > Project: TOREE > Issue Type: Bug > Components: Kernel >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > > We currently have spark-context-initialization-mode that supports eager and > lazy options. This ticket is to deprecate the nonsparkcontext in favor of > adding a 'none' option to spark-context-initialization-mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TOREE-483) Add %showOutput magic to disable console output
Luciano Resende created TOREE-483: - Summary: Add %showOutput magic to disable console output Key: TOREE-483 URL: https://issues.apache.org/jira/browse/TOREE-483 Project: TOREE Issue Type: Improvement Components: Kernel Reporter: Luciano Resende Assignee: Luciano Resende -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TOREE-482) Truncation magic does not work
Luciano Resende created TOREE-482: - Summary: Truncation magic does not work Key: TOREE-482 URL: https://issues.apache.org/jira/browse/TOREE-482 Project: TOREE Issue Type: Bug Components: Kernel Affects Versions: 0.2.0 Reporter: Luciano Resende -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TOREE-474) support for custom resolvers (for %AddDeps)
[ https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488163#comment-16488163 ] Luciano Resende edited comment on TOREE-474 at 5/23/18 10:59 PM: - Would you be willing to help port the fix from 0.1 to 0.2? We would appreciate the patch. was (Author: luciano resende): Would you be willing to port the fix from 0.1 to 0.2? We would appreciate the patch. > support for custom resolvers (for %AddDeps) > --- > > Key: TOREE-474 > URL: https://issues.apache.org/jira/browse/TOREE-474 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Evgeniy Tokarev >Priority: Major > > Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 > which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality > anymore. > Lot of libs using custom repositories like Bintray therefore seem like is a > big limitation of project > which doesn't allow to use custom repositories (Zeppelin and spark-notebooks > has that functionality) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-474) support for custom resolvers (for %AddDeps)
[ https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488163#comment-16488163 ] Luciano Resende commented on TOREE-474: --- Would you be willing to port the fix from 0.1 to 0.2? We would appreciate the patch. > support for custom resolvers (for %AddDeps) > --- > > Key: TOREE-474 > URL: https://issues.apache.org/jira/browse/TOREE-474 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Evgeniy Tokarev >Priority: Major > > Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 > which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality > anymore. > Lot of libs using custom repositories like Bintray therefore seem like is a > big limitation of project > which doesn't allow to use custom repositories (Zeppelin and spark-notebooks > has that functionality) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (TOREE-436) variable assignments generate output in notebook
[ https://issues.apache.org/jira/browse/TOREE-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende closed TOREE-436. - Resolution: Duplicate Fix Version/s: Not Applicable > variable assignments generate output in notebook > > > Key: TOREE-436 > URL: https://issues.apache.org/jira/browse/TOREE-436 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.2.0 > Environment: Ubuntu 16.04 > Spark 2.2 > notebook 5.0.0 >Reporter: Paulo Villegas >Priority: Minor > Fix For: Not Applicable > > Attachments: Toree.png > > > I'm having a somehow strange behaviour with the latest Toree kernel (checked > out from github master branch). Every assignment on a cell generates notebook > output, as it can be seen in the attached figure > !Toree.png|thumbnail! > The output produces the value of the variable (but apparently not at the time > of assignment, but the final value in the cell). I see no means of switching > that off (I've tried to find a %%magic to activate/deactivate it, but found > no one). Is that expected behaviour? If I try an earlier version of the > kernel on the same system, no output is generated (except the one explicitly > generated by printing, or the usual last value in the cell) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-367) Document make tasks available in Toree
[ https://issues.apache.org/jira/browse/TOREE-367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-367. --- Resolution: Not A Problem Assignee: Luciano Resende Fix Version/s: 0.2.0 The README already describes that build is done using make, and document various if not all the possible make targets. > Document make tasks available in Toree > -- > > Key: TOREE-367 > URL: https://issues.apache.org/jira/browse/TOREE-367 > Project: TOREE > Issue Type: Improvement >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > It seems that the project has various, and mostly undocumented, tasks done > via make foo. We should document them all to make it more easily for > outsiders to come and participate in the project -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-471) Typo in spark_context_intialization_timeout command line option
[ https://issues.apache.org/jira/browse/TOREE-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-471. --- Resolution: Fixed Fix Version/s: 0.2.0 > Typo in spark_context_intialization_timeout command line option > --- > > Key: TOREE-471 > URL: https://issues.apache.org/jira/browse/TOREE-471 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > spark_context_intialization_timeout => spark_context_initialization_timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-470) Add option to disable SparkContext lazy initialization
[ https://issues.apache.org/jira/browse/TOREE-470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-470. --- Resolution: Fixed Fix Version/s: 0.2.0 > Add option to disable SparkContext lazy initialization > -- > > Key: TOREE-470 > URL: https://issues.apache.org/jira/browse/TOREE-470 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Critical > Fix For: 0.2.0 > > > We are looking into a scenario where we need to disable the lazy > initialization of SparkContext and a startup parameter or some sort of > configuration might be helpful on this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TOREE-471) Typo in spark_context_intialization_timeout command line option
Luciano Resende created TOREE-471: - Summary: Typo in spark_context_intialization_timeout command line option Key: TOREE-471 URL: https://issues.apache.org/jira/browse/TOREE-471 Project: TOREE Issue Type: Improvement Components: Kernel Affects Versions: 0.2.0 Reporter: Luciano Resende Assignee: Luciano Resende spark_context_intialization_timeout => spark_context_initialization_timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TOREE-470) Add option to disable SparkContext lazy initialization
Luciano Resende created TOREE-470: - Summary: Add option to disable SparkContext lazy initialization Key: TOREE-470 URL: https://issues.apache.org/jira/browse/TOREE-470 Project: TOREE Issue Type: Improvement Components: Kernel Affects Versions: 0.2.0 Reporter: Luciano Resende Assignee: Luciano Resende We are looking into a scenario where we need to disable the lazy initialization of SparkContext and a startup parameter or some sort of configuration might be helpful on this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-366) Prepare a Toree release supporting latest Spark 2.x release
[ https://issues.apache.org/jira/browse/TOREE-366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-366. --- Resolution: Fixed Fix Version/s: 0.2.0 > Prepare a Toree release supporting latest Spark 2.x release > --- > > Key: TOREE-366 > URL: https://issues.apache.org/jira/browse/TOREE-366 > Project: TOREE > Issue Type: Bug >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Critical > Fix For: 0.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-469) Remove binary dependencies from source distribution
[ https://issues.apache.org/jira/browse/TOREE-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-469. --- Resolution: Fixed > Remove binary dependencies from source distribution > --- > > Key: TOREE-469 > URL: https://issues.apache.org/jira/browse/TOREE-469 > Project: TOREE > Issue Type: Task > Components: Build >Affects Versions: 0.2.0 >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Blocker > Fix For: 0.2.0 > > > The Toree release vote failed due to some test jars added to source code > repository. We need to remove these test jars and update the test cases to > create test jars on the fly or download from a known place (e.g. maven) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-355) java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
[
https://issues.apache.org/jira/browse/TOREE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399032#comment-16399032
]
Luciano Resende commented on TOREE-355:
---
Please try the latest RC:
https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0-incubating-rc3/toree-pip/toree-0.2.0.tar.gz
> java.lang.IncompatibleClassChangeError: class
> org.clapper.classutil.asm.ASMEmptyVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> ---
>
> Key: TOREE-355
> URL: https://issues.apache.org/jira/browse/TOREE-355
> Project: TOREE
> Issue Type: Bug
> Environment: Toree 0.2.0.dev1
> CDH 5.9's embedded spark-2.0.0-beta2
>Reporter: Adrien Lavoillotte
>Assignee: Luciano Resende
>Priority: Major
> Labels: easyfix
> Fix For: 0.2.0
>
> Attachments: toree shade.png
>
>
> Using Toree 0.2.0.dev1 with CDH's embedded spark 2 did not cause TOREE-327
> for me. Instead, I got this error:
> {code}
> Exception in thread "main" java.lang.IncompatibleClassChangeError: class
> org.clapper.classutil.asm.ASMEmptyVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
> at
> org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
> at
> org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at
> org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at scala.collection.immutable.Stream.map(Stream.scala:418)
> at
> org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
> at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
> at
> org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
> at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
> at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
> at
> org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at
> org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at scala.util.Try$.apply(Try.scala:192)
> at
> org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
> at
> org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
> at
> org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
> at
> org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
> at
> org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
> at
> org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
> at
> org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
> at
> org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
> at
>
[jira] [Resolved] (TOREE-468) Build Error
[
https://issues.apache.org/jira/browse/TOREE-468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-468.
---
Resolution: Fixed
> Build Error
> ---
>
> Key: TOREE-468
> URL: https://issues.apache.org/jira/browse/TOREE-468
> Project: TOREE
> Issue Type: Bug
> Components: Build
>Affects Versions: Not Applicable
> Environment: Ubuntu 14.04 LTS
>Reporter: David Laxer
>Assignee: Luciano Resende
>Priority: Major
> Fix For: Not Applicable
>
>
> I'm getting the following eror(s) running $ make release on Ubuntu 14.04LTS
>
> ubuntu@ip-10-0-1-112:~/incubator-toree$ make release
> APACHE_SPARK_VERSION=2.2.0 VERSION=0.3.0-dev1-incubating IS_SNAPSHOT=true sbt
> root/assembly
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024M;
> support was removed in 8.0
> Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in
> the future
> [info] Loading settings from plugins.sbt ...
> [info] Loading project definition from /home/ubuntu/incubator-toree/project
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Resolving key references (12568 settings) ...
> [info] Using Apache Spark version 2.2.0, provided from APACHE_SPARK_VERSION
> [info] Set current project to toree (in build
> file:/home/ubuntu/incubator-toree/)
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}macros...
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Done updating.
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}protocol...
> [info] Updating \{file:/home/ubuntu/incubator-toree/}plugins...
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}kernelApi...
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}communication...
> [info] Done updating.
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}pysparkInterpreter...
> [info] Updating \{file:/home/ubuntu/incubator-toree/}sparkrInterpreter...
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}scalaInterpreter...
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}sqlInterpreter...
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}client...
> [info] Done updating.
> [info] Done updating.
> [info] Done updating.
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}kernel...
> [info] Done updating.
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}root...
> [info] Done updating.
> [info] :: delivering ::
> org.apache.toree.kernel#toree_2.11;0.3.0-dev1-incubating-SNAPSHOT ::
> 0.3.0-dev1-incubating-SNAPSHOT :: integration :: Mon Mar 05 18:55:23 UTC 2018
> [info] delivering ivy file to
> /home/ubuntu/incubator-toree/target/scala-2.11/ivy-0.3.0-dev1-incubating-SNAPSHOT.xml
> [info] :: delivering ::
> org.apache.toree.kernel#toree-communication_2.11;0.3.0-dev1-incubating-SNAPSHOT
> :: 0.3.0-dev1-incubating-SNAPSHOT :: integration :: Mon Mar 05 18:55:23 UTC
> 2018
> [info] delivering ivy file to
> /home/ubuntu/incubator-toree/communication/target/scala-2.11/ivy-0.3.0-dev1-incubating-SNAPSHOT.xml
> [info] :: delivering ::
> org.apache.toree.kernel#toree-kernel-api_2.11;0.3.0-dev1-incubating-SNAPSHOT
> :: 0.3.0-dev1-incubating-SNAPSHOT :: integration :: Mon Mar 05 18:55:23 UTC
> 2018
> [info] delivering ivy file to
> /home/ubuntu/incubator-toree/kernel-api/target/scala-2.11/ivy-0.3.0-dev1-incubating-SNAPSHOT.xml
> [info] Copying
>
[jira] [Assigned] (TOREE-468) Build Error
[
https://issues.apache.org/jira/browse/TOREE-468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende reassigned TOREE-468:
-
Assignee: Luciano Resende
> Build Error
> ---
>
> Key: TOREE-468
> URL: https://issues.apache.org/jira/browse/TOREE-468
> Project: TOREE
> Issue Type: Bug
> Components: Build
>Affects Versions: Not Applicable
> Environment: Ubuntu 14.04 LTS
>Reporter: David Laxer
>Assignee: Luciano Resende
>Priority: Major
> Fix For: Not Applicable
>
>
> I'm getting the following eror(s) running $ make release on Ubuntu 14.04LTS
>
> ubuntu@ip-10-0-1-112:~/incubator-toree$ make release
> APACHE_SPARK_VERSION=2.2.0 VERSION=0.3.0-dev1-incubating IS_SNAPSHOT=true sbt
> root/assembly
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024M;
> support was removed in 8.0
> Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in
> the future
> [info] Loading settings from plugins.sbt ...
> [info] Loading project definition from /home/ubuntu/incubator-toree/project
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Resolving key references (12568 settings) ...
> [info] Using Apache Spark version 2.2.0, provided from APACHE_SPARK_VERSION
> [info] Set current project to toree (in build
> file:/home/ubuntu/incubator-toree/)
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}macros...
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Done updating.
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}protocol...
> [info] Updating \{file:/home/ubuntu/incubator-toree/}plugins...
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}kernelApi...
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}communication...
> [info] Done updating.
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}pysparkInterpreter...
> [info] Updating \{file:/home/ubuntu/incubator-toree/}sparkrInterpreter...
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}scalaInterpreter...
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}sqlInterpreter...
> [info] Done updating.
> [info] Updating \{file:/home/ubuntu/incubator-toree/}client...
> [info] Done updating.
> [info] Done updating.
> [info] Done updating.
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}kernel...
> [info] Done updating.
> [warn] Credentials file /home/ubuntu/.ivy2/.credentials does not exist
> [info] Updating \{file:/home/ubuntu/incubator-toree/}root...
> [info] Done updating.
> [info] :: delivering ::
> org.apache.toree.kernel#toree_2.11;0.3.0-dev1-incubating-SNAPSHOT ::
> 0.3.0-dev1-incubating-SNAPSHOT :: integration :: Mon Mar 05 18:55:23 UTC 2018
> [info] delivering ivy file to
> /home/ubuntu/incubator-toree/target/scala-2.11/ivy-0.3.0-dev1-incubating-SNAPSHOT.xml
> [info] :: delivering ::
> org.apache.toree.kernel#toree-communication_2.11;0.3.0-dev1-incubating-SNAPSHOT
> :: 0.3.0-dev1-incubating-SNAPSHOT :: integration :: Mon Mar 05 18:55:23 UTC
> 2018
> [info] delivering ivy file to
> /home/ubuntu/incubator-toree/communication/target/scala-2.11/ivy-0.3.0-dev1-incubating-SNAPSHOT.xml
> [info] :: delivering ::
> org.apache.toree.kernel#toree-kernel-api_2.11;0.3.0-dev1-incubating-SNAPSHOT
> :: 0.3.0-dev1-incubating-SNAPSHOT :: integration :: Mon Mar 05 18:55:23 UTC
> 2018
> [info] delivering ivy file to
> /home/ubuntu/incubator-toree/kernel-api/target/scala-2.11/ivy-0.3.0-dev1-incubating-SNAPSHOT.xml
> [info] Copying
>
[jira] [Assigned] (TOREE-466) Function taking a function as parameter, Kernel Crashes
[
https://issues.apache.org/jira/browse/TOREE-466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende reassigned TOREE-466:
-
Assignee: Luciano Resende
> Function taking a function as parameter, Kernel Crashes
> ---
>
> Key: TOREE-466
> URL: https://issues.apache.org/jira/browse/TOREE-466
> Project: TOREE
> Issue Type: Bug
>Affects Versions: 0.2.0
> Environment: jre-8u162, scala-2.12.4, spark-2.2.1-bin-hadoop2.7,
> 0.2.0-incubating-rc3 in a Ubuntu 17.10 based Docker Container
>Reporter: Hans
>Assignee: Luciano Resende
>Priority: Major
>
> Dear Toree Developers
> The following simple code (in a Jupyter notebook cell) crashes the kernel:
>
>
> {code:java}
> def f(x: Int, l: (Int) => Int): Int = { l(x) }
> f(2, x => x*x)
> {code}
>
> This works in the Scala REPL.
> With best wishes & thanks for your great efforts on Toree
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-344) No module named pyspark
[
https://issues.apache.org/jira/browse/TOREE-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-344.
---
Resolution: Won't Fix
Fix Version/s: 0.2.0
This is true for any python dependencies and I believe its more like a runtime
feature request rather than a kernel issue.
> No module named pyspark
> ---
>
> Key: TOREE-344
> URL: https://issues.apache.org/jira/browse/TOREE-344
> Project: TOREE
> Issue Type: Bug
>Reporter: haniar
>Priority: Major
> Fix For: 0.2.0
>
>
> I have installed toree to my jupyter environment
> (https://github.com/apache/incubator-toree) and written a piece of code that
> works with pyspark. Yarn starts properly and I can see the containers running
> in the queue,
> When I run the code, I get the following error
> Error from python worker:
> /usr/local/bin/python2.7: No module named pyspark
> the kernel is set-up as follows:
> {
> "language": "python",
> "display_name": "Apache Toree - PySpark",
> "env": {
> "__TOREE_SPARK_OPTS__": " --master yarn",
> "SPARK_HOME": "/usr/hdp/2.4.2.0-258/spark",
> "__TOREE_OPTS__": "",
> "DEFAULT_INTERPRETER": "PySpark",
> "PYTHONPATH":
> "/usr/hdp/2.4.2.0-258/spark/python:/usr/hdp/2.4.2.0-258/spark/python/lib/py4j-0.9-src.zip",
> "PYTHON_EXEC": "python",
> "PYTHONSTARTUP": "/usr/hdp/2.4.2.0-258/spark/python/pyspark/shell.py",
> "PYSPARK_PYTHON": "/usr/local/bin/python2.7",
>"PYSPARK_DRIVER_PYTHON": "/usr/local/bin/python2.7"
> },
> "argv": [
> "/usr/local/share/jupyter/kernels/apache_toree_pyspark/bin/run.sh",
> "--profile",
> "{connection_file}"
> ]
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules
[
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-420.
---
Resolution: Fixed
Assignee: Kalvin Chau
Fix Version/s: 0.2.0
> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id:
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>Assignee: Kalvin Chau
> Fix For: 0.2.0
>
>
> When using the AddDeps magic, if the dependency includes
> org.scala-lang.modules it fails with a reflection error, this kafka depdency
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose
> {code}
> {code}
> {'buffers': [],
> 'content': {'ename': 'scala.reflect.internal.FatalError',
> 'evalue': 'object Predef does not have a member classOf',
> 'traceback':
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)',
>
> 'scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)',
>
> 'scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)',
>
> 'scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)',
>
[jira] [Resolved] (TOREE-407) Improve Branding on Site
[ https://issues.apache.org/jira/browse/TOREE-407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-407. --- Resolution: Fixed Fix Version/s: Not Applicable > Improve Branding on Site > > > Key: TOREE-407 > URL: https://issues.apache.org/jira/browse/TOREE-407 > Project: TOREE > Issue Type: Improvement >Reporter: Kyle Kelley >Assignee: Luciano Resende > Fix For: Not Applicable > > > I want to recommend Toree to others as a Scala kernel. The messaging on > https://toree.incubator.apache.org/ is all about "remote spark" which muddies > what it actually does, which is provide kernels that are connected to Spark. > Toree, standalone, doesn't do anything without Jupyter. > Here's what I wish it read: > ``` > Apache Toree > Spark connected kernels for Jupyter projects - Scala, Python, and R > ``` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (TOREE-407) Improve Branding on Site
[ https://issues.apache.org/jira/browse/TOREE-407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-407: - Assignee: Luciano Resende > Improve Branding on Site > > > Key: TOREE-407 > URL: https://issues.apache.org/jira/browse/TOREE-407 > Project: TOREE > Issue Type: Improvement >Reporter: Kyle Kelley >Assignee: Luciano Resende > > I want to recommend Toree to others as a Scala kernel. The messaging on > https://toree.incubator.apache.org/ is all about "remote spark" which muddies > what it actually does, which is provide kernels that are connected to Spark. > Toree, standalone, doesn't do anything without Jupyter. > Here's what I wish it read: > ``` > Apache Toree > Spark connected kernels for Jupyter projects - Scala, Python, and R > ``` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TOREE-392) Update list of dependencies on LICENSE file
[ https://issues.apache.org/jira/browse/TOREE-392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-392. --- Resolution: Fixed Fix Version/s: 0.2.0 > Update list of dependencies on LICENSE file > --- > > Key: TOREE-392 > URL: https://issues.apache.org/jira/browse/TOREE-392 > Project: TOREE > Issue Type: Bug >Reporter: Luciano Resende >Assignee: Luciano Resende > Fix For: 0.2.0 > > > More details on the thread: > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01576.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TOREE-456) Convert env variable TOREE_ALTERNATE_SIGINT to command-line option
[
https://issues.apache.org/jira/browse/TOREE-456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-456.
---
Resolution: Fixed
Fix Version/s: 0.2.0
> Convert env variable TOREE_ALTERNATE_SIGINT to command-line option
> --
>
> Key: TOREE-456
> URL: https://issues.apache.org/jira/browse/TOREE-456
> Project: TOREE
> Issue Type: Improvement
> Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Kevin Bates
>Assignee: Sanjay Saxena
>Priority: Minor
> Fix For: 0.2.0
>
>
> The recently added support (TOREE-437) for an alternate interrupt signal was
> implemented using the environment variable {{TOREE_ALTERNATE_SIGINT}}.
> However, we've recently learned that support for {{SPARK_YARN_USER_ENV}} is
> being removed in Spark 2.2 and its via that variable that
> {{TOREE_ALTERNATE_SIGINT}} is conveyed to the Toree kernel when launched as a
> Yarn cluster application (_cluster mode_).
> As a result, and based on similar discussion on TOREE-443, it would be better
> to convey the need for an alternate interrupt signal via a command line
> option. This way, all launch modes of Toree could set this option in the
> same manner - via {{___TOREE_OPTS___}} in {{kernel.json}}, for example.
> The command-line option should be something like:
> {{--alternate-sigint=}}
> with a description of something like:
> {{Specifies the signal to use instead of SIGINT for interrupting a
> long-running cell. The value is a string and does not include the SIG
> prefix. Use of USR2 is recommended.}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TOREE-392) Update list of dependencies on LICENSE file
[ https://issues.apache.org/jira/browse/TOREE-392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-392: -- Summary: Update list of dependencies on LICENSE file (was: Update scala version on list of dependencies on LICENSE file) > Update list of dependencies on LICENSE file > --- > > Key: TOREE-392 > URL: https://issues.apache.org/jira/browse/TOREE-392 > Project: TOREE > Issue Type: Bug >Reporter: Luciano Resende >Assignee: Luciano Resende > > More details on the thread: > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01576.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TOREE-392) Update scala version on list of dependencies on LICENSE file
[ https://issues.apache.org/jira/browse/TOREE-392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-392: -- Description: More details on the thread: https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01576.html > Update scala version on list of dependencies on LICENSE file > > > Key: TOREE-392 > URL: https://issues.apache.org/jira/browse/TOREE-392 > Project: TOREE > Issue Type: Bug >Reporter: Luciano Resende >Assignee: Luciano Resende > > More details on the thread: > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01576.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TOREE-460) Verbose output for variable declaration after fix for TOREE-380
[ https://issues.apache.org/jira/browse/TOREE-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-460. --- Resolution: Not A Problem No more comments after my previous investigation, please reopen if you disagree. > Verbose output for variable declaration after fix for TOREE-380 > --- > > Key: TOREE-460 > URL: https://issues.apache.org/jira/browse/TOREE-460 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Vinayak Joshi >Assignee: Luciano Resende > Fix For: 0.2.0 > > > Currently, when executing a cell with a variable declaration in Toree (0.2.0) > & Scala 2.11 the cell result contains verbose output which was not present > prior to the fix for TOREE-380: > Before TOREE-380 there was not output when a cell was only declaring/defining > a variable > ``` > var v = "abc" > ``` > After TOREE-380 > ``` > v = abc > Out[2]: abc > ``` > This is breaking consistency/compatibility with previous versions of Toree. > Also there are users that use some kind of code cell hiding to prevent > credentials etc being displayed in a notebook - this use case now breaks > since the cell output now displays any credential values that are assigned to > variables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TOREE-444) Startup performance: collapse multiple startup code evals into fewer calls
[ https://issues.apache.org/jira/browse/TOREE-444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-444: -- Attachment: Toree - Kernel Startup.pdf > Startup performance: collapse multiple startup code evals into fewer calls > -- > > Key: TOREE-444 > URL: https://issues.apache.org/jira/browse/TOREE-444 > Project: TOREE > Issue Type: Bug >Reporter: Luciano Resende > Attachments: Toree - Kernel Startup.pdf > > > During kernel startup, various one line code is submitted for evaluation > (e.g. import, definition of helper variables, etc). It seems that when > executed in multiple blocks the time taken is greater then if evaluated in > fewer blocks. We should investigate and possibly aggregate these blocks as > possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TOREE-443) Make delayed kernel execution timeout configurable
[
https://issues.apache.org/jira/browse/TOREE-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-443.
---
Resolution: Fixed
Fix Version/s: 0.2.0
> Make delayed kernel execution timeout configurable
> --
>
> Key: TOREE-443
> URL: https://issues.apache.org/jira/browse/TOREE-443
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Reporter: Luciano Resende
>Assignee: Sanjay Saxena
> Fix For: 0.2.0
>
>
> In the *kernel* sub-project's {{org.apache.toree.kernel.api.Kernel.scala}},
> when we do the spark context delayed initialization, there is hardcoded wait
> of 100 ms. This usually works ok, but in slow machines or busy nodes it's not
> enough and cause confusion to users as it's going to try to initialize the
> context and display the message on the notebook ui twice.
> We should enable users to configure the timeout via an environment variable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (TOREE-462) %%dataframe magic fails with df containing nulls and arrays
[ https://issues.apache.org/jira/browse/TOREE-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-462. --- Resolution: Fixed > %%dataframe magic fails with df containing nulls and arrays > --- > > Key: TOREE-462 > URL: https://issues.apache.org/jira/browse/TOREE-462 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.1.0, 0.2.0 >Reporter: Luciano Resende > Fix For: 0.2.0 > > > When a DF contains null, it breaks with server error: > [Stage 2:===> (17 + 2) / > 19]18/01/01 17:09:46 WARN TaskSetManager: Lost task 6.0 in stage 2.0 (TID 11, > 192.168.1.126, executor 1): java.lang.NullPointerException > at > org.apache.toree.utils.DataFrameConverter$$anonfun$3$$anonfun$4.apply(DataFrameConverter.scala:55) > at > org.apache.toree.utils.DataFrameConverter$$anonfun$3$$anonfun$4.apply(DataFrameConverter.scala:54) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.toree.utils.DataFrameConverter$$anonfun$3.apply(DataFrameConverter.scala:54) > at > org.apache.toree.utils.DataFrameConverter$$anonfun$3.apply(DataFrameConverter.scala:53) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > at scala.collection.Iterator$$anon$10.next(Iterator.scala:393) > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) > at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) > at scala.collection.AbstractIterator.to(Iterator.scala:1336) > at > scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) > at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336) > at > scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) > at scala.collection.AbstractIterator.toArray(Iterator.scala:1336) > at > org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1354) > at > org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1354) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2062) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2062) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:108) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TOREE-462) %%dataframe magic fails with df containing nulls and arrays
Luciano Resende created TOREE-462: - Summary: %%dataframe magic fails with df containing nulls and arrays Key: TOREE-462 URL: https://issues.apache.org/jira/browse/TOREE-462 Project: TOREE Issue Type: Bug Components: Kernel Affects Versions: 0.1.0, 0.2.0 Reporter: Luciano Resende Fix For: 0.2.0 When a DF contains null, it breaks with server error: [Stage 2:===> (17 + 2) / 19]18/01/01 17:09:46 WARN TaskSetManager: Lost task 6.0 in stage 2.0 (TID 11, 192.168.1.126, executor 1): java.lang.NullPointerException at org.apache.toree.utils.DataFrameConverter$$anonfun$3$$anonfun$4.apply(DataFrameConverter.scala:55) at org.apache.toree.utils.DataFrameConverter$$anonfun$3$$anonfun$4.apply(DataFrameConverter.scala:54) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at org.apache.toree.utils.DataFrameConverter$$anonfun$3.apply(DataFrameConverter.scala:54) at org.apache.toree.utils.DataFrameConverter$$anonfun$3.apply(DataFrameConverter.scala:53) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at scala.collection.Iterator$$anon$10.next(Iterator.scala:393) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) at scala.collection.AbstractIterator.to(Iterator.scala:1336) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) at scala.collection.AbstractIterator.toArray(Iterator.scala:1336) at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1354) at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1354) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2062) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2062) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (TOREE-460) Verbose output for variable declaration after fix for TOREE-380
[ https://issues.apache.org/jira/browse/TOREE-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307037#comment-16307037 ] Luciano Resende edited comment on TOREE-460 at 12/31/17 2:04 AM: - I was investigating the issue, more towards how these statements work in different languages (Python and Scala) ``` Python Console | iPython Kernel |Scala Console | Toree Kernel a = 5 | a = 5 | var a = 5 | var a = 5 no output| no output| a: Int = 5 | Out[n]: 5 ``` So, Although there was a change in behavior for Toree, I believe the change is making it consistent with the different ways to perform interactive development on the Scala world and I am inclined to close this was working as expected. was (Author: luciano resende): I was investigating the issue, more towards how these statements work in different languages (Python and Scala) Python Console | iPython Kernel |Scala Console | Toree Kernel a = 5 | a = 5 | var a = 5 | var a = 5 no output| no output| a: Int = 5 | Out[n]: 5 So, Although there was a change in behavior for Toree, I believe the change is making it consistent with the different ways to perform interactive development on the Scala world and I am inclined to close this was working as expected. > Verbose output for variable declaration after fix for TOREE-380 > --- > > Key: TOREE-460 > URL: https://issues.apache.org/jira/browse/TOREE-460 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Vinayak Joshi >Assignee: Luciano Resende > Fix For: 0.2.0 > > > Currently, when executing a cell with a variable declaration in Toree (0.2.0) > & Scala 2.11 the cell result contains verbose output which was not present > prior to the fix for TOREE-380: > Before TOREE-380 there was not output when a cell was only declaring/defining > a variable > ``` > var v = "abc" > ``` > After TOREE-380 > ``` > v = abc > Out[2]: abc > ``` > This is breaking consistency/compatibility with previous versions of Toree. > Also there are users that use some kind of code cell hiding to prevent > credentials etc being displayed in a notebook - this use case now breaks > since the cell output now displays any credential values that are assigned to > variables. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
