Re: Removing support for Scala 2.10
+1 On Mon, Nov 6, 2017 at 6:04 AM, Marius van Niekerk < marius.v.niek...@gmail.com> wrote: > I don't think that the current support for 2.10 in master even works > reliably. Remove it as it will make the maintenance of this simpler. > > On Sat, 4 Nov 2017 at 18:56 Chip Senkbeil wrote: > > > Go for it. > > > > On Sat, Nov 4, 2017, 3:08 PM Luciano Resende > wrote: > > > > > The current master only works with Apache Spark 2.x which is based on > > Scala > > > 2.11 and moving towards Scala 2.12. > > > > > > Anyone opposed on removing Scala 2.10 from master ? > > > > > > -- > > > Luciano Resende > > > http://twitter.com/lresende1975 > > > http://lresende.blogspot.com/ > > > > > > -- > regards > Marius van Niekerk > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.2.0 RC2
I think the tag and commit hash are incorrect. The tag is v0.2.0-incubating-rc2 and the commit hash is 32bbefa121aafd8713afab81516917234d72d690 See https://github.com/apache/incubator-toree/commit/32bbefa. rb On Sun, Nov 5, 2017 at 10:27 AM, Luciano Resende wrote: > Please vote to approve the release of Apache Toree 0.2.0-incubating (RC2). > > Tag: v0.2.0-incubating-rc1 (01cd97e9bad04878a8014016c154a50e2a00f21d) > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc2 > > All distribution packages, including signatures, digests, etc. can be found > at: > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2. > 0-incubating-rc2/ > > Staging artifacts can be found at: > > https://repository.apache.org/content/repositories/orgapachetoree-1008 > > ## Testing Instructions > > The fastest way to get up and running is to using Jupyter. > > 1. Install Jupyter if you haven't already (http://jupyter.org/install.html > ) > > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ > dist/dev/incubator/toree/0.2.0-incubating-rc2/toree-pip/ > toree-0.2.0.tar.gz` > followed by `jupyter toree install` > > - You need to set a valid Apache Spark 2.x home, which can be done via > `jupyter toree install --spark_home=/usr/local/spark` > > - You may need to run with `sudo` for installation permission > > - For all installation options, run `jupyter toree install --help-all` > > 4. Run a Jupyter notebook server via `jupyter notebook` > > - If the notebook portion of Jupyter is not installed but Jupyter is, > you can install via `pip install notebook` > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under > new > dev > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the > notebook > > ## Voting Instructions > > The vote is open for at least 72 hours and passes if a majority of at least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > [ ] -1 Do not release this package because ... > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.2.0 RC2
-1 Looks like license documentation is out of date. The source tarball looks fine, but the binary one has problems. The LICENSE file in the bin tarball appears to be missing some dependencies that are included in the assembly Jar: * Akka (Apache 2 - https://github.com/akka/akka/blob/master/LICENSE) * Play (Apache 2 - https://github.com/playframework/playframework/blob/master/LICENSE) * Spring framework (Apache 2 - https://github.com/spring-projects/spring-framework) * Joda time (Apache 2 - http://www.joda.org/joda-time/license.html) * Coursier (Apache 2 - https://github.com/coursier/coursier/blob/master/LICENSE) * Typesafe config (Apache 2 - https://github.com/lightbend/config/blob/master/LICENSE-2.0.txt) * Guava (Apache 2 - https://github.com/google/guava/blob/master/COPYING) Coursier shades other libraries, including parts of Maven (ALv2), fastparse (MIT - https://github.com/lihaoyi/fastparse), jsoup (MIT - https://jsoup.org/license), something under "sourcecode", and Ivy. The LICENSE file needs to have entries for any dependencies that are distributed with Toree because those projects aren't licensed to users by the ASF, they are licensed by the copyright owners for those projects. We also need to make sure any entries in the NOTICE files from other Apache projects that apply to this Toree release are included in Toree's NOTICE file. Also, I noticed that the license file includes 2 entries for Joda time and lists depenencies on Scala 2.10 libs and Ammonite (which isn't used as far as I know). I think we need to regenerate this file with the current dependency set and possibly update it. One last point is that if we don't already include the binary tarball's license and notice in the assembly Jar when it is published to Maven central, we'll need to add it there as well. rb On Mon, Nov 6, 2017 at 11:05 AM, Ryan Blue wrote: > I think the tag and commit hash are incorrect. The tag is > v0.2.0-incubating-rc2 and the commit hash is > 32bbefa121aafd8713afab81516917234d72d690 > > See https://github.com/apache/incubator-toree/commit/32bbefa. > > rb > > On Sun, Nov 5, 2017 at 10:27 AM, Luciano Resende > wrote: > >> Please vote to approve the release of Apache Toree 0.2.0-incubating (RC2). >> >> Tag: v0.2.0-incubating-rc1 (01cd97e9bad04878a8014016c154a50e2a00f21d) >> >> https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc2 >> >> All distribution packages, including signatures, digests, etc. can be >> found >> at: >> >> https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0 >> -incubating-rc2/ >> >> Staging artifacts can be found at: >> >> https://repository.apache.org/content/repositories/orgapachetoree-1008 >> >> ## Testing Instructions >> >> The fastest way to get up and running is to using Jupyter. >> >> 1. Install Jupyter if you haven't already (http://jupyter.org/install.ht >> ml) >> >> 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ >> dist/dev/incubator/toree/0.2.0-incubating-rc2/toree-pip/tore >> e-0.2.0.tar.gz` >> <https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0-incubating-rc2/toree-pip/toree-0.2.0.tar.gz> >> followed by `jupyter toree install` >> >> - You need to set a valid Apache Spark 2.x home, which can be done via >> `jupyter toree install --spark_home=/usr/local/spark` >> >> - You may need to run with `sudo` for installation permission >> >> - For all installation options, run `jupyter toree install --help-all` >> >> 4. Run a Jupyter notebook server via `jupyter notebook` >> >> - If the notebook portion of Jupyter is not installed but Jupyter is, >> you can install via `pip install notebook` >> >> 5. Create a notebook using "Apache Toree - Scala" from the dropdown under >> new >> dev >> 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in >> the >> notebook >> >> ## Voting Instructions >> >> The vote is open for at least 72 hours and passes if a majority of at >> least >> 3 +1 PMC votes are cast. >> >> [ ] +1 Release this package as Apache Toree 0.2.0-incubating >> [ ] -1 Do not release this package because ... >> >> -- >> Luciano Resende >> http://twitter.com/lresende1975 >> http://lresende.blogspot.com/ >> > > > > -- > Ryan Blue > Software Engineer > Netflix > -- Ryan Blue Software Engineer Netflix
Re: Podling Report Reminder - February 2018
Thanks for putting that together. It looks good to me and I added my sign-off. On Thu, Feb 1, 2018 at 7:32 PM, Luciano Resende wrote: > I have added a draft report at > https://wiki.apache.org/incubator/February2018 > > Please feel free to provide feedback and other additions and I can update > it (or feel free to update it directly on the wiki page) > > On Thu, Feb 1, 2018 at 7:16 PM, wrote: > > > Dear podling, > > > > This email was sent by an automated system on behalf of the Apache > > Incubator PMC. It is an initial reminder to give you plenty of time to > > prepare your quarterly board report. > > > > The board meeting is scheduled for Wed, 21 February 2018, 10:30 am PDT. > > The report for your podling will form a part of the Incubator PMC > > report. The Incubator PMC requires your report to be submitted 2 weeks > > before the board meeting, to allow sufficient time for review and > > submission (Wed, February 07). > > > > Please submit your report with sufficient time to allow the Incubator > > PMC, and subsequently board members to review and digest. Again, the > > very latest you should submit your report is 2 weeks prior to the board > > meeting. > > > > Thanks, > > > > The Apache Incubator PMC > > > > Submitting your Report > > > > -- > > > > Your report should contain the following: > > > > * Your project name > > * A brief description of your project, which assumes no knowledge of > > the project or necessarily of its field > > * A list of the three most important issues to address in the move > > towards graduation. > > * Any issues that the Incubator PMC or ASF Board might wish/need to be > > aware of > > * How has the community developed since the last report > > * How has the project developed since the last report. > > * How does the podling rate their own maturity. > > > > This should be appended to the Incubator Wiki page at: > > > > https://wiki.apache.org/incubator/February2018 > > > > Note: This is manually populated. You may need to wait a little before > > this page is created from a template. > > > > Mentors > > --- > > > > Mentors should review reports for their project(s) and sign them off on > > the Incubator wiki page. Signing off reports shows that you are > > following the project - projects that are not signed may raise alarms > > for the Incubator PMC. > > > > Incubator PMC > > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.2.0 RC4
+1 (binding) Checked licenses in src tarball, built and tested using `make test`, validated checksums and signature. Thanks for working on this, Luciano! On Sat, Mar 31, 2018 at 9:46 AM, Luciano Resende wrote: > My +1. > > Also, notice that the only change between RC3 and RC4 is to remove the > binary test jars from the source code, which should make the review a > little easier for those that already reviewed the previous release > candidate. > > On Fri, Mar 30, 2018 at 10:28 PM, Luciano Resende > wrote: > > > Please vote to approve the release of Apache Toree 0.2.0-incubating > > (RC4). > > > > Tag: v0.2.0-incubating-rc4 (8849271e28a4ea4e77c57ef8fb3da77b73469799) > > > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc4 > > > > All distribution packages, including signatures, digests, etc. can be > > found at: > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0 > > -incubating-rc4/ > > > > Staging artifacts can be found at: > > > > https://repository.apache.org/content/repositories/orgapachetoree-1011 > > > > ## Testing Instructions > > > > The fastest way to get up and running is to using Jupyter. > > > > 1. Install Jupyter if you haven't already (http://jupyter.org/install. > html > > ) > > > > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ > > dist/dev/incubator/toree/0.2.0-incubating-rc4/toree-pip/toree > > -0.2.0.tar.gz` > > followed by `jupyter toree install` > > > > - You need to set a valid Apache Spark 2.x home, which can be done via > > `jupyter toree install --spark_home=/usr/local/spark` > > > > - You may need to run with `sudo` for installation permission > > > > - For all installation options, run `jupyter toree install --help-all` > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > you can install via `pip install notebook` > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under > > new > > dev > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in > the > > notebook > > > > ## Voting Instructions > > > > The vote is open for at least 72 hours and passes if a majority of at > > least > > 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > > [ ] -1 Do not release this package because ... > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: Draft report posted, was Re: Podling Report Reminder - May 2018
+1 from me (and signed off). Thanks for writing it up! On Sun, Apr 29, 2018 at 10:12 AM, Luciano Resende wrote: > I have posted a draft report to the Incubator Wiki. > > https://wiki.apache.org/incubator/May2018#preview > > Please feel free to use this thread for comments/questions. > > On Fri, Apr 27, 2018 at 8:01 PM, wrote: > > > Dear podling, > > > > This email was sent by an automated system on behalf of the Apache > > Incubator PMC. It is an initial reminder to give you plenty of time to > > prepare your quarterly board report. > > > > The board meeting is scheduled for Wed, 16 May 2018, 10:30 am PDT. > > The report for your podling will form a part of the Incubator PMC > > report. The Incubator PMC requires your report to be submitted 2 weeks > > before the board meeting, to allow sufficient time for review and > > submission (Wed, May 02). > > > > Please submit your report with sufficient time to allow the Incubator > > PMC, and subsequently board members to review and digest. Again, the > > very latest you should submit your report is 2 weeks prior to the board > > meeting. > > > > Candidate names should not be made public before people are actually > > elected, so please do not include the names of potential committers or > > PPMC members in your report. > > > > Thanks, > > > > The Apache Incubator PMC > > > > Submitting your Report > > > > -- > > > > Your report should contain the following: > > > > * Your project name > > * A brief description of your project, which assumes no knowledge of > > the project or necessarily of its field > > * A list of the three most important issues to address in the move > > towards graduation. > > * Any issues that the Incubator PMC or ASF Board might wish/need to be > > aware of > > * How has the community developed since the last report > > * How has the project developed since the last report. > > * How does the podling rate their own maturity. > > > > This should be appended to the Incubator Wiki page at: > > > > https://wiki.apache.org/incubator/May2018 > > > > Note: This is manually populated. You may need to wait a little before > > this page is created from a template. > > > > Mentors > > --- > > > > Mentors should review reports for their project(s) and sign them off on > > the Incubator wiki page. Signing off reports shows that you are > > following the project - projects that are not signed may raise alarms > > for the Incubator PMC. > > > > Incubator PMC > > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.2.0 RC5
-1 I'm hitting an error in integration.interpreter.scala.AddExternalJarMagicSpecForIntegration, which I think was the same error someone on the IPMC list hit in the last RC. Not sure what's happening yet, but I'll try to take some time to dig into it soon. I'm on Ubuntu 18.04 and I ran `make test`, if you want to try to reproduce it. rb On Sun, Apr 29, 2018 at 7:59 PM, Marius van Niekerk < marius.v.niek...@gmail.com> wrote: > Tested this on binderhub and seems to work fine +1 > > On Wed, 25 Apr 2018 at 15:32 Luciano Resende wrote: > > > On Sat, Apr 21, 2018 at 3:58 PM, Luciano Resende > > wrote: > > > > > Please vote to approve the release of Apache Toree 0.2.0-incubating > > > (RC5). > > > > > > Tag: v0.2.0-incubating-rc5 (1b2d4aac2113bf59bbb4ec1cff75adbe127f221a) > > > > > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc5 > > > > > > All distribution packages, including signatures, digests, etc. can be > > > found at: > > > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0 > > > -incubating-rc5/ > > > > > > Staging artifacts can be found at: > > > > > > https://repository.apache.org/content/repositories/orgapachetoree-1012 > > > > > > ## Testing Instructions > > > > > > The fastest way to get up and running is to using Jupyter. > > > > > > 1. Install Jupyter if you haven't already ( > > http://jupyter.org/install.html > > > ) > > > > > > 2. Install Apache Toree via `pip install > https://dist.apache.org/repos/ > > > dist/dev/incubator/toree/0.2.0-incubating-rc5/toree-pip/toree > > > -0.2.0.tar.gz` > > > followed by `jupyter toree install` > > > > > > - You need to set a valid Apache Spark 2.x home, which can be done via > > > `jupyter toree install --spark_home=/usr/local/spark` > > > > > > - You may need to run with `sudo` for installation permission > > > > > > - For all installation options, run `jupyter toree install --help-all` > > > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > > you can install via `pip install notebook` > > > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown > under > > > new > > > dev > > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in > > the > > > notebook > > > > > > ## Voting Instructions > > > > > > The vote is open for at least 72 hours and passes if a majority of at > > > least > > > 3 +1 PMC votes are cast. > > > > > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > > > [ ] -1 Do not release this package because ... > > > > > > > > > > Off course, here is my +1. > > > > Other committers and mentors, please vote here. > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > -- > regards > Marius van Niekerk > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.2.0 RC6
+1 Tests succeed, signature and checksum are good. Thanks for getting this ready, Luciano! On Thu, Aug 9, 2018 at 8:18 AM Luciano Resende wrote: > Have been using RC6 internally for a few days already without any issues. > Of course, my +1 > > On Mon, Jul 30, 2018 at 10:39 AM Luciano Resende > wrote: > > > Please vote to approve the release of Apache Toree 0.2.0-incubating > > (RC6). > > > > Tag: v0.2.0-incubating-rc6 (eae2da61828a0c88e2ab64a7ffb9ff7647909a2c) > > > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc6 > > > > All distribution packages, including signatures, digests, etc. can be > > found at: > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0 > > -incubating-rc6/ > > > > Staging artifacts can be found at: > > > > https://repository.apache.org/content/repositories/orgapachetoree-1013 > > > > ## Testing Instructions > > > > The fastest way to get up and running is to using Jupyter. > > > > 1. Install Jupyter if you haven't already ( > http://jupyter.org/install.html > > ) > > > > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ > > dist/dev/incubator/toree/0.2.0-incubating-rc6/toree-pip/toree > > -0.2.0.tar.gz` > > followed by `jupyter toree install` > > > > - You need to set a valid Apache Spark 2.x home, which can be done via > > `jupyter toree install --spark_home=/usr/local/spark` > > > > - You may need to run with `sudo` for installation permission > > > > - For all installation options, run `jupyter toree install --help-all` > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > you can install via `pip install notebook` > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under > > new > > dev > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in > the > > notebook > > > > ## Voting Instructions > > > > The vote is open for at least 72 hours and passes if a majority of at > > least > > 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > > [ ] -1 Do not release this package because ... > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: Apache Toree 0.2.0-incubating release status
Great news! Thank you for making this happen, Luciano. On Tue, Aug 14, 2018 at 3:29 AM Chip Senkbeil wrote: > Who has ownership rights? Corey and Go no? I don't think I have them, but > I'll check later today. > > On Mon, Aug 13, 2018, 11:23 PM Luciano Resende > wrote: > > > The RC6 has been approved by the IPMC and I have started the process of > > publishing the release which usually takes a day or so for replicating to > > the mirrors. After that, I will update the website with the new > artifacts. > > > > I will need permissions to push the artifacts to PyPI. Could someone with > > ownership rights to the repository please add me there as well. > > > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > -- Ryan Blue Software Engineer Netflix
Re: Removing Toree specific support for Python and R
+1 After investigating the python support, we concluded that it's better to use a normal ipython kernel. I think that the python Toree kernel would take a lot of work to be on par with what already exists in Python. On Tue, Sep 25, 2018 at 9:04 AM Luciano Resende wrote: > Toree currently has some support for Python and R which are a thin > wrapper around what is provided in Spark. While this enables sharing > the Spark session and SQL Context between Scala and Python code, it > brings a huge gap on functionality when comparing with other Python > kernels like IPython and IRKernel. Adding to that, there is no > community member that is actively enhancing and contributing to these > two areas. > > Based on these, I would like to suggest the following to be done for > Toree 0.3.0 release timeframe: > - Remove Python and R support from Toree > - Document possible suggestions on Toree website (e.g. IPython, > IRKernel or others more active on their respective community) > > Some of the benefits: > - Avoid user confusion and frustration > - Improve kernel startup performance > - Cleaner code to maintain > > Please let me know your thoughts. > > [1] > https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/sqlcontext_sharing.ipynb > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.3.0 RC1
+1 Checked source build, signatures, checksums, and ran tests. On Fri, Oct 26, 2018 at 7:24 AM Marius van Niekerk < marius.v.niek...@gmail.com> wrote: > +1 > > Tested a sample notebook in a binderhub instance running toree. > > On Wed, 24 Oct 2018 at 03:27 Manu Zhang wrote: > > > +1 (non-binding) > > > > > > I downloaded the toree-0.3.0.tar.gz, installed with pip and checked the > > following functionalities with jupyterlab-0.28.2. > > > > 1. Created a SparkSession, read from a HDFS parquet file into Spark > > DataFrame and showed some rows. > > 2. Disabled console output with showOutput magic > > 3. SQL magic with variables in sql > > > > > > Thanks, > > Manu Zhang > > On Oct 18, 2018, 11:16 AM +0800, Luciano Resende , > > wrote: > > > On Tue, Oct 16, 2018 at 9:30 AM Luciano Resende > > wrote: > > > > > > > > Please vote to approve the release of Apache Toree 0.3.0-incubating > > (RC1). > > > > > > > > Tag: v0.3.0-incubating-rc1 (3ff5df33cc459ea214e542a4798fd9ba6e0a2ef8) > > > > > > > > https://github.com/apache/incubator-toree/tree/v0.3.0-incubating-rc1 > > > > > > > > All distribution packages, including signatures, digests, etc. can be > > found at: > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/ > > > > > > > > Staging artifacts can be found at: > > > > > > > > > https://repository.apache.org/content/repositories/orgapachetoree-1014 > > > > > > > > ## Testing Instructions > > > > > > > > The fastest way to get up and running is to using Jupyter. > > > > > > > > 1. Install Jupyter if you haven't already ( > > http://jupyter.org/install.html) > > > > > > > > 2. Install Apache Toree via `pip install > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz` > <https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz> > > < > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz > > > > > > followed by `jupyter toree install` > > > > > > > > - You need to set a valid Apache Spark 2.x home, which can be done > via > > > > `jupyter toree install --spark_home=/usr/local/spark` > > > > > > > > - You may need to run with `sudo` for installation permission > > > > > > > > - For all installation options, run `jupyter toree install > --help-all` > > > > > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > > > you can install via `pip install notebook` > > > > > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown > > under new > > > > dev > > > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` > > in the > > > > notebook > > > > > > > > ## Voting Instructions > > > > > > > > The vote is open for at least 72 hours and passes if a majority of at > > least > > > > 3 +1 PMC votes are cast. > > > > > > > > [ ] +1 Release this package as Apache Toree 0.3.0-incubating > > > > [ ] -1 Do not release this package because ... > > > > > > > > > > > > > Off course, my +1 > > > > > > -- > > > Luciano Resende > > > http://twitter.com/lresende1975 > > > http://lresende.blogspot.com/ > > > -- > regards > Marius van Niekerk > -- Ryan Blue Software Engineer Netflix
Interpreter outputs
Hi everyone, I've been putting together a proof-of-concept Toree deployment for our Spark users to test out. One of the things I needed to change was how interpreter outputs are displayed, allowing interpreters to show HTML or other representations of output objects. For anyone else interested, I've put up a PR with the changes for discussion: https://github.com/apache/incubator-toree/pull/104 There are also a few other PRs that fix tab completion, code completeness checks, syntax highlighting, and other usability issues. It would be great to get feedback from the community. Thanks! rb -- Ryan Blue Software Engineer Netflix
SQL interpreter
I've been working on improvements to the SQL interpreter, but I'm not quite sure why it works the way it currently does. Right now, the interpreter is associated with a SQL service that is a BrokerService. BrokerService says that it is a high-level interface between the JVM and another process, like R or Python, so it isn't clear why SQL uses it. The version I have working is considerably simpler and uses the lazy `kernel.sparkSession` accessor. Is there a reason why the current one doesn't use this approach? rb -- Ryan Blue Software Engineer Netflix
Incubator report this month
Hi everyone, The Toree report to the incubator was scheduled for last month, but since we missed it we're supposed to report this month. Below is a start with PPMC additions and JIRA activity. What else should we add? Also, would anyone like to take a shot at the health report? Also, I went to check the number of PPMC members and I'm not in the list. Could someone add me? Is anyone else missing? I'm looking here: http://people.apache.org/committers-by-project.html#toree rb Toree Report June 2017 ## Description: - Toree is a Jupyter kernel for Scala and Apache Spark. ## Issues: - No issues require board attention at this time. ## Activity: - Apache Toree 0.1.0-incubating release on 2017-02-21 - Apache Toree website updated ## Health report: - ## PPMC and committer changes: - Currently 9 PPMC members. - Ryan Blue was added to the PMC on 2017-04-03 ## Releases: - Apache Toree 0.1.0-incubating release on 2017-02-21 ## JIRA activity: - 43 JIRA tickets created in the last 4 months - 32 JIRA tickets closed/resolved in the last 4 months -- Ryan Blue Software Engineer Netflix
[jira] [Commented] (TOREE-355) java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
[ https://issues.apache.org/jira/browse/TOREE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114522#comment-16114522 ] Ryan Blue commented on TOREE-355: - I think we fixed this by running with the JVM option {{-noverify}}. We pass that through {{--driver-java-options}}. That might be an easy way to solve this. > java.lang.IncompatibleClassChangeError: class > org.clapper.classutil.asm.ASMEmptyVisitor has interface > org.objectweb.asm.ClassVisitor as super class > --- > > Key: TOREE-355 > URL: https://issues.apache.org/jira/browse/TOREE-355 > Project: TOREE > Issue Type: Bug > Environment: Toree 0.2.0.dev1 > CDH 5.9's embedded spark-2.0.0-beta2 >Reporter: Adrien Lavoillotte >Assignee: Luciano Resende > Labels: easyfix > Fix For: 0.2.0 > > Attachments: toree shade.png > > > Using Toree 0.2.0.dev1 with CDH's embedded spark 2 did not cause TOREE-327 > for me. Instead, I got this error: > {code} > Exception in thread "main" java.lang.IncompatibleClassChangeError: class > org.clapper.classutil.asm.ASMEmptyVisitor has interface > org.objectweb.asm.ClassVisitor as super class > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250) > at > org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427) > at > org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385) > at > org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385) > at scala.collection.immutable.Stream.map(Stream.scala:418) > at > org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385) > at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340) > at > org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329) > at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320) > at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311) > at > org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73) > at > org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73) > at scala.util.Try$.apply(Try.scala:192) > at > org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73) > at > org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35) > at > org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34) > at > org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38) > at > org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38) > at > org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45) > a
[jira] [Commented] (TOREE-355) java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
[ https://issues.apache.org/jira/browse/TOREE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180975#comment-16180975 ] Ryan Blue commented on TOREE-355: - Looks like the formatting was messed up. The option is -noverify. > java.lang.IncompatibleClassChangeError: class > org.clapper.classutil.asm.ASMEmptyVisitor has interface > org.objectweb.asm.ClassVisitor as super class > --- > > Key: TOREE-355 > URL: https://issues.apache.org/jira/browse/TOREE-355 > Project: TOREE > Issue Type: Bug > Environment: Toree 0.2.0.dev1 > CDH 5.9's embedded spark-2.0.0-beta2 >Reporter: Adrien Lavoillotte >Assignee: Luciano Resende > Labels: easyfix > Fix For: 0.2.0 > > Attachments: toree shade.png > > > Using Toree 0.2.0.dev1 with CDH's embedded spark 2 did not cause TOREE-327 > for me. Instead, I got this error: > {code} > Exception in thread "main" java.lang.IncompatibleClassChangeError: class > org.clapper.classutil.asm.ASMEmptyVisitor has interface > org.objectweb.asm.ClassVisitor as super class > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250) > at > org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427) > at > org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385) > at > org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385) > at scala.collection.immutable.Stream.map(Stream.scala:418) > at > org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385) > at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340) > at > org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329) > at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320) > at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311) > at > org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73) > at > org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73) > at scala.util.Try$.apply(Try.scala:192) > at > org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73) > at > org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35) > at > org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34) > at > org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38) > at > org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38) > at > org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45) > at > org.apache.toree.plugins.PluginManager.i
[jira] [Created] (TOREE-454) Use release audit tool to validate source release files.
Ryan Blue created TOREE-454: --- Summary: Use release audit tool to validate source release files. Key: TOREE-454 URL: https://issues.apache.org/jira/browse/TOREE-454 Project: TOREE Issue Type: Bug Reporter: Ryan Blue -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TOREE-454) Use release audit tool to validate source release files.
[ https://issues.apache.org/jira/browse/TOREE-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved TOREE-454. - Resolution: Not A Problem Fix Version/s: Not Applicable Already exists. Use {{make audit-licenses}}. > Use release audit tool to validate source release files. > > > Key: TOREE-454 > URL: https://issues.apache.org/jira/browse/TOREE-454 > Project: TOREE > Issue Type: Bug > Reporter: Ryan Blue > Fix For: Not Applicable > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TOREE-428) Can't use case class in the Scala notebook
[ https://issues.apache.org/jira/browse/TOREE-428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286283#comment-16286283 ] Ryan Blue commented on TOREE-428: - We've noticed this problem as well. It is something to do with the way implicits are used to create decoders for case classes. *The work-around is to create the case class in a separate cell.* I'm not sure why this works, but given the work-around is so easy, we are planning to wait for Scala 2.12 or just move away from the standard Scala interpreter. This affects all of the REPLs that are based on the Scala interpreter, by the way. You can get it to happen in {{spark-shell}} by ensuring the case class and its use happen in the same eval call in the interpreter. Just run this line: {code} case class IdClass(id: String); sc.textFile("arraystore.csv").map(IdClass).collect() {code} > Can't use case class in the Scala notebook > -- > > Key: TOREE-428 > URL: https://issues.apache.org/jira/browse/TOREE-428 > Project: TOREE > Issue Type: Bug > Components: Build >Reporter: Haifeng Li > > the version of docker: > jupyter/all-spark-notebook:lastest > the way to start docker: > docker run -it --rm -p : jupyter/all-spark-notebook:latest > or > docker ps -a > docker start -i containerID > the steps: > Visit http://localhost: > Start an toree notebook > input code above > {code:java} > import spark.implicits._ > val p = spark.sparkContext.textFile ("../Data/person.txt") > val pmap = p.map ( _.split (",")) > pmap.collect() > {code} > the output:res0: Array[Array[String]] = Array(Array(Barack, Obama, 53), > Array(George, Bush, 68), Array(Bill, Clinton, 68)) > {code:java} > case class Persons (first_name:String,last_name: String,age:Int) > val personRDD = pmap.map ( p => Persons (p(0), p(1), p(2).toInt)) > personRDD.take(1) > {code} > the error message: > {code:java} > org.apache.spark.SparkDriverExecutionException: Execution error > at > org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1186) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1711) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:2043) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:2062) > at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1354) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) > at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) > at org.apache.spark.rdd.RDD.take(RDD.scala:1327) > ... 39 elided > Caused by: java.lang.ArrayStoreException: [LPersons; > at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:90) > at > org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:2043) > at > org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:2043) > at org.apache.spark.scheduler.JobWaiter.taskSucceeded(JobWaiter.scala:59) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1182) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1711) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > {code} > The above code is working with the spark-shell. From error message, I > speculated that the driver program didn't correctly handle case class Persons > to RDD partition. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TOREE-427) Unable to use display() or magics in Toree-PySpark
[ https://issues.apache.org/jira/browse/TOREE-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306482#comment-16306482 ] Ryan Blue commented on TOREE-427: - Magics currently get translated into Scala code that gets run in the current interpreter. We need to move magics to get invoked directly instead of going through the interpreter. That would also help debugging because you could have magics that, for example, print logs when the interpreter has crashed. Patches are welcome for this work, and I'll help review them if anyone wants to pick this up. > Unable to use display() or magics in Toree-PySpark > -- > > Key: TOREE-427 > URL: https://issues.apache.org/jira/browse/TOREE-427 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Environment: Linux >Reporter: Mark Gritter >Priority: Minor > > I am trying to use PySpark in a Jupyter notebook using PySpark. Plain-text > interaction is working fine. When I tried to include HTML or graphical > output, though I get the following: > display(sc.getConf().getAll()) > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in > eval(compiled_code) > File "", line 1, in > File "/usr/local/lib/python2.7/dist-packages/IPython/core/display.py", line > 300, in display > format = InteractiveShell.instance().display_formatter.format > File > "/usr/local/lib/python2.7/dist-packages/traitlets/config/configurable.py", > line 412, in instance > inst = cls(*args, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 500, in __init__ > self.init_io() > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 660, in init_io > io.stdout = io.IOStream(sys.stdout) > File "/usr/local/lib/python2.7/dist-packages/IPython/utils/io.py", line 34, > in __init__ > raise ValueError("fallback required, but not specified") > ValueError: fallback required, but not specified > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.CallCommand.execute(CallCommand.java:79) > py4j.GatewayConnection.run(GatewayConnection.java:214) > java.lang.Thread.run(Thread.java:748) > I tried using magics to display raw HTML (following the example here: > https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/magic-tutorial.ipynb) > and get the following backtrace: > %%dataframe > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in > eval(compiled_code) > File "", line 1, in > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 107, in __getattr__ > return self._jvm_kernel.__getattribute__(name) > AttributeError: 'JavaObject' object has no attribute 'magics' > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
[jira] [Created] (TOREE-376) Scala interpreter should make Spark SQL implicits available
Ryan Blue created TOREE-376: --- Summary: Scala interpreter should make Spark SQL implicits available Key: TOREE-376 URL: https://issues.apache.org/jira/browse/TOREE-376 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue The scala interpreter doesn't define Spark SQL functions or implicits that are commonly used and available by default in Spark's shell. Relevant test cases: {code} scala> val c = $"test" c: org.apache.spark.sql.ColumnName = test scala> val df = Seq((1, "a"), (2, "b")).toDF("id", "data") df: org.apache.spark.sql.DataFrame = [id: int, data: string] scala> df.withColumn("m", map(lit("id"), $"id")).printSchema root |-- id: integer (nullable = false) |-- data: string (nullable = true) |-- m: map (nullable = false) ||-- key: string ||-- value: integer (valueContainsNull = false) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-376) Scala interpreter should make Spark SQL implicits available
[ https://issues.apache.org/jira/browse/TOREE-376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864591#comment-15864591 ] Ryan Blue commented on TOREE-376: - This is implemented in PR #96. > Scala interpreter should make Spark SQL implicits available > --- > > Key: TOREE-376 > URL: https://issues.apache.org/jira/browse/TOREE-376 > Project: TOREE > Issue Type: Improvement > Reporter: Ryan Blue > > The scala interpreter doesn't define Spark SQL functions or implicits that > are commonly used and available by default in Spark's shell. > Relevant test cases: > {code} > scala> val c = $"test" > c: org.apache.spark.sql.ColumnName = test > scala> val df = Seq((1, "a"), (2, "b")).toDF("id", "data") > df: org.apache.spark.sql.DataFrame = [id: int, data: string] > scala> df.withColumn("m", map(lit("id"), $"id")).printSchema > root > |-- id: integer (nullable = false) > |-- data: string (nullable = true) > |-- m: map (nullable = false) > ||-- key: string > ||-- value: integer (valueContainsNull = false) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-377) When magic fails, the error is swallowed
Ryan Blue created TOREE-377: --- Summary: When magic fails, the error is swallowed Key: TOREE-377 URL: https://issues.apache.org/jira/browse/TOREE-377 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-378) Usability improvements in jupyter console
Ryan Blue created TOREE-378: --- Summary: Usability improvements in jupyter console Key: TOREE-378 URL: https://issues.apache.org/jira/browse/TOREE-378 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue Using Toree with jupyter console has some bugs and unexpected behavior. These include: * IsCompleteHandler sends "incomplete" for indentation * A syntactically complete multi-line block will be run immediately, unlike python which waits for a blank line * Indentation is based on the previous line, but is not anchored -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-379) Tab completion doesn't replace partial words
Ryan Blue created TOREE-379: --- Summary: Tab completion doesn't replace partial words Key: TOREE-379 URL: https://issues.apache.org/jira/browse/TOREE-379 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue Tab completion in both notebooks and the console is incorrect and causes the front-end to add the selected option after an incomplete word instead of replacing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-380) Interpreters should be allowed to send non-text results.
Ryan Blue created TOREE-380: --- Summary: Interpreters should be allowed to send non-text results. Key: TOREE-380 URL: https://issues.apache.org/jira/browse/TOREE-380 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue Jupyter allows kernels to send results using different content types, but this isn't allowed by Toree for its interpreters. This prevents custom display logic. The basic problem is that `ExecuteOutput` is a `String` and not a `Map[String, String]` like `CellMagicOutput`. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TOREE-380) Interpreters should be allowed to send non-text results.
[ https://issues.apache.org/jira/browse/TOREE-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated TOREE-380: Description: Jupyter allows kernels to send results using different content types, but this isn't allowed by Toree for its interpreters. This prevents custom display logic. The basic problem is that {{ExecuteOutput}} is a {{String}} and not a {{Map[String, String]}} like {{CellMagicOutput}}. (was: Jupyter allows kernels to send results using different content types, but this isn't allowed by Toree for its interpreters. This prevents custom display logic. The basic problem is that `ExecuteOutput` is a `String` and not a `Map[String, String]` like `CellMagicOutput`.) > Interpreters should be allowed to send non-text results. > > > Key: TOREE-380 > URL: https://issues.apache.org/jira/browse/TOREE-380 > Project: TOREE > Issue Type: Improvement >Reporter: Ryan Blue > > Jupyter allows kernels to send results using different content types, but > this isn't allowed by Toree for its interpreters. This prevents custom > display logic. The basic problem is that {{ExecuteOutput}} is a {{String}} > and not a {{Map[String, String]}} like {{CellMagicOutput}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TOREE-380) Interpreters should be allowed to send results other than text/plain.
[ https://issues.apache.org/jira/browse/TOREE-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated TOREE-380: Summary: Interpreters should be allowed to send results other than text/plain. (was: Interpreters should be allowed to send non-text results.) > Interpreters should be allowed to send results other than text/plain. > - > > Key: TOREE-380 > URL: https://issues.apache.org/jira/browse/TOREE-380 > Project: TOREE > Issue Type: Improvement > Reporter: Ryan Blue > > Jupyter allows kernels to send results using different content types, but > this isn't allowed by Toree for its interpreters. This prevents custom > display logic. The basic problem is that {{ExecuteOutput}} is a {{String}} > and not a {{Map[String, String]}} like {{CellMagicOutput}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-387) Kernel should not store SparkSession
Ryan Blue created TOREE-387: --- Summary: Kernel should not store SparkSession Key: TOREE-387 URL: https://issues.apache.org/jira/browse/TOREE-387 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue Currently, the kernel creates and stores the SparkSession in a field to share between interpreters. If the user closes a SparkSession and creates a new one, then the Kernel still returns the original. Users may need to restart Spark sessions for long-running notebooks or to deal with Spark errors without losing datasets that have been pulled back to the notebook. I think that Toree should always return the current Spark session by calling {{SparkSession.builder.getOrCreate}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-388) Add optional gradle build script
Ryan Blue created TOREE-388: --- Summary: Add optional gradle build script Key: TOREE-388 URL: https://issues.apache.org/jira/browse/TOREE-388 Project: TOREE Issue Type: New Feature Reporter: Ryan Blue We've been using a gradle build instead of sbt. I'll open a PR with it, in case anyone else is interested. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-389) Scala interpreter hangs if Spark is referenced before it is bound
Ryan Blue created TOREE-389: --- Summary: Scala interpreter hangs if Spark is referenced before it is bound Key: TOREE-389 URL: https://issues.apache.org/jira/browse/TOREE-389 Project: TOREE Issue Type: Bug Reporter: Ryan Blue Some users run cells in a notebook before the kernel is finished starting. The messages are queued and run after the KernelMessageRelay is enabled, but any messages that reference {{spark}} or bound variables cause IMain to throw an exception. That exception causes the interpreter to hang. It seems odd that a missing variable reference causes IMain to throw an exception, so there may be a threading issue, a race between binding {{spark}} and the cell that references it. The problem can be avoided by moving {{postInit}} tasks into init. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-390) Lazily start Spark sessions
Ryan Blue created TOREE-390: --- Summary: Lazily start Spark sessions Key: TOREE-390 URL: https://issues.apache.org/jira/browse/TOREE-390 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue In our deployment, more than half of the startup time for a Toree notebook is taken by starting a Spark session and waiting for containers. Lazily starting Spark sessions helps the notebook environment feel faster, even if the user is waiting on Spark to start up because the time waiting for Spark to start is clearly Spark, not Toree, and is initiated by the user. Also, lazily starting a Spark session allows users to change settings that can't be changed in a Spark context. It also enables the same startup code that would be used in a spark-submit application: {code:lang=java} SparkSession.builder .config(...) .setAppName(...) .getOrCreate() {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934825#comment-15934825 ] Ryan Blue commented on TOREE-395: - Thanks for continuing the discussion on this, Kevin. A few weeks ago, I posted a [WIP pull request|https://github.com/apache/incubator-toree/pull/104] with what we're currently using that you might be interested in. That addresses a few issues, including this one. The approach we took was to get rid of multiOutputStream and just use lastResultOut. Normal stderr and stdout printing happens just fine, so I think that the only output that gets printed the writer passed to IMain is the summary of what ran (e.g. import scala...). Your idea to conditionally print the output to the logger should work fine, but is it really necessary for people to see this output in the logs? I see no reason to keep it this way. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934883#comment-15934883 ] Ryan Blue commented on TOREE-395: - That PR probably won't be committed on its own. It has a few parts, one of which is this issue. The main two are: 1. How does an interpreter provide rich output to Jupyter notebooks? Right now, they can only provide text so I think we need to change the API. That PR changes the API so that interpreters return a {{Map[String, String]}} that is the representation of an object by mime type. 2. How can an interpreter inspect Java objects to create rich output? Python uses {{_repr_html_}} methods, but defining magic method names isn't a good practice in JVM languages. The PR introduces an API for registering type-safe functions that can turn an object into the {{Map[String, String]}} set of representations. I pulled out the display API to a separate repo here: https://github.com/rdblue/jupyter-repr-api The idea is to get the Jupyter community to recommend a minimal API that can be an intermediary between Jupyter kernels and libraries. Next steps are to get feedback on that API and to get the change for #1 committed. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935254#comment-15935254 ] Ryan Blue commented on TOREE-395: - Yeah, please do. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936567#comment-15936567 ] Ryan Blue commented on TOREE-395: - I don't think there is a reason to keep it. When using Toree with Jupyter notebooks, the user doesn't see the second copy of results. That opens up the possibility to print sensitive data as was already pointed out, but, more importantly, it pollutes the notebook server log with results that aren't useful when you go to look at it. Is there a use case for notebooks that you can think of that makes this a useful feature? The other environment to think about is Jupyter console. When you use Toree with the console, everything gets printed as cell output (which my PR addresses) and is printed to stdout. That it produces double output is the main reason why we disabled it. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-390) Lazily start Spark sessions
[ https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990490#comment-15990490 ] Ryan Blue commented on TOREE-390: - [~luciano resende], we aren't having the weird issues with requests any more, since the commit you linked to and [8f1775d|https://github.com/apache/incubator-toree/commit/8f1775d9dcdc56dd4f3a7c44abf0c36ae0417444]. The problem this fixes is the initial wait time is both startup for Toree and Spark. This separates the two waits, making it clear which one is due to Spark starting up and getting resources from YARN (in our deployment). This is better for the user experience because people don't think Toree is slow, they correctly attribute it to Spark. The secondary benefit is that users can customize their Spark session before it starts by creating one in the first cell, and Toree works just fine with that. > Lazily start Spark sessions > --- > > Key: TOREE-390 > URL: https://issues.apache.org/jira/browse/TOREE-390 > Project: TOREE > Issue Type: Improvement >Reporter: Ryan Blue > > In our deployment, more than half of the startup time for a Toree notebook is > taken by starting a Spark session and waiting for containers. Lazily starting > Spark sessions helps the notebook environment feel faster, even if the user > is waiting on Spark to start up because the time waiting for Spark to start > is clearly Spark, not Toree, and is initiated by the user. > Also, lazily starting a Spark session allows users to change settings that > can't be changed in a Spark context. It also enables the same startup code > that would be used in a spark-submit application: > {code:lang=java} > SparkSession.builder > .config(...) > .setAppName(...) > .getOrCreate() > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[ https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045657#comment-16045657 ] Ryan Blue commented on TOREE-411: - The solution to this problem that we use is to set the catalog implementation to "hive", as suggested by Xin, in our Spark defaults. Also, TOREE-390 was just committed, which changes the Spark session so that it is lazily created when you reference {{spark}} or picks up an existing context. That means you can either use the default session or create your own and call {{enableHiveSupport}}, among other configuration options. This should address the problem without needing to set the catalog implementation outside of Toree. > Hive tables are not visible from Toree > -- > > Key: TOREE-411 > URL: https://issues.apache.org/jira/browse/TOREE-411 > Project: TOREE > Issue Type: Bug > Components: Kernel > Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1 >Reporter: Alex Adria Cuadripani > Labels: hadoop, hive, spark > Fix For: 0.2.0 > > > I am trying to work with Hive tables but is impossible to see it with toree. > I do: > from pyspark.sql import SparkSession > spark = > SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate() > spark.sql("show tables").show() > It works perfectly in pyspark kernel (ipykernel), but in toree kernel it > returns an empty list. > I have seen issue 353, but I don't know if it is the same issue, or what I > have to do exactly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-408) AddJAR magic should support HDFS file format
[ https://issues.apache.org/jira/browse/TOREE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045658#comment-16045658 ] Ryan Blue commented on TOREE-408: - I have a patch for this in our version that supports Hadoop file systems like HDFS and S3. I'll open a PR with it. > AddJAR magic should support HDFS file format > > > Key: TOREE-408 > URL: https://issues.apache.org/jira/browse/TOREE-408 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Should implement HDFS downloader on Toree > Related source codes: > Download utils: > https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/utils/DownloadSupport.scala > AddJar magic: > https://github.com/apache/incubator-toree/blob/master/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-407) Improve Branding on Site
[ https://issues.apache.org/jira/browse/TOREE-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045660#comment-16045660 ] Ryan Blue commented on TOREE-407: - +1 [~luciano resende], are there docs on how to edit the site? > Improve Branding on Site > > > Key: TOREE-407 > URL: https://issues.apache.org/jira/browse/TOREE-407 > Project: TOREE > Issue Type: Improvement >Reporter: Kyle Kelley > > I want to recommend Toree to others as a Scala kernel. The messaging on > https://toree.incubator.apache.org/ is all about "remote spark" which muddies > what it actually does, which is provide kernels that are connected to Spark. > Toree, standalone, doesn't do anything without Jupyter. > Here's what I wish it read: > ``` > Apache Toree > Spark connected kernels for Jupyter projects - Scala, Python, and R > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-412) AddDeps should default to the default configuration, not compile
Ryan Blue created TOREE-412: --- Summary: AddDeps should default to the default configuration, not compile Key: TOREE-412 URL: https://issues.apache.org/jira/browse/TOREE-412 Project: TOREE Issue Type: Improvement Components: Kernel Reporter: Ryan Blue The AddDeps magic (using Coursier) currently requests the compile configuration instead of the default configuration. The dependencies shouldn't be those needed to compile, but should be the runtime dependencies, which are normally pulled in by the default configuration. We ran into issues with this because runtime dependencies are not included. In addition, I'd like to add --ivy-configuration and --classifier options to give users control over what gets used. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-413) Add support for session timeout and exit
Ryan Blue created TOREE-413: --- Summary: Add support for session timeout and exit Key: TOREE-413 URL: https://issues.apache.org/jira/browse/TOREE-413 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue When using Toree with Jupyter console, the only way to exit is to use CTRL+D to send EOF and then confirm. Toree should include support for "exit", "quit", and ":q" in any interpreter. In addition, because Toree runs in a separate process and can start up slowly, the default Jupyter console timeout (60s) will sometimes cause the console to exit but leave a Toree kernel running. Similarly, losing a SSH session running Jupyter console with Toree leaves the kernel running. Toree should have a 24h idle timeout to clean up these zombie kernels. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-414) Non-daemon thread pools prevent graceful shutdown
Ryan Blue created TOREE-414: --- Summary: Non-daemon thread pools prevent graceful shutdown Key: TOREE-414 URL: https://issues.apache.org/jira/browse/TOREE-414 Project: TOREE Issue Type: Improvement Affects Versions: 0.1.0 Reporter: Ryan Blue TaskManager and ScheduledTaskManager create threads without using ThreadFactoryBuilder, which can set the thread name and whether the thread is a daemon. Not using daemon threads prevents graceful shutdown (TOREE-413). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-415) Interrupting tasks should cancel running Spark jobs.
Ryan Blue created TOREE-415: --- Summary: Interrupting tasks should cancel running Spark jobs. Key: TOREE-415 URL: https://issues.apache.org/jira/browse/TOREE-415 Project: TOREE Issue Type: Improvement Components: Kernel Affects Versions: 0.1.0 Reporter: Ryan Blue The most common use of CTRL+C in Jupyter console is to cancel the running Spark stage, but the current implementation only interrupts the monitoring thread. Interrupting a running cell should cancel running Spark jobs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-416) Flaky test: KernelInputStreamSpec
Ryan Blue created TOREE-416: --- Summary: Flaky test: KernelInputStreamSpec Key: TOREE-416 URL: https://issues.apache.org/jira/browse/TOREE-416 Project: TOREE Issue Type: Bug Affects Versions: 0.1.0 Reporter: Ryan Blue When running tests in gradle, KernelInputStreamSpec is flaky. Calls to the InputStream.read method hang forever. Sleeping for ~100ms at the end of the before block seems to avoid the problem. The likely cause is an actor thread or ZMQ consumer that needs to start up before running tests, or messages get lost. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TOREE-417) Flaky test: ExecuteActorRelaySpec
Ryan Blue created TOREE-417: --- Summary: Flaky test: ExecuteActorRelaySpec Key: TOREE-417 URL: https://issues.apache.org/jira/browse/TOREE-417 Project: TOREE Issue Type: Bug Affects Versions: 0.1.0 Reporter: Ryan Blue ExecuteActorRelaySpec is flaky when running in gradle. Some {{expectMsgClass}} calls take more than the default 3s timeout. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TOREE-418) Improve use of Jupyter's jvm-repr-api
Ryan Blue created TOREE-418: --- Summary: Improve use of Jupyter's jvm-repr-api Key: TOREE-418 URL: https://issues.apache.org/jira/browse/TOREE-418 Project: TOREE Issue Type: Bug Reporter: Ryan Blue TOREE-380 uses the [Jupyter jvm-repr API|https://github.com/jupyter/jvm-repr] to show rich representations of JVM objects. Libraries can register methods to display their classes by MIME type. Currently, the use is limited to the Scala interpreter, but it should probably be used to display any JVM object that interpreters don't handle. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TOREE-380) Interpreters should be allowed to send results other than text/plain.
[ https://issues.apache.org/jira/browse/TOREE-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved TOREE-380. - Resolution: Fixed Fix Version/s: 0.2.0 Merged PR #104. Thanks to [~mariusvniekerk] and [~chipsenkbeil] for reviewing! > Interpreters should be allowed to send results other than text/plain. > - > > Key: TOREE-380 > URL: https://issues.apache.org/jira/browse/TOREE-380 > Project: TOREE > Issue Type: Improvement > Reporter: Ryan Blue >Assignee: Ryan Blue > Fix For: 0.2.0 > > > Jupyter allows kernels to send results using different content types, but > this isn't allowed by Toree for its interpreters. This prevents custom > display logic. The basic problem is that {{ExecuteOutput}} is a {{String}} > and not a {{Map[String, String]}} like {{CellMagicOutput}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TOREE-414) Non-daemon thread pools prevent graceful shutdown
[ https://issues.apache.org/jira/browse/TOREE-414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved TOREE-414. - Resolution: Fixed Assignee: Ryan Blue Fix Version/s: 0.2.0 Merged #122. Thanks to [~senkwich] for reviewing! > Non-daemon thread pools prevent graceful shutdown > - > > Key: TOREE-414 > URL: https://issues.apache.org/jira/browse/TOREE-414 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.1.0 >Reporter: Ryan Blue > Assignee: Ryan Blue > Fix For: 0.2.0 > > > TaskManager and ScheduledTaskManager create threads without using > ThreadFactoryBuilder, which can set the thread name and whether the thread is > a daemon. Not using daemon threads prevents graceful shutdown (TOREE-413). -- This message was sent by Atlassian JIRA (v6.4.14#64029)