[jira] [Commented] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059902#comment-17059902
 ] 

ASF GitHub Bot commented on TOREE-508:
--

lresende commented on pull request #180: [TOREE-508] Reply errors to 
iopub.error instead of iopub.exec_result
URL: https://github.com/apache/incubator-toree/pull/180
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reply errors to iopub.error instead of iopub.exec_result
> 
>
> Key: TOREE-508
> URL: https://issues.apache.org/jira/browse/TOREE-508
> Project: TOREE
>  Issue Type: Bug
>Reporter: Fabiano V. Santos
>Priority: Major
>
> Errors of any kind (compilation, exceptions, etc) returns as normal results 
> instead of using iopub.error, so jupyter interpret the cell execution as 
> success which lead to problems if you use tools like papermill to schedule 
> notebooks, as you never know if the notebook execution was successfully or 
> not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019410#comment-17019410
 ] 

ASF GitHub Bot commented on TOREE-508:
--

gmcoringa commented on issue #180: [TOREE-508] Reply errors to iopub.error 
instead of iopub.exec_result
URL: https://github.com/apache/incubator-toree/pull/180#issuecomment-576235407
 
 
   > @lresende fix done, also I did another commit to fix some unit tests, but 
there are other tests failing, that I couldn't understand why.
   
   Another point is that I was not able to reproduce the error.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reply errors to iopub.error instead of iopub.exec_result
> 
>
> Key: TOREE-508
> URL: https://issues.apache.org/jira/browse/TOREE-508
> Project: TOREE
>  Issue Type: Bug
>Reporter: Fabiano V. Santos
>Priority: Major
>
> Errors of any kind (compilation, exceptions, etc) returns as normal results 
> instead of using iopub.error, so jupyter interpret the cell execution as 
> success which lead to problems if you use tools like papermill to schedule 
> notebooks, as you never know if the notebook execution was successfully or 
> not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018142#comment-17018142
 ] 

ASF GitHub Bot commented on TOREE-508:
--

gmcoringa commented on issue #180: [TOREE-508] Reply errors to iopub.error 
instead of iopub.exec_result
URL: https://github.com/apache/incubator-toree/pull/180#issuecomment-575652569
 
 
   @lresende fix done, also I did another commit to fix some unit tests, but 
there are other tests failing, that I couldn't understand why. 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reply errors to iopub.error instead of iopub.exec_result
> 
>
> Key: TOREE-508
> URL: https://issues.apache.org/jira/browse/TOREE-508
> Project: TOREE
>  Issue Type: Bug
>Reporter: Fabiano V. Santos
>Priority: Major
>
> Errors of any kind (compilation, exceptions, etc) returns as normal results 
> instead of using iopub.error, so jupyter interpret the cell execution as 
> success which lead to problems if you use tools like papermill to schedule 
> notebooks, as you never know if the notebook execution was successfully or 
> not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018080#comment-17018080
 ] 

ASF GitHub Bot commented on TOREE-508:
--

gmcoringa commented on issue #180: [TOREE-508] Reply errors to iopub.error 
instead of iopub.exec_result
URL: https://github.com/apache/incubator-toree/pull/180#issuecomment-575652569
 
 
   @lresende fix done, also I did another commit to fix some unit tests, but 
there are other testing failing, that I couldn't understand why. 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reply errors to iopub.error instead of iopub.exec_result
> 
>
> Key: TOREE-508
> URL: https://issues.apache.org/jira/browse/TOREE-508
> Project: TOREE
>  Issue Type: Bug
>Reporter: Fabiano V. Santos
>Priority: Major
>
> Errors of any kind (compilation, exceptions, etc) returns as normal results 
> instead of using iopub.error, so jupyter interpret the cell execution as 
> success which lead to problems if you use tools like papermill to schedule 
> notebooks, as you never know if the notebook execution was successfully or 
> not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017455#comment-17017455
 ] 

ASF GitHub Bot commented on TOREE-508:
--

lresende commented on issue #180: [TOREE-508] Reply errors to iopub.error 
instead of iopub.exec_result
URL: https://github.com/apache/incubator-toree/pull/180#issuecomment-575329162
 
 
   @gmcoringa Yes, please update with the fix for the removal of the message.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reply errors to iopub.error instead of iopub.exec_result
> 
>
> Key: TOREE-508
> URL: https://issues.apache.org/jira/browse/TOREE-508
> Project: TOREE
>  Issue Type: Bug
>Reporter: Fabiano V. Santos
>Priority: Major
>
> Errors of any kind (compilation, exceptions, etc) returns as normal results 
> instead of using iopub.error, so jupyter interpret the cell execution as 
> success which lead to problems if you use tools like papermill to schedule 
> notebooks, as you never know if the notebook execution was successfully or 
> not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016228#comment-17016228
 ] 

ASF GitHub Bot commented on TOREE-508:
--

gmcoringa commented on issue #180: [TOREE-508] Reply errors to iopub.error 
instead of iopub.exec_result
URL: https://github.com/apache/incubator-toree/pull/180#issuecomment-574800328
 
 
   > I believe we are going in the right direction here, but with the change on 
behavior the errors are not completely showing up on the notebook interface.
   > 
   > Original behavior:
   > 
![toree-508-original](https://user-images.githubusercontent.com/382917/72102913-e9819900-32dc-11ea-803d-4175c986128a.png)
   > 
   > And with the changes from this pr:
   > 
![toree-508-updated](https://user-images.githubusercontent.com/382917/72102929-f30b0100-32dc-11ea-8262-93ea5c7646e6.png)
   > 
   > As shown in the picture above, the contents are not all there, and the 
cell is not scrollable anymore.
   
   The error is not completely showing because of the code below found in 
[ScalaInterpreterSpecific.scala#L429](https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala-2.11/org/apache/toree/kernel/interpreter/scala/ScalaInterpreterSpecific.scala#L429):
   ```java
   ExecuteError(
 ex.getClass.getName,
 ex.getLocalizedMessage,
 formattedException.slice(1, formattedException.size - 1).toList
   )
   ```
   Which removes the first line from the stack, I don't know why, but I believe 
that this should not be done.
   
   About the scrollable, testing with python kernels seems to have the same 
behaviour (errors are not scrollable).
   
   So @lresende if you think its better I can fix the removal of messages from 
the stack trace.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reply errors to iopub.error instead of iopub.exec_result
> 
>
> Key: TOREE-508
> URL: https://issues.apache.org/jira/browse/TOREE-508
> Project: TOREE
>  Issue Type: Bug
>Reporter: Fabiano V. Santos
>Priority: Major
>
> Errors of any kind (compilation, exceptions, etc) returns as normal results 
> instead of using iopub.error, so jupyter interpret the cell execution as 
> success which lead to problems if you use tools like papermill to schedule 
> notebooks, as you never know if the notebook execution was successfully or 
> not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012214#comment-17012214
 ] 

ASF GitHub Bot commented on TOREE-508:
--

lresende commented on issue #180: [TOREE-508] Reply errors to iopub.error 
instead of iopub.exec_result
URL: https://github.com/apache/incubator-toree/pull/180#issuecomment-572744750
 
 
   I believe we are going in the right direction here, but with the change on 
behavior the errors are not completely showing up on the notebook interface.
   
   Original behavior:
   
![toree-508-original](https://user-images.githubusercontent.com/382917/72102913-e9819900-32dc-11ea-803d-4175c986128a.png)
   
   And with the changes from this pr:
   
![toree-508-updated](https://user-images.githubusercontent.com/382917/72102929-f30b0100-32dc-11ea-8262-93ea5c7646e6.png)
   
   As shown in the picture above, the contents are not all there, and the cell 
is not scrollable anymore.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reply errors to iopub.error instead of iopub.exec_result
> 
>
> Key: TOREE-508
> URL: https://issues.apache.org/jira/browse/TOREE-508
> Project: TOREE
>  Issue Type: Bug
>Reporter: Fabiano V. Santos
>Priority: Major
>
> Errors of any kind (compilation, exceptions, etc) returns as normal results 
> instead of using iopub.error, so jupyter interpret the cell execution as 
> success which lead to problems if you use tools like papermill to schedule 
> notebooks, as you never know if the notebook execution was successfully or 
> not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-514) Update dependencies in preparation for 0.4.0 release

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010130#comment-17010130
 ] 

ASF GitHub Bot commented on TOREE-514:
--

lresende commented on pull request #183: [TOREE-514] Update Toree kernel 
dependencies
URL: https://github.com/apache/incubator-toree/pull/183
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update dependencies in preparation for 0.4.0 release
> 
>
> Key: TOREE-514
> URL: https://issues.apache.org/jira/browse/TOREE-514
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Affects Versions: 0.4.0
>Reporter: Luciano Resende
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-513) Add logo resource for Toree Kernel

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010127#comment-17010127
 ] 

ASF GitHub Bot commented on TOREE-513:
--

lresende commented on pull request #182: [TOREE-513] Adds logo resource for 
installed Toree kernels
URL: https://github.com/apache/incubator-toree/pull/182
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add logo resource for Toree Kernel
> --
>
> Key: TOREE-513
> URL: https://issues.apache.org/jira/browse/TOREE-513
> Project: TOREE
>  Issue Type: Task
>Affects Versions: 0.4.0
>Reporter: Luciano Resende
>Priority: Major
> Attachments: jupyterlab-launcher-with-kernel-logo.png
>
>
> JupyterLab exposes logos for kernels in the new launcher widget, this adds a 
> logo for Toree installed kernels. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-514) Update dependencies in preparation for 0.4.0 release

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010075#comment-17010075
 ] 

ASF GitHub Bot commented on TOREE-514:
--

lresende commented on pull request #183: [TOREE-514] Update Toree kernel 
dependencies
URL: https://github.com/apache/incubator-toree/pull/183
 
 
   Update the following dependencies:
- Akka 2.5.27
- 0MQ 0.4.3
- Spring Framework 5.2.2.RELEASE
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update dependencies in preparation for 0.4.0 release
> 
>
> Key: TOREE-514
> URL: https://issues.apache.org/jira/browse/TOREE-514
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Affects Versions: 0.4.0
>Reporter: Luciano Resende
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-513) Add logo resource for Toree Kernel

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010023#comment-17010023
 ] 

ASF GitHub Bot commented on TOREE-513:
--

lresende commented on pull request #182: [TOREE-513] Adds logo resource for 
installed Toree kernels
URL: https://github.com/apache/incubator-toree/pull/182
 
 
   With the new enhancements, the installed Toree kernels will have a logo in 
the JupyterLab launcher
   
   
![jupyterlab-launcher-with-kernel-logo](https://user-images.githubusercontent.com/382917/71923433-e00ff980-3141-11ea-931f-b7b76ef626c2.png)
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add logo resource for Toree Kernel
> --
>
> Key: TOREE-513
> URL: https://issues.apache.org/jira/browse/TOREE-513
> Project: TOREE
>  Issue Type: Task
>Affects Versions: 0.4.0
>Reporter: Luciano Resende
>Priority: Major
> Attachments: jupyterlab-launcher-with-kernel-logo.png
>
>
> JupyterLab exposes logos for kernels in the new launcher widget, this adds a 
> logo for Toree installed kernels. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009951#comment-17009951
 ] 

ASF GitHub Bot commented on TOREE-511:
--

lresende commented on pull request #181: [TOREE-511] Rework make jupyter build 
tasks
URL: https://github.com/apache/incubator-toree/pull/181
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009828#comment-17009828
 ] 

ASF GitHub Bot commented on TOREE-511:
--

lresende commented on pull request #181: [TOREE-511] Rework make jupyter build 
tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r363805182
 
 

 ##
 File path: Makefile
 ##
 @@ -15,7 +15,10 @@
 # limitations under the License
 #
 
-.PHONY: help clean clean-dist build dev test system-test test-travis release 
pip-release bin-release dev-binder .binder-image audit audit-licenses
+.PHONY: help clean clean-dist .clean-binder-image .clean-toree-dev-image \
+ build dev test system-test test-travis release pip-release bin-release \
+ dev-binder .binder-image .toree-dev-image \
 
 Review comment:
   regarding the dot, I was just following the same pattern as others
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009780#comment-17009780
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r363778165
 
 

 ##
 File path: Makefile
 ##
 @@ -83,21 +87,19 @@ clean: clean-dist
rm -r `find . -name target -type d`
-rm -r `find . -name .ipynb_checkpoints -type d`
 
-.example-image: EXTRA_CMD?=printf "deb 
http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> 
/etc/apt/sources.list; apt-key adv --keyserver keys.gnupg.net --recv-key 
381BA480; apt-get update; pip install jupyter_declarativewidgets==0.4.4; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs r-base r-base-dev; npm install -g bower;
-.example-image:
-   @-docker rm -f examples_image
-   @docker run -t --user root --name examples_image \
-   $(IMAGE) bash -c '$(EXTRA_CMD)'
-   @docker commit examples_image $(EXAMPLE_IMAGE)
-   @-docker rm -f examples_image
-   touch $@
+.clean-toree-dev-image:
+   if [ -f .toree-dev-image ]; then rm -r .toree-dev-image; fi
 
 Review comment:
   Since `.toree-dev-image` is just a file and not a directory, this line can 
be replaced with...
   
   `@rm -f .toree-dev-image`
   
   `-f` removes the need to test of existence.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009782#comment-17009782
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r363777123
 
 

 ##
 File path: Makefile
 ##
 @@ -83,21 +87,19 @@ clean: clean-dist
rm -r `find . -name target -type d`
-rm -r `find . -name .ipynb_checkpoints -type d`
 
-.example-image: EXTRA_CMD?=printf "deb 
http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> 
/etc/apt/sources.list; apt-key adv --keyserver keys.gnupg.net --recv-key 
381BA480; apt-get update; pip install jupyter_declarativewidgets==0.4.4; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs r-base r-base-dev; npm install -g bower;
-.example-image:
-   @-docker rm -f examples_image
-   @docker run -t --user root --name examples_image \
-   $(IMAGE) bash -c '$(EXTRA_CMD)'
-   @docker commit examples_image $(EXAMPLE_IMAGE)
-   @-docker rm -f examples_image
-   touch $@
+.clean-toree-dev-image:
+   if [ -f .toree-dev-image ]; then rm -r .toree-dev-image; fi
+   @-docker rmi -f $(TOREE_DEV_IMAGE)
 
-.system-test-image:
-   @-docker rm -f $(SYSTEM_TEST_IMAGE)
-   @docker build -t $(SYSTEM_TEST_IMAGE) -f Dockerfile.system-test .
+.toree-dev-image:
+   @docker build -t $(TOREE_DEV_IMAGE) -f Dockerfile.toree-dev .
touch $@
 
-.binder-image:
+.clean-binder-image:
+   if [ -f .binder-image ]; then rm -r .binder-image; fi
+   @-docker rmi -f $(BINDER_IMAGE)
+
+.binder-image: .clean-binder-image
@docker build --rm -t $(BINDER_IMAGE) .
 
 Review comment:
   This target needs `touch $@` to create the `.binder-image` file.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009779#comment-17009779
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r363776548
 
 

 ##
 File path: Makefile
 ##
 @@ -15,7 +15,10 @@
 # limitations under the License
 #
 
-.PHONY: help clean clean-dist build dev test system-test test-travis release 
pip-release bin-release dev-binder .binder-image audit audit-licenses
+.PHONY: help clean clean-dist .clean-binder-image .clean-toree-dev-image \
+ build dev test system-test test-travis release pip-release bin-release \
+ dev-binder .binder-image .toree-dev-image \
 
 Review comment:
   `.binder-image` and `.toree-dev-image` should not be listed here as they 
correspond to files that _are_ the target.
   
   Also, do we need the '.' prefix on `.clean-binder-image` and 
`.clean-toree-dev-image`?  I think the dotted notation is purely because we 
don't want the files visible by default.  Because these are PHONY targets, I 
think removing the dotted prefix is appropriate.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009781#comment-17009781
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r363778369
 
 

 ##
 File path: Makefile
 ##
 @@ -83,21 +87,19 @@ clean: clean-dist
rm -r `find . -name target -type d`
-rm -r `find . -name .ipynb_checkpoints -type d`
 
-.example-image: EXTRA_CMD?=printf "deb 
http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> 
/etc/apt/sources.list; apt-key adv --keyserver keys.gnupg.net --recv-key 
381BA480; apt-get update; pip install jupyter_declarativewidgets==0.4.4; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs r-base r-base-dev; npm install -g bower;
-.example-image:
-   @-docker rm -f examples_image
-   @docker run -t --user root --name examples_image \
-   $(IMAGE) bash -c '$(EXTRA_CMD)'
-   @docker commit examples_image $(EXAMPLE_IMAGE)
-   @-docker rm -f examples_image
-   touch $@
+.clean-toree-dev-image:
+   if [ -f .toree-dev-image ]; then rm -r .toree-dev-image; fi
+   @-docker rmi -f $(TOREE_DEV_IMAGE)
 
-.system-test-image:
-   @-docker rm -f $(SYSTEM_TEST_IMAGE)
-   @docker build -t $(SYSTEM_TEST_IMAGE) -f Dockerfile.system-test .
+.toree-dev-image:
+   @docker build -t $(TOREE_DEV_IMAGE) -f Dockerfile.toree-dev .
touch $@
 
-.binder-image:
+.clean-binder-image:
+   if [ -f .binder-image ]; then rm -r .binder-image; fi
 
 Review comment:
   Same comment as `.toree-dev-image` removal above.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007112#comment-17007112
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r362649415
 
 

 ##
 File path: Makefile
 ##
 @@ -63,12 +63,14 @@ help:
@echo ' '
@echo ' audit - run audit tools against the source code'
@echo ' clean - clean build files'
-   @echo ' dev - starts ipython'
+   @echo ' clean-images - clean docker images used in examples and '
 
 Review comment:
   The command's description is incomplete.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007110#comment-17007110
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r362650343
 
 

 ##
 File path: Makefile
 ##
 @@ -81,19 +83,15 @@ clean: VM_WORKDIR=/src/toree-kernel
 clean: clean-dist
$(call RUN,$(ENV_OPTS) sbt clean)
rm -r `find . -name target -type d`
+   -rm -r `find . -name .ipynb_checkpoints -type d`
 
-.example-image: EXTRA_CMD?=printf "deb 
http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> 
/etc/apt/sources.list; apt-key adv --keyserver keys.gnupg.net --recv-key 
381BA480; apt-get update; pip install jupyter_declarativewidgets==0.4.4; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs r-base r-base-dev; npm install -g bower;
-.example-image:
-   @-docker rm -f examples_image
-   @docker run -t --user root --name examples_image \
-   $(IMAGE) bash -c '$(EXTRA_CMD)'
-   @docker commit examples_image $(EXAMPLE_IMAGE)
-   @-docker rm -f examples_image
-   touch $@
+clean-images:
+   -rm -r .toree-dev-image
+   -rm -r .binder-image
 
 Review comment:
   Should this target actually remove the images as well?  If so, 
`clean-images` would depend on `clean-toree-dev-image` and `clean-binder-image` 
- with those added to `.PHONY` and ignore the same comment regarding 
`clean-images` since it now depends on other targets.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007115#comment-17007115
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r362651592
 
 

 ##
 File path: .gitignore
 ##
 @@ -51,7 +51,7 @@ apache-rat-*.jar
 **.swp
 **/*ivy.xml
 .vagrant
-.example-image
+.toree-dev-image
 
 Review comment:
   Will likely need to add `.binder-image` once the `touch` is added (per other 
comment).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007113#comment-17007113
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r362650724
 
 

 ##
 File path: Makefile
 ##
 @@ -81,19 +83,15 @@ clean: VM_WORKDIR=/src/toree-kernel
 clean: clean-dist
$(call RUN,$(ENV_OPTS) sbt clean)
rm -r `find . -name target -type d`
+   -rm -r `find . -name .ipynb_checkpoints -type d`
 
-.example-image: EXTRA_CMD?=printf "deb 
http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> 
/etc/apt/sources.list; apt-key adv --keyserver keys.gnupg.net --recv-key 
381BA480; apt-get update; pip install jupyter_declarativewidgets==0.4.4; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs r-base r-base-dev; npm install -g bower;
-.example-image:
-   @-docker rm -f examples_image
-   @docker run -t --user root --name examples_image \
-   $(IMAGE) bash -c '$(EXTRA_CMD)'
-   @docker commit examples_image $(EXAMPLE_IMAGE)
-   @-docker rm -f examples_image
-   touch $@
+clean-images:
+   -rm -r .toree-dev-image
+   -rm -r .binder-image
 
-.system-test-image:
-   @-docker rm -f $(SYSTEM_TEST_IMAGE)
-   @docker build -t $(SYSTEM_TEST_IMAGE) -f Dockerfile.system-test .
+.toree-dev-image:
+   @-docker rm -f $(TOREE_DEV_IMAGE)
 
 Review comment:
   If `clean-images` removes the images, then this target would depend on 
`clean-toree-dev-image` and that target would include the `docker rm` command.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007114#comment-17007114
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r362651030
 
 

 ##
 File path: Makefile
 ##
 @@ -81,19 +83,15 @@ clean: VM_WORKDIR=/src/toree-kernel
 clean: clean-dist
$(call RUN,$(ENV_OPTS) sbt clean)
rm -r `find . -name target -type d`
+   -rm -r `find . -name .ipynb_checkpoints -type d`
 
-.example-image: EXTRA_CMD?=printf "deb 
http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> 
/etc/apt/sources.list; apt-key adv --keyserver keys.gnupg.net --recv-key 
381BA480; apt-get update; pip install jupyter_declarativewidgets==0.4.4; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs r-base r-base-dev; npm install -g bower;
-.example-image:
-   @-docker rm -f examples_image
-   @docker run -t --user root --name examples_image \
-   $(IMAGE) bash -c '$(EXTRA_CMD)'
-   @docker commit examples_image $(EXAMPLE_IMAGE)
-   @-docker rm -f examples_image
-   touch $@
+clean-images:
+   -rm -r .toree-dev-image
+   -rm -r .binder-image
 
-.system-test-image:
-   @-docker rm -f $(SYSTEM_TEST_IMAGE)
-   @docker build -t $(SYSTEM_TEST_IMAGE) -f Dockerfile.system-test .
+.toree-dev-image:
+   @-docker rm -f $(TOREE_DEV_IMAGE)
+   @docker build -t $(TOREE_DEV_IMAGE) -f Dockerfile.toree-dev .
touch $@
 
 .binder-image:
 
 Review comment:
   I think this target needs a `touch $@` (and image removal - similar to 
`.toree-dev-image`).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007111#comment-17007111
 ] 

ASF GitHub Bot commented on TOREE-511:
--

kevin-bates commented on pull request #181: [TOREE-511] Rework make jupyter 
build tasks
URL: https://github.com/apache/incubator-toree/pull/181#discussion_r362649694
 
 

 ##
 File path: Makefile
 ##
 @@ -81,19 +83,15 @@ clean: VM_WORKDIR=/src/toree-kernel
 clean: clean-dist
$(call RUN,$(ENV_OPTS) sbt clean)
rm -r `find . -name target -type d`
+   -rm -r `find . -name .ipynb_checkpoints -type d`
 
-.example-image: EXTRA_CMD?=printf "deb 
http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> 
/etc/apt/sources.list; apt-key adv --keyserver keys.gnupg.net --recv-key 
381BA480; apt-get update; pip install jupyter_declarativewidgets==0.4.4; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs r-base r-base-dev; npm install -g bower;
-.example-image:
-   @-docker rm -f examples_image
-   @docker run -t --user root --name examples_image \
-   $(IMAGE) bash -c '$(EXTRA_CMD)'
-   @docker commit examples_image $(EXAMPLE_IMAGE)
-   @-docker rm -f examples_image
-   touch $@
+clean-images:
 
 Review comment:
   Since this target doesn't include a dependency, I think it should be added 
to the `.PHONY` target.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-511) Rework "make dev/make jupyter" build tasks

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002129#comment-17002129
 ] 

ASF GitHub Bot commented on TOREE-511:
--

lresende commented on pull request #181: [TOREE-511] Rework make jupyter build 
tasks
URL: https://github.com/apache/incubator-toree/pull/181
 
 
   - Create a proper docker image for dev
   - Combine dev/jupyter build tasks into jupyter
   - Use JupyterLab as default UI
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Rework "make dev/make jupyter" build tasks
> --
>
> Key: TOREE-511
> URL: https://issues.apache.org/jira/browse/TOREE-511
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> Toree has `make dev` and `make jupyter` commands that are using some old 
> dependencies and some weird way to build them.  We should create a docker 
> file that extends from `jupyter/all-spark-notebook` to make maintenance 
> easier. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995558#comment-16995558
 ] 

ASF GitHub Bot commented on TOREE-508:
--

gmcoringa commented on pull request #180: [TOREE-508] Reply errors to 
iopub.error instead of iopub.exec_result
URL: https://github.com/apache/incubator-toree/pull/180
 
 
   Fix string values within Metadata, the values needs to be json string.
   Reply errors to iopub.error instead of iopub.exec_result
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reply errors to iopub.error instead of iopub.exec_result
> 
>
> Key: TOREE-508
> URL: https://issues.apache.org/jira/browse/TOREE-508
> Project: TOREE
>  Issue Type: Bug
>Reporter: Fabiano V. Santos
>Priority: Major
>
> Errors of any kind (compilation, exceptions, etc) returns as normal results 
> instead of using iopub.error, so jupyter interpret the cell execution as 
> success which lead to problems if you use tools like papermill to schedule 
> notebooks, as you never know if the notebook execution was successfully or 
> not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965037#comment-16965037
 ] 

ASF GitHub Bot commented on TOREE-421:
--

lresende commented on pull request #178: [TOREE-421] Allow thread groups
URL: https://github.com/apache/incubator-toree/pull/178
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
>  Issue Type: Bug
>Reporter: Piyush Narang
>Priority: Major
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some 
> issues as the code in our job calls into one of our libraries which tries to 
> create threads in its own ThreadGroup: 
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip: 
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace:   at 
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
>   at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
>   at java.lang.Thread.init(Thread.java:394)
>   at java.lang.Thread.init(Thread.java:349)
>   at java.lang.Thread.(Thread.java:599)
>   at 
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new 
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
>   }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this? 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958060#comment-16958060
 ] 

ASF GitHub Bot commented on TOREE-421:
--

kevin-bates commented on pull request #178: [TOREE-421] Allow thread groups
URL: https://github.com/apache/incubator-toree/pull/178
 
 
   Remove the portions of Security Manager that restricted the creation of 
thread groups.  This change also preserves the prevention of calls to 
`System.exit()` from sub-threads/groups (including those derived from the 
restricted group) - otherwise such calls from threads in a new thread group can 
terminate the kernel.
   
   I chose **not** to implement the ability to preserve the existing logic 
whereby the creation of new groups are prevented (i.e., via a new boolean 
option `--prevent-new-thread-groups`) since it felt like adding something that 
probably isn't necessary.  If others disagree or we (eventually) find a good 
use-case for restoring that kind of functionality, we can add such an option at 
that time.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
>  Issue Type: Bug
>Reporter: Piyush Narang
>Priority: Major
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some 
> issues as the code in our job calls into one of our libraries which tries to 
> create threads in its own ThreadGroup: 
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip: 
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace:   at 
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
>   at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
>   at java.lang.Thread.init(Thread.java:394)
>   at java.lang.Thread.init(Thread.java:349)
>   at java.lang.Thread.(Thread.java:599)
>   at 
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new 
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
>   }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this? 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-505) Travis CI builds are failing due to issues installing JDK8

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957162#comment-16957162
 ] 

ASF GitHub Bot commented on TOREE-505:
--

kevin-bates commented on pull request #177: [TOREE-505] Fix Oracle JDK8 in 
Travis CI
URL: https://github.com/apache/incubator-toree/pull/177
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Travis CI builds are failing due to issues installing JDK8
> --
>
> Key: TOREE-505
> URL: https://issues.apache.org/jira/browse/TOREE-505
> Project: TOREE
>  Issue Type: Bug
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
>
> $ ~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" --workspace 
> "/home/travis/.cache/install-jdk" --feature "8" --license "BCL"
> Ignoring license option: BCL -- using GPLv2+CE by default
> install-jdk.sh 2019-10-16
> Expected feature release number in range of 9 to 14, but got: 8
> The command "~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" 
> --workspace "/home/travis/.cache/install-jdk" --feature "8" --license "BCL"" 
> failed and exited with 3 during .
> Details on the following build log:
> https://travis-ci.org/apache/incubator-toree?utm_medium=notification_source=email



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-505) Travis CI builds are failing due to issues installing JDK8

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955608#comment-16955608
 ] 

ASF GitHub Bot commented on TOREE-505:
--

lresende commented on pull request #177: [TOREE-505] Fix Oracle JDK8 in Travis 
CI
URL: https://github.com/apache/incubator-toree/pull/177
 
 
   Install of Oracle JDK 8 Failing in Travis CI and as a result,
   build is failing for new pull requests.
   
   We just need to add `dist: trusty` in the .travis.yml file
   as mentioned in the issue below:
   https://travis-ci.community/t/install-of-oracle-jdk-8-failing/3038
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Travis CI builds are failing due to issues installing JDK8
> --
>
> Key: TOREE-505
> URL: https://issues.apache.org/jira/browse/TOREE-505
> Project: TOREE
>  Issue Type: Bug
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
>
> $ ~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" --workspace 
> "/home/travis/.cache/install-jdk" --feature "8" --license "BCL"
> Ignoring license option: BCL -- using GPLv2+CE by default
> install-jdk.sh 2019-10-16
> Expected feature release number in range of 9 to 14, but got: 8
> The command "~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" 
> --workspace "/home/travis/.cache/install-jdk" --feature "8" --license "BCL"" 
> failed and exited with 3 during .
> Details on the following build log:
> https://travis-ci.org/apache/incubator-toree?utm_medium=notification_source=email



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-505) Travis CI builds are failing due to issues installing JDK8

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955606#comment-16955606
 ] 

ASF GitHub Bot commented on TOREE-505:
--

lresende commented on pull request #176: [TOREE-505] Fix Oracle JDK8 in Travis 
CI
URL: https://github.com/apache/incubator-toree/pull/176
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Travis CI builds are failing due to issues installing JDK8
> --
>
> Key: TOREE-505
> URL: https://issues.apache.org/jira/browse/TOREE-505
> Project: TOREE
>  Issue Type: Bug
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
>
> $ ~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" --workspace 
> "/home/travis/.cache/install-jdk" --feature "8" --license "BCL"
> Ignoring license option: BCL -- using GPLv2+CE by default
> install-jdk.sh 2019-10-16
> Expected feature release number in range of 9 to 14, but got: 8
> The command "~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" 
> --workspace "/home/travis/.cache/install-jdk" --feature "8" --license "BCL"" 
> failed and exited with 3 during .
> Details on the following build log:
> https://travis-ci.org/apache/incubator-toree?utm_medium=notification_source=email



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-506) Enable TravisCI to build using Open JDK

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955605#comment-16955605
 ] 

ASF GitHub Bot commented on TOREE-506:
--

lresende commented on pull request #175: [TOREE-506] Enable Toree to build with 
openJDK8
URL: https://github.com/apache/incubator-toree/pull/175
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Enable TravisCI to build using Open JDK
> ---
>
> Key: TOREE-506
> URL: https://issues.apache.org/jira/browse/TOREE-506
> Project: TOREE
>  Issue Type: Task
>  Components: Build
>Reporter: Luciano Resende
>Priority: Minor
>
> Recently TravisCI build on JDK 8 started failing, and running the tests on 
> openJDK enables a safe net in case of JDK not being available on the user 
> env. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-504) Version missing from shutdown_reply header

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931927#comment-16931927
 ] 

ASF GitHub Bot commented on TOREE-504:
--

kevin-bates commented on pull request #172: [TOREE-504] Add protocol version to 
shutdown_reply header
URL: https://github.com/apache/incubator-toree/pull/172
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Version missing from shutdown_reply header
> --
>
> Key: TOREE-504
> URL: https://issues.apache.org/jira/browse/TOREE-504
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Kevin Bates
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.4.0
>
>
> The 5.0 Jupyter message protocol adds a {{version}} field to [message 
> headers|https://jupyter-client.readthedocs.io/en/latest/messaging.html#general-message-format].
>   However, shutdown_reply headers are not setting the protocol version.  As a 
> result, clients that adhere to the protocol can have issues when {{version}} 
> is present in the response header as an empty string, leading to noise and 
> possible incorrect behaviors following Toree's shutdown.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TOREE-492) Add DataFrame displayer

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920520#comment-16920520
 ] 

ASF GitHub Bot commented on TOREE-492:
--

mohammedi-haroune commented on issue #167: [TOREE-492] Add DataFrame displayer
URL: https://github.com/apache/incubator-toree/pull/167#issuecomment-526955835
 
 
   @manuzhang Thanks for you reactivity. I will check it out.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add DataFrame displayer
> ---
>
> Key: TOREE-492
> URL: https://issues.apache.org/jira/browse/TOREE-492
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Manu Zhang
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (TOREE-492) Add DataFrame displayer

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TOREE-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920349#comment-16920349
 ] 

ASF GitHub Bot commented on TOREE-492:
--

mohammedi-haroune commented on issue #167: [TOREE-492] Add DataFrame displayer
URL: https://github.com/apache/incubator-toree/pull/167#issuecomment-526901964
 
 
   @manuzhang It would be great if the displayer works by default (the same way 
as a pandas data frame). Is that working for you ?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add DataFrame displayer
> ---
>
> Key: TOREE-492
> URL: https://issues.apache.org/jira/browse/TOREE-492
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Manu Zhang
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (TOREE-502) Toree TravisCI fails installing OpenJDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866974#comment-16866974
 ] 

ASF GitHub Bot commented on TOREE-502:
--

lresende commented on pull request #171: [TOREE-502] Fix travis build failures
URL: https://github.com/apache/incubator-toree/pull/171
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Toree TravisCI fails installing OpenJDK
> ---
>
> Key: TOREE-502
> URL: https://issues.apache.org/jira/browse/TOREE-502
> Project: TOREE
>  Issue Type: Bug
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
>
> Here is the logs from Travis CI
> https://travis-ci.org/apache/incubator-toree/jobs/546650927
> Removing intermediate container e1ac74d99565
> Step 4/6 : RUN echo 'deb http://ftp.debian.org/debian jessie-backports main' 
> > /etc/apt/sources.list.d/jessie-backports.list && apt-get -y update &&   
>   apt-get install -y --no-install-recommends -t jessie-backports 
> openjdk-8-jre-headless && rm 
> /etc/apt/sources.list.d/jessie-backports.list && apt-get clean && rm 
> -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/* && 
> update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
>  ---> Running in a548a8ea42e3
> Ign http://ftp.debian.org jessie-backports InRelease
> Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
> Get:2 http://repos.mesosphere.io wheezy InRelease [5,279 B]
> Ign http://httpredir.debian.org jessie InRelease
> Get:3 http://httpredir.debian.org jessie-updates InRelease [7,340 B]
> Get:4 http://httpredir.debian.org jessie Release.gpg [2,420 B]
> Get:5 http://httpredir.debian.org jessie Release [148 kB]
> Ign http://ftp.debian.org jessie-backports Release.gpg
> Get:6 http://security.debian.org jessie/updates/main amd64 Packages [851 kB]
> Ign http://ftp.debian.org jessie-backports Release
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   
> Get:7 http://repos.mesosphere.io wheezy/main amd64 Packages [3,625 B]
> Get:8 http://httpredir.debian.org jessie/main amd64 Packages [9,098 kB]
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   404  Not Found
> W: Failed to fetch 
> http://httpredir.debian.org/debian/dists/jessie-updates/InRelease  Unable to 
> find expected entry 'main/binary-amd64/Packages' in Release file (Wrong 
> sources.list entry or malformed file)
> W: Failed to fetch 
> http://ftp.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages
>   404  Not Found
> E: Some index files failed to download. They have been ignored, or old ones 
> used instead.
> Fetched 10.2 MB in 1s (5,361 kB/s)
> The command '/bin/sh -c echo 'deb http://ftp.debian.org/debian 
> jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list && 
> apt-get -y update && apt-get install -y --no-install-recommends -t 
> jessie-backports openjdk-8-jre-headless && rm 
> /etc/apt/sources.list.d/jessie-backports.list && apt-get clean && rm 
> -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/* && 
> update-alternatives --set java 
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java' returned a non-zero code: 100
> make: *** [.system-test-image] Error 100
> The command "make all-travis" exited with 2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-502) Toree TravisCI fails installing OpenJDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866776#comment-16866776
 ] 

ASF GitHub Bot commented on TOREE-502:
--

lresende commented on pull request #171: [TOREE-502] Fix travis build failures
URL: https://github.com/apache/incubator-toree/pull/171
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Toree TravisCI fails installing OpenJDK
> ---
>
> Key: TOREE-502
> URL: https://issues.apache.org/jira/browse/TOREE-502
> Project: TOREE
>  Issue Type: Bug
>  Components: Build
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
>
> Here is the logs from Travis CI
> https://travis-ci.org/apache/incubator-toree/jobs/546650927
> Removing intermediate container e1ac74d99565
> Step 4/6 : RUN echo 'deb http://ftp.debian.org/debian jessie-backports main' 
> > /etc/apt/sources.list.d/jessie-backports.list && apt-get -y update &&   
>   apt-get install -y --no-install-recommends -t jessie-backports 
> openjdk-8-jre-headless && rm 
> /etc/apt/sources.list.d/jessie-backports.list && apt-get clean && rm 
> -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/* && 
> update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
>  ---> Running in a548a8ea42e3
> Ign http://ftp.debian.org jessie-backports InRelease
> Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
> Get:2 http://repos.mesosphere.io wheezy InRelease [5,279 B]
> Ign http://httpredir.debian.org jessie InRelease
> Get:3 http://httpredir.debian.org jessie-updates InRelease [7,340 B]
> Get:4 http://httpredir.debian.org jessie Release.gpg [2,420 B]
> Get:5 http://httpredir.debian.org jessie Release [148 kB]
> Ign http://ftp.debian.org jessie-backports Release.gpg
> Get:6 http://security.debian.org jessie/updates/main amd64 Packages [851 kB]
> Ign http://ftp.debian.org jessie-backports Release
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   
> Get:7 http://repos.mesosphere.io wheezy/main amd64 Packages [3,625 B]
> Get:8 http://httpredir.debian.org jessie/main amd64 Packages [9,098 kB]
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   
> Err http://ftp.debian.org jessie-backports/main amd64 Packages
>   404  Not Found
> W: Failed to fetch 
> http://httpredir.debian.org/debian/dists/jessie-updates/InRelease  Unable to 
> find expected entry 'main/binary-amd64/Packages' in Release file (Wrong 
> sources.list entry or malformed file)
> W: Failed to fetch 
> http://ftp.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages
>   404  Not Found
> E: Some index files failed to download. They have been ignored, or old ones 
> used instead.
> Fetched 10.2 MB in 1s (5,361 kB/s)
> The command '/bin/sh -c echo 'deb http://ftp.debian.org/debian 
> jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list && 
> apt-get -y update && apt-get install -y --no-install-recommends -t 
> jessie-backports openjdk-8-jre-headless && rm 
> /etc/apt/sources.list.d/jessie-backports.list && apt-get clean && rm 
> -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/* && 
> update-alternatives --set java 
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java' returned a non-zero code: 100
> make: *** [.system-test-image] Error 100
> The command "make all-travis" exited with 2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-487) Remove PySpark Interpreter support in Toree

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859871#comment-16859871
 ] 

ASF GitHub Bot commented on TOREE-487:
--

echarles commented on issue #166: [TOREE-487][TOREE-488] Remove PySpark and 
SparkR interpreters
URL: https://github.com/apache/incubator-toree/pull/166#issuecomment-500356895
 
 
   @lresende @kevin-bates 
   
   Seems like I have missed the [mailing list 
conversation](https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html)
 and the JIRA tickets (TOREE-487 and TOREE-488).
   
   Is there still a way to benefit from sparksession and variables/dataframes 
sharing between scala/python/R with Toree? Zeppelin and PixieDust provide this 
and I am interested having that feature.
   
   Also, you point in the mailing limitations and user confusion: I see 
reported issues on pyspark on the mailinglist, but do they come from Toree 
code/design or simply from config/env issues. In other words, what were those 
design limits?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove PySpark Interpreter support in Toree
> ---
>
> Key: TOREE-487
> URL: https://issues.apache.org/jira/browse/TOREE-487
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.3.0
>
>
> Detailed discussion on the mailing-list
> https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-495) Add deps-dir option to toree configuration options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787902#comment-16787902
 ] 

ASF GitHub Bot commented on TOREE-495:
--

tiki-sk commented on pull request #168: [TOREE-495] Add deps_dir option to 
toree_opts command line option
URL: https://github.com/apache/incubator-toree/pull/168
 
 
   The deps_dir option configure destination path of dependencies downloaded 
with AddDeps magic.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
>  Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with 
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then 
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-492) Add DataFrame displayer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696356#comment-16696356
 ] 

ASF GitHub Bot commented on TOREE-492:
--

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-toree/pull/167

[TOREE-492] Add DataFrame displayer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-toree df_displayer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/167.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #167


commit 6b527686f67ab2357a7d6cb682cc80a13ce7a64e
Author: manuzhang 
Date:   2018-10-25T03:03:30Z

[TOREE-492] Add DataFrame displayer




> Add DataFrame displayer
> ---
>
> Key: TOREE-492
> URL: https://issues.apache.org/jira/browse/TOREE-492
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Manu Zhang
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-487) Remove PySpark Interpreter support in Toree

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639859#comment-16639859
 ] 

ASF GitHub Bot commented on TOREE-487:
--

Github user lresende closed the pull request at:

https://github.com/apache/incubator-toree/pull/166


> Remove PySpark Interpreter support in Toree
> ---
>
> Key: TOREE-487
> URL: https://issues.apache.org/jira/browse/TOREE-487
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.3.0
>
>
> Detailed discussion on the mailing-list
> https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-487) Remove PySpark Interpreter support in Toree

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637691#comment-16637691
 ] 

ASF GitHub Bot commented on TOREE-487:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/166

[TOREE-487][TOREE-488] Remove PySpark and SparkR interpreters

Instead, please use a supported kernel such IPython or IRKernel

Closes #166

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree 
toree-487-488-clean-python-r

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/166.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #166


commit 1459403fc8ad5b4bb1e956513ef4190a0fc5b6f4
Author: Luciano Resende 
Date:   2018-10-03T19:26:14Z

[TOREE-487][TOREE-488] Remove PySpark and SparkR interpreters

Instead, please use a supported kernel such IPython or IRKernel

Closes #166




> Remove PySpark Interpreter support in Toree
> ---
>
> Key: TOREE-487
> URL: https://issues.apache.org/jira/browse/TOREE-487
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Reporter: Luciano Resende
>Priority: Major
>
> Detailed discussion on the mailing-list
> https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-482) Truncation magic does not work

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637454#comment-16637454
 ] 

ASF GitHub Bot commented on TOREE-482:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/160


> Truncation magic does not work
> --
>
> Key: TOREE-482
> URL: https://issues.apache.org/jira/browse/TOREE-482
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-486) Typo in %AddJar magic in Scala notebook ==> Strange classloader errors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635833#comment-16635833
 ] 

ASF GitHub Bot commented on TOREE-486:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/164


> Typo in %AddJar magic in Scala notebook ==> Strange classloader errors 
> ---
>
> Key: TOREE-486
> URL: https://issues.apache.org/jira/browse/TOREE-486
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.2.0
>Reporter: SHOBHIT SHUKLA
>Priority: Major
> Attachments: 5cd4964a-9b1d-11e8-95fe-ec9b604cd332.png
>
>
> If the user makes a typo while entering an %AddJar magic with an HTTP URL, 
> the Spark classloader ends up in a strange state that produces errors.
> Attachment have the error logs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-485) Not able to run Toree kernel in JupyterLab 0.34

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633927#comment-16633927
 ] 

ASF GitHub Bot commented on TOREE-485:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/165


> Not able to run Toree kernel in JupyterLab 0.34
> ---
>
> Key: TOREE-485
> URL: https://issues.apache.org/jira/browse/TOREE-485
> Project: TOREE
>  Issue Type: Bug
> Environment: Ubuntu 18.04, Python 3.6.6, JupyterLab v0.34.12, Apache 
> Toree 0.2.0-rc6
>Reporter: Sergey Kuznetsov
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.3.0
>
>
> Hello! I've been using Toree kernel in JupyterLab (Python 2.7) versions 0.32 
> and 0.33 for some time and it worked great. Recently I updated JupyterLab to 
> version 0.34 that supports only Python 3 and now I'm not ably to run Scala 
> code in Toree. The kernel starts fine, without any warnings, but once I 
> executed the cell with simple import I get the following error:
> {noformat}
> ERROR OneForOneStrategy:159 - 
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException: 
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
>   at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
>   at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
>   at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
>   at play.api.libs.json.JsError.fold(JsResult.scala:13)
>   at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
>   at play.api.libs.json.JsObject.as(JsValue.scala:166)
>   at 
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
>   at 
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
>   at 
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:495)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:224)
>   at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-09-26 14:45:29 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in 
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
>   at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
>   at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
>   at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
>   at 
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
>   at 
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
>   at 
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
>   at 
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
>   at java.lang.Thread.run(Thread.java:748){noformat}
> I tried to update Toree: install with pip3 the latest rc6 release, but the 
> issue remains. Is there any way to fix this?
>  
> Thanks,
> Sergey



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-485) Not able to run Toree kernel in JupyterLab 0.34

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633510#comment-16633510
 ] 

ASF GitHub Bot commented on TOREE-485:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/165

[TOREE-485] Properly parse updated JupyterLab cell metadata



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-485-jupyterlab

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #165


commit fe9c434cd0998b6654a677fc60fb5d66a32c8a3f
Author: Luciano Resende 
Date:   2018-09-30T20:36:34Z

[TOREE-485] Properly parse updated JupyterLab cell metadata




> Not able to run Toree kernel in JupyterLab 0.34
> ---
>
> Key: TOREE-485
> URL: https://issues.apache.org/jira/browse/TOREE-485
> Project: TOREE
>  Issue Type: Bug
> Environment: Ubuntu 18.04, Python 3.6.6, JupyterLab v0.34.12, Apache 
> Toree 0.2.0-rc6
>Reporter: Sergey Kuznetsov
>Priority: Major
>
> Hello! I've been using Toree kernel in JupyterLab (Python 2.7) versions 0.32 
> and 0.33 for some time and it worked great. Recently I updated JupyterLab to 
> version 0.34 that supports only Python 3 and now I'm not ably to run Scala 
> code in Toree. The kernel starts fine, without any warnings, but once I 
> executed the cell with simple import I get the following error:
> {noformat}
> ERROR OneForOneStrategy:159 - 
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException: 
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
>   at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
>   at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
>   at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
>   at play.api.libs.json.JsError.fold(JsResult.scala:13)
>   at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
>   at play.api.libs.json.JsObject.as(JsValue.scala:166)
>   at 
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
>   at 
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
>   at 
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:495)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:224)
>   at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-09-26 14:45:29 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in 
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
>   at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
>   at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
>   at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
>   at 
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
>   at 
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
>   at 
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
>   at 
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
>   at java.lang.Thread.run(Thread.java:748){noformat}
> I tried to update Toree: install with pip3 the latest rc6 release, but the 
> issue remains. Is there any way to fix this?
>  
> Thanks,
> Sergey



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-475) Sql statement can't resolve predefined variables in scala cells

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631950#comment-16631950
 ] 

ASF GitHub Bot commented on TOREE-475:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/155


> Sql statement can't resolve predefined variables in scala cells
> ---
>
> Key: TOREE-475
> URL: https://issues.apache.org/jira/browse/TOREE-475
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Reporter: Huafeng Wang
>Assignee: Huafeng Wang
>Priority: Major
> Fix For: 0.3.0
>
>
> Current the SQL cell does not support code like
> "
> {{%%sql}}
> {{select * from ${tableName} limit ${limitNum}}}
> "



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-486) Typo in %AddJar magic in Scala notebook ==> Strange classloader errors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630544#comment-16630544
 ] 

ASF GitHub Bot commented on TOREE-486:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/164

[TOREE-486] Raise exception when %AddJar called with invalid jar



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree 
toree-486-throw-exception-invalid-jar

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/164.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #164


commit a0f72820e2250889b6d6576ffa21bd23c8b25f4e
Author: Luciano Resende 
Date:   2018-09-25T02:29:26Z

[TOREE-486] Raise exception when %AddJar called with invalid jar




> Typo in %AddJar magic in Scala notebook ==> Strange classloader errors 
> ---
>
> Key: TOREE-486
> URL: https://issues.apache.org/jira/browse/TOREE-486
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.2.0
>Reporter: SHOBHIT SHUKLA
>Priority: Major
> Attachments: 5cd4964a-9b1d-11e8-95fe-ec9b604cd332.png
>
>
> If the user makes a typo while entering an %AddJar magic with an HTTP URL, 
> the Spark classloader ends up in a strange state that produces errors.
> Attachment have the error logs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-474) support for custom resolvers (for %AddDeps)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627606#comment-16627606
 ] 

ASF GitHub Bot commented on TOREE-474:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/162


> support for custom resolvers (for %AddDeps)
> ---
>
> Key: TOREE-474
> URL: https://issues.apache.org/jira/browse/TOREE-474
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Evgeniy Tokarev
>Priority: Major
>
> Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 
> which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality 
> anymore.
> Lot of libs using custom repositories like Bintray therefore seem like is a 
> big limitation of project 
> which doesn't allow to use custom repositories (Zeppelin and spark-notebooks 
> has that functionality) 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-483) Add %showOutput magic to disable console output

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626696#comment-16626696
 ] 

ASF GitHub Bot commented on TOREE-483:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/161


> Add %showOutput magic to disable console output
> ---
>
> Key: TOREE-483
> URL: https://issues.apache.org/jira/browse/TOREE-483
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-474) support for custom resolvers (for %AddDeps)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622551#comment-16622551
 ] 

ASF GitHub Bot commented on TOREE-474:
--

GitHub user rmcentyre opened a pull request:

https://github.com/apache/incubator-toree/pull/162

[TOREE-474] support for custom resolvers (for %AddDeps)

The code path where `downloadTotal` is set is inconsistently reached for 
certain repositories, like Artifactory. This should fix the masked issue 
(`java.util.NoSuchElementException: key not found` in `downloadTotal`) that was 
derailing the directory traversal process specified in TOREE-474. The visible 
error in the logs was the generic `An error occurred while downloading` 
specified in the Logger overrides.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmcentyre/incubator-toree patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/162.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #162


commit f3f5899111e8f100a4c75f3a128959975275f6ac
Author: Russell McEntyre 
Date:   2018-09-20T19:06:49Z

downloadTotal key error

the code path where downloadTotal is set is inconsistently reached for 
certain repositories, like Artifactory




> support for custom resolvers (for %AddDeps)
> ---
>
> Key: TOREE-474
> URL: https://issues.apache.org/jira/browse/TOREE-474
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Evgeniy Tokarev
>Priority: Major
>
> Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 
> which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality 
> anymore.
> Lot of libs using custom repositories like Bintray therefore seem like is a 
> big limitation of project 
> which doesn't allow to use custom repositories (Zeppelin and spark-notebooks 
> has that functionality) 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-467) Showtypes magic doesn't work

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581891#comment-16581891
 ] 

ASF GitHub Bot commented on TOREE-467:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/158


> Showtypes magic doesn't work
> 
>
> Key: TOREE-467
> URL: https://issues.apache.org/jira/browse/TOREE-467
> Project: TOREE
>  Issue Type: Bug
>Reporter: Maciej Bryński
>Priority: Major
> Fix For: 0.3.0
>
> Attachments: image-2018-02-21-12-02-56-348.png
>
>
> !image-2018-02-21-12-02-56-348.png!
>  
> I'm using Toree version 0.2.0-incubating-rc3.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-480) Deprecate nosparkcontext in favor of spark-context-initialization-mode none

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581883#comment-16581883
 ] 

ASF GitHub Bot commented on TOREE-480:
--

Github user lresende closed the pull request at:

https://github.com/apache/incubator-toree/pull/157


> Deprecate nosparkcontext in favor of spark-context-initialization-mode none
> ---
>
> Key: TOREE-480
> URL: https://issues.apache.org/jira/browse/TOREE-480
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
>
> We currently have spark-context-initialization-mode that supports eager and 
> lazy options. This ticket is to deprecate the nonsparkcontext in favor of 
> adding a 'none' option to spark-context-initialization-mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-483) Add %showOutput magic to disable console output

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577659#comment-16577659
 ] 

ASF GitHub Bot commented on TOREE-483:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/161

[TOREE-483] Add %showOutput magic to disable console output



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-483-showoutput

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #161


commit 89153725b4eb15ea69512471ffd5c4111b030292
Author: Luciano Resende 
Date:   2018-08-01T13:59:33Z

[TOREE-467] Fix ShowTypes magic functionality

There was a regression where the actual configuration for ShowTypes
and Truncation was not wired with the code that perform the output
format, thus making those magics to stop working.

commit e793d2a56c7f20a10b3d1ef8c078bef787ec4aa0
Author: Luciano Resende 
Date:   2018-08-12T16:27:06Z

[TOREE-482] Fix %Truncation magic

Refactored the configuration storing the truncation magic option
and wire it again with the runtime logic

commit 35c49baad63e7711d5f5a662fedd523495d41a35
Author: Luciano Resende 
Date:   2018-08-12T16:52:29Z

[TOREE-483] Add %showOutput magic to disable console output




> Add %showOutput magic to disable console output
> ---
>
> Key: TOREE-483
> URL: https://issues.apache.org/jira/browse/TOREE-483
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-482) Truncation magic does not work

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577658#comment-16577658
 ] 

ASF GitHub Bot commented on TOREE-482:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/160

[TOREE-482] Fix %Truncation magic

Refactored the configuration storing the truncation magic option and wire 
it again with the runtime logic

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-482-truncation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/160.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #160


commit 89153725b4eb15ea69512471ffd5c4111b030292
Author: Luciano Resende 
Date:   2018-08-01T13:59:33Z

[TOREE-467] Fix ShowTypes magic functionality

There was a regression where the actual configuration for ShowTypes
and Truncation was not wired with the code that perform the output
format, thus making those magics to stop working.

commit e793d2a56c7f20a10b3d1ef8c078bef787ec4aa0
Author: Luciano Resende 
Date:   2018-08-12T16:27:06Z

[TOREE-482] Fix %Truncation magic

Refactored the configuration storing the truncation magic option
and wire it again with the runtime logic




> Truncation magic does not work
> --
>
> Key: TOREE-482
> URL: https://issues.apache.org/jira/browse/TOREE-482
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-467) Showtypes magic doesn't work

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567471#comment-16567471
 ] 

ASF GitHub Bot commented on TOREE-467:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/158

[TOREE-467] Fix ShowTypes magic functionality

There was a regression where the actual configuration for ShowTypes
and Truncation was not wired with the code that perform the output
format, thus making those magics to stop working.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-467-showtypes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/158.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #158


commit 72ce68ae338a42a9645940621ac9a80237e5afd7
Author: Luciano Resende 
Date:   2018-08-01T13:59:33Z

[TOREE-467] Fix ShowTypes magic functioanlity

There was a regression where the actual configuration for ShowTypes
and Truncation was not wired with the code that perform the output
format, thus making those magics to stop working.




> Showtypes magic doesn't work
> 
>
> Key: TOREE-467
> URL: https://issues.apache.org/jira/browse/TOREE-467
> Project: TOREE
>  Issue Type: Bug
>Reporter: Maciej Bryński
>Priority: Major
> Attachments: image-2018-02-21-12-02-56-348.png
>
>
> !image-2018-02-21-12-02-56-348.png!
>  
> I'm using Toree version 0.2.0-incubating-rc3.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-475) Sql statement can't resolve predefined variables in scala cells

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488594#comment-16488594
 ] 

ASF GitHub Bot commented on TOREE-475:
--

GitHub user huafengw opened a pull request:

https://github.com/apache/incubator-toree/pull/155

[TOREE-475] Resolve variables in sql statement



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huafengw/incubator-toree sql

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/155.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #155


commit 64b029f70cf30b8ab6d5eb4640ef2c25b53e2c89
Author: huafengw 
Date:   2018-05-24T07:50:35Z

[TOREE-475] Resolve variables in sql statement




> Sql statement can't resolve predefined variables in scala cells
> ---
>
> Key: TOREE-475
> URL: https://issues.apache.org/jira/browse/TOREE-475
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Reporter: Huafeng Wang
>Priority: Major
>
> Current the SQL cell does not support code like
> "
> {{%%sql}}
> {{select * from ${tableName} limit ${limitNum}}}
> "



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-470) Add option to disable SparkContext lazy initialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446979#comment-16446979
 ] 

ASF GitHub Bot commented on TOREE-470:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/154


> Add option to disable SparkContext lazy initialization
> --
>
> Key: TOREE-470
> URL: https://issues.apache.org/jira/browse/TOREE-470
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
>
> We are looking into a scenario where we need to disable the lazy 
> initialization of SparkContext and a startup parameter or some sort of 
> configuration might be helpful on this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-470) Add option to disable SparkContext lazy initialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445250#comment-16445250
 ] 

ASF GitHub Bot commented on TOREE-470:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/154

[TOREE-470] Config option do control SparkContext initialization



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree 
toree-470-config-lazy-context-init

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/154.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #154


commit 5f4d77aa434db464ee630befe4342c65af6b0149
Author: Luciano Resende 
Date:   2018-04-19T19:23:58Z

[TOREE-471] Fix typo in command line configuration option

Fix typo in spark_context_intialization_timeout config option
spark_context_intialization_timeout => spark_context_initialization_timeout

commit 5e835a518b86b5663aaef616c24bccf996194835
Author: Luciano Resende 
Date:   2018-04-20T03:12:59Z

[TOREE-470] Config option do control SparkContext initialization

--spark-context-initialization-mode eager (disable lazy initialization)
--spark-context-initialization-mode lazy (default, enable lazy 
initialization)




> Add option to disable SparkContext lazy initialization
> --
>
> Key: TOREE-470
> URL: https://issues.apache.org/jira/browse/TOREE-470
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
>
> We are looking into a scenario where we need to disable the lazy 
> initialization of SparkContext and a startup parameter or some sort of 
> configuration might be helpful on this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-466) Function taking a function as parameter, Kernel Crashes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386417#comment-16386417
 ] 

ASF GitHub Bot commented on TOREE-466:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/153

[TOREE-466] Properly recognize higher order functions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree higher-order-function

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/153.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #153


commit b8872a4cb29de61f145255d8c0f4e44f485b57f6
Author: Luciano Resende 
Date:   2018-03-05T17:42:23Z

[TOREE-466] Properly recognize higher order functions




> Function taking a function as parameter, Kernel Crashes
> ---
>
> Key: TOREE-466
> URL: https://issues.apache.org/jira/browse/TOREE-466
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.2.0
> Environment: jre-8u162, scala-2.12.4, spark-2.2.1-bin-hadoop2.7, 
> 0.2.0-incubating-rc3 in a Ubuntu 17.10 based Docker Container
>Reporter: Hans
>Assignee: Luciano Resende
>Priority: Major
>
> Dear Toree Developers
> The following simple code (in a Jupyter notebook cell) crashes the kernel:
>  
>  
> {code:java}
> def f(x: Int, l: (Int) => Int): Int = { l(x) }
> f(2, x => x*x)
> {code}
>  
> This works in the Scala REPL.
> With best wishes & thanks for your great efforts on Toree



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-463) Startup performance: initialize ScalaInterpreter earlier and in its own thread

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356156#comment-16356156
 ] 

ASF GitHub Bot commented on TOREE-463:
--

Github user sanjay-saxena commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/151#discussion_r166772771
  
--- Diff: 
kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala 
---
@@ -28,6 +28,7 @@ import org.apache.toree.comm.{CommManager, CommRegistrar, 
CommStorage, KernelCom
 import org.apache.toree.dependencies.{CoursierDependencyDownloader, 
Credentials, DependencyDownloader}
 import org.apache.toree.interpreter._
 import org.apache.toree.kernel.api.Kernel
+import org.apache.toree.kernel.interpreter.scala.ScalaInterpreter
--- End diff --

Based on my limited understanding of the code, picking up a dependency on 
an implementation such as `ScalaInterpreter` from `scala-interpreter` project 
in the generic `kernel` project maybe an abstraction violation. It may also 
lead to circular dependencies. Maybe, you should look at either reusing the 
existing APIs on the generic trait `Interpreter`. And, if the existing APIs are 
not sufficient, then maybe add new APIs to the generic trait `Interpreter`.


> Startup performance: initialize ScalaInterpreter earlier and in its own thread
> --
>
> Key: TOREE-463
> URL: https://issues.apache.org/jira/browse/TOREE-463
> Project: TOREE
>  Issue Type: Bug
>Reporter: Brad
>Priority: Major
>
> The Scala Interpreter is the slowest thing to start up, have it start earlier 
> and run concurrently with the rest of Toree



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-463) Startup performance: initialize ScalaInterpreter earlier and in its own thread

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338245#comment-16338245
 ] 

ASF GitHub Bot commented on TOREE-463:
--

GitHub user brad-kaiser opened a pull request:

https://github.com/apache/incubator-toree/pull/151

[TOREE-463] break out scala interpreter initialization to speed up startup

The goal of this PR is to decrease Toree's startup time by breaking out the 
ScalaInterpreter initialization and running it in its own thread. If you count 
the startup time as the time it takes KernelBootstrap.initizalize to complete, 
I saw the average startup time go from 8.16s to 5.69s on my laptop. I started 
the kernel ten times with the original code and my changes.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brad-kaiser/incubator-toree fast-start

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/151.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #151


commit 725cc1983c008531b293ef7249ece1a1b57330e2
Author: Brad Kaiser 
Date:   2017-12-19T19:14:30Z

[TOREE-463] break out scala interpreter initialization to speed up startup.




> Startup performance: initialize ScalaInterpreter earlier and in its own thread
> --
>
> Key: TOREE-463
> URL: https://issues.apache.org/jira/browse/TOREE-463
> Project: TOREE
>  Issue Type: Bug
>Reporter: Brad
>Priority: Major
>
> The Scala Interpreter is the slowest thing to start up, have it start earlier 
> and run concurrently with the rest of Toree



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325726#comment-16325726
 ] 

ASF GitHub Bot commented on TOREE-420:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/126


> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)',
>
> 'scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)',
>
> 'scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)',
>
> 'scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)',
>

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325727#comment-16325727
 ] 

ASF GitHub Bot commented on TOREE-420:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/146


> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)',
>
> 'scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)',
>
> 'scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)',
>
> 'scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)',
>

[jira] [Commented] (TOREE-456) Convert env variable TOREE_ALTERNATE_SIGINT to command-line option

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324366#comment-16324366
 ] 

ASF GitHub Bot commented on TOREE-456:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree-website/pull/7


> Convert env variable TOREE_ALTERNATE_SIGINT to command-line option
> --
>
> Key: TOREE-456
> URL: https://issues.apache.org/jira/browse/TOREE-456
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Kevin Bates
>Assignee: Sanjay Saxena
>Priority: Minor
> Fix For: 0.2.0
>
>
> The recently added support (TOREE-437) for an alternate interrupt signal was 
> implemented using the environment variable {{TOREE_ALTERNATE_SIGINT}}.  
> However, we've recently learned that support for {{SPARK_YARN_USER_ENV}} is 
> being removed in Spark 2.2 and its via that variable that 
> {{TOREE_ALTERNATE_SIGINT}} is conveyed to the Toree kernel when launched as a 
> Yarn cluster application (_cluster mode_).
> As a result, and based on similar discussion on TOREE-443, it would be better 
> to convey the need for an alternate interrupt signal via a command line 
> option.  This way, all launch modes of Toree could set this option in the 
> same manner - via {{___TOREE_OPTS___}} in {{kernel.json}}, for example.
> The command-line option should be something like:
> {{--alternate-sigint=}}
> with a description of something like:
> {{Specifies the signal to use instead of SIGINT for interrupting a 
> long-running cell.  The value is a string and does not include the SIG 
> prefix.  Use of USR2 is recommended.}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-456) Convert env variable TOREE_ALTERNATE_SIGINT to command-line option

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323271#comment-16323271
 ] 

ASF GitHub Bot commented on TOREE-456:
--

GitHub user sanjay-saxena opened a pull request:

https://github.com/apache/incubator-toree-website/pull/7

Added --alternate-sigint command-line option

Updated the doc to include information about the
newly added --alternate-sigint command-line option.

Addresses TOREE-456.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sanjay-saxena/incubator-toree-website 
alternate-sigint-option

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree-website/pull/7.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #7


commit 1348e5f1c5d33d24a7ecd2f1a58b9cc6275c02df
Author: Sanjay Saxena 
Date:   2018-01-12T00:07:35Z

Added --alternate-sigint command-line option

Updated the doc to include information about the
newly added --alternate-sigint command-line option.




> Convert env variable TOREE_ALTERNATE_SIGINT to command-line option
> --
>
> Key: TOREE-456
> URL: https://issues.apache.org/jira/browse/TOREE-456
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Kevin Bates
>Assignee: Sanjay Saxena
>Priority: Minor
>
> The recently added support (TOREE-437) for an alternate interrupt signal was 
> implemented using the environment variable {{TOREE_ALTERNATE_SIGINT}}.  
> However, we've recently learned that support for {{SPARK_YARN_USER_ENV}} is 
> being removed in Spark 2.2 and its via that variable that 
> {{TOREE_ALTERNATE_SIGINT}} is conveyed to the Toree kernel when launched as a 
> Yarn cluster application (_cluster mode_).
> As a result, and based on similar discussion on TOREE-443, it would be better 
> to convey the need for an alternate interrupt signal via a command line 
> option.  This way, all launch modes of Toree could set this option in the 
> same manner - via {{___TOREE_OPTS___}} in {{kernel.json}}, for example.
> The command-line option should be something like:
> {{--alternate-sigint=}}
> with a description of something like:
> {{Specifies the signal to use instead of SIGINT for interrupting a 
> long-running cell.  The value is a string and does not include the SIG 
> prefix.  Use of USR2 is recommended.}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-456) Convert env variable TOREE_ALTERNATE_SIGINT to command-line option

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323264#comment-16323264
 ] 

ASF GitHub Bot commented on TOREE-456:
--

GitHub user sanjay-saxena opened a pull request:

https://github.com/apache/incubator-toree/pull/150

Support for --alternate-sigint command-line option

Added support for --alternate-sigint command-line option and
removed the usage of TOREE_ALTERNATE_SIGINT environment variable.
Changed the signature of HookInitialization.initializeHooks() to
pass in the Config. Added unit tests for the new command-line.
option.

Addresses issue TOREE-456.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sanjay-saxena/incubator-toree 
alternate-sigint-option

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/150.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #150


commit 54aa686168a6cfc34475a0a5eb6bf41a254ba477
Author: Sanjay Saxena 
Date:   2018-01-12T00:05:49Z

Support for --alternate-sigint command-line option

Added support for --alternate-sigint command-line option and
removed the usage of TOREE_ALTERNATE_SIGINT environment variable.
Changed the signature of HookInitialization.initializeHooks() to
pass in the Config. Added unit tests for the new command-line.
option.

Addresses issue TOREE-456.




> Convert env variable TOREE_ALTERNATE_SIGINT to command-line option
> --
>
> Key: TOREE-456
> URL: https://issues.apache.org/jira/browse/TOREE-456
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Kevin Bates
>Assignee: Sanjay Saxena
>Priority: Minor
>
> The recently added support (TOREE-437) for an alternate interrupt signal was 
> implemented using the environment variable {{TOREE_ALTERNATE_SIGINT}}.  
> However, we've recently learned that support for {{SPARK_YARN_USER_ENV}} is 
> being removed in Spark 2.2 and its via that variable that 
> {{TOREE_ALTERNATE_SIGINT}} is conveyed to the Toree kernel when launched as a 
> Yarn cluster application (_cluster mode_).
> As a result, and based on similar discussion on TOREE-443, it would be better 
> to convey the need for an alternate interrupt signal via a command line 
> option.  This way, all launch modes of Toree could set this option in the 
> same manner - via {{___TOREE_OPTS___}} in {{kernel.json}}, for example.
> The command-line option should be something like:
> {{--alternate-sigint=}}
> with a description of something like:
> {{Specifies the signal to use instead of SIGINT for interrupting a 
> long-running cell.  The value is a string and does not include the SIG 
> prefix.  Use of USR2 is recommended.}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-443) Make delayed kernel execution timeout configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308963#comment-16308963
 ] 

ASF GitHub Bot commented on TOREE-443:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/147


> Make delayed kernel execution timeout configurable
> --
>
> Key: TOREE-443
> URL: https://issues.apache.org/jira/browse/TOREE-443
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Reporter: Luciano Resende
>Assignee: Sanjay Saxena
>
> In the *kernel* sub-project's {{org.apache.toree.kernel.api.Kernel.scala}}, 
> when we do the spark context delayed initialization, there is hardcoded wait 
> of 100 ms. This usually works ok, but in slow machines or busy nodes it's not 
> enough and cause confusion to users as it's going to try to initialize the 
> context and display the message on the notebook ui twice.
> We should enable users to configure the timeout via an environment variable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-443) Make delayed kernel execution timeout configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259859#comment-16259859
 ] 

ASF GitHub Bot commented on TOREE-443:
--

GitHub user sanjay-saxena opened a pull request:

https://github.com/apache/incubator-toree-website/pull/6

Updated doc for --spark-context-initialization-timeout option

To address TOREE-443, a new command-line option,
--spark-context-initialization-timeout, has been introduced.
Cleaned up the list of command-line options specified in
site/docs/current/user/installation.md and also added information
about the newly added command-line option.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sanjay-saxena/incubator-toree-website 
configurable-kernel-execution-timeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree-website/pull/6.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6


commit d035963e585f558900968c2dbe7e4573072a0af7
Author: Sanjay Saxena 
Date:   2017-11-20T21:15:25Z

Updated doc for --spark-context-initialization-timeout option

To address TOREE-443, a new command-line option,
--spark-context-initialization-timeout, has been introduced.
Cleaned up the list of command-line options specified in
site/docs/current/user/installation.md and also added information
about the newly added command-line option.




> Make delayed kernel execution timeout configurable
> --
>
> Key: TOREE-443
> URL: https://issues.apache.org/jira/browse/TOREE-443
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Reporter: Luciano Resende
>Assignee: Sanjay Saxena
>
> In the *kernel* sub-project's {{org.apache.toree.kernel.api.Kernel.scala}}, 
> when we do the spark context delayed initialization, there is hardcoded wait 
> of 100 ms. This usually works ok, but in slow machines or busy nodes it's not 
> enough and cause confusion to users as it's going to try to initialize the 
> context and display the message on the notebook ui twice.
> We should enable users to configure the timeout via an environment variable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-443) Make delayed kernel execution timeout configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259544#comment-16259544
 ] 

ASF GitHub Bot commented on TOREE-443:
--

GitHub user sanjay-saxena opened a pull request:

https://github.com/apache/incubator-toree/pull/147

Support for --spark-context-initialization-timeout command-line option

This change addresses TOREE-443. Instead of hardcoding the
timeout to 100ms, --spark-context-initialization-timeout
command-line option has been introduced to make it configurable.
On slow/busy nodes, an appropriate timeout can be specified
to avoid the confusing messages in the notebook UI. Added
unit tests to CommandLineOptionsSpec.scala.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sanjay-saxena/incubator-toree 
configurable-kernel-execution-timeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/147.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #147


commit 0cb32a98fae08a8eae3575c1bc6f9e080bc59aca
Author: Sanjay Saxena 
Date:   2017-11-15T21:52:30Z

Support for --spark-context-initialization-timeout command-line option

This change addresses TOREE-443. Instead of hardcoding the
timeout to 100ms, --spark-context-initialization-timeout
command-line option has been introduced to make it configurable.
On slow/busy nodes, an appropriate timeout can be specified
to avoid the confusing messages in the notebook UI. Added
unit tests to CommandLineOptionsSpec.scala.




> Make delayed kernel execution timeout configurable
> --
>
> Key: TOREE-443
> URL: https://issues.apache.org/jira/browse/TOREE-443
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Reporter: Luciano Resende
>Assignee: Sanjay Saxena
>
> In the *kernel* sub-project's {{org.apache.toree.kernel.api.Kernel.scala}}, 
> when we do the spark context delayed initialization, there is hardcoded wait 
> of 100 ms. This usually works ok, but in slow machines or busy nodes it's not 
> enough and cause confusion to users as it's going to try to initialize the 
> context and display the message on the notebook ui twice.
> We should enable users to configure the timeout via an environment variable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253448#comment-16253448
 ] 

ASF GitHub Bot commented on TOREE-420:
--

GitHub user kervel opened a pull request:

https://github.com/apache/incubator-toree/pull/146

Add --exclude option to AddDeps and exclude org.scala-lang.modules by 
default

This PR includes the (in my eyes valid) fix to TOREE-420 by Kalvin Chau, 
and extends on it by adding a way to work around other dependency conflicts by 
excluding specific dependencies. This is already possible in other scala-based 
interactive notebooks.

For example, look at 
https://zeppelin.apache.org/docs/0.6.2/manual/dependencymanagement.html



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Kapernikov/incubator-toree depExcludes_rebased

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/146.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #146


commit db48bf33e867c05b4bb560a3f787153d86c1c393
Author: Kalvin Chau 
Date:   2017-06-22T20:26:41Z

updated depdency downloaders to exclude org.scala-lang.modules to prevent 
scala refelection issues TOREE-420

commit 2094e0187543ee111c7ee332d95d613b53a568ef
Author: Frank Dekervel 
Date:   2017-09-11T07:58:04Z

Add --exclude option to AddDeps in order to be able to exclude certain 
modules

The exclude syntax is follows "organisation:name" or "organisation" when 
everything from a certain organisation is to be excluded




> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
> 

[jira] [Commented] (TOREE-97) How to support Spark on Yarn model?

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164950#comment-16164950
 ] 

ASF GitHub Bot commented on TOREE-97:
-

Github user ribamar-santarosa commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/141#discussion_r138678529
  
--- Diff: etc/pip_install/toree/toreeapp.py ---
@@ -57,6 +59,12 @@ class ToreeInstall(InstallKernelSpec):
 spark_home = Unicode(os.getenv(SPARK_HOME, '/usr/local/spark'), 
config=True,
--- End diff --

If they're set to be empty, it's just like they're not set -- so it's true 
that probably they're better empty. 


> How to support Spark on Yarn model?
> ---
>
> Key: TOREE-97
> URL: https://issues.apache.org/jira/browse/TOREE-97
> Project: TOREE
>  Issue Type: Bug
>Reporter: Anne623
>
> Hi, All 
>   Now I test spark-kernel in IPython3.0 released and Spark On Yarn model. 
>  kernel.json like as below
> {code}
> {
> "display_name": "SparkOnYarn",
> "language": "scala",
> "argv": [
>   "/root/local/bin/sparkkernel",
>"--master",
>"yarn-client",
> "--profile",
> "{connection_file}"
>  ],
>  "codemirror_mode": "scala"
> }
> {code}
> while kernel can not be started.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-97) How to support Spark on Yarn model?

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164948#comment-16164948
 ] 

ASF GitHub Bot commented on TOREE-97:
-

Github user ribamar-santarosa commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/141#discussion_r138678195
  
--- Diff: etc/pip_install/toree/toreeapp.py ---
@@ -57,6 +59,12 @@ class ToreeInstall(InstallKernelSpec):
 spark_home = Unicode(os.getenv(SPARK_HOME, '/usr/local/spark'), 
config=True,
 help='''Specify where the spark files can be found.'''
 )
+hadoop_conf_dir = Unicode(os.getenv(HADOOP_CONF_DIR, 
'/usr/local/hadoop'), config=True,
+help='''Specify where the hadoop config files can be found.'''
+)
+spark_conf_dir = Unicode(os.getenv(SPARK_CONF_DIR, 
'/usr/local/spark'), config=True,
+help='''Specify where the spark config files can be found.'''
--- End diff --

sure, it has to be the place where `spark-env.sh` is found.  


> How to support Spark on Yarn model?
> ---
>
> Key: TOREE-97
> URL: https://issues.apache.org/jira/browse/TOREE-97
> Project: TOREE
>  Issue Type: Bug
>Reporter: Anne623
>
> Hi, All 
>   Now I test spark-kernel in IPython3.0 released and Spark On Yarn model. 
>  kernel.json like as below
> {code}
> {
> "display_name": "SparkOnYarn",
> "language": "scala",
> "argv": [
>   "/root/local/bin/sparkkernel",
>"--master",
>"yarn-client",
> "--profile",
> "{connection_file}"
>  ],
>  "codemirror_mode": "scala"
> }
> {code}
> while kernel can not be started.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-437) Cell interrupts do not occur in background

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164731#comment-16164731
 ] 

ASF GitHub Bot commented on TOREE-437:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/140


> Cell interrupts do not occur in background
> --
>
> Key: TOREE-437
> URL: https://issues.apache.org/jira/browse/TOREE-437
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Kevin Bates
>Assignee: Kevin Bates
>  Labels: usability
>
> Whenever Toree is run in the background - either directly from the shell or 
> indirectly from the Jupyter stack that is started in the background - cell 
> interrupts (via ctrl-C or SIGINT from parent) are not received - resulting in 
> the inability to interrupt long-running cells.
> This can be most simply demonstrated by invoking run.sh into the background 
> (e.g., run.sh &) then issue ctrl-C (or `kill -2 `) to no avail.
> This is related to TOREE-33 but only pertains to cell interrupt functionality 
> since complete life-cycle management is assumed by the parent for the 
> shutdown (double ctrl-C) scenario.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-437) Cell interrupts do not occur in background

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161510#comment-16161510
 ] 

ASF GitHub Bot commented on TOREE-437:
--

GitHub user kevin-bates opened a pull request:

https://github.com/apache/incubator-toree/pull/140

[TOREE-437] Establish alternate interrupt handler for background execution

Cell interrupts do not take place if Toree is running in the background 
(typically due to its parent being in the background).  This PR adds support 
for the caller of Toree to specify an alternate interrupt signal via the 
environment variable `TOREE_ALTERNATE_SIGINT` - the value of which should be 
the string indicating the alternate signal name's suffix portion (e.g., "USR2" 
for `SIGUSR2`).
The common scenario would be to specify this env value in the kernel.json's 
`env:` stanza when necessary ...
```json
  "display_name": "Apache Toree - Scala",
  "env": {
"DEFAULT_INTERPRETER": "Scala",
"PYTHON_EXEC": "python",
"__TOREE_SPARK_OPTS__": "",
"__TOREE_OPTS__": "",
"TOREE_ALTERNATE_SIGINT": "USR2",
```

If the value of `TOREE_ALTERNATE_SIGINT` is not recognized as a valid 
signal name, a warning message will be logged, but the kernel's launch will 
proceed:
```
17/09/11 08:44:37 [WARN] o.a.t.Main$$anon$1 - Error occurred establishing 
alternate signal handler.  Value of TOREE_ALTERNATE_SIGINT is probably bad: 
FOO.  Error: Unknown signal: FOO
```

Fixes TOREE-437

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kevin-bates/incubator-toree 
alt-interrupt-signal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/140.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #140


commit be78e5474b0b97a87a88ba35098af3aeb381edf4
Author: Kevin Bates 
Date:   2017-09-11T16:09:53Z

[TOREE-437] Establish alternate interrupt handler for background execution




> Cell interrupts do not occur in background
> --
>
> Key: TOREE-437
> URL: https://issues.apache.org/jira/browse/TOREE-437
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Kevin Bates
>Assignee: Kevin Bates
>  Labels: usability
>
> Whenever Toree is run in the background - either directly from the shell or 
> indirectly from the Jupyter stack that is started in the background - cell 
> interrupts (via ctrl-C or SIGINT from parent) are not received - resulting in 
> the inability to interrupt long-running cells.
> This can be most simply demonstrated by invoking run.sh into the background 
> (e.g., run.sh &) then issue ctrl-C (or `kill -2 `) to no avail.
> This is related to TOREE-33 but only pertains to cell interrupt functionality 
> since complete life-cycle management is assumed by the parent for the 
> shutdown (double ctrl-C) scenario.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-396) SparkR interpeter does not connect to Spark 2.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156291#comment-16156291
 ] 

ASF GitHub Bot commented on TOREE-396:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/138


> SparkR interpeter does not connect to Spark 2.1
> ---
>
> Key: TOREE-396
> URL: https://issues.apache.org/jira/browse/TOREE-396
> Project: TOREE
>  Issue Type: Bug
>Reporter: Mario Briggs
>Assignee: Michele Lamarca
> Fix For: 0.2.0
>
>
> I was trying the SparkR interpreter and trying to connect to a Spark 2.1 
> cluster. I get the following failure when execute any cell
> "Failed to connect JVM: Error in socketConnection(host = hostname, port = 
> port, server = FALSE, : argument "timeout" is missing, with no default
> Calls: sparkR.connect ... tryCatch -> tryCatchList -> tryCatchOne -> 
> 
> "
> Looks like in Spark2.1, the 
> [timeout|https://github.com/apache/spark/blob/branch-2.1/R/pkg/R/client.R#L22]
>  param to sparkRConnect no longer has a default value. 
> So this needs to fixed in 
> [here|https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/resources/kernelR/sparkr_runner.R#L65]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-396) SparkR interpeter does not connect to Spark 2.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152734#comment-16152734
 ] 

ASF GitHub Bot commented on TOREE-396:
--

GitHub user lammic opened a pull request:

https://github.com/apache/incubator-toree/pull/138

Fix for TOREE-396

Added timeout parameter to call to `connectBackend`. Tested with Spark 
2.2.0.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Bright-Computing/incubator-toree master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/138.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #138


commit cc2e6ed25eb66653adb4bcecc147e0965fd6429e
Author: Michele Lamarca 
Date:   2017-09-04T15:38:28Z

fix for TOREE-396




> SparkR interpeter does not connect to Spark 2.1
> ---
>
> Key: TOREE-396
> URL: https://issues.apache.org/jira/browse/TOREE-396
> Project: TOREE
>  Issue Type: Bug
>Reporter: Mario Briggs
>
> I was trying the SparkR interpreter and trying to connect to a Spark 2.1 
> cluster. I get the following failure when execute any cell
> "Failed to connect JVM: Error in socketConnection(host = hostname, port = 
> port, server = FALSE, : argument "timeout" is missing, with no default
> Calls: sparkR.connect ... tryCatch -> tryCatchList -> tryCatchOne -> 
> 
> "
> Looks like in Spark2.1, the 
> [timeout|https://github.com/apache/spark/blob/branch-2.1/R/pkg/R/client.R#L22]
>  param to sparkRConnect no longer has a default value. 
> So this needs to fixed in 
> [here|https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/resources/kernelR/sparkr_runner.R#L65]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151248#comment-16151248
 ] 

ASF GitHub Bot commented on TOREE-420:
--

GitHub user kalvinnchau reopened a pull request:

https://github.com/apache/incubator-toree/pull/126

TOREE-420 Exclude org.scala-lang.modules from AddDeps

https://issues.apache.org/jira/browse/TOREE-420

Include `org.scala-lang.modules` as part of the always exclude list when 
using the `%AddDeps` magic.

This prevents the `scala.reflect.internal.FatalError` that crashes the 
kernel, when a new scala-lang module is included and reloaded.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kalvinnchau/incubator-toree TOREE-420

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #126


commit 6c6c7262ceffc89a251cfc00565933eabc01ec81
Author: Kalvin Chau 
Date:   2017-06-22T20:26:41Z

updated depdency downloaders to exclude org.scala-lang.modules to prevent 
scala refelection issues TOREE-420




> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151247#comment-16151247
 ] 

ASF GitHub Bot commented on TOREE-420:
--

Github user kalvinnchau closed the pull request at:

https://github.com/apache/incubator-toree/pull/126


> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)',
>
> 'scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)',
>
> 'scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)',
>
> 'scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)',
>

[jira] [Commented] (TOREE-324) java.lang.NoSuchMethodError: joptsimple.OptionParser.acceptsAll

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137492#comment-16137492
 ] 

ASF GitHub Bot commented on TOREE-324:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/133


> java.lang.NoSuchMethodError: joptsimple.OptionParser.acceptsAll
> ---
>
> Key: TOREE-324
> URL: https://issues.apache.org/jira/browse/TOREE-324
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
> Environment: RHEL 7.2
> CDH 5.7 Spark Gateway Role (Spark 1.6.0)
> Jupyter 4.1.0
>Reporter: William Lau
>Assignee: Aldrin Leal
>
> Encountered the following error when starting the Toree kernel in the Jupyter 
> notebook:
> {code}
> Starting Spark Kernel with SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
> [I 02:12:40.663 NotebookApp] Kernel started: 
> 595279c3-d5b1-48aa-8a24-5bfeb27bc181
> WARNING: User-defined SPARK_HOME 
> (/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark) overrides 
> detected (/opt/cloudera/parcels/CDH/lib/spark).
> WARNING: Running spark-class from user-defined location.
> Exception in thread "main" java.lang.NoSuchMethodError: 
> joptsimple.OptionParser.acceptsAll(Ljava/util/Collection;Ljava/lang/String;)Ljoptsimple/OptionSpecBuilder;
>   at 
> org.apache.toree.boot.CommandLineOptions.(CommandLineOptions.scala:37)
>   at org.apache.toree.Main$delayedInit$body.apply(Main.scala:25)
>   at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>   at 
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>   at scala.App$$anonfun$main$1.apply(App.scala:71)
>   at scala.App$$anonfun$main$1.apply(App.scala:71)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>   at scala.App$class.main(App.scala:71)
>   at org.apache.toree.Main$.main(Main.scala:24)
>   at org.apache.toree.Main.main(Main.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>   at 
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {code}
> I used the documented command to install the Toree kernel:
> {code}
> pip install --pre toree
> jupyter toree install
> {code}
> Before this I was trying a self-compiled build from the spark-kernel project 
> and I encountered the same exception when starting the kernel in the notebook.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-324) java.lang.NoSuchMethodError: joptsimple.OptionParser.acceptsAll

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123757#comment-16123757
 ] 

ASF GitHub Bot commented on TOREE-324:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/133

[TOREE-324] Upgrade jopt-simple dependency to solve NoSuchMethodError



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-324

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/133.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #133


commit cdb4fb7d3ec3d3ed7d477f65deb3aa94debcab58
Author: Luciano Resende 
Date:   2017-08-11T17:47:16Z

[TOREE-324] Upgrade jopt-simple dependency to solve NoSuchMethodError




> java.lang.NoSuchMethodError: joptsimple.OptionParser.acceptsAll
> ---
>
> Key: TOREE-324
> URL: https://issues.apache.org/jira/browse/TOREE-324
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
> Environment: RHEL 7.2
> CDH 5.7 Spark Gateway Role (Spark 1.6.0)
> Jupyter 4.1.0
>Reporter: William Lau
>
> Encountered the following error when starting the Toree kernel in the Jupyter 
> notebook:
> {code}
> Starting Spark Kernel with SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
> [I 02:12:40.663 NotebookApp] Kernel started: 
> 595279c3-d5b1-48aa-8a24-5bfeb27bc181
> WARNING: User-defined SPARK_HOME 
> (/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark) overrides 
> detected (/opt/cloudera/parcels/CDH/lib/spark).
> WARNING: Running spark-class from user-defined location.
> Exception in thread "main" java.lang.NoSuchMethodError: 
> joptsimple.OptionParser.acceptsAll(Ljava/util/Collection;Ljava/lang/String;)Ljoptsimple/OptionSpecBuilder;
>   at 
> org.apache.toree.boot.CommandLineOptions.(CommandLineOptions.scala:37)
>   at org.apache.toree.Main$delayedInit$body.apply(Main.scala:25)
>   at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>   at 
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>   at scala.App$$anonfun$main$1.apply(App.scala:71)
>   at scala.App$$anonfun$main$1.apply(App.scala:71)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>   at scala.App$class.main(App.scala:71)
>   at org.apache.toree.Main$.main(Main.scala:24)
>   at org.apache.toree.Main.main(Main.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>   at 
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
>   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> {code}
> I used the documented command to install the Toree kernel:
> {code}
> pip install --pre toree
> jupyter toree install
> {code}
> Before this I was trying a self-compiled build from the spark-kernel project 
> and I encountered the same exception when starting the kernel in the notebook.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-412) AddDeps should default to the default configuration, not compile

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123729#comment-16123729
 ] 

ASF GitHub Bot commented on TOREE-412:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/120


> AddDeps should default to the default configuration, not compile
> 
>
> Key: TOREE-412
> URL: https://issues.apache.org/jira/browse/TOREE-412
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Reporter: Ryan Blue
>
> The AddDeps magic (using Coursier) currently requests the compile 
> configuration instead of the default configuration. The dependencies 
> shouldn't be those needed to compile, but should be the runtime dependencies, 
> which are normally pulled in by the default configuration. We ran into issues 
> with this because runtime dependencies are not included.
> In addition, I'd like to add --ivy-configuration and --classifier options to 
> give users control over what gets used.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-430) ClassNotFoundException with lambdas on CDH 5.10.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123256#comment-16123256
 ] 

ASF GitHub Bot commented on TOREE-430:
--

Github user Myllyenko commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/129#discussion_r132670358
  
--- Diff: 
scala-interpreter/src/main/scala-2.10/org/apache/toree/kernel/interpreter/scala/ScalaInterpreterSpecific.scala
 ---
@@ -61,6 +61,7 @@ trait ScalaInterpreterSpecific { this: ScalaInterpreter =>
   ): SparkIMain = {
 val s = new SparkIMain(settings, out)
 s.initializeSynchronous()
+System.setProperty("spark.repl.class.outputDir", 
s.getClassOutputDirectory.getAbsolutePath)
--- End diff --

I've tried to figure out how to do that without heavy code modifications 
but haven't succeeded.

It worth noting that currently in a Scala 2.11 environment Toree [sets this 
property in a similar 
manner](https://github.com/apache/incubator-toree/blob/master/kernel/src/main/scala/org/apache/toree/boot/KernelBootstrap.scala#L70).
 And in Scala 2.11 we can pass it directly to `SparkConf` with ease because 
`spark.repl.class.outputDir`'s required value becomes known very early ([when 
`org.apache.spark.repl.Main` is 
initialised](https://github.com/apache/spark/blob/branch-1.6/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala#L33)).

But in a Scala 2.10 environment, things are different. 
`spark.repl.class.outputDir`' [is initialised only with 
`org.apache.spark.repl.SparkIMain`](https://github.com/apache/spark/blob/branch-2.2/repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkIMain.scala#L100),
 that is after creation of `ScalaInterpreter` in Toree. But in order to pass 
`spark.repl.class.outputDir` directly to `SparkConf` we have to 
initialise`SparkIMain` earlier than Toree's `Kernel`.

This is applicable for both Spark 2 and Spark 1.6.


> ClassNotFoundException with lambdas on CDH 5.10.0
> -
>
> Key: TOREE-430
> URL: https://issues.apache.org/jira/browse/TOREE-430
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.1.0
> Environment: CDH 5.10.0 (Spark 1.6.0, Scala 2.10.5)
>Reporter: Igor Melnichenko
>
> In the specified environment, Toree classloader fails to load classes defined 
> and compiled in REPL. For instance, the following Spark example fails with 
> ClassNotFoundException and works fine if the lambda is removed:
> {code:title=Example.scala|borderStyle=solid}
> val NUM_SAMPLES = 100
> val count = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
>   val x = math.random
>   val y = math.random
>   x*x + y*y < 1
> }.count()
> println(s"Pi is roughly ${4.0 * count / NUM_SAMPLES}")
> {code}
> Cause: a location of a directory used by a REPL compiler to store 
> runtime-compiled classes is different from a location where classloader tries 
> to find these classes.
> PR with fix will be opened soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-431) Toree build fails with Spark 2.2.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122218#comment-16122218
 ] 

ASF GitHub Bot commented on TOREE-431:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/131


> Toree build fails with Spark 2.2.0
> --
>
> Key: TOREE-431
> URL: https://issues.apache.org/jira/browse/TOREE-431
> Project: TOREE
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
> Fix For: 0.2.0
>
>
> Build steps:
> APACHE_SPARK_VERSION=2.2.0 make clean dist
> Build issue:
> [info] Compiling 1 Scala source to 
> /Users/lresende/opensource/jupyter/incubator-toree/protocol/target/scala-2.11/classes...
> [info] Compiling 63 Scala sources to 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/target/scala-2.11/classes...
> [error] 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/src/main/scala/org/apache/toree/magic/builtin/BuiltinLoader.scala:20:
>  object reflect is not a member of package com.google.common
> [error] import com.google.common.reflect.ClassPath
> [error]  ^
> [error] 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/src/main/scala/org/apache/toree/magic/builtin/BuiltinLoader.scala:21:
>  object reflect is not a member of package com.google.common
> [error] import com.google.common.reflect.ClassPath.ClassInfo
> [error]  ^
> [error] 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/src/main/scala/org/apache/toree/magic/builtin/BuiltinLoader.scala:40:
>  not found: type ClassInfo
> [error]   def getClasses(pkg: String = pkgName): List[ClassInfo] = {
> [error]   ^
> [error] 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/src/main/scala/org/apache/toree/magic/builtin/BuiltinLoader.scala:46:
>  not found: value ClassPath
> [error] val classPath = ClassPath.from(this.getClass.getClassLoader)
> [error] ^
> [error] four errors found
> [error] (kernel/compile:compileIncremental) Compilation failed
> [error] Total time: 7 s, completed Aug 8, 2017 5:02:41 PM
> make: *** 
> [target/scala-2.11/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar] Error 1



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-432) Update Akka dependency to version 2.4.20

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122219#comment-16122219
 ] 

ASF GitHub Bot commented on TOREE-432:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/132


> Update Akka dependency to version 2.4.20
> 
>
> Key: TOREE-432
> URL: https://issues.apache.org/jira/browse/TOREE-432
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.1.0, 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Blocker
> Fix For: 0.2.0
>
>
> This address CVE-2017-5643.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-432) Update Akka dependency to version 2.4.20

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122199#comment-16122199
 ] 

ASF GitHub Bot commented on TOREE-432:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/132

[TOREE-432] Update Akka to version 2.4.20

Address akka vulnerability: CVE-2017-5643

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-432

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #132


commit 15b9ba63848e8d432ff942eca82107ddcbc569e1
Author: Luciano Resende 
Date:   2017-08-10T19:47:26Z

[TOREE-432] Update Akka to version 2.4.20

Address akka vulnerability: CVE-2017-5643




> Update Akka dependency to version 2.4.20
> 
>
> Key: TOREE-432
> URL: https://issues.apache.org/jira/browse/TOREE-432
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.1.0, 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Blocker
> Fix For: 0.2.0
>
>
> This address CVE-2017-5643.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-431) Toree build fails with Spark 2.2.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120914#comment-16120914
 ] 

ASF GitHub Bot commented on TOREE-431:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/131

[TOREE-431] Explicit add Guava dependency to kernel module



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-431

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/131.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #131


commit bb5f7ce2f21b1ae2d86458b7c45c3165769fde88
Author: Luciano Resende 
Date:   2017-08-10T01:13:53Z

[TOREE-431] Explicit add Guava dependency to kernel module




> Toree build fails with Spark 2.2.0
> --
>
> Key: TOREE-431
> URL: https://issues.apache.org/jira/browse/TOREE-431
> Project: TOREE
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
> Fix For: 0.2.0
>
>
> Build steps:
> APACHE_SPARK_VERSION=2.2.0 make clean dist
> Build issue:
> [info] Compiling 1 Scala source to 
> /Users/lresende/opensource/jupyter/incubator-toree/protocol/target/scala-2.11/classes...
> [info] Compiling 63 Scala sources to 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/target/scala-2.11/classes...
> [error] 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/src/main/scala/org/apache/toree/magic/builtin/BuiltinLoader.scala:20:
>  object reflect is not a member of package com.google.common
> [error] import com.google.common.reflect.ClassPath
> [error]  ^
> [error] 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/src/main/scala/org/apache/toree/magic/builtin/BuiltinLoader.scala:21:
>  object reflect is not a member of package com.google.common
> [error] import com.google.common.reflect.ClassPath.ClassInfo
> [error]  ^
> [error] 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/src/main/scala/org/apache/toree/magic/builtin/BuiltinLoader.scala:40:
>  not found: type ClassInfo
> [error]   def getClasses(pkg: String = pkgName): List[ClassInfo] = {
> [error]   ^
> [error] 
> /Users/lresende/opensource/jupyter/incubator-toree/kernel/src/main/scala/org/apache/toree/magic/builtin/BuiltinLoader.scala:46:
>  not found: value ClassPath
> [error] val classPath = ClassPath.from(this.getClass.getClassLoader)
> [error] ^
> [error] four errors found
> [error] (kernel/compile:compileIncremental) Compilation failed
> [error] Total time: 7 s, completed Aug 8, 2017 5:02:41 PM
> make: *** 
> [target/scala-2.11/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar] Error 1



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-430) ClassNotFoundException with lambdas on CDH 5.10.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120375#comment-16120375
 ] 

ASF GitHub Bot commented on TOREE-430:
--

Github user lresende commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/129#discussion_r132261271
  
--- Diff: 
scala-interpreter/src/main/scala-2.10/org/apache/toree/kernel/interpreter/scala/ScalaInterpreterSpecific.scala
 ---
@@ -61,6 +61,7 @@ trait ScalaInterpreterSpecific { this: ScalaInterpreter =>
   ): SparkIMain = {
 val s = new SparkIMain(settings, out)
 s.initializeSynchronous()
+System.setProperty("spark.repl.class.outputDir", 
s.getClassOutputDirectory.getAbsolutePath)
--- End diff --

Could we pass this as conf while creating the Spark Session? 


> ClassNotFoundException with lambdas on CDH 5.10.0
> -
>
> Key: TOREE-430
> URL: https://issues.apache.org/jira/browse/TOREE-430
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.1.0
> Environment: CDH 5.10.0 (Spark 1.6.0, Scala 2.10.5)
>Reporter: Igor Melnichenko
>
> In the specified environment, Toree classloader fails to load classes defined 
> and compiled in REPL. For instance, the following Spark example fails with 
> ClassNotFoundException and works fine if the lambda is removed:
> {code:title=Example.scala|borderStyle=solid}
> val NUM_SAMPLES = 100
> val count = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
>   val x = math.random
>   val y = math.random
>   x*x + y*y < 1
> }.count()
> println(s"Pi is roughly ${4.0 * count / NUM_SAMPLES}")
> {code}
> Cause: a location of a directory used by a REPL compiler to store 
> runtime-compiled classes is different from a location where classloader tries 
> to find these classes.
> PR with fix will be opened soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-425) sparkContext lazy initiation causes some issues when Toree is running on Yarn Cluster mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113317#comment-16113317
 ] 

ASF GitHub Bot commented on TOREE-425:
--

Github user lresende closed the pull request at:

https://github.com/apache/incubator-toree/pull/128


> sparkContext lazy initiation causes some issues when Toree is running on Yarn 
> Cluster mode
> --
>
> Key: TOREE-425
> URL: https://issues.apache.org/jira/browse/TOREE-425
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
> Fix For: 0.2.0
>
>
> Kernels running in yarn-cluster mode (when launched via spark-submit) must 
> initialize a SparkContext in order for the Spark Yarn code to register the 
> application as RUNNING: 
> https://github.com/apache/spark/blob/3d4d11a80fe8953d48d8bfac2ce112e37d38dc90/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L405



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-408) AddJAR magic should support HDFS file format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095248#comment-16095248
 ] 

ASF GitHub Bot commented on TOREE-408:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/125


> AddJAR magic should support HDFS file format
> 
>
> Key: TOREE-408
> URL: https://issues.apache.org/jira/browse/TOREE-408
> Project: TOREE
>  Issue Type: Improvement
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Should implement HDFS downloader on Toree
> Related source codes:
> Download utils: 
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/utils/DownloadSupport.scala
> AddJar magic: 
> https://github.com/apache/incubator-toree/blob/master/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-425) sparkContext lazy initiation causes some issues when Toree is running on Yarn Cluster mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090868#comment-16090868
 ] 

ASF GitHub Bot commented on TOREE-425:
--

Github user rdblue commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/128#discussion_r127859204
  
--- Diff: 
kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala 
---
@@ -82,6 +84,8 @@ trait StandardComponentInitialization extends 
ComponentInitialization {
 
 initializePlugins(config, pluginManager)
 
+initializeSparkContext(config, kernel)
--- End diff --

Should this be here, or in the kernel's initialization? I think we 
eventually want Toree to not require Spark, in which case we would have a 
SparkKernel and a regular Kernel. Then it would be the responsibility of the 
SparkKernel to detect that the application is in cluster mode and initialize. 
Doing this sooner rather than later would avoid more changes.


> sparkContext lazy initiation causes some issues when Toree is running on Yarn 
> Cluster mode
> --
>
> Key: TOREE-425
> URL: https://issues.apache.org/jira/browse/TOREE-425
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
> Fix For: 0.2.0
>
>
> Kernels running in yarn-cluster mode (when launched via spark-submit) must 
> initialize a SparkContext in order for the Spark Yarn code to register the 
> application as RUNNING: 
> https://github.com/apache/spark/blob/3d4d11a80fe8953d48d8bfac2ce112e37d38dc90/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L405



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-425) sparkContext lazy initiation causes some issues when Toree is running on Yarn Cluster mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090866#comment-16090866
 ] 

ASF GitHub Bot commented on TOREE-425:
--

Github user rdblue commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/128#discussion_r127858771
  
--- Diff: kernel/src/main/scala/org/apache/toree/kernel/api/Kernel.scala ---
@@ -20,8 +20,11 @@ package org.apache.toree.kernel.api
 import java.io.{InputStream, PrintStream}
 import java.net.URI
 import java.util.concurrent.{ConcurrentHashMap, TimeUnit, TimeoutException}
+
--- End diff --

Nit: unnecessary changes in imports make backports and branch maintenance 
harder. I'd prefer not to have these changes unless there is a stated style 
that this changes conforms to.


> sparkContext lazy initiation causes some issues when Toree is running on Yarn 
> Cluster mode
> --
>
> Key: TOREE-425
> URL: https://issues.apache.org/jira/browse/TOREE-425
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
> Fix For: 0.2.0
>
>
> Kernels running in yarn-cluster mode (when launched via spark-submit) must 
> initialize a SparkContext in order for the Spark Yarn code to register the 
> application as RUNNING: 
> https://github.com/apache/spark/blob/3d4d11a80fe8953d48d8bfac2ce112e37d38dc90/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L405



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-425) sparkContext lazy initiation causes some issues when Toree is running on Yarn Cluster mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090759#comment-16090759
 ] 

ASF GitHub Bot commented on TOREE-425:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/128

[TOREE-425] Force sparkContext initialization in cluster mode

Kernels running in yarn-cluster mode (when launched via spark-submit)
must initialize a SparkContext in order for the Spark Yarn code
to register the application as RUNNING

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-425

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #128


commit f80a5484eb0a100b66227ee1e3b4968c568a978b
Author: Luciano Resende 
Date:   2017-07-17T22:56:56Z

[TOREE-425] Force sparkContext initialization in cluster mode

Kernels running in yarn-cluster mode (when launched via spark-submit)
must initialize a SparkContext in order for the Spark Yarn code
to register the application as RUNNING




> sparkContext lazy initiation causes some issues when Toree is running on Yarn 
> Cluster mode
> --
>
> Key: TOREE-425
> URL: https://issues.apache.org/jira/browse/TOREE-425
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
>Assignee: Luciano Resende
>Priority: Critical
> Fix For: 0.2.0
>
>
> Kernels running in yarn-cluster mode (when launched via spark-submit) must 
> initialize a SparkContext in order for the Spark Yarn code to register the 
> application as RUNNING: 
> https://github.com/apache/spark/blob/3d4d11a80fe8953d48d8bfac2ce112e37d38dc90/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L405



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-422) Toree pip install downgrades Jupyter Client to 4.0 version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078999#comment-16078999
 ] 

ASF GitHub Bot commented on TOREE-422:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-toree/pull/127


> Toree pip install downgrades Jupyter Client to 4.0 version
> --
>
> Key: TOREE-422
> URL: https://issues.apache.org/jira/browse/TOREE-422
> Project: TOREE
>  Issue Type: Bug
>  Components: Build, Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
> Fix For: 0.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-422) Toree pip install downgrades Jupyter Client to 4.0 version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071877#comment-16071877
 ] 

ASF GitHub Bot commented on TOREE-422:
--

GitHub user lresende opened a pull request:

https://github.com/apache/incubator-toree/pull/127

[TOREE-422] Enable Jupyter 5.1.0 dependencies

Enable Jupyter 5.1.0 dependencies to avoid issues with
downgrading Jupyter Client to 4.x in distributions such as
anaconda.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lresende/incubator-toree toree-422

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/127.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #127


commit e6b9eaa9adba194d7644f4af6351d109649656f6
Author: Luciano Resende 
Date:   2017-07-03T03:01:28Z

[TOREE-422] Enable Jupyter 5.1.0 dependencies

Enable Jupyter 5.1.0 dependencies to avoid issues with
downgrading Jupyter Client to 4.x in distributions such as
anaconda.




> Toree pip install downgrades Jupyter Client to 4.0 version
> --
>
> Key: TOREE-422
> URL: https://issues.apache.org/jira/browse/TOREE-422
> Project: TOREE
>  Issue Type: Bug
>  Components: Build, Kernel
>Affects Versions: 0.2.0
>Reporter: Luciano Resende
> Fix For: 0.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063666#comment-16063666
 ] 

ASF GitHub Bot commented on TOREE-420:
--

Github user kalvinnchau closed the pull request at:

https://github.com/apache/incubator-toree/pull/126


> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)',
>
> 'scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)',
>
> 'scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)',
>
> 'scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)',
>

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063667#comment-16063667
 ] 

ASF GitHub Bot commented on TOREE-420:
--

GitHub user kalvinnchau reopened a pull request:

https://github.com/apache/incubator-toree/pull/126

TOREE-420 Exclude org.scala-lang.modules from AddDeps

https://issues.apache.org/jira/browse/TOREE-420

Include `org.scala-lang.modules` as part of the always exclude list when 
using the `%AddDeps` magic.

This prevents the `scala.reflect.internal.FatalError` that crashes the 
kernel, when a new scala-lang module is included and reloaded.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kalvinnchau/incubator-toree TOREE-420

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #126


commit 6c6c7262ceffc89a251cfc00565933eabc01ec81
Author: Kalvin Chau 
Date:   2017-06-22T20:26:41Z

updated depdency downloaders to exclude org.scala-lang.modules to prevent 
scala refelection issues TOREE-420




> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061471#comment-16061471
 ] 

ASF GitHub Bot commented on TOREE-420:
--

GitHub user kalvinnchau reopened a pull request:

https://github.com/apache/incubator-toree/pull/126

TOREE-420 Exclude org.scala-lang.modules from AddDeps

https://issues.apache.org/jira/browse/TOREE-420

Include `org.scala-lang.modules` as part of the always exclude list when 
using the `%AddDeps` magic.

This prevents the `scala.reflect.internal.FatalError` that crashes the 
kernel, when a new scala-lang module is included and reloaded.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kalvinnchau/incubator-toree TOREE-420

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #126


commit 6c6c7262ceffc89a251cfc00565933eabc01ec81
Author: Kalvin Chau 
Date:   2017-06-22T20:26:41Z

updated depdency downloaders to exclude org.scala-lang.modules to prevent 
scala refelection issues TOREE-420




> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061470#comment-16061470
 ] 

ASF GitHub Bot commented on TOREE-420:
--

Github user kalvinnchau closed the pull request at:

https://github.com/apache/incubator-toree/pull/126


> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)',
>
> 'scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)',
>
> 'scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)',
>
> 'scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)',
>

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061403#comment-16061403
 ] 

ASF GitHub Bot commented on TOREE-420:
--

GitHub user kalvinnchau reopened a pull request:

https://github.com/apache/incubator-toree/pull/126

TOREE-420 Exclude org.scala-lang.modules from AddDeps

https://issues.apache.org/jira/browse/TOREE-420

Include `org.scala-lang.modules` as part of the always exclude list when 
using the `%AddDeps` magic.

This prevents the `scala.reflect.internal.FatalError` that crashes the 
kernel, when a new scala-lang module is included and reloaded.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kalvinnchau/incubator-toree TOREE-420

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #126


commit 6c6c7262ceffc89a251cfc00565933eabc01ec81
Author: Kalvin Chau 
Date:   2017-06-22T20:26:41Z

updated depdency downloaders to exclude org.scala-lang.modules to prevent 
scala refelection issues TOREE-420




> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 

[jira] [Commented] (TOREE-420) AddDeps fails if transitive dependency includes org.scala-lang.modules

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16061402#comment-16061402
 ] 

ASF GitHub Bot commented on TOREE-420:
--

Github user kalvinnchau closed the pull request at:

https://github.com/apache/incubator-toree/pull/126


> AddDeps fails if transitive dependency includes org.scala-lang.modules
> --
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: 
> c7b008506fa07538b42aa94b287a361a64f71ee6
>Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes 
> org.scala-lang.modules it fails with a reflection error, this kafka depdency 
> for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose 
> {code}
> {code}
> {'buffers': [],
>  'content': {'ename': 'scala.reflect.internal.FatalError',
>  'evalue': 'object Predef does not have a member classOf',
>  'traceback': 
> ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
>
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
>
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
>
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)',
>
> 'scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)',
>
> 'scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)',
>
> 'scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)',
>
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)',
>