[jira] [Commented] (TOREE-427) Unable to use display() or magics in Toree-PySpark

2017-12-29 Thread Ryan Blue (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306482#comment-16306482
 ] 

Ryan Blue commented on TOREE-427:
-

Magics currently get translated into Scala code that gets run in the current 
interpreter. We need to move magics to get invoked directly instead of going 
through the interpreter. That would also help debugging because you could have 
magics that, for example, print logs when the interpreter has crashed. Patches 
are welcome for this work, and I'll help review them if anyone wants to pick 
this up.

> Unable to use display() or magics in Toree-PySpark
> --
>
> Key: TOREE-427
> URL: https://issues.apache.org/jira/browse/TOREE-427
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Linux
>Reporter: Mark Gritter
>Priority: Minor
>
> I am trying to use PySpark in a Jupyter notebook using PySpark.  Plain-text 
> interaction is working fine.  When I tried to include HTML or graphical 
> output, though I get the following:
> display(sc.getConf().getAll())
> Name: org.apache.toree.interpreter.broker.BrokerException
> Message: Traceback (most recent call last):
>   File 
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", 
> line 189, in 
> eval(compiled_code)
>   File "", line 1, in 
>   File "/usr/local/lib/python2.7/dist-packages/IPython/core/display.py", line 
> 300, in display
> format = InteractiveShell.instance().display_formatter.format
>   File 
> "/usr/local/lib/python2.7/dist-packages/traitlets/config/configurable.py", 
> line 412, in instance
> inst = cls(*args, **kwargs)
>   File 
> "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", 
> line 500, in __init__
> self.init_io()
>   File 
> "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", 
> line 660, in init_io
> io.stdout = io.IOStream(sys.stdout)
>   File "/usr/local/lib/python2.7/dist-packages/IPython/utils/io.py", line 34, 
> in __init__
> raise ValueError("fallback required, but not specified")
> ValueError: fallback required, but not specified
> StackTrace: 
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> scala.Option.foreach(Option.scala:257)
> org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> py4j.Gateway.invoke(Gateway.java:280)
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> py4j.commands.CallCommand.execute(CallCommand.java:79)
> py4j.GatewayConnection.run(GatewayConnection.java:214)
> java.lang.Thread.run(Thread.java:748)
> I tried using magics to display raw HTML (following the example here: 
> https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/magic-tutorial.ipynb)
>  and get the following backtrace:
> %%dataframe
> Name: org.apache.toree.interpreter.broker.BrokerException
> Message: Traceback (most recent call last):
>   File 
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", 
> line 189, in 
> eval(compiled_code)
>   File "", line 1, in 
>   File 
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", 
> line 107, in __getattr__
> return self._jvm_kernel.__getattribute__(name)
> AttributeError: 'JavaObject' object has no attribute 'magics'
> StackTrace: 
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> scala.Option.foreach(Option.scala:257)
> org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> py4j.Gateway.invoke(Gateway.java:280)
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>

[jira] [Comment Edited] (TOREE-427) Unable to use display() or magics in Toree-PySpark

2017-12-29 Thread Huadong Liu (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306065#comment-16306065
 ] 

Huadong Liu edited comment on TOREE-427 at 12/29/17 8:09 AM:
-

Yes, magic is available in the scala kernel, but not the pyspark kernel. Looks 
like it used to work 
https://www.mail-archive.com/dev@toree.incubator.apache.org/msg00823.html. 
[~luciano resende] can we bump up priority?


was (Author: huadongliu):
Yes, magic is available in the scala kernel, but not the pyspark kernel. Looks 
like it used to work 
https://www.mail-archive.com/dev@toree.incubator.apache.org/msg00823.html. 

> Unable to use display() or magics in Toree-PySpark
> --
>
> Key: TOREE-427
> URL: https://issues.apache.org/jira/browse/TOREE-427
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
> Environment: Linux
>Reporter: Mark Gritter
>Priority: Minor
>
> I am trying to use PySpark in a Jupyter notebook using PySpark.  Plain-text 
> interaction is working fine.  When I tried to include HTML or graphical 
> output, though I get the following:
> display(sc.getConf().getAll())
> Name: org.apache.toree.interpreter.broker.BrokerException
> Message: Traceback (most recent call last):
>   File 
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", 
> line 189, in 
> eval(compiled_code)
>   File "", line 1, in 
>   File "/usr/local/lib/python2.7/dist-packages/IPython/core/display.py", line 
> 300, in display
> format = InteractiveShell.instance().display_formatter.format
>   File 
> "/usr/local/lib/python2.7/dist-packages/traitlets/config/configurable.py", 
> line 412, in instance
> inst = cls(*args, **kwargs)
>   File 
> "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", 
> line 500, in __init__
> self.init_io()
>   File 
> "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", 
> line 660, in init_io
> io.stdout = io.IOStream(sys.stdout)
>   File "/usr/local/lib/python2.7/dist-packages/IPython/utils/io.py", line 34, 
> in __init__
> raise ValueError("fallback required, but not specified")
> ValueError: fallback required, but not specified
> StackTrace: 
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> scala.Option.foreach(Option.scala:257)
> org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> py4j.Gateway.invoke(Gateway.java:280)
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> py4j.commands.CallCommand.execute(CallCommand.java:79)
> py4j.GatewayConnection.run(GatewayConnection.java:214)
> java.lang.Thread.run(Thread.java:748)
> I tried using magics to display raw HTML (following the example here: 
> https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/magic-tutorial.ipynb)
>  and get the following backtrace:
> %%dataframe
> Name: org.apache.toree.interpreter.broker.BrokerException
> Message: Traceback (most recent call last):
>   File 
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", 
> line 189, in 
> eval(compiled_code)
>   File "", line 1, in 
>   File 
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", 
> line 107, in __getattr__
> return self._jvm_kernel.__getattribute__(name)
> AttributeError: 'JavaObject' object has no attribute 'magics'
> StackTrace: 
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> scala.Option.foreach(Option.scala:257)
> org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> py4j.Gateway.invoke(Gateway.java:280)
>