[jira] [Commented] (TOREE-427) Unable to use display() or magics in Toree-PySpark
[ https://issues.apache.org/jira/browse/TOREE-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153622#comment-16153622 ] Ribamar Santarosa commented on TOREE-427: - [~markgritter] can you use magics using the scala kernel instead? For me, the magics are available with the scala kernel, but not with the pyspark one. > Unable to use display() or magics in Toree-PySpark > -- > > Key: TOREE-427 > URL: https://issues.apache.org/jira/browse/TOREE-427 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Environment: Linux >Reporter: Mark Gritter >Priority: Minor > > I am trying to use PySpark in a Jupyter notebook using PySpark. Plain-text > interaction is working fine. When I tried to include HTML or graphical > output, though I get the following: > display(sc.getConf().getAll()) > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in > eval(compiled_code) > File "", line 1, in > File "/usr/local/lib/python2.7/dist-packages/IPython/core/display.py", line > 300, in display > format = InteractiveShell.instance().display_formatter.format > File > "/usr/local/lib/python2.7/dist-packages/traitlets/config/configurable.py", > line 412, in instance > inst = cls(*args, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 500, in __init__ > self.init_io() > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 660, in init_io > io.stdout = io.IOStream(sys.stdout) > File "/usr/local/lib/python2.7/dist-packages/IPython/utils/io.py", line 34, > in __init__ > raise ValueError("fallback required, but not specified") > ValueError: fallback required, but not specified > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.CallCommand.execute(CallCommand.java:79) > py4j.GatewayConnection.run(GatewayConnection.java:214) > java.lang.Thread.run(Thread.java:748) > I tried using magics to display raw HTML (following the example here: > https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/magic-tutorial.ipynb) > and get the following backtrace: > %%dataframe > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in > eval(compiled_code) > File "", line 1, in > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 107, in __getattr__ > return self._jvm_kernel.__getattribute__(name) > AttributeError: 'JavaObject' object has no attribute 'magics' > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.CallCommand.execute(CallCommand.java:79) > py4j.GatewayConnection.run(GatewayConnection.java:214) > java.lang.Thread.run(Thread.java:748) > VERSION: 0.2.0.dev1-incubating > COMMIT: 9b577f19df83 -- This message was sent by Atlassia
[jira] [Commented] (TOREE-427) Unable to use display() or magics in Toree-PySpark
[ https://issues.apache.org/jira/browse/TOREE-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306065#comment-16306065 ] Huadong Liu commented on TOREE-427: --- Yes, magic is available in the scala kernel, but not the pyspark kernel. Looks like it used to work https://www.mail-archive.com/dev@toree.incubator.apache.org/msg00823.html. > Unable to use display() or magics in Toree-PySpark > -- > > Key: TOREE-427 > URL: https://issues.apache.org/jira/browse/TOREE-427 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Environment: Linux >Reporter: Mark Gritter >Priority: Minor > > I am trying to use PySpark in a Jupyter notebook using PySpark. Plain-text > interaction is working fine. When I tried to include HTML or graphical > output, though I get the following: > display(sc.getConf().getAll()) > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in > eval(compiled_code) > File "", line 1, in > File "/usr/local/lib/python2.7/dist-packages/IPython/core/display.py", line > 300, in display > format = InteractiveShell.instance().display_formatter.format > File > "/usr/local/lib/python2.7/dist-packages/traitlets/config/configurable.py", > line 412, in instance > inst = cls(*args, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 500, in __init__ > self.init_io() > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 660, in init_io > io.stdout = io.IOStream(sys.stdout) > File "/usr/local/lib/python2.7/dist-packages/IPython/utils/io.py", line 34, > in __init__ > raise ValueError("fallback required, but not specified") > ValueError: fallback required, but not specified > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.CallCommand.execute(CallCommand.java:79) > py4j.GatewayConnection.run(GatewayConnection.java:214) > java.lang.Thread.run(Thread.java:748) > I tried using magics to display raw HTML (following the example here: > https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/magic-tutorial.ipynb) > and get the following backtrace: > %%dataframe > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in > eval(compiled_code) > File "", line 1, in > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 107, in __getattr__ > return self._jvm_kernel.__getattribute__(name) > AttributeError: 'JavaObject' object has no attribute 'magics' > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.CallCommand.execute(CallCommand.java:79) > py4j.GatewayConnection.run(GatewayConnection.java:214) > java.lang.Thread.run(Thread.java:748) > VERSION: 0.2.0.dev1-incubating > COMMIT: 9b577f19df83 -- This message was sent
[jira] [Commented] (TOREE-427) Unable to use display() or magics in Toree-PySpark
[ https://issues.apache.org/jira/browse/TOREE-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306482#comment-16306482 ] Ryan Blue commented on TOREE-427: - Magics currently get translated into Scala code that gets run in the current interpreter. We need to move magics to get invoked directly instead of going through the interpreter. That would also help debugging because you could have magics that, for example, print logs when the interpreter has crashed. Patches are welcome for this work, and I'll help review them if anyone wants to pick this up. > Unable to use display() or magics in Toree-PySpark > -- > > Key: TOREE-427 > URL: https://issues.apache.org/jira/browse/TOREE-427 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Environment: Linux >Reporter: Mark Gritter >Priority: Minor > > I am trying to use PySpark in a Jupyter notebook using PySpark. Plain-text > interaction is working fine. When I tried to include HTML or graphical > output, though I get the following: > display(sc.getConf().getAll()) > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in > eval(compiled_code) > File "", line 1, in > File "/usr/local/lib/python2.7/dist-packages/IPython/core/display.py", line > 300, in display > format = InteractiveShell.instance().display_formatter.format > File > "/usr/local/lib/python2.7/dist-packages/traitlets/config/configurable.py", > line 412, in instance > inst = cls(*args, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 500, in __init__ > self.init_io() > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 660, in init_io > io.stdout = io.IOStream(sys.stdout) > File "/usr/local/lib/python2.7/dist-packages/IPython/utils/io.py", line 34, > in __init__ > raise ValueError("fallback required, but not specified") > ValueError: fallback required, but not specified > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.CallCommand.execute(CallCommand.java:79) > py4j.GatewayConnection.run(GatewayConnection.java:214) > java.lang.Thread.run(Thread.java:748) > I tried using magics to display raw HTML (following the example here: > https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/magic-tutorial.ipynb) > and get the following backtrace: > %%dataframe > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in > eval(compiled_code) > File "", line 1, in > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 107, in __getattr__ > return self._jvm_kernel.__getattribute__(name) > AttributeError: 'JavaObject' object has no attribute 'magics' > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.C