[ https://issues.apache.org/jira/browse/TOREE-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153622#comment-16153622 ]
Ribamar Santarosa commented on TOREE-427: ----------------------------------------- [~markgritter] can you use magics using the scala kernel instead? For me, the magics are available with the scala kernel, but not with the pyspark one. > Unable to use display() or magics in Toree-PySpark > -------------------------------------------------- > > Key: TOREE-427 > URL: https://issues.apache.org/jira/browse/TOREE-427 > Project: TOREE > Issue Type: Bug > Components: Kernel > Affects Versions: 0.2.0 > Environment: Linux > Reporter: Mark Gritter > Priority: Minor > > I am trying to use PySpark in a Jupyter notebook using PySpark. Plain-text > interaction is working fine. When I tried to include HTML or graphical > output, though I get the following: > display(sc.getConf().getAll()) > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in <module> > eval(compiled_code) > File "<string>", line 1, in <module> > File "/usr/local/lib/python2.7/dist-packages/IPython/core/display.py", line > 300, in display > format = InteractiveShell.instance().display_formatter.format > File > "/usr/local/lib/python2.7/dist-packages/traitlets/config/configurable.py", > line 412, in instance > inst = cls(*args, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 500, in __init__ > self.init_io() > File > "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", > line 660, in init_io > io.stdout = io.IOStream(sys.stdout) > File "/usr/local/lib/python2.7/dist-packages/IPython/utils/io.py", line 34, > in __init__ > raise ValueError("fallback required, but not specified") > ValueError: fallback required, but not specified > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.CallCommand.execute(CallCommand.java:79) > py4j.GatewayConnection.run(GatewayConnection.java:214) > java.lang.Thread.run(Thread.java:748) > I tried using magics to display raw HTML (following the example here: > https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/magic-tutorial.ipynb) > and get the following backtrace: > %%dataframe > Name: org.apache.toree.interpreter.broker.BrokerException > Message: Traceback (most recent call last): > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 189, in <module> > eval(compiled_code) > File "<string>", line 1, in <module> > File > "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py", > line 107, in __getattr__ > return self._jvm_kernel.__getattribute__(name) > AttributeError: 'JavaObject' object has no attribute 'magics' > StackTrace: > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163) > scala.Option.foreach(Option.scala:257) > org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > java.lang.reflect.Method.invoke(Method.java:498) > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > py4j.Gateway.invoke(Gateway.java:280) > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > py4j.commands.CallCommand.execute(CallCommand.java:79) > py4j.GatewayConnection.run(GatewayConnection.java:214) > java.lang.Thread.run(Thread.java:748) > VERSION: 0.2.0.dev1-incubating > COMMIT: 9b577f19df83 -- This message was sent by Atlassian JIRA (v6.4.14#64029)