[ https://issues.apache.org/jira/browse/LIVY-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366469#comment-17366469 ]
Moritz commented on LIVY-863: ----------------------------- I proposed a fix here, it's missing a test and potentially needs some code to account for different spark versions. If you can help me with this and give some guidance, that would be great and I can add the additional things :) https://github.com/apache/incubator-livy/pull/322 > Missing JVM class imports for Spark3 > ------------------------------------ > > Key: LIVY-863 > URL: https://issues.apache.org/jira/browse/LIVY-863 > Project: Livy > Issue Type: Bug > Components: REPL > Reporter: Moritz > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Livy creates a fake spark shell initiating its own java gateway. This gateway > imports some java classes for the user which have diverged from the ones in > Spark 3. Some imports are missing which lead to methods like > `dataframe.explain()` to be broken: > {code:java} > result.explain(True) > An error was encountered: > 'JavaPackage' object is not callable > Traceback (most recent call last): > File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", > line 386, in explain > > print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(), > explain_mode)) > TypeError: 'JavaPackage' object is not callable > {code} > Spark is importing: > [https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153] > Livy is currently importing: > [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581] -- This message was sent by Atlassian Jira (v8.3.4#803005)