Hi Jeff,
I'm building my custom zeppelin:
mvn clean package -DskipTests -Pweb-angular -Pspark-scala-2.12
-Pinclude-hadoop -Pspark-3.0 -Phadoop2 -Pbuild-distr -pl
zeppelin-interpreter,zeppelin-zengine,spark/interpreter,spark/spark-dependencies,zeppelin-web-angular,zeppelin-server,zeppelin-distribution,jdbc,zeppelin-plugins/notebookrepo/filesystem
-am
Running the following code:
%spark.pyspark
import pandas as pd
from pyspark.sql.types import StringType
import numpy as np
def np_sqrt(v):
return np.__path__
spark.udf.register("np_sqrt", np_sqrt, StringType())
df = spark.range(10).createOrReplaceTempView("d")
spark.sql("select np_sqrt(id) as arr from d").show(truncate=False)
Thanks
On Sun, Sep 12, 2021 at 6:09 PM Jeff Zhang <[email protected]> wrote:
> We have integration test for spark 3.1.2, it is supposed to work. Could
> you let us know how to reproduce it ?
>
> Lior Chaga <[email protected]> 于2021年9月12日周日 下午8:17写道：
>
>> Hi,
>>
>> I'm running zeppelin 0.10 with spark 3.1.2
>> When trying to run %spark.pyspark paragraph, I
>> get ClassNotFoundException: py4j.GatewayServer$GatewayServerBuilder
>>
>> I see that py4j is a provided scope dependency of spark interpreter,
>> should I explicitly add py4j to the interpreter dependencies?
>> Setting zeppelin.py4j.useAuth to false doesn't help, as the code
>> uses sparkInterpreter.getSparkVersion().isSecretSocketSupported(), and
>> doesn't allow manual control over it.
>>
>> Thanks,
>> Lior
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
