Re: NoClassDefFoundError: scala/Product$class

Hi Pol, thanks for your suggestion, I am going to use Spark-3.0.0 for GPU acceleration,so I update the scala to the *version 2.12.11* and the latest *2.13* ,but the error is still there, and by the way , the Spark version is *spark-3.0.0-preview2-bin-without-hadoop* Caused by:

Re: Add python library with native code

Not totally sure it's gonna help your use case, but I'd recommend that you consider these too: * pex A library and tool for generating .pex (Python EXecutable) files * cluster-pack cluster-pack is a library on

Re: How to set Description in UI SQL tab

I don't think this option is available for SQL UI tableSearched through the official docs https://spark.apache.org/docs/latest/configuration.html#spark-uiand the codehttps://github.com/apache/spark/search?l=Scala=9=spark.sql.uispark.job.description is available in SparkContext but for UI didn't

Re: Add python library with native code

Thanks Dark. Looked at that article. I think the article described approach B, let me summary both approach A and approach B A) Put libraries in a network share, mount on each node, and in your code, manually set PYTHONPATH B) In your code, manually install the necessary package using "pip install

Re: Add python library with native code

Hi Stone, Have you looked into this article? https://medium.com/@SSKahani/pyspark-applications-dependencies-99415e0df987 I haven't tried it with .so files however I did use the approach he recommends to install my other dependencies. I Hope it helps. On Fri, Jun 5, 2020 at 1:12 PM Stone Zhong

Add python library with native code

Hi, So my pyspark app depends on some python libraries, it is not a problem, I pack all the dependencies into a file libs.zip, and then call *sc.addPyFile("libs.zip")* and it works pretty well for a while. Then I encountered a problem, if any of my library has any binary file dependency (like