[
https://issues.apache.org/jira/browse/SPARK-45255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Faiz Halde updated SPARK-45255:
-------------------------------
Description:
java 1.8, sbt 1.9, scala 2.12
I have a very simple repo with the following dependency in `build.sbt`
```
{{libraryDependencies ++= Seq("org.apache.spark" %% "spark-connect-client-jvm"
% "3.5.0")}}
```
A simple application
```
{{object Main extends App {}}
{{ val s = SparkSession.builder().remote("sc://localhost").getOrCreate()}}
{{ s.read.json("/tmp/input.json").repartition(10).show(false)}}
{{}}}
```
But when I run it, I get the following error
```
{{Exception in thread "main" java.lang.NoClassDefFoundError:
org/sparkproject/connect/client/com/google/common/cache/CacheLoader}}
{{ at Main$.delayedEndpoint$Main$1(Main.scala:4)}}
{{ at Main$delayedInit$body.apply(Main.scala:3)}}
{{ at scala.Function0.apply$mcV$sp(Function0.scala:39)}}
{{ at scala.Function0.apply$mcV$sp$(Function0.scala:39)}}
{{ at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}}
{{ at scala.App.$anonfun$main$1$adapted(App.scala:80)}}
{{ at scala.collection.immutable.List.foreach(List.scala:431)}}
{{ at scala.App.main(App.scala:80)}}
{{ at scala.App.main$(App.scala:78)}}
{{ at Main$.main(Main.scala:3)}}
{{ at Main.main(Main.scala)}}
{{Caused by: java.lang.ClassNotFoundException:
org.sparkproject.connect.client.com.google.common.cache.CacheLoader}}
{{ at java.net.URLClassLoader.findClass(URLClassLoader.java:387)}}
{{ at java.lang.ClassLoader.loadClass(ClassLoader.java:418)}}
{{ at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)}}
{{ at java.lang.ClassLoader.loadClass(ClassLoader.java:351)}}
{{ ... 11 more}}
```
I know the connect does a bunch of shading during assembly so it could be
related to that. This application is not started via spark-submit or anything.
It's not run neither under a `SPARK_HOME` ( I guess that's the whole point of
connect client )
EDIT
Not sure if it's the right mitigation but explicitly adding guava worked but
now I am in the 2nd territory of error
{{Sep 21, 2023 8:21:59 PM
org.sparkproject.connect.client.io.grpc.NameResolverRegistry
getDefaultRegistry}}
{{WARNING: No NameResolverProviders found via ServiceLoader, including for DNS.
This is probably due to a broken build. If using ProGuard, check your
configuration}}
{{Exception in thread "main"
org.sparkproject.connect.client.com.google.common.util.concurrent.UncheckedExecutionException:
org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry$ProviderNotFoundException:
No functional channel service provider found. Try adding a dependency on the
grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact}}
{{ at
org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085)}}
{{ at
org.sparkproject.connect.client.com.google.common.cache.LocalCache.get(LocalCache.java:4011)}}
{{ at
org.sparkproject.connect.client.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4034)}}
{{ at
org.sparkproject.connect.client.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5010)}}
{{ at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:945)}}
{{ at scala.Option.getOrElse(Option.scala:189)}}
{{ at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:945)}}
{{ at Main$.delayedEndpoint$Main$1(Main.scala:4)}}
{{ at Main$delayedInit$body.apply(Main.scala:3)}}
{{ at scala.Function0.apply$mcV$sp(Function0.scala:39)}}
{{ at scala.Function0.apply$mcV$sp$(Function0.scala:39)}}
{{ at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}}
{{ at scala.App.$anonfun$main$1$adapted(App.scala:80)}}
{{ at scala.collection.immutable.List.foreach(List.scala:431)}}
{{ at scala.App.main(App.scala:80)}}
{{ at scala.App.main$(App.scala:78)}}
{{ at Main$.main(Main.scala:3)}}
{{ at Main.main(Main.scala)}}
{{Caused by:
org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry$ProviderNotFoundException:
No functional channel service provider found. Try adding a dependency on the
grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact}}
{{ at
org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:179)}}
{{ at
org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:155)}}
{{ at
org.sparkproject.connect.client.io.grpc.Grpc.newChannelBuilder(Grpc.java:101)}}
{{ at
org.sparkproject.connect.client.io.grpc.Grpc.newChannelBuilderForAddress(Grpc.java:111)}}
{{ at
org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.createChannel(SparkConnectClient.scala:633)}}
{{ at
org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.toSparkConnectClient(SparkConnectClient.scala:645)}}
{{ at org.apache.spark.sql.SparkSession$.create(SparkSession.scala:760)}}
{{ at
org.apache.spark.sql.SparkSession$$anon$1.load(SparkSession.scala:736)}}
{{ at
org.apache.spark.sql.SparkSession$$anon$1.load(SparkSession.scala:735)}}
{{ at
org.sparkproject.connect.client.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3570)}}
{{ at
org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2312)}}
{{ at
org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)}}
{{ at
org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)}}
{{ ... 17 more}}
The fix as per the error suggestion isn't working :/
I followed the doc exactly as described. Can somebody help
was:
java 1.8, sbt 1.9, scala 2.12
I have a very simple repo with the following dependency in `build.sbt`
```
{{libraryDependencies ++= Seq("org.apache.spark" %% "spark-connect-client-jvm"
% "3.5.0")}}
```
A simple application
```
{{object Main extends App {}}
{{ val s = SparkSession.builder().remote("sc://localhost").getOrCreate()}}
{{ s.read.json("/tmp/input.json").repartition(10).show(false)}}
{{}}}
```
But when I run it, I get the following error
```
{{Exception in thread "main" java.lang.NoClassDefFoundError:
org/sparkproject/connect/client/com/google/common/cache/CacheLoader}}
{{ at Main$.delayedEndpoint$Main$1(Main.scala:4)}}
{{ at Main$delayedInit$body.apply(Main.scala:3)}}
{{ at scala.Function0.apply$mcV$sp(Function0.scala:39)}}
{{ at scala.Function0.apply$mcV$sp$(Function0.scala:39)}}
{{ at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}}
{{ at scala.App.$anonfun$main$1$adapted(App.scala:80)}}
{{ at scala.collection.immutable.List.foreach(List.scala:431)}}
{{ at scala.App.main(App.scala:80)}}
{{ at scala.App.main$(App.scala:78)}}
{{ at Main$.main(Main.scala:3)}}
{{ at Main.main(Main.scala)}}
{{Caused by: java.lang.ClassNotFoundException:
org.sparkproject.connect.client.com.google.common.cache.CacheLoader}}
{{ at java.net.URLClassLoader.findClass(URLClassLoader.java:387)}}
{{ at java.lang.ClassLoader.loadClass(ClassLoader.java:418)}}
{{ at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)}}
{{ at java.lang.ClassLoader.loadClass(ClassLoader.java:351)}}
{{ ... 11 more}}
```
I know the connect does a bunch of shading during assembly so it could be
related to that. This application is not started via spark-submit or anything.
It's not run neither under a `SPARK_HOME` ( I guess that's the whole point of
connect client )
I followed the doc exactly as described. Can somebody help
> Spark connect client failing with java.lang.NoClassDefFoundError
> ----------------------------------------------------------------
>
> Key: SPARK-45255
> URL: https://issues.apache.org/jira/browse/SPARK-45255
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 3.5.0
> Reporter: Faiz Halde
> Priority: Major
>
> java 1.8, sbt 1.9, scala 2.12
>
> I have a very simple repo with the following dependency in `build.sbt`
> ```
> {{libraryDependencies ++= Seq("org.apache.spark" %%
> "spark-connect-client-jvm" % "3.5.0")}}
> ```
> A simple application
> ```
> {{object Main extends App {}}
> {{ val s = SparkSession.builder().remote("sc://localhost").getOrCreate()}}
> {{ s.read.json("/tmp/input.json").repartition(10).show(false)}}
> {{}}}
> ```
> But when I run it, I get the following error
>
> ```
> {{Exception in thread "main" java.lang.NoClassDefFoundError:
> org/sparkproject/connect/client/com/google/common/cache/CacheLoader}}
> {{ at Main$.delayedEndpoint$Main$1(Main.scala:4)}}
> {{ at Main$delayedInit$body.apply(Main.scala:3)}}
> {{ at scala.Function0.apply$mcV$sp(Function0.scala:39)}}
> {{ at scala.Function0.apply$mcV$sp$(Function0.scala:39)}}
> {{ at
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}}
> {{ at scala.App.$anonfun$main$1$adapted(App.scala:80)}}
> {{ at scala.collection.immutable.List.foreach(List.scala:431)}}
> {{ at scala.App.main(App.scala:80)}}
> {{ at scala.App.main$(App.scala:78)}}
> {{ at Main$.main(Main.scala:3)}}
> {{ at Main.main(Main.scala)}}
> {{Caused by: java.lang.ClassNotFoundException:
> org.sparkproject.connect.client.com.google.common.cache.CacheLoader}}
> {{ at java.net.URLClassLoader.findClass(URLClassLoader.java:387)}}
> {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:418)}}
> {{ at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)}}
> {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:351)}}
> {{ ... 11 more}}
> ```
> I know the connect does a bunch of shading during assembly so it could be
> related to that. This application is not started via spark-submit or
> anything. It's not run neither under a `SPARK_HOME` ( I guess that's the
> whole point of connect client )
>
> EDIT
> Not sure if it's the right mitigation but explicitly adding guava worked but
> now I am in the 2nd territory of error
> {{Sep 21, 2023 8:21:59 PM
> org.sparkproject.connect.client.io.grpc.NameResolverRegistry
> getDefaultRegistry}}
> {{WARNING: No NameResolverProviders found via ServiceLoader, including for
> DNS. This is probably due to a broken build. If using ProGuard, check your
> configuration}}
> {{Exception in thread "main"
> org.sparkproject.connect.client.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry$ProviderNotFoundException:
> No functional channel service provider found. Try adding a dependency on the
> grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact}}
> {{ at
> org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085)}}
> {{ at
> org.sparkproject.connect.client.com.google.common.cache.LocalCache.get(LocalCache.java:4011)}}
> {{ at
> org.sparkproject.connect.client.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4034)}}
> {{ at
> org.sparkproject.connect.client.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5010)}}
> {{ at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:945)}}
> {{ at scala.Option.getOrElse(Option.scala:189)}}
> {{ at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:945)}}
> {{ at Main$.delayedEndpoint$Main$1(Main.scala:4)}}
> {{ at Main$delayedInit$body.apply(Main.scala:3)}}
> {{ at scala.Function0.apply$mcV$sp(Function0.scala:39)}}
> {{ at scala.Function0.apply$mcV$sp$(Function0.scala:39)}}
> {{ at
> scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}}
> {{ at scala.App.$anonfun$main$1$adapted(App.scala:80)}}
> {{ at scala.collection.immutable.List.foreach(List.scala:431)}}
> {{ at scala.App.main(App.scala:80)}}
> {{ at scala.App.main$(App.scala:78)}}
> {{ at Main$.main(Main.scala:3)}}
> {{ at Main.main(Main.scala)}}
> {{Caused by:
> org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry$ProviderNotFoundException:
> No functional channel service provider found. Try adding a dependency on the
> grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact}}
> {{ at
> org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:179)}}
> {{ at
> org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:155)}}
> {{ at
> org.sparkproject.connect.client.io.grpc.Grpc.newChannelBuilder(Grpc.java:101)}}
> {{ at
> org.sparkproject.connect.client.io.grpc.Grpc.newChannelBuilderForAddress(Grpc.java:111)}}
> {{ at
> org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.createChannel(SparkConnectClient.scala:633)}}
> {{ at
> org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.toSparkConnectClient(SparkConnectClient.scala:645)}}
> {{ at org.apache.spark.sql.SparkSession$.create(SparkSession.scala:760)}}
> {{ at
> org.apache.spark.sql.SparkSession$$anon$1.load(SparkSession.scala:736)}}
> {{ at
> org.apache.spark.sql.SparkSession$$anon$1.load(SparkSession.scala:735)}}
> {{ at
> org.sparkproject.connect.client.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3570)}}
> {{ at
> org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2312)}}
> {{ at
> org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)}}
> {{ at
> org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)}}
> {{ ... 17 more}}
>
> The fix as per the error suggestion isn't working :/
> I followed the doc exactly as described. Can somebody help
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
