[ https://issues.apache.org/jira/browse/SPARK-45255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Herman van Hövell resolved SPARK-45255. --------------------------------------- Fix Version/s: 4.0.0 3.5.1 Resolution: Duplicate > Spark connect client failing with java.lang.NoClassDefFoundError > ---------------------------------------------------------------- > > Key: SPARK-45255 > URL: https://issues.apache.org/jira/browse/SPARK-45255 > Project: Spark > Issue Type: Bug > Components: Connect > Affects Versions: 3.5.0 > Reporter: Faiz Halde > Assignee: Herman van Hövell > Priority: Major > Fix For: 4.0.0, 3.5.1 > > > java 1.8, sbt 1.9, scala 2.12 > > I have a very simple repo with the following dependency in `build.sbt` > ``` > {{libraryDependencies ++= Seq("org.apache.spark" %% > "spark-connect-client-jvm" % "3.5.0")}} > ``` > A simple application > ``` > {{object Main extends App {}} > {{ val s = SparkSession.builder().remote("sc://localhost").getOrCreate()}} > {{ s.read.json("/tmp/input.json").repartition(10).show(false)}} > {{}}} > ``` > But when I run it, I get the following error > > ``` > {{Exception in thread "main" java.lang.NoClassDefFoundError: > org/sparkproject/connect/client/com/google/common/cache/CacheLoader}} > {{ at Main$.delayedEndpoint$Main$1(Main.scala:4)}} > {{ at Main$delayedInit$body.apply(Main.scala:3)}} > {{ at scala.Function0.apply$mcV$sp(Function0.scala:39)}} > {{ at scala.Function0.apply$mcV$sp$(Function0.scala:39)}} > {{ at > scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}} > {{ at scala.App.$anonfun$main$1$adapted(App.scala:80)}} > {{ at scala.collection.immutable.List.foreach(List.scala:431)}} > {{ at scala.App.main(App.scala:80)}} > {{ at scala.App.main$(App.scala:78)}} > {{ at Main$.main(Main.scala:3)}} > {{ at Main.main(Main.scala)}} > {{Caused by: java.lang.ClassNotFoundException: > org.sparkproject.connect.client.com.google.common.cache.CacheLoader}} > {{ at java.net.URLClassLoader.findClass(URLClassLoader.java:387)}} > {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:418)}} > {{ at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)}} > {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:351)}} > {{ ... 11 more}} > ``` > I know the connect does a bunch of shading during assembly so it could be > related to that. This application is not started via spark-submit or > anything. It's not run neither under a `SPARK_HOME` ( I guess that's the > whole point of connect client ) > > EDIT > Not sure if it's the right mitigation but explicitly adding guava worked but > now I am in the 2nd territory of error > {{Sep 21, 2023 8:21:59 PM > org.sparkproject.connect.client.io.grpc.NameResolverRegistry > getDefaultRegistry}} > {{WARNING: No NameResolverProviders found via ServiceLoader, including for > DNS. This is probably due to a broken build. If using ProGuard, check your > configuration}} > {{Exception in thread "main" > org.sparkproject.connect.client.com.google.common.util.concurrent.UncheckedExecutionException: > > org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry$ProviderNotFoundException: > No functional channel service provider found. Try adding a dependency on the > grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact}} > {{ at > org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085)}} > {{ at > org.sparkproject.connect.client.com.google.common.cache.LocalCache.get(LocalCache.java:4011)}} > {{ at > org.sparkproject.connect.client.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4034)}} > {{ at > org.sparkproject.connect.client.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5010)}} > {{ at > org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:945)}} > {{ at scala.Option.getOrElse(Option.scala:189)}} > {{ at > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:945)}} > {{ at Main$.delayedEndpoint$Main$1(Main.scala:4)}} > {{ at Main$delayedInit$body.apply(Main.scala:3)}} > {{ at scala.Function0.apply$mcV$sp(Function0.scala:39)}} > {{ at scala.Function0.apply$mcV$sp$(Function0.scala:39)}} > {{ at > scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}} > {{ at scala.App.$anonfun$main$1$adapted(App.scala:80)}} > {{ at scala.collection.immutable.List.foreach(List.scala:431)}} > {{ at scala.App.main(App.scala:80)}} > {{ at scala.App.main$(App.scala:78)}} > {{ at Main$.main(Main.scala:3)}} > {{ at Main.main(Main.scala)}} > {{Caused by: > org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry$ProviderNotFoundException: > No functional channel service provider found. Try adding a dependency on the > grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact}} > {{ at > org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:179)}} > {{ at > org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:155)}} > {{ at > org.sparkproject.connect.client.io.grpc.Grpc.newChannelBuilder(Grpc.java:101)}} > {{ at > org.sparkproject.connect.client.io.grpc.Grpc.newChannelBuilderForAddress(Grpc.java:111)}} > {{ at > org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.createChannel(SparkConnectClient.scala:633)}} > {{ at > org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.toSparkConnectClient(SparkConnectClient.scala:645)}} > {{ at org.apache.spark.sql.SparkSession$.create(SparkSession.scala:760)}} > {{ at > org.apache.spark.sql.SparkSession$$anon$1.load(SparkSession.scala:736)}} > {{ at > org.apache.spark.sql.SparkSession$$anon$1.load(SparkSession.scala:735)}} > {{ at > org.sparkproject.connect.client.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3570)}} > {{ at > org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2312)}} > {{ at > org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)}} > {{ at > org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)}} > {{ ... 17 more}} > > The fix as per the error suggestion isn't working :/ > I followed the doc exactly as described. Can somebody help -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org