Hey guys,
I’ve just started looking at the excellent spark-solr project (thanks Tim
Potter, Kiran Chitturi, Kevin Risden and Jason Gerlowski for their efforts with
this project it looks really neat!!).
I’m only at the initial stages of my exploration but I’m running into a class
not found exception when connecting to a secure solr cloud instance (basic
auth, ssl). Everything is working as expected on a non-secure solr cloud
instance.
The process looks pretty straightforward according to the doco so I’m wondering
if I’m missing anything obvious or if I need to bring any extra classes to the
classpath when using this project?
Any advice would be greatly appreciated.
Thanks,
Dwane
Environments tried
7.6 and 8.1.1 solr cloud
SSL, Basic Auth Plugin, Rules Based Authorisation Plugin enabled
Spark v 2.4.3
Spark-Solr build spark-solr-3.7.0-20190619.153847-16-shaded.jar
./spark-2.4.3-bin-hado./spark-2.4.3-bin-hadoop2.7/bin/spark-shell --master
local[*] --jars spark-solr-3.7.0-20190619.153847-16-shaded.jar --conf
'spark.driver.extraJavaOptions=-Dbasicauth=solr:SolrRocks'
val options = Map(
"collection" -> "My_Collection",
"zkhost" -> "zkn1:2181,zkn2:2181,zkn3:2181/solr/SPARKTEST"
)
val df = spark.read.format("solr").options(options).load
com.google.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError: org/eclipse/jetty/client/api/Authentication
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at
com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:244)
at
com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:248)
at
com.lucidworks.spark.SolrRelation.dynamicSuffixes$lzycompute(SolrRelation.scala:100)
at com.lucidworks.spark.SolrRelation.dynamicSuffixes(SolrRelation.scala:98)
at
com.lucidworks.spark.SolrRelation.getBaseSchemaFromConfig(SolrRelation.scala:299)
at
com.lucidworks.spark.SolrRelation.querySchema$lzycompute(SolrRelation.scala:239)
at com.lucidworks.spark.SolrRelation.querySchema(SolrRelation.scala:108)
at com.lucidworks.spark.SolrRelation.schema(SolrRelation.scala:428)
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:403)
at
org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
... 49 elided
Caused by: java.lang.NoClassDefFoundError:
org/eclipse/jetty/client/api/Authentication
at
com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:214)
at
com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:240)
at
com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:38)
at
com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:36)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
... 64 more
Caused by: java.lang.ClassNotFoundException:
org.eclipse.jetty.client.api.Authentication
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 72 more
