I think I'm going to have to rebuild Spark with commons.httpclient.version set to 4.3.1 which looks to be the version chosen by Solrj, rather than the 4.2.6 that Spark's pom mentions. Might work.
On Wed, Feb 18, 2015 at 1:37 AM, Arush Kharbanda <ar...@sigmoidanalytics.com > wrote: > Hi > > Did you try to make maven pick the latest version > > > http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management > > That way solrj won't cause any issue, you can try this and check if the > part of your code where you access HDFS works fine? > > > > On Wed, Feb 18, 2015 at 10:23 AM, dgoldenberg <dgoldenberg...@gmail.com> > wrote: > >> I'm getting the below error when running spark-submit on my class. This >> class >> has a transitive dependency on HttpClient v.4.3.1 since I'm calling SolrJ >> 4.10.3 from within the class. >> >> This is in conflict with the older version, HttpClient 3.1 that's a >> dependency of Hadoop 2.4 (I'm running Spark 1.2.1 built for Hadoop 2.4). >> >> I've tried setting spark.files.userClassPathFirst to true in SparkConf in >> my >> program, also setting it to true in $SPARK-HOME/conf/spark-defaults.conf >> as >> >> spark.files.userClassPathFirst true >> >> No go, I'm still getting the error, as below. Is there anything else I can >> try? Are there any plans in Spark to support multiple class loaders? >> >> Exception in thread "main" java.lang.NoSuchMethodError: >> >> org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry; >> at >> >> org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121) >> at >> >> org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:445) >> at >> >> org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:206) >> at >> >> org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:35) >> at >> >> org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:142) >> at >> >> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:118) >> at >> >> org.apache.solr.client.solrj.impl.HttpSolrServer.<init>(HttpSolrServer.java:168) >> at >> >> org.apache.solr.client.solrj.impl.HttpSolrServer.<init>(HttpSolrServer.java:141) >> ....................... >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Class-loading-issue-spark-files-userClassPathFirst-doesn-t-seem-to-be-working-tp21693.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > > -- > > [image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com> > > *Arush Kharbanda* || Technical Teamlead > > ar...@sigmoidanalytics.com || www.sigmoidanalytics.com >
