Are you proposing I downgrade Solrj's httpclient dependency to be on par with that of Spark/Hadoop? Or upgrade Spark/Hadoop's httpclient to the latest?
Solrj has to stay with its selected version. I could try and rebuild Spark with the latest httpclient but I've no idea what effects that may cause on Spark. Sent from my iPhone > On Feb 18, 2015, at 1:37 AM, Arush Kharbanda <ar...@sigmoidanalytics.com> > wrote: > > Hi > > Did you try to make maven pick the latest version > > http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management > > That way solrj won't cause any issue, you can try this and check if the part > of your code where you access HDFS works fine? > > > >> On Wed, Feb 18, 2015 at 10:23 AM, dgoldenberg <dgoldenberg...@gmail.com> >> wrote: >> I'm getting the below error when running spark-submit on my class. This class >> has a transitive dependency on HttpClient v.4.3.1 since I'm calling SolrJ >> 4.10.3 from within the class. >> >> This is in conflict with the older version, HttpClient 3.1 that's a >> dependency of Hadoop 2.4 (I'm running Spark 1.2.1 built for Hadoop 2.4). >> >> I've tried setting spark.files.userClassPathFirst to true in SparkConf in my >> program, also setting it to true in $SPARK-HOME/conf/spark-defaults.conf as >> >> spark.files.userClassPathFirst true >> >> No go, I'm still getting the error, as below. Is there anything else I can >> try? Are there any plans in Spark to support multiple class loaders? >> >> Exception in thread "main" java.lang.NoSuchMethodError: >> org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry; >> at >> org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121) >> at >> org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:445) >> at >> org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:206) >> at >> org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:35) >> at >> org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:142) >> at >> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:118) >> at >> org.apache.solr.client.solrj.impl.HttpSolrServer.<init>(HttpSolrServer.java:168) >> at >> org.apache.solr.client.solrj.impl.HttpSolrServer.<init>(HttpSolrServer.java:141) >> ....................... >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Class-loading-issue-spark-files-userClassPathFirst-doesn-t-seem-to-be-working-tp21693.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org > > > > -- > > > Arush Kharbanda || Technical Teamlead > > ar...@sigmoidanalytics.com || www.sigmoidanalytics.com
