I have submitted a defect in JIRA for this:
https://issues.apache.org/jira/browse/SPARK-3638 and have submitted a PR (
https://github.com/apache/spark/pull/2489) that temporarily fixes the
issue. Users would have to build spark with kinesis-asl to get the
compatible httpclient added to spark assemb
Hello,
In my case, I manually deleted org/apache/http directory in the
spark-assembly jar file..
I think if we use the latest version of httpclient (httpcore) library, we
can resolve the problem.
How about upgrading httpclient? (or jets3t?)
2014-09-11 19:09 GMT+09:00 Aniket Bhatnagar :
> Thanks
Thanks everyone for weighing in on this.
I had backported kinesis module from master to spark 1.0.2 so just to
confirm if I am not missing anything, I did a dependency graph compare of
my spark build with spark-master
and org.apache.httpcomponents:httpclient:jar does seem to resolve to 4.1.2
depen
>From output of dependency:tree:
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @
spark-streaming_2.10 ---
[INFO] org.apache.spark:spark-streaming_2.10:jar:1.1.0-SNAPSHOT
INFO] +- org.apache.spark:spark-core_2.10:jar:1.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-client:ja
If httpClient dependency is coming from Hive, you could build Spark without
Hive. Alternatively, have you tried excluding httpclient from
spark-streaming dependency in your sbt/maven project?
TD
On Thu, Sep 4, 2014 at 6:42 AM, Koert Kuipers wrote:
> custom spark builds should not be the answe
custom spark builds should not be the answer. at least not if spark ever
wants to have a vibrant community for spark apps.
spark does support a user-classpath-first option, which would deal with
some of these issues, but I don't think it works.
On Sep 4, 2014 9:01 AM, "Felix Garcia Borrego" wrote
Hi,
I run into the same issue and apart from the ideas Aniket said, I only
could find a nasty workaround. Add my custom PoolingClientConnectionManager
to my classpath.
http://stackoverflow.com/questions/24788949/nosuchmethoderror-while-running-aws-s3-client-on-spark-while-javap-shows-otherwi/25488
Dumb question -- are you using a Spark build that includes the Kinesis
dependency? that build would have resolved conflicts like this for
you. Your app would need to use the same version of the Kinesis client
SDK, ideally.
All of these ideas are well-known, yes. In cases of super-common
dependenci
I am trying to use Kinesis as source to Spark Streaming and have run into a
dependency issue that can't be resolved without making my own custom Spark
build. The issue is that Spark is transitively dependent
on org.apache.httpcomponents:httpclient:jar:4.1.2 (I think because of
libfb303 coming from
