Hi Folks,
I have a situation where I am getting a version conflict between java
libraries that is used by my application and ones used by spark.
Following are the details -
I use spark provided by Cloudera running on the CDH5.3.2 cluster (Spark
1.2.0-cdh5.3.2). The library that is causing the conflict is commons-net.
In our spark application we use commons-net with version 3.3.
However I found out that spark uses commons-net version 2.2.
Hence when we try to submit our application using spark-submit, I end up
getting, a NoSuchMethodError()
Error starting receiver 5 -
java.lang.NoSuchMethodError:
org.apache.commons.net.ftp.FTPClient.setAutodetectUTF8(Z)V
at ZipStream.onStart(ZipStream.java:55)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:277)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:269)
.....
Now, if I change the commons-net version to 2.2, the job runs fine
(expect for the fact that some of the features we use from the
commons-net 3.3 are not there).
How does one resolve such an issue where sparks uses one set of
libraries and our user application requires the same set of libraries,
but just a different version of it (In my case commons-net 2.2 vs
3.3).
I see that there is a setting that I can supply -
"spark.files.userClassPathFirst", but the documentation says that it
is experimental and for us this did not work at all.
Thanks in advance.
Regards,
-Jacob
