Hi Sean,
Thanks a ton for you reply.
The particular situation I have is case (3) that you have mentioned. The
class that I am using from commons-net is FTPClient(). This class is
present in both the 2.2 version and the 3.3 version. However, in the 3.3
version there are two additional methods (among several others)
setAutoDetectUTF8() and setControlKeepAliveTimeout() that we require to
use.
The class FTPClient() and the two methods are a part of a custom receiver
(ZipStream), that we wrote. Our spark app is deployed on YARN. Looking
online and doing more research I found this link - SPARK-939
https://issues.apache.org/jira/browse/SPARK-939
In the comments section, I found a mention that there is already a flag
spark.yarn.user.classpath.first to make this happen. I have not yet tried
it out. I guess, this should do the trick right? Also, there are some more
JIRA items I see that, indicate combining the
spark.files.userClassPathFirst and spark.yarn.user.classpath.first. It
has been marked as resolved. However I am a bit confused about the state of
the JIRA as far as Cloudera version of spark. We are using spark that comes
with CDH 5.3.2 (Spark 1.2.0). I think this combined flag is marked for
spark 1.3.
If all else fails shade 3.3... :)
Thanks again,
-Jacob
On Fri, Mar 20, 2015 at 10:07 AM, Sean Owen so...@cloudera.com wrote:
It's not a crazy question, no. I'm having a bit of trouble figuring
out what's happening. Commons Net 2.2 is what's used by Spark. The
error appears to come from Spark. But the error is not finding a
method that did not exist in 2.2. I am not sure what ZipStream is, for
example. This could be a bizarre situation where classloader rules
mean that part of 2.2 and part of 3.3 are being used. For example,
let's say:
- your receiver uses 3.3 classes that are only in 3.3, so they are
found in your user classloader
- 3.3 classes call some class that also existed in 2.2, but those are
found in the Spark classloader.
- 2.2 class doesn't have methods that 3.3 expects
userClassPathFirst is often a remedy. There are several versions of
this flag though. For example you need a different one if on YARN to
have it take effect.
It's worth ruling that out first. If all else fails you can shade 3.3.
On Fri, Mar 20, 2015 at 11:44 AM, Jacob Abraham abe.jac...@gmail.com
wrote:
Anyone? or is this question nonsensical... and I am doing something
fundamentally wrong?
On Mon, Mar 16, 2015 at 5:33 PM, Jacob Abraham abe.jac...@gmail.com
wrote:
Hi Folks,
I have a situation where I am getting a version conflict between java
libraries that is used by my application and ones used by spark.
Following are the details -
I use spark provided by Cloudera running on the CDH5.3.2 cluster (Spark
1.2.0-cdh5.3.2). The library that is causing the conflict is
commons-net.
In our spark application we use commons-net with version 3.3.
However I found out that spark uses commons-net version 2.2.
Hence when we try to submit our application using spark-submit, I end up
getting, a NoSuchMethodError()
Error starting receiver 5 -
java.lang.NoSuchMethodError:
org.apache.commons.net.ftp.FTPClient.setAutodetectUTF8(Z)V
at ZipStream.onStart(ZipStream.java:55)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:277)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:269)
.
Now, if I change the commons-net version to 2.2, the job runs fine
(expect
for the fact that some of the features we use from the commons-net 3.3
are
not there).
How does one resolve such an issue where sparks uses one set of
libraries
and our user application requires the same set of libraries, but just a
different version of it (In my case commons-net 2.2 vs 3.3).
I see that there is a setting that I can supply -
spark.files.userClassPathFirst, but the documentation says that it is
experimental and for us this did not work at all.
Thanks in advance.
Regards,
-Jacob
