Hi Sean,
Thanks a ton for you reply.
The particular situation I have is case (3) that you have mentioned. The
class that I am using from commons-net is FTPClient(). This class is
present in both the 2.2 version and the 3.3 version. However, in the 3.3
version there are two additional methods (among several others)
"setAutoDetectUTF8()" and "setControlKeepAliveTimeout()" that we require to
use.
The class FTPClient() and the two methods are a part of a custom receiver
("ZipStream"), that we wrote. Our spark app is deployed on YARN. Looking
online and doing more research I found this link - SPARK-939
<https://issues.apache.org/jira/browse/SPARK-939>
In the comments section, I found a mention that there is already a flag "
spark.yarn.user.classpath.first" to make this happen. I have not yet tried
it out. I guess, this should do the trick right? Also, there are some more
JIRA items I see that, indicate combining the "
spark.files.userClassPathFirst" and "spark.yarn.user.classpath.first". It
has been marked as resolved. However I am a bit confused about the state of
the JIRA as far as Cloudera version of spark. We are using spark that comes
with CDH 5.3.2 (Spark 1.2.0). I think this combined flag is marked for
spark 1.3.
If all else fails shade 3.3... :)
Thanks again,
-Jacob
On Fri, Mar 20, 2015 at 10:07 AM, Sean Owen <so...@cloudera.com> wrote:
> It's not a crazy question, no. I'm having a bit of trouble figuring
> out what's happening. Commons Net 2.2 is what's used by Spark. The
> error appears to come from Spark. But the error is not finding a
> method that did not exist in 2.2. I am not sure what ZipStream is, for
> example. This could be a bizarre situation where classloader rules
> mean that part of 2.2 and part of 3.3 are being used. For example,
> let's say:
>
> - your receiver uses 3.3 classes that are only in 3.3, so they are
> found in your user classloader
> - 3.3 classes call some class that also existed in 2.2, but those are
> found in the Spark classloader.
> - 2.2 class doesn't have methods that 3.3 expects
>
> userClassPathFirst is often a remedy. There are several versions of
> this flag though. For example you need a different one if on YARN to
> have it take effect.
>
> It's worth ruling that out first. If all else fails you can shade 3.3.
>
> On Fri, Mar 20, 2015 at 11:44 AM, Jacob Abraham <abe.jac...@gmail.com>
> wrote:
> > Anyone? or is this question nonsensical... and I am doing something
> > fundamentally wrong?
> >
> >
> >
> > On Mon, Mar 16, 2015 at 5:33 PM, Jacob Abraham <abe.jac...@gmail.com>
> wrote:
> >>
> >> Hi Folks,
> >>
> >> I have a situation where I am getting a version conflict between java
> >> libraries that is used by my application and ones used by spark.
> >>
> >> Following are the details -
> >>
> >> I use spark provided by Cloudera running on the CDH5.3.2 cluster (Spark
> >> 1.2.0-cdh5.3.2). The library that is causing the conflict is
> commons-net.
> >>
> >> In our spark application we use commons-net with version 3.3.
> >>
> >> However I found out that spark uses commons-net version 2.2.
> >>
> >> Hence when we try to submit our application using spark-submit, I end up
> >> getting, a NoSuchMethodError()
> >>
> >> Error starting receiver 5 -
> >>
> >>
> >> java.lang.NoSuchMethodError:
> >> org.apache.commons.net.ftp.FTPClient.setAutodetectUTF8(Z)V
> >>
> >> at ZipStream.onStart(ZipStream.java:55)
> >> at
> >>
> org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
> >> at
> >>
> org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
> >> at
> >>
> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:277)
> >> at
> >>
> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:269)
> >>
> >> .....
> >>
> >>
> >>
> >>
> >>
> >>
> >> Now, if I change the commons-net version to 2.2, the job runs fine
> (expect
> >> for the fact that some of the features we use from the commons-net 3.3
> are
> >> not there).
> >>
> >>
> >>
> >> How does one resolve such an issue where sparks uses one set of
> libraries
> >> and our user application requires the same set of libraries, but just a
> >> different version of it (In my case commons-net 2.2 vs 3.3).
> >>
> >>
> >> I see that there is a setting that I can supply -
> >> "spark.files.userClassPathFirst", but the documentation says that it is
> >> experimental and for us this did not work at all.
> >>
> >>
> >> Thanks in advance.
> >>
> >>
> >> Regards,
> >>
> >> -Jacob
> >>
> >>
> >>
> >>
> >
>
