Thanks for prompt response, Sean. The issue is that we are restricted on dependencies we can include in our project.
There are 2 issues while including dependencies: 1) there are several dependencies which we and Spark has, but the versions are conflicting. 2) there are dependencies Spark has, and our project does not have. How do we handle these 2 cases differently for including Spark dependencies (direct and transitive ones)? We need to include all dependencies (so there should not be 3rd party transitive dependency) in our POM file, more like Apache Ivy style of management of dependencies (which includes all transitive dependencies in POM file) rather than Maven style. Our main goal is: We want to integrate Spark in our Java application using the Spark Java APIi and run then on the Yarn clusters. Thanks a lot. On Fri, Jun 12, 2015 at 11:17 AM, Sean Owen <so...@cloudera.com> wrote: > You don't add dependencies to your app -- you mark Spark as 'provided' > in the build and you rely on the deployed Spark environment to provide > it. > > On Fri, Jun 12, 2015 at 7:14 PM, Elkhan Dadashov <elkhan8...@gmail.com> > wrote: > > Hi all, > > > > We want to integrate Spark in our Java application using the Spark Java > Api > > and run then on the Yarn clusters. > > > > If i want to run Spark on Yarn, which dependencies are must for > including ? > > > > I looked at Spark POM which lists that Spark requires 50+ 3rd party > > dependencies. > > > > Is there minimum set of Spark dependencies which are necessary for Spark > > Java API (for Spark client run on Yarn cluster) ? > > > > Thanks in advance. > > > -- Best regards, Elkhan Dadashov
