Yes, spark-core will depend on Hadoop and several other jars. Here’s the list of dependencies: https://github.com/apache/spark/blob/master/core/pom.xml#L35 <https://github.com/apache/spark/blob/master/core/pom.xml#L35>
Whether you need spark-sql depends on whether you will use the DataFrame API. Without spark-sql, you will just have the RDD API. > On Jul 19, 2016, at 7:09 AM, Sachin Mittal <sjmit...@gmail.com> wrote: > > > Hi, > Can someone please guide me what all jars I need to place in my lib folder of > the project to build a standalone scala application via sbt. > > Note I need to provide static dependencies and I cannot download the jars > using libraryDependencies. > So I need to provide all the jars upfront. > > So far I found that we need: > spark-core_<version>.jar > > Do we also need > spark-sql_<version>.jar > and > hadoop-core-<version>.jar > > Is there any jar from spark side I may be missing? What I found that > spark-core needs hadoop-core classes and if I don't add them then sbt was > giving me this error: > [error] bad symbolic reference. A signature in SparkContext.class refers to > term hadoop > [error] in package org.apache which is not available. > > So I was just confused on library dependency part when building an > application via sbt. Any inputs here would be helpful. > > Thanks > Sachin > > >
