[
https://issues.apache.org/jira/browse/SPARK-19245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-19245.
-------------------------------
Resolution: Not A Problem
There aren't assemblies anymore:
https://issues.apache.org/jira/browse/SPARK-11157
Questions about how to use it should go to the mailing list.
> Cannot build spark-assembly jar
> -------------------------------
>
> Key: SPARK-19245
> URL: https://issues.apache.org/jira/browse/SPARK-19245
> Project: Spark
> Issue Type: Documentation
> Components: Build
> Affects Versions: 2.1.0
> Reporter: Barry Becker
> Priority: Minor
>
> This may just be an issue with documentation, but I am not sure.
> In previous versions of spark (spark 2.0.2 and earlier) one could execute a
> command like
> build/mvn -Pmesos -Phadoop-2.6 -Dhadoop.version=2.6.5 -Dscala-2.11
> -DskipTests clean package
> and have it produce an assembly jar called something like
> spark-assembly-x.y.z-hadoop2.6.5.jar. This no longer seems possible with
> spark 2.1.0.
> There is a -bigtop-dist option (not documented on the main 2.1 build page)
> that allows you to
> create a spark-assembly_2.11-2.1.0-dist.tar.gzip in
> spark-2.1.0/assembly/target.
> If I unpack it, it contains a lot of jars and a README that states:
> {quote}
> This is an assembly module for Spark project.
> It creates a single tar.gz file that includes all needed dependency of the
> project except for org.apache.hadoop.* jars that are supposed to be available
> from the deployed Hadoop cluster.
> This module is off by default. To activate it specify the profile in the
> command line
> -Pbigtop-dist
> If you need to build an assembly for a different version of Hadoop the
> hadoop-version system property needs to be set as in this example:
> -Dhadoop.version=2.0.6-alpha
> {quote}
> My question is, how can I produce a single
> mineset-assembly-x.y.y-hadoopA.B.C.jar? If that it no longer possible, how
> are we supposed to use the bigtop-dist.tar.gzip file? Do we copy it to the
> same lib directory and unpack it there? In any case, some clarification on
> the build page (http://spark.apache.org/docs/latest/building-spark.html)
> would help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
