@Sean, here is where I think it's a little misleading (underlining is mine):
Building a Runnable Distribution To create a *Spark distribution like those distributed by the Spark Downloads <http://spark.apache.org/downloads.html> page*, and that is laid out so as to be runnable, use make-distribution.sh in the project root directory. It can be configured with Maven profile settings and so on like the direct Maven build. Example: Agreed that "like" doesn't necessarily imply "exactly the same". On the other hand, if I go to the download page all I select is a hadoop version and distribution, so it's not super-intuitive that -Phive was used to produce these. I don't have a strong opinion on whether this should be a fix to the script or the docs but now that it's bitten me twice I'm very appreciative of either :) Thanks On Mon, Oct 26, 2015 at 1:29 PM, Sean Owen <so...@cloudera.com> wrote: > I don't think the page suggests that gives you any of the tarballs on the > downloads page, and -Phive does not by itself do so either. > > On Mon, Oct 26, 2015 at 4:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> I logged SPARK-11318 with a PR. >> >> I verified that by adding -Phive the datanucleus jars are included: >> >> tar tzvf spark-1.6.0-SNAPSHOT-bin-custom-spark.tgz | grep datanucleus >> -rw-r--r-- hbase/hadoop 1890075 2015-10-26 09:52 >> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-core-3.2.10.jar >> -rw-r--r-- hbase/hadoop 339666 2015-10-26 09:52 >> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-api-jdo-3.2.6.jar >> -rw-r--r-- hbase/hadoop 1809447 2015-10-26 09:52 >> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-rdbms-3.2.9.jar >> >> Cheers >> >> On Mon, Oct 26, 2015 at 8:52 AM, Yana Kadiyska <yana.kadiy...@gmail.com> >> wrote: >> >>> thank you so much! You are correct. This is the second time I've made >>> this mistake :( >>> >>> On Mon, Oct 26, 2015 at 11:36 AM, java8964 <java8...@hotmail.com> wrote: >>> >>>> Maybe you need the Hive part? >>>> >>>> Yong >>>> >>>> ------------------------------ >>>> Date: Mon, 26 Oct 2015 11:34:30 -0400 >>>> Subject: Problem with make-distribution.sh >>>> From: yana.kadiy...@gmail.com >>>> To: user@spark.apache.org >>>> >>>> >>>> Hi folks, >>>> >>>> building spark instructions ( >>>> http://spark.apache.org/docs/latest/building-spark.html) suggest that >>>> >>>> >>>> ./make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Pyarn >>>> >>>> >>>> >>>> should produce a distribution similar to the ones found on the >>>> "Downloads" page. >>>> >>>> I noticed that the tgz I built using the above command does not produce >>>> the datanucleus jars which are included in the "boxed" spark distributions. >>>> What is the best-practice advice here? >>>> >>>> I would like my distribution to match the official one as closely as >>>> possible. >>>> >>>> Thanks >>>> >>> >>> >> >
