git commit: SPARK-1911: Emphasize that Spark jars should be built with Java 6.
Repository: spark Updated Branches: refs/heads/master 5081a0a9d - 75a032777 SPARK-1911: Emphasize that Spark jars should be built with Java 6. This commit requires the user to manually say yes when buiding Spark without Java 6. The prompt can be bypassed with a flag (e.g. if the user is scripting around make-distribution). Author: Patrick Wendell pwend...@gmail.com Closes #859 from pwendell/java6 and squashes the following commits: 4921133 [Patrick Wendell] Adding Pyspark Notice fee8c9e [Patrick Wendell] SPARK-1911: Emphasize that Spark jars should be built with Java 6. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/75a03277 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/75a03277 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/75a03277 Branch: refs/heads/master Commit: 75a03277704f8618a0f1c41aecfb1ebd24a8ac1a Parents: 5081a0a Author: Patrick Wendell pwend...@gmail.com Authored: Sat May 24 18:27:00 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Sat May 24 18:27:00 2014 -0700 -- make-distribution.sh | 52 --- 1 file changed, 31 insertions(+), 21 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/75a03277/make-distribution.sh -- diff --git a/make-distribution.sh b/make-distribution.sh index 1c89027..ae52b49 100755 --- a/make-distribution.sh +++ b/make-distribution.sh @@ -46,27 +46,6 @@ set -e FWDIR=$(cd `dirname $0`; pwd) DISTDIR=$FWDIR/dist -if [ -z $JAVA_HOME ]; then - echo Error: JAVA_HOME is not set, cannot proceed. - exit -1 -fi - -JAVA_CMD=$JAVA_HOME/bin/java -JAVA_VERSION=$($JAVA_CMD -version 21) -if ! [[ $JAVA_VERSION =~ 1.6 ]]; then - echo ***NOTE***: JAVA_HOME is not set to a JDK 6 installation. The resulting - echo distribution will not support Java 6. See SPARK-1703. - echo Output from 'java -version' was: - echo $JAVA_VERSION -fi - -VERSION=$(mvn help:evaluate -Dexpression=project.version 2/dev/null | grep -v INFO | tail -n 1) -if [ $? != 0 ]; then -echo -e You need Maven installed to build Spark. -echo -e Download Maven from https://maven.apache.org/; -exit -1; -fi - # Initialize defaults SPARK_HADOOP_VERSION=1.0.4 SPARK_YARN=false @@ -88,6 +67,9 @@ while (( $# )); do --with-hive) SPARK_HIVE=true ;; +--skip-java-test) + SKIP_JAVA_TEST=true + ;; --with-tachyon) SPARK_TACHYON=true ;; @@ -102,6 +84,34 @@ while (( $# )); do shift done +if [ -z $JAVA_HOME ]; then + echo Error: JAVA_HOME is not set, cannot proceed. + exit -1 +fi + +VERSION=$(mvn help:evaluate -Dexpression=project.version 2/dev/null | grep -v INFO | tail -n 1) +if [ $? != 0 ]; then +echo -e You need Maven installed to build Spark. +echo -e Download Maven from https://maven.apache.org/; +exit -1; +fi + +JAVA_CMD=$JAVA_HOME/bin/java +JAVA_VERSION=$($JAVA_CMD -version 21) +if [[ ! $JAVA_VERSION =~ 1.6 -z $SKIP_JAVA_TEST ]]; then + echo ***NOTE***: JAVA_HOME is not set to a JDK 6 installation. The resulting + echo distribution may not work well with PySpark and will not run + echo with Java 6 (See SPARK-1703 and SPARK-1911). + echo This test can be disabled by adding --skip-java-test. + echo Output from 'java -version' was: + echo $JAVA_VERSION + read -p Would you like to continue anyways? [y,n]: -r + if [[ ! $REPLY =~ ^[Yy]$ ]]; then +echo Okay, exiting. +exit 1 + fi +fi + if [ $NAME == none ]; then NAME=$SPARK_HADOOP_VERSION fi
git commit: Updated CHANGES.txt
Repository: spark Updated Branches: refs/heads/branch-1.0 217bd5627 - 840609272 Updated CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/84060927 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/84060927 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/84060927 Branch: refs/heads/branch-1.0 Commit: 84060927262159c4c847034a50ff8092f0a72a44 Parents: 217bd56 Author: Tathagata Das tathagata.das1...@gmail.com Authored: Sun May 25 02:20:13 2014 + Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Sun May 25 02:20:13 2014 + -- CHANGES.txt | 87 +++- 1 file changed, 86 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/84060927/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d21f0ac..b097339 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,91 @@ Spark Change Log Release 1.0.0 + SPARK-1911: Emphasize that Spark jars should be built with Java 6. + Patrick Wendell pwend...@gmail.com + 2014-05-24 18:27:00 -0700 + Commit: 217bd56, github.com/apache/spark/pull/859 + + [SPARK-1900 / 1918] PySpark on YARN is broken + Andrew Or andrewo...@gmail.com + 2014-05-24 18:01:49 -0700 + Commit: 12f5ecc, github.com/apache/spark/pull/853 + + Update LBFGSSuite.scala + baishuo(ç½ç¡) vc_j...@hotmail.com + 2014-05-23 13:02:40 -0700 + Commit: 9be103a, github.com/apache/spark/pull/815 + + Updated scripts for auditing releases + Tathagata Das tathagata.das1...@gmail.com + 2014-05-22 20:48:55 -0700 + Commit: 6541ca2, github.com/apache/spark/pull/844 + + [SPARK-1896] Respect spark.master (and --master) before MASTER in spark-shell + Andrew Or andrewo...@gmail.com + 2014-05-22 20:32:27 -0700 + Commit: c3b4065, github.com/apache/spark/pull/846 + + [SPARK-1897] Respect spark.jars (and --jars) in spark-shell + Andrew Or andrewo...@gmail.com + 2014-05-22 20:25:41 -0700 + Commit: 23cc40e, github.com/apache/spark/pull/849 + + Fix UISuite unit test that fails under Jenkins contention + Aaron Davidson aa...@databricks.com + 2014-05-22 15:11:05 -0700 + Commit: a566216, github.com/apache/spark/pull/857 + + [SPARK-1870] Make spark-submit --jars work in yarn-cluster mode. + Xiangrui Meng m...@databricks.com + 2014-05-22 01:52:50 -0700 + Commit: 79cd26c, github.com/apache/spark/pull/848 + + Configuration documentation updates + Reynold Xin r...@apache.org + 2014-05-21 18:49:12 -0700 + Commit: 75af8bd, github.com/apache/spark/pull/851 + + [SPARK-1889] [SQL] Apply splitConjunctivePredicates to join condition while finding join ke... + Takuya UESHIN ues...@happy-camper.st + 2014-05-21 15:37:47 -0700 + Commit: 6e7934e, github.com/apache/spark/pull/836 + + [SPARK-1519] Support minPartitions param of wholeTextFiles() in PySpark + Kan Zhang kzh...@apache.org + 2014-05-21 13:26:53 -0700 + Commit: 30d1df5, github.com/apache/spark/pull/697 + + [Typo] Stoped - Stopped + Andrew Or andrewo...@gmail.com + 2014-05-21 11:59:05 -0700 + Commit: 9b8f772, github.com/apache/spark/pull/847 + + [Minor] Move JdbcRDDSuite to the correct package + Andrew Or andrewo...@gmail.com + 2014-05-21 01:25:10 -0700 + Commit: bc6bbfa, github.com/apache/spark/pull/839 + + [Docs] Correct example of creating a new SparkConf + Andrew Or andrewo...@gmail.com + 2014-05-21 01:23:34 -0700 + Commit: 7295dd9, github.com/apache/spark/pull/842 + + [SPARK-1250] Fixed misleading comments in bin/pyspark, bin/spark-class + Sumedh Mungee smun...@gmail.com + 2014-05-21 01:22:25 -0700 + Commit: 364c14a, github.com/apache/spark/pull/843 + + [Hotfix] Blacklisted flaky HiveCompatibility test + Tathagata Das tathagata.das1...@gmail.com + 2014-05-20 10:27:12 -0700 + Commit: b4d93d3, github.com/apache/spark/pull/838 + + Updated CHANGES.txt + Tathagata Das tathagata.das1...@gmail.com + 2014-05-19 23:12:24 -0700 + Commit: 1c00f2a + [Spark 1877] ClassNotFoundException when loading RDD with serialized objects Tathagata Das tathagata.das1...@gmail.com, Ghidireac bogd...@u448a5b0a73d45358d94a.ant.amazon.com 2014-05-19 22:36:24 -0700 @@ -676,7 +761,7 @@ Release 1.0.0 Proposal: clarify Scala programming guide on caching ... Ethan Jewett esjew...@gmail.com 2014-05-06 20:50:08 -0700 - Commit: 51e27755, github.com/apache/spark/pull/668 + Commit: 51e2775, github.com/apache/spark/pull/668 SPARK-1727. Correct small compile errors, typos, and markdown issues in (primarly) MLlib docs Sean Owen so...@cloudera.com
git commit: spark-submit: add exec at the end of the script
Repository: spark
Updated Branches:
refs/heads/branch-1.0 b5e968696 - 7e59335eb
spark-submit: add exec at the end of the script
Add an 'exec' at the end of the spark-submit script, to avoid keeping a
bash process hanging around while it runs. This makes ps look a little
bit nicer.
Author: Colin Patrick Mccabe cmcc...@cloudera.com
Closes #858 from cmccabe/SPARK-1907 and squashes the following commits:
7023b64 [Colin Patrick Mccabe] spark-submit: add exec at the end of the script
(cherry picked from commit 6e9fb6320bec3371bc9c010ccbc1b915f500486b)
Signed-off-by: Reynold Xin r...@apache.org
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7e59335e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7e59335e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7e59335e
Branch: refs/heads/branch-1.0
Commit: 7e59335eb958bea495c43e687a1b425b28121d75
Parents: b5e9686
Author: Colin Patrick Mccabe cmcc...@cloudera.com
Authored: Sat May 24 22:39:27 2014 -0700
Committer: Reynold Xin r...@apache.org
Committed: Sat May 24 22:39:34 2014 -0700
--
bin/spark-submit | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/7e59335e/bin/spark-submit
--
diff --git a/bin/spark-submit b/bin/spark-submit
index 63903b1..9e7cece 100755
--- a/bin/spark-submit
+++ b/bin/spark-submit
@@ -41,5 +41,5 @@ if [ -n $DRIVER_MEMORY ] [ $DEPLOY_MODE == client ];
then
export SPARK_DRIVER_MEMORY=$DRIVER_MEMORY
fi
-$SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit
${ORIG_ARGS[@]}
+exec $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit
${ORIG_ARGS[@]}
