git commit: [SPARK-1876] Windows fixes to deal with latest distribution layout changes
Repository: spark Updated Branches: refs/heads/master df0aa8353 - 7b70a7071 [SPARK-1876] Windows fixes to deal with latest distribution layout changes - Look for JARs in the right place - Launch examples the same way as on Unix - Load datanucleus JARs if they exist - Don't attempt to parse local paths as URIs in SparkSubmit, since paths with C:\ are not valid URIs - Also fixed POM exclusion rules for datanucleus (it wasn't properly excluding it, whereas SBT was) Author: Matei Zaharia ma...@databricks.com Closes #819 from mateiz/win-fixes and squashes the following commits: d558f96 [Matei Zaharia] Fix comment 228577b [Matei Zaharia] Review comments d3b71c7 [Matei Zaharia] Properly exclude datanucleus files in Maven assembly 144af84 [Matei Zaharia] Update Windows scripts to match latest binary package layout Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7b70a707 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7b70a707 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7b70a707 Branch: refs/heads/master Commit: 7b70a7071894dd90ea1d0091542b3e13e7ef8d3a Parents: df0aa83 Author: Matei Zaharia ma...@databricks.com Authored: Mon May 19 15:02:35 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 19 15:02:35 2014 -0700 -- README.md | 7 +-- assembly/pom.xml| 2 +- bin/compute-classpath.cmd | 24 - bin/run-example | 23 + bin/run-example2.cmd| 51 +++- bin/spark-class2.cmd| 2 + .../org/apache/spark/deploy/SparkSubmit.scala | 2 +- 7 files changed, 81 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/7b70a707/README.md -- diff --git a/README.md b/README.md index 9c2e32b..6211a58 100644 --- a/README.md +++ b/README.md @@ -9,13 +9,14 @@ You can find the latest Spark documentation, including a programming guide, on the project webpage at http://spark.apache.org/documentation.html. This README file only contains basic setup instructions. - ## Building Spark Spark is built on Scala 2.10. To build Spark and its example programs, run: ./sbt/sbt assembly +(You do not need to do this if you downloaded a pre-built package.) + ## Interactive Scala Shell The easiest way to start using Spark is through the Scala shell: @@ -41,9 +42,9 @@ And run the following command, which should also return 1000: Spark also comes with several sample programs in the `examples` directory. To run one of them, use `./bin/run-example class [params]`. For example: -./bin/run-example org.apache.spark.examples.SparkLR +./bin/run-example SparkPi -will run the Logistic Regression example locally. +will run the Pi example locally. You can set the MASTER environment variable when running examples to submit examples to a cluster. This can be a mesos:// or spark:// URL, http://git-wip-us.apache.org/repos/asf/spark/blob/7b70a707/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index abd8935..963357b 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -96,7 +96,7 @@ filter artifact*:*/artifact excludes -excludeorg.datanucleus:*/exclude +excludeorg/datanucleus/**/exclude excludeMETA-INF/*.SF/exclude excludeMETA-INF/*.DSA/exclude excludeMETA-INF/*.RSA/exclude http://git-wip-us.apache.org/repos/asf/spark/blob/7b70a707/bin/compute-classpath.cmd -- diff --git a/bin/compute-classpath.cmd b/bin/compute-classpath.cmd index 065553e..58710cd 100644 --- a/bin/compute-classpath.cmd +++ b/bin/compute-classpath.cmd @@ -20,6 +20,13 @@ rem rem This script computes Spark's classpath and prints it to stdout; it's used by both the run rem script and the ExecutorRunner in standalone cluster mode. +rem If we're called from spark-class2.cmd, it already set enabledelayedexpansion and setting +rem it here would stop us from affecting its copy of the CLASSPATH variable; otherwise we +rem need to set it here because we use !datanucleus_jars! below. +if %DONT_PRINT_CLASSPATH%==1 goto skip_delayed_expansion +setlocal enabledelayedexpansion +:skip_delayed_expansion + set SCALA_VERSION=2.10 rem Figure out where the Spark framework is installed @@ -31,7 +38,7 @@ if exist %FWDIR%conf\spark-env.cmd call %FWDIR%conf\spark-env.cmd rem Build up classpath set
git commit: SPARK-1878: Fix the incorrect initialization order
Repository: spark
Updated Branches:
refs/heads/branch-1.0 111c121ae - 901102c1b
SPARK-1878: Fix the incorrect initialization order
JIRA: https://issues.apache.org/jira/browse/SPARK-1878
Author: zsxwing zsxw...@gmail.com
Closes #822 from zsxwing/SPARK-1878 and squashes the following commits:
4a47e27 [zsxwing] SPARK-1878: Fix the incorrect initialization order
(cherry picked from commit 1811ba8ccb580979aa2e12019e6a82805f09ab53)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/901102c1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/901102c1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/901102c1
Branch: refs/heads/branch-1.0
Commit: 901102c1ba5f800705819916f2b7a38b6750cffb
Parents: 111c121
Author: zsxwing zsxw...@gmail.com
Authored: Mon May 19 16:41:31 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 16:41:53 2014 -0700
--
.../apache/spark/streaming/api/java/JavaStreamingContext.scala | 6 +++---
.../src/test/java/org/apache/spark/streaming/JavaAPISuite.java | 4
2 files changed, 7 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/901102c1/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
--
diff --git
a/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
b/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
index 75a3e93..18605ca 100644
---
a/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
+++
b/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
@@ -142,12 +142,12 @@ class JavaStreamingContext(val ssc: StreamingContext) {
*/
def this(path: String, hadoopConf: Configuration) = this(new
StreamingContext(path, hadoopConf))
- @deprecated(use sparkContext, 0.9.0)
- val sc: JavaSparkContext = sparkContext
-
/** The underlying SparkContext */
val sparkContext = new JavaSparkContext(ssc.sc)
+ @deprecated(use sparkContext, 0.9.0)
+ val sc: JavaSparkContext = sparkContext
+
/**
* Create an input stream from network source hostname:port. Data is
received using
* a TCP socket and the receive bytes is interpreted as UTF8 encoded \n
delimited
http://git-wip-us.apache.org/repos/asf/spark/blob/901102c1/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
--
diff --git
a/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
b/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
index ce58cb1..4efeb8d 100644
--- a/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
+++ b/streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
@@ -55,6 +55,10 @@ public class JavaAPISuite extends LocalJavaStreamingContext
implements Serializa
equalIterator(a.iterator(), b.iterator());
}
+ @Test
+ public void testInitialization() {
+Assert.assertNotNull(ssc.sc());
+ }
@SuppressWarnings(unchecked)
@Test
git commit: SPARK-1879. Increase MaxPermSize since some of our builds have many classes
Repository: spark
Updated Branches:
refs/heads/master 1811ba8cc - 5af99d761
SPARK-1879. Increase MaxPermSize since some of our builds have many classes
See https://issues.apache.org/jira/browse/SPARK-1879 -- builds with Hadoop2 and
Hive ran out of PermGen space in spark-shell, when those things added up with
the Scala compiler.
Note that users can still override it by setting their own Java options with
this change. Their options will come later in the command string than the
-XX:MaxPermSize=128m.
Author: Matei Zaharia ma...@databricks.com
Closes #823 from mateiz/spark-1879 and squashes the following commits:
6bc0ee8 [Matei Zaharia] Increase MaxPermSize to 128m since some of our builds
have lots of classes
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5af99d76
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5af99d76
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5af99d76
Branch: refs/heads/master
Commit: 5af99d7617ba3b9fbfdb345ef9571b7dd41f45a1
Parents: 1811ba8
Author: Matei Zaharia ma...@databricks.com
Authored: Mon May 19 18:42:28 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 18:42:28 2014 -0700
--
bin/spark-class | 4 ++--
bin/spark-class2.cmd | 4 ++--
.../main/scala/org/apache/spark/deploy/worker/CommandUtils.scala | 4 +++-
3 files changed, 7 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/5af99d76/bin/spark-class
--
diff --git a/bin/spark-class b/bin/spark-class
index 6480ccb..2e57295 100755
--- a/bin/spark-class
+++ b/bin/spark-class
@@ -99,14 +99,14 @@ else
fi
# Set JAVA_OPTS to be able to load native libraries and to set heap size
-JAVA_OPTS=$OUR_JAVA_OPTS
+JAVA_OPTS=-XX:MaxPermSize=128m $OUR_JAVA_OPTS
JAVA_OPTS=$JAVA_OPTS -Xms$OUR_JAVA_MEM -Xmx$OUR_JAVA_MEM
# Load extra JAVA_OPTS from conf/java-opts, if it exists
if [ -e $FWDIR/conf/java-opts ] ; then
JAVA_OPTS=$JAVA_OPTS `cat $FWDIR/conf/java-opts`
fi
export JAVA_OPTS
-# Attention: when changing the way the JAVA_OPTS are assembled, the change
must be reflected in ExecutorRunner.scala!
+# Attention: when changing the way the JAVA_OPTS are assembled, the change
must be reflected in CommandUtils.scala!
if [ ! -f $FWDIR/RELEASE ]; then
# Exit if the user hasn't compiled Spark
http://git-wip-us.apache.org/repos/asf/spark/blob/5af99d76/bin/spark-class2.cmd
--
diff --git a/bin/spark-class2.cmd b/bin/spark-class2.cmd
index 266edd9..e420eb4 100755
--- a/bin/spark-class2.cmd
+++ b/bin/spark-class2.cmd
@@ -77,8 +77,8 @@ rem All drivers use SPARK_JAVA_OPTS + SPARK_DRIVER_MEMORY.
The repl also uses SP
)
rem Set JAVA_OPTS to be able to load native libraries and to set heap size
-set JAVA_OPTS=%OUR_JAVA_OPTS% -Djava.library.path=%SPARK_LIBRARY_PATH%
-Xms%OUR_JAVA_MEM% -Xmx%OUR_JAVA_MEM%
-rem Attention: when changing the way the JAVA_OPTS are assembled, the change
must be reflected in ExecutorRunner.scala!
+set JAVA_OPTS=-XX:MaxPermSize=128m %OUR_JAVA_OPTS%
-Djava.library.path=%SPARK_LIBRARY_PATH% -Xms%OUR_JAVA_MEM% -Xmx%OUR_JAVA_MEM%
+rem Attention: when changing the way the JAVA_OPTS are assembled, the change
must be reflected in CommandUtils.scala!
rem Test whether the user has built Spark
if exist %FWDIR%RELEASE goto skip_build_test
http://git-wip-us.apache.org/repos/asf/spark/blob/5af99d76/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
index c7f0f24..4af5bc3 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
@@ -65,6 +65,8 @@ object CommandUtils extends Logging {
Seq()
}
+val permGenOpt = Seq(-XX:MaxPermSize=128m)
+
// Figure out our classpath with the external compute-classpath script
val ext = if (System.getProperty(os.name).startsWith(Windows)) .cmd
else .sh
val classPath = Utils.executeAndGetOutput(
@@ -73,7 +75,7 @@ object CommandUtils extends Logging {
val userClassPath = command.classPathEntries ++ Seq(classPath)
Seq(-cp,
userClassPath.filterNot(_.isEmpty).mkString(File.pathSeparator)) ++
- libraryOpts ++ extraOpts ++ workerLocalOpts ++ memoryOpts
+ permGenOpt ++ libraryOpts ++ extraOpts ++ workerLocalOpts ++ memoryOpts
}
/** Spawn a thread that will
git commit: SPARK-1689: Spark application should die when removed by Master
Repository: spark
Updated Branches:
refs/heads/master 6a2c5c610 - b0ce22e07
SPARK-1689: Spark application should die when removed by Master
scheduler.error() will mask the error if there are active tasks. Being removed
is a cataclysmic event for Spark applications, and should probably be treated
as such.
Author: Aaron Davidson aa...@databricks.com
Closes #832 from aarondav/i-love-u and squashes the following commits:
9f1200f [Aaron Davidson] SPARK-1689: Spark application should die when removed
by Master
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b0ce22e0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b0ce22e0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b0ce22e0
Branch: refs/heads/master
Commit: b0ce22e071da4cc62ec5e29abf7b1299b8e4a6b0
Parents: 6a2c5c6
Author: Aaron Davidson aa...@databricks.com
Authored: Mon May 19 20:55:26 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 20:55:26 2014 -0700
--
.../spark/scheduler/cluster/SparkDeploySchedulerBackend.scala | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b0ce22e0/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
index 9768670..9c07b3f 100644
---
a/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
+++
b/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
@@ -88,6 +88,8 @@ private[spark] class SparkDeploySchedulerBackend(
if (!stopping) {
logError(Application has been killed. Reason: + reason)
scheduler.error(reason)
+ // Ensure the application terminates, as we can no longer run jobs.
+ sc.stop()
}
}
git commit: SPARK-1689: Spark application should die when removed by Master
Repository: spark
Updated Branches:
refs/heads/branch-1.0 875c54fb3 - 78b6e6f1e
SPARK-1689: Spark application should die when removed by Master
scheduler.error() will mask the error if there are active tasks. Being removed
is a cataclysmic event for Spark applications, and should probably be treated
as such.
Author: Aaron Davidson aa...@databricks.com
Closes #832 from aarondav/i-love-u and squashes the following commits:
9f1200f [Aaron Davidson] SPARK-1689: Spark application should die when removed
by Master
(cherry picked from commit b0ce22e071da4cc62ec5e29abf7b1299b8e4a6b0)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/78b6e6f1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/78b6e6f1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/78b6e6f1
Branch: refs/heads/branch-1.0
Commit: 78b6e6f1e8ee6a27ef4eed93aac6eba716b5ffce
Parents: 875c54f
Author: Aaron Davidson aa...@databricks.com
Authored: Mon May 19 20:55:26 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 21:01:18 2014 -0700
--
.../spark/scheduler/cluster/SparkDeploySchedulerBackend.scala | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/78b6e6f1/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
index 9768670..9c07b3f 100644
---
a/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
+++
b/core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
@@ -88,6 +88,8 @@ private[spark] class SparkDeploySchedulerBackend(
if (!stopping) {
logError(Application has been killed. Reason: + reason)
scheduler.error(reason)
+ // Ensure the application terminates, as we can no longer run jobs.
+ sc.stop()
}
}
[4/4] git commit: [SPARK-1874][MLLIB] Clean up MLlib sample data
[SPARK-1874][MLLIB] Clean up MLlib sample data 1. Added synthetic datasets for `MovieLensALS`, `LinearRegression`, `BinaryClassification`. 2. Embedded instructions in the help message of those example apps. Per discussion with Matei on the JIRA page, new example data is under `data/mllib`. Author: Xiangrui Meng m...@databricks.com Closes #833 from mengxr/mllib-sample-data and squashes the following commits: 59f0a18 [Xiangrui Meng] add sample binary classification data 3c2f92f [Xiangrui Meng] add linear regression data 050f1ca [Xiangrui Meng] add a sample dataset for MovieLensALS example Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bcb9dce6 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bcb9dce6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bcb9dce6 Branch: refs/heads/master Commit: bcb9dce6f444a977c714117811bce0c54b417650 Parents: b0ce22e Author: Xiangrui Meng m...@databricks.com Authored: Mon May 19 21:29:33 2014 -0700 Committer: Tathagata Das tathagata.das1...@gmail.com Committed: Mon May 19 21:29:33 2014 -0700 -- .../mllib/sample_binary_classification_data.txt | 100 ++ data/mllib/sample_linear_regression_data.txt| 501 ++ data/mllib/sample_movielens_data.txt| 1501 ++ .../examples/mllib/BinaryClassification.scala | 12 +- .../spark/examples/mllib/LinearRegression.scala | 11 +- .../spark/examples/mllib/MovieLensALS.scala | 15 + 6 files changed, 2138 insertions(+), 2 deletions(-) --
[1/4] [SPARK-1874][MLLIB] Clean up MLlib sample data
Repository: spark Updated Branches: refs/heads/branch-1.0 78b6e6f1e - 1c6c8b5bd http://git-wip-us.apache.org/repos/asf/spark/blob/1c6c8b5b/data/mllib/sample_movielens_data.txt -- diff --git a/data/mllib/sample_movielens_data.txt b/data/mllib/sample_movielens_data.txt new file mode 100644 index 000..f0eee19 --- /dev/null +++ b/data/mllib/sample_movielens_data.txt @@ -0,0 +1,1501 @@ +0::2::3 +0::3::1 +0::5::2 +0::9::4 +0::11::1 +0::12::2 +0::15::1 +0::17::1 +0::19::1 +0::21::1 +0::23::1 +0::26::3 +0::27::1 +0::28::1 +0::29::1 +0::30::1 +0::31::1 +0::34::1 +0::37::1 +0::41::2 +0::44::1 +0::45::2 +0::46::1 +0::47::1 +0::48::1 +0::50::1 +0::51::1 +0::54::1 +0::55::1 +0::59::2 +0::61::2 +0::64::1 +0::67::1 +0::68::1 +0::69::1 +0::71::1 +0::72::1 +0::77::2 +0::79::1 +0::83::1 +0::87::1 +0::89::2 +0::91::3 +0::92::4 +0::94::1 +0::95::2 +0::96::1 +0::98::1 +0::99::1 +1::2::2 +1::3::1 +1::4::2 +1::6::1 +1::9::3 +1::12::1 +1::13::1 +1::14::1 +1::16::1 +1::19::1 +1::21::3 +1::27::1 +1::28::3 +1::33::1 +1::36::2 +1::37::1 +1::40::1 +1::41::2 +1::43::1 +1::44::1 +1::47::1 +1::50::1 +1::54::1 +1::56::2 +1::57::1 +1::58::1 +1::60::1 +1::62::4 +1::63::1 +1::67::1 +1::68::4 +1::70::2 +1::72::1 +1::73::1 +1::74::2 +1::76::1 +1::77::3 +1::78::1 +1::81::1 +1::82::1 +1::85::3 +1::86::2 +1::88::2 +1::91::1 +1::92::2 +1::93::1 +1::94::2 +1::96::1 +1::97::1 +2::4::3 +2::6::1 +2::8::5 +2::9::1 +2::10::1 +2::12::3 +2::13::1 +2::15::2 +2::18::2 +2::19::4 +2::22::1 +2::26::1 +2::28::1 +2::34::4 +2::35::1 +2::37::5 +2::38::1 +2::39::5 +2::40::4 +2::47::1 +2::50::1 +2::52::2 +2::54::1 +2::55::1 +2::57::2 +2::58::2 +2::59::1 +2::61::1 +2::62::1 +2::64::1 +2::65::1 +2::66::3 +2::68::1 +2::71::3 +2::76::1 +2::77::1 +2::78::1 +2::80::1 +2::83::5 +2::85::1 +2::87::2 +2::88::1 +2::89::4 +2::90::1 +2::92::4 +2::93::5 +3::0::1 +3::1::1 +3::2::1 +3::7::3 +3::8::3 +3::9::1 +3::14::1 +3::15::1 +3::16::1 +3::18::4 +3::19::1 +3::24::3 +3::26::1 +3::29::3 +3::33::1 +3::34::3 +3::35::1 +3::36::3 +3::37::1 +3::38::2 +3::43::1 +3::44::1 +3::46::1 +3::47::1 +3::51::5 +3::52::3 +3::56::1 +3::58::1 +3::60::3 +3::62::1 +3::65::2 +3::66::1 +3::67::1 +3::68::2 +3::70::1 +3::72::2 +3::76::3 +3::79::3 +3::80::4 +3::81::1 +3::83::1 +3::84::1 +3::86::1 +3::87::2 +3::88::4 +3::89::1 +3::91::1 +3::94::3 +4::1::1 +4::6::1 +4::8::1 +4::9::1 +4::10::1 +4::11::1 +4::12::1 +4::13::1 +4::14::2 +4::15::1 +4::17::1 +4::20::1 +4::22::1 +4::23::1 +4::24::1 +4::29::4 +4::30::1 +4::31::1 +4::34::1 +4::35::1 +4::36::1 +4::39::2 +4::40::3 +4::41::4 +4::43::2 +4::44::1 +4::45::1 +4::46::1 +4::47::1 +4::49::2 +4::50::1 +4::51::1 +4::52::4 +4::54::1 +4::55::1 +4::60::3 +4::61::1 +4::62::4 +4::63::3 +4::65::1 +4::67::2 +4::69::1 +4::70::4 +4::71::1 +4::73::1 +4::78::1 +4::84::1 +4::85::1 +4::87::3 +4::88::3 +4::89::2 +4::96::1 +4::97::1 +4::98::1 +4::99::1 +5::0::1 +5::1::1 +5::4::1 +5::5::1 +5::8::1 +5::9::3 +5::10::2 +5::13::3 +5::15::1 +5::19::1 +5::20::3 +5::21::2 +5::23::3 +5::27::1 +5::28::1 +5::29::1 +5::31::1 +5::36::3 +5::38::2 +5::39::1 +5::42::1 +5::48::3 +5::49::4 +5::50::3 +5::51::1 +5::52::1 +5::54::1 +5::55::5 +5::56::3 +5::58::1 +5::60::1 +5::61::1 +5::64::3 +5::65::2 +5::68::4 +5::70::1 +5::71::1 +5::72::1 +5::74::1 +5::79::1 +5::81::2 +5::84::1 +5::85::1 +5::86::1 +5::88::1 +5::90::4 +5::91::2 +5::95::2 +5::99::1 +6::0::1 +6::1::1 +6::2::3 +6::5::1 +6::6::1 +6::9::1 +6::10::1 +6::15::2 +6::16::2 +6::17::1 +6::18::1 +6::20::1 +6::21::1 +6::22::1 +6::24::1 +6::25::5 +6::26::1 +6::28::1 +6::30::1 +6::33::1 +6::38::1 +6::39::1 +6::43::4 +6::44::1 +6::45::1 +6::48::1 +6::49::1 +6::50::1 +6::53::1 +6::54::1 +6::55::1 +6::56::1 +6::58::4 +6::59::1 +6::60::1 +6::61::3 +6::63::3 +6::66::1 +6::67::3 +6::68::1 +6::69::1 +6::71::2 +6::73::1 +6::75::1 +6::77::1 +6::79::1 +6::81::1 +6::84::1 +6::85::3 +6::86::1 +6::87::1 +6::88::1 +6::89::1 +6::91::2 +6::94::1 +6::95::2 +6::96::1 +7::1::1 +7::2::2 +7::3::1 +7::4::1 +7::7::1 +7::10::1 +7::11::2 +7::14::2 +7::15::1 +7::16::1 +7::18::1 +7::21::1 +7::22::1 +7::23::1 +7::25::5 +7::26::1 +7::29::4 +7::30::1 +7::31::3 +7::32::1 +7::33::1 +7::35::1 +7::37::2 +7::39::3 +7::40::2 +7::42::2 +7::44::1 +7::45::2 +7::47::4 +7::48::1 +7::49::1 +7::53::1 +7::54::1 +7::55::1 +7::56::1 +7::59::1 +7::61::2 +7::62::3 +7::63::2 +7::66::1 +7::67::3 +7::74::1 +7::75::1 +7::76::3 +7::77::1 +7::81::1 +7::82::1 +7::84::2 +7::85::4 +7::86::1 +7::92::2 +7::96::1 +7::97::1 +7::98::1 +8::0::1 +8::2::4 +8::3::2 +8::4::2 +8::5::1 +8::7::1 +8::9::1 +8::11::1 +8::15::1 +8::18::1 +8::19::1 +8::21::1 +8::29::5 +8::31::3 +8::33::1 +8::35::1 +8::36::1 +8::40::2 +8::44::1 +8::45::1 +8::50::1 +8::51::1 +8::52::5 +8::53::5 +8::54::1 +8::55::1 +8::56::1 +8::58::4 +8::60::3 +8::62::4 +8::64::1 +8::67::3 +8::69::1 +8::71::1 +8::72::3 +8::77::3 +8::78::1 +8::79::1 +8::83::1 +8::85::5 +8::86::1 +8::88::1 +8::90::1 +8::92::2 +8::95::4 +8::96::3 +8::97::1 +8::98::1 +8::99::1 +9::2::3 +9::3::1
git commit: [Spark 1877] ClassNotFoundException when loading RDD with serialized objects
Repository: spark
Updated Branches:
refs/heads/branch-1.0 1c6c8b5bd - 6cbe2a37c
[Spark 1877] ClassNotFoundException when loading RDD with serialized objects
Updated version of #821
Author: Tathagata Das tathagata.das1...@gmail.com
Author: Ghidireac bogd...@u448a5b0a73d45358d94a.ant.amazon.com
Closes #835 from tdas/SPARK-1877 and squashes the following commits:
f346f71 [Tathagata Das] Addressed Patrick's comments.
fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with
serialized objects
(cherry picked from commit 52eb54d02403a3c37d84b9da7cc1cdb261048cf8)
Signed-off-by: Tathagata Das tathagata.das1...@gmail.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6cbe2a37
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6cbe2a37
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6cbe2a37
Branch: refs/heads/branch-1.0
Commit: 6cbe2a37ccb14f65b6d6b813a585adbbc43684c4
Parents: 1c6c8b5
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Mon May 19 22:36:24 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 22:36:37 2014 -0700
--
core/src/main/scala/org/apache/spark/SparkContext.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6cbe2a37/core/src/main/scala/org/apache/spark/SparkContext.scala
--
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 634c10c..49737fa 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -718,7 +718,7 @@ class SparkContext(config: SparkConf) extends Logging {
minPartitions: Int = defaultMinPartitions
): RDD[T] = {
sequenceFile(path, classOf[NullWritable], classOf[BytesWritable],
minPartitions)
- .flatMap(x = Utils.deserialize[Array[T]](x._2.getBytes))
+ .flatMap(x = Utils.deserialize[Array[T]](x._2.getBytes,
Utils.getContextOrSparkClassLoader))
}
protected[spark] def checkpointFile[T: ClassTag](
git commit: [Spark 1877] ClassNotFoundException when loading RDD with serialized objects
Repository: spark
Updated Branches:
refs/heads/master bcb9dce6f - 52eb54d02
[Spark 1877] ClassNotFoundException when loading RDD with serialized objects
Updated version of #821
Author: Tathagata Das tathagata.das1...@gmail.com
Author: Ghidireac bogd...@u448a5b0a73d45358d94a.ant.amazon.com
Closes #835 from tdas/SPARK-1877 and squashes the following commits:
f346f71 [Tathagata Das] Addressed Patrick's comments.
fee0c5d [Ghidireac] SPARK-1877: ClassNotFoundException when loading RDD with
serialized objects
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/52eb54d0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/52eb54d0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/52eb54d0
Branch: refs/heads/master
Commit: 52eb54d02403a3c37d84b9da7cc1cdb261048cf8
Parents: bcb9dce
Author: Tathagata Das tathagata.das1...@gmail.com
Authored: Mon May 19 22:36:24 2014 -0700
Committer: Tathagata Das tathagata.das1...@gmail.com
Committed: Mon May 19 22:36:24 2014 -0700
--
core/src/main/scala/org/apache/spark/SparkContext.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/52eb54d0/core/src/main/scala/org/apache/spark/SparkContext.scala
--
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 634c10c..49737fa 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -718,7 +718,7 @@ class SparkContext(config: SparkConf) extends Logging {
minPartitions: Int = defaultMinPartitions
): RDD[T] = {
sequenceFile(path, classOf[NullWritable], classOf[BytesWritable],
minPartitions)
- .flatMap(x = Utils.deserialize[Array[T]](x._2.getBytes))
+ .flatMap(x = Utils.deserialize[Array[T]](x._2.getBytes,
Utils.getContextOrSparkClassLoader))
}
protected[spark] def checkpointFile[T: ClassTag](
