GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/10614
[SPARK-12433][CORE][TEST][RFC/WIP] Make parts of spark core testing api
public
Apache Spark has a wide variety of internal tests, but many of the APIs
used for testing are private. To allow external developers to easily test their
Spark programs we should expose a minimal set of testing APIs.
Since Sparkâs current tests are primarily written with scalatest &
junit - these are the two testing library bases that we will support. Much of
the desired functionality already exists in the Spark code base, all that is
required is making slimmed down versions of existing test bases (so that the
API to support remains small) and making them visible.
For core the primary thing we want to abstract away is creating &
cleaning up a SparkContext. If we assume that each test will not do anything
destructive to the SparkContext this is perhaps unnecessary. This is based on
the current LocalSparkContext in testing, we would just need to change its
visibility.
This PR moves the SharedSparkContext to main in
org.apache.spark.utils.testing and introduces a Java version of the API
(SharedJavaSparkContext) for Java tests (while also updating JavaAPISuite to
use the new SharedJavaSparkContext).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark
SPARK-12433-make-parts-of-spark-core-testing-api-public
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10614.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10614
----
commit 8c063274a8c05ec67151b73c39f02715299cb484
Author: Holden Karau <hol...@us.ibm.com>
Date: 2015-12-30T19:43:58Z
Move the spark core testing components into main
commit 7a779c46bae523200acc33b39a059bcd40c7f72c
Author: Holden Karau <hol...@us.ibm.com>
Date: 2015-12-30T20:13:05Z
move to the correct sub director
commit c147b27315f4cf4ad3da7f9357046ed8bf81186d
Author: Holden Karau <hol...@us.ibm.com>
Date: 2015-12-30T20:23:45Z
Use a package object to allow the current tests to use the old imports for
the new location of the test libraries
commit 175751ff97ed2226d49b1182296128541aed1cde
Author: Holden Karau <hol...@us.ibm.com>
Date: 2015-12-30T20:28:18Z
Start adding more comments
commit ac973303488983163835a196750d512086dcef50
Author: Holden Karau <hol...@us.ibm.com>
Date: 2015-12-30T21:02:16Z
Expand on the comments a bit mor
commit 4ec035484507683aad1f5f9d106ebc3c8a67c245
Author: Holden Karau <hol...@us.ibm.com>
Date: 2015-12-31T07:05:06Z
Merge branch 'master' into
SPARK-12433-make-parts-of-spark-core-testing-api-public
commit 7f6fcf0edfd4eac2ae2d8dd1cc815211a0d8be09
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-04T22:42:49Z
Merge branch 'master' into
SPARK-12433-make-parts-of-spark-core-testing-api-public
commit dad8f7f1070e470bca908cdd79059b23338559cf
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-04T23:25:51Z
Add the SPARK_VERSION to the spark.scala in the test one and remove the
comment that its for package docs only (this is not the case anymore since its
referenced inside of the SparkContext.)
commit bf58514e2ee3a3a9e11587185b2c8e691fef8376
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-05T02:01:02Z
Add a Local version of JavaSparkContext (e.g. non-shared). TODO: factor the
java tests for shared/non-shared
commit 2b3916c27f1bc6e4abcb02bc3ec609043f172c39
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-05T06:42:21Z
Move the Gauva test to a seperate suite so we can share the SparkContext
for the rest of the JavaAPISuite
commit b0be7d433e1ba96b8225df1954a360dcfbcf718f
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-05T06:53:38Z
Move some of the code around, remove println, and fix use of shared context
to only have 1 thread as required in the JavaAPISuite
commit 93696fd913142e5cc5932bb65832edaac896a47e
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-05T06:54:23Z
Don't both with the LocalJavaSparkContext
commit b21d6545e71cdd1a7f4f368d93841a41a72dd792
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-05T06:55:35Z
simplify imports
commit cc4d37d056fd060f8245aaf16398fe8ba3ae6855
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-05T06:58:09Z
Move to correct new target and fix imports in GuavaSuite
commit 8c718602c2dca9c818d9cc78b8f75b54829042b3
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-05T22:50:07Z
Merge in master
commit 852425f9dae83e63f742d0eb98372de13e0a57b8
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-05T23:23:31Z
Change the testing imports for SQL to match the new locations
commit 5ac6d329795930417bbdeb1c154e64de726c26f4
Author: Holden Karau <hol...@us.ibm.com>
Date: 2016-01-06T06:17:17Z
merge in master
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
