[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19529 Maven is the build of reference... YMMV but I also have a bunch of trouble with SBT whereas not with Maven. I'd honestly prefer we not support an SBT build explicitly for reasons like this, though make sure it's not hard to let it work if people want to. at least the PR builder should be Maven. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 hm... I was always testing with sbt, because maven was so slow to do anything. Will do --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19529 I just reverted this PR. @nkronenfeld Could you submit another PR and update the title to `[SPARK-22308][test-maven] Support alternative unit testing styles in external applications`? Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19529 @nkronenfeld @gatorsmile I think this has been failing the master build (Maven only) for a few days: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/3981/consoleFull ``` SQLQuerySuite: *** RUN ABORTED *** org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at: org.apache.spark.sql.test.GenericFunSpecSuite.(GenericFunSpecSuite.scala:28) sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) java.lang.reflect.Constructor.newInstance(Constructor.java:422) java.lang.Class.newInstance(Class.java:442) org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:66) org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:38) org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:37) scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) scala.collection.Iterator$class.foreach(Iterator.scala:893) scala.collection.AbstractIterator.foreach(Iterator.scala:1336) scala.collection.IterableLike$class.foreach(IterableLike.scala:72) scala.collection.AbstractIterable.foreach(Iterable.scala:54) scala.collection.TraversableLike$class.map(TraversableLike.scala:234) scala.collection.AbstractTraversable.map(Traversable.scala:104) org.scalatest.tools.DiscoverySuite.(DiscoverySuite.scala:37) org.scalatest.tools.Runner$.genDiscoSuites$1(Runner.scala:1165) org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1250) ``` I suspect that there's some slightly different way that the tests execute in Maven that may highlight a problem with how the SQL tests use and reuse SparkContexts. It's likely this change did cause it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19529 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83074/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #83074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83074/testReport)** for PR 19529 at commit [`6c0b0d5`](https://github.com/apache/spark/commit/6c0b0d569ae1d779fd9253da0c7e97d12634063c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19529 LGTM pending Jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #83074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83074/testReport)** for PR 19529 at commit [`6c0b0d5`](https://github.com/apache/spark/commit/6c0b0d569ae1d779fd9253da0c7e97d12634063c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83046/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #83046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83046/testReport)** for PR 19529 at commit [`24fc4a3`](https://github.com/apache/spark/commit/24fc4a324008b2acfcf5a2617eb7cc320565e83c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #83046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83046/testReport)** for PR 19529 at commit [`24fc4a3`](https://github.com/apache/spark/commit/24fc4a324008b2acfcf5a2617eb7cc320565e83c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #3960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3960/testReport)** for PR 19529 at commit [`2d927e9`](https://github.com/apache/spark/commit/2d927e94f627919ac1546b47072276b23d3e8da2). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait PlanTestBase extends PredicateHelper ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #3960 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3960/testReport)** for PR 19529 at commit [`2d927e9`](https://github.com/apache/spark/commit/2d927e94f627919ac1546b47072276b23d3e8da2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19529 The Jenkins jobs get killed not-infrequently for various reasons. You can ignore that and retest. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 I assume the "unknown error code, -9" is: [error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt -Phadoop-2.6 -Pflume -Phive-thriftserver -Pyarn -Pkafka-0-8 -Phive -Pkinesis-asl -Pmesos -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest test ; process was terminated by signal 9 It started popping up between build 82895 and 82969 - between which all I did was eliminate the documentation on unit testing. I wouldn't think this should affect the build. Signal 9 is a kill signal - which means something external killed the build, I think. Any idea why this is happening for the last several builds? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83006/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #83006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83006/testReport)** for PR 19529 at commit [`2d927e9`](https://github.com/apache/spark/commit/2d927e94f627919ac1546b47072276b23d3e8da2). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait PlanTestBase extends PredicateHelper ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 Yeah, as predicted, that made PlanTest very easy to review, but didn't do as well with SQLTestUtils. I suspect I reordered functions and what-not when I was moving stuff around. If this is still too confusing to deal with, just let me know. Even if I can't make the end diffs of the entire PR non-trivial, I could certainly re-implement this in a way that the individual commits would each be trivial; then it'd just be a question of following along commit-by-commit, and shouldn't be too bad. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #83006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83006/testReport)** for PR 19529 at commit [`2d927e9`](https://github.com/apache/spark/commit/2d927e94f627919ac1546b47072276b23d3e8da2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 @gatorsmile sounds good, giving that a try now... assuming tests pass, I'll check it in and see if it's any better. I've so far done this for PlanTest and SQLTestUtils PlanTest I suspect it will make much cleaner. In SQLTestUtils I suspect it won't help as much, as it was more a pick-and-choose (this function goes in base, this doesn't) I haven't done it at all for SharedSQLContext/SharedSparkSession... that one seems more deserving of a first-level place to me, so I'm more hesitant to, but if you want, I'll do that one too. I suspect the correct answer is going to be redoing the PR, with careful commits that are clearer about what each does... but I'll try this first anyway :-) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19529
I found a very simple way to reduce the line of changes.
Could you put the PlanTest and PlanTestBase in the same file? We can
refactor it later, if necessary. For example, in `PlanTest.scala`
```Scala
/**
* Provides helper methods for comparing plans.
*/
trait PlanTest extends SparkFunSuite with PlanTestBase
/**
* Provides helper methods for comparing plans, but without the overhead of
* mandating a FunSuite.
*/
trait PlanTestBase extends PredicateHelper { self: Suite =>
```
You also can do the same thing for the other files.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 @gatorsmile the code changes aren't huge - there's almost no new code here, it's all just moving code around from one file to another in order to expose a SharedSparkSession with no dependence on FunSuite. I don't *think* that could be broken up into multiple PRs - half-done, it won't compile. As for taking over the code movement, I'm not sure what you mean; please explain further? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19529 Generally, it makes sense to me. Since the code changes are pretty large here, it is not very straightforward for us to review it. Do you mind if I taking over some of them? Or could you split the PR to multiple smaller ones? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82969/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #82969 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82969/testReport)** for PR 19529 at commit [`4218b86`](https://github.com/apache/spark/commit/4218b86d5a8ff2321232ff38ed3e1b217ff7db2a). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 Documentation removed as per @srowen 's request in the associated JIRA issue [SPARK-22308] --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #82969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82969/testReport)** for PR 19529 at commit [`4218b86`](https://github.com/apache/spark/commit/4218b86d5a8ff2321232ff38ed3e1b217ff7db2a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user nkronenfeld commented on the issue: https://github.com/apache/spark/pull/19529 nope, using lazy val initialization won't work - at the very least, UnsafeKryoSerializerSuite modifies conf before context construction --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82895/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19529 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19529 **[Test build #82895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82895/testReport)** for PR 19529 at commit [`802a958`](https://github.com/apache/spark/commit/802a958b640067b99fda0b2c8587dea5b8000495). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
