[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283941#comment-14283941 ] Sean Owen commented on SPARK-4442: -- [~matthewcornell] Normally I'd say you don't need to build any JARs yourself, and shouldn't bother manually managing JARs; just use Maven or SBT and write in the dependencies you want. But I see that Spark doesn't actually publish test artifacts. (Which to be fair would be unusual. But [~joshrosen] is that not the simplest way to expose this?). You can mvn package as shown on the Building Spark documentation, and you'll end up with a bunch of artifacts in core/target, including the test JAR file containing Spark's test code and thus any utility code you want from there. Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283993#comment-14283993 ] Matthew Cornell commented on SPARK-4442: [~srowen] Thanks for the tip. I tried compiled 1.2.0 using this command: $ mvn package -DskipTests But I could not find 'LocalSparkContext' in any jar: $ find . -iname '*.jar' | xargs grep -i 'LocalSparkContext' I'm recompiling without -DskipTests (it's taking a while) - would that cause anything to be added? Once the build is done I'll past the output. Until then - am I missing something that would cause the tests to be excluded? Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[
https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284054#comment-14284054
]
Sean Owen commented on SPARK-4442:
--
Hm, no works for me. Maybe {{mvn -DskipTests install}} the entire project
first? although I wouldn't think that's necessary. Also I'm working off
{{master}} although again should be the same thing from any release.
Move common unit test utilities into their own package / module
---
Key: SPARK-4442
URL: https://issues.apache.org/jira/browse/SPARK-4442
Project: Spark
Issue Type: Improvement
Reporter: Josh Rosen
Priority: Minor
We should move generally-useful unit test fixtures / utility methods to their
own test utilities set package / module to make them easier to find / use.
See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for
one example of this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284069#comment-14284069 ] Matthew Cornell commented on SPARK-4442: Thanks for sticking with me on this, Sean! I tried again from scratch with no luck. Maybe the downloaded sources have something crucial missing from master? Here's what I did: # start with extracting http://apache.spinellicreations.com/spark/spark-1.2.0/spark-1.2.0.tgz $ cd /Users/cornell/Downloads/spark-1.2.0/ $ mvn -DskipTests install $ cd core $ mvn jar:test-jar - same warning: [WARNING] JAR will be empty - no content was marked for inclusion! Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[
https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284013#comment-14284013
]
Sean Owen commented on SPARK-4442:
--
[~matthewcornell] Oops, I missed again. The test JARs aren't configured to be
generated by the build as-is. But you can simply do this in {{core/}}:
{code}
mvn jar:test-jar
...
jar tf target/spark-core_2.10-1.3.0-SNAPSHOT-tests.jar | grep
LocalSparkContext
org/apache/spark/LocalSparkContext$.class
org/apache/spark/LocalSparkContext$class.class
org/apache/spark/LocalSparkContext.class
{code}
Move common unit test utilities into their own package / module
---
Key: SPARK-4442
URL: https://issues.apache.org/jira/browse/SPARK-4442
Project: Spark
Issue Type: Improvement
Reporter: Josh Rosen
Priority: Minor
We should move generally-useful unit test fixtures / utility methods to their
own test utilities set package / module to make them easier to find / use.
See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for
one example of this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284052#comment-14284052 ] Matthew Cornell commented on SPARK-4442: [~srowen] I might have misunderstood. I tried this: $ cd dir/spark-1.2.0/core/ $ mvn jar:test-jar But it says it created an empty jar (see output below). Any ideas re: what I'm doing wrong? [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building Spark Project Core 1.2.0 [INFO] [INFO] [INFO] --- maven-jar-plugin:2.4:test-jar (default-cli) @ spark-core_2.10 --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 1.585 s [INFO] Finished at: 2015-01-20T12:13:57-05:00 [INFO] Final Memory: 10M/81M [INFO] Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283920#comment-14283920 ] Matthew Cornell commented on SPARK-4442: Please, as a new Spark (and Maven and SBT) user, having a jar I could simply drop into my IntelliJ project would be a life saver. Until then, would someone please sketch a little detail on how I could build the jar using the 1.2.0 sources? Thanks! Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284210#comment-14284210 ] Matthew Cornell commented on SPARK-4442: OK, progress: I cloned master and re-ran the commands and did end up getting spark-core_2.10-1.3.0-SNAPSHOT-tests.jar, which does contain LocalSparkContext. So I guess I've upgraded to 1.3.0 :-) Question, please: There is as second LocalSparkContext defined in: graphx/src/test/scala/org/apache/spark/graphx/LocalSparkContext.scala that did not get included in the mvn jar:test-jar command. I looked at pom.xml to try to figure out what that argument does, but all I found was a profile called 'java8-tests'. I couldn't find anywhere that mentioned: dirspark/core/src/test/scala/org/apache/spark/LocalSparkContext.scala Any pointers would be appreciated! Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284216#comment-14284216 ] Sean Owen commented on SPARK-4442: -- I don't think 1.3.0 should be different in this regard, or any release. The command is going to JAR up compiled test classes, so it's necessary for tests to be compiled first, but, install should have done that. You're referring to another class in the graphx module, so that won't be part of core's test code. java8-tests is not related. I'm not sure what you are looking for in the POM? Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284251#comment-14284251 ] Matthew Cornell commented on SPARK-4442: I agree they should be the same, but I got different results for whatever reason. Maybe I'll try it again, but the commands were the same. I looked in the POM because I thought that's were 'mvn jar:test-jar' goes. I've never used maven before, so I was just trying to figure it out before asking. I guess it's a command to a Maven jar plugin, but I don't understand how it knows where the test sources are. Because the core module isn't passed explicitly to the jar command, I'm guessing it's a default somewhere (in the POM?) How would I tell the jar command to run on the graphx module? Thank you. Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284259#comment-14284259 ] Sean Owen commented on SPARK-4442: -- That's just standard Maven stuff -- standard JAR plugin and its goals. It uses the pom in the current directory by default. You can just go to the graphx/ dir. Running from the top level dir will invoke the target for everything since the parent has refs to all child modules. Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218704#comment-14218704 ] Josh Rosen commented on SPARK-4442: --- There's a few generally useful test utility functions / classes that are parts of individual suites rather than being located in easier-to-find places. This proposal was to find and relocate a bunch of those small utilities so that they get re-used. For example, I have a fixture for modifying and re-setting system properties in unit tests. Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4442) Move common unit test utilities into their own package / module
[ https://issues.apache.org/jira/browse/SPARK-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214461#comment-14214461 ] Sean Owen commented on SPARK-4442: -- You can already depend on just core's test code from other modules' test code. Is this more than that? Move common unit test utilities into their own package / module --- Key: SPARK-4442 URL: https://issues.apache.org/jira/browse/SPARK-4442 Project: Spark Issue Type: Improvement Reporter: Josh Rosen Priority: Minor We should move generally-useful unit test fixtures / utility methods to their own test utilities set package / module to make them easier to find / use. See https://github.com/apache/spark/pull/3121#discussion-diff-20413659 for one example of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
