[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587627#comment-15587627 ]
Saikat Kanjilal edited comment on SPARK-9487 at 10/19/16 4:24 AM: ------------------------------------------------------------------ [~holdenk] finally getting time to look at this, so I am starting small, I made the change inside ContextCleanerSuite and HeartbeatReceiverSuite from local[2] tp local[4], per the documentation here (http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version) I ran mvn -P hadoop2 -Dsuites= org.apache.spark.HeartbeatReceiverSuite test--- looks like everything worked I then ran mvn -P hadoop2 -Dsuites= org.apache.spark.ContextCleanerSuite test-- looks like everything worked as well See the attachments and let me know if this is the right process to run single unit tests, I'll start making changes to the other Suites , how would you like to see the output, should I just have attachments or just do a pull request from the new branch that I created? Thanks PS Another question, running single unit tests like this takes forever, are there flags I can set to speed up the builds, even on my 15 inch macbook pro with SSD the builds shouldnt take this long :(. Let me know next steps to get this into a PR. was (Author: kanjilal): [~holdenk] finally getting time to look at this, so I am starting small, I made the change inside ContextCleanerSuite and HeartbeatReceiverSuite from local[2] tp local[4], per the documentation here (http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version) I ran mvn -P hadoop2 -Dsuites= org.apache.spark.HeartbeatReceiverSuite test--- looks like everything worked I then ran mvn -P hadoop2 -Dsuites= org.apache.spark.ContextCleanerSuite test-- looks like everything worked as well See the attachments and let me know if this is this is the right process to run single unit tests, if not I'll start making changes to the other Suites , how would you like to see the output, should I just have attachments or just do a pull request from the new branch that I created? Thanks PS Another question, running single unit tests like this takes forever, are there flags I can set to speed up the builds, even on my 15 inch macbook pro with SSD the builds shouldnt take this long :(. Let me know next steps to get this into a PR. > Use the same num. worker threads in Scala/Python unit tests > ----------------------------------------------------------- > > Key: SPARK-9487 > URL: https://issues.apache.org/jira/browse/SPARK-9487 > Project: Spark > Issue Type: Improvement > Components: PySpark, Spark Core, SQL, Tests > Affects Versions: 1.5.0 > Reporter: Xiangrui Meng > Labels: starter > Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults > > > In Python we use `local[4]` for unit tests, while in Scala/Java we use > `local[2]` and `local` for some unit tests in SQL, MLLib, and other > components. If the operation depends on partition IDs, e.g., random number > generator, this will lead to different result in Python and Scala/Java. It > would be nice to use the same number in all unit tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org