[
https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658479#comment-15658479
]
Saikat Kanjilal commented on SPARK-9487:
----------------------------------------
ok so I've spent the last hour or so doing deeper investigations into the
failures, I used this
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68531/ as a
point of reference, listed below is what I found
java/scala test my master
branch
my feature branch
LogisticRegressionSuite success
success
OneVsRestSuite success
success
DataFrameStatSuite success
success
DataFrameSuite success
success
SQLQueryTestSuite success
success
ForeachSinkSuite success
success
JavaAPISuite
failure
failure
The master branch does not have any code changes from me and the feature branch
of course does
I am running individual tests by issuing commands like the following from the
root directory based on the documentation: ./build/mvn test -P...
-DwildcardSuites=none -Dtest=org.apache.spark.streaming.JavaAPISuite
Therefore my conclusion so far based on the above jenkins report is that my
changes have not introduced any new failures that were not already there,
[~srowen] please let me know if my methodology is off anywhere
> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>
> Key: SPARK-9487
> URL: https://issues.apache.org/jira/browse/SPARK-9487
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, Spark Core, SQL, Tests
> Affects Versions: 1.5.0
> Reporter: Xiangrui Meng
> Labels: starter
> Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
>
>
> In Python we use `local[4]` for unit tests, while in Scala/Java we use
> `local[2]` and `local` for some unit tests in SQL, MLLib, and other
> components. If the operation depends on partition IDs, e.g., random number
> generator, this will lead to different result in Python and Scala/Java. It
> would be nice to use the same number in all unit tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
