[
https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15651496#comment-15651496
]
Saikat Kanjilal commented on SPARK-9487:
----------------------------------------
ok I have moved onto python, I am attaching a log that contains test errors
upon changing local[2] to local[4] on the ml module in python
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
[Stage 49:> (0 + 3) /
3]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
**********************************************************************
File "/Users/skanjila/code/opensource/spark/python/pyspark/ml/clustering.py",
line 98, in __main__.GaussianMixture
Failed example:
model.gaussiansDF.show()
Expected:
+--------------------+--------------------+
| mean| cov|
+--------------------+--------------------+
|[0.82500000140229...|0.005625000000006...|
|[-0.4777098016092...|0.167969502720916...|
|[-0.4472625243352...|0.167304119758233...|
+--------------------+--------------------+
...
Got:
+--------------------+--------------------+
| mean| cov|
+--------------------+--------------------+
|[-0.6158006194417...|0.132188091748508...|
|[0.54523101952701...|0.159129291449328...|
|[0.54042985246699...|0.161430620150745...|
+--------------------+--------------------+
<BLANKLINE>
**********************************************************************
File "/Users/skanjila/code/opensource/spark/python/pyspark/ml/clustering.py",
line 123, in __main__.GaussianMixture
Failed example:
model2.gaussiansDF.show()
Expected:
+--------------------+--------------------+
| mean| cov|
+--------------------+--------------------+
|[0.82500000140229...|0.005625000000006...|
|[-0.4777098016092...|0.167969502720916...|
|[-0.4472625243352...|0.167304119758233...|
+--------------------+--------------------+
...
Got:
+--------------------+--------------------+
| mean| cov|
+--------------------+--------------------+
|[-0.6158006194417...|0.132188091748508...|
|[0.54523101952701...|0.159129291449328...|
|[0.54042985246699...|0.161430620150745...|
+--------------------+--------------------+
<BLANKLINE>
**********************************************************************
File "/Users/skanjila/code/opensource/spark/python/pyspark/ml/clustering.py",
line 656, in __main__.LDA
Failed example:
model.describeTopics().show()
Expected:
+-----+-----------+--------------------+
|topic|termIndices| termWeights|
+-----+-----------+--------------------+
| 0| [1, 0]|[0.50401530077160...|
| 1| [0, 1]|[0.50401530077160...|
+-----+-----------+--------------------+
...
Got:
+-----+-----------+--------------------+
|topic|termIndices| termWeights|
+-----+-----------+--------------------+
| 0| [1, 0]|[0.50010191915681...|
| 1| [0, 1]|[0.50010191915681...|
+-----+-----------+--------------------+
<BLANKLINE>
**********************************************************************
File "/Users/skanjila/code/opensource/spark/python/pyspark/ml/clustering.py",
line 664, in __main__.LDA
Failed example:
model.topicsMatrix()
Expected:
DenseMatrix(2, 2, [0.496, 0.504, 0.504, 0.496], 0)
Got:
DenseMatrix(2, 2, [0.4999, 0.5001, 0.5001, 0.4999], 0)
**********************************************************************
2 items had failures:
2 of 21 in __main__.GaussianMixture
2 of 20 in __main__.LDA
***Test Failed*** 4 failures.
[~srowen][~holdenk] thoughts on next steps, should this pull request also
contain code fixes to fix the errors that occur when changing local[2] to
local[4] or should we break up the pull request into subcomponents, one focused
on the scala pieces already submitted and the next focused on fixing the python
code to work with local[4], thoughts on next steps?
> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>
> Key: SPARK-9487
> URL: https://issues.apache.org/jira/browse/SPARK-9487
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, Spark Core, SQL, Tests
> Affects Versions: 1.5.0
> Reporter: Xiangrui Meng
> Labels: starter
> Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
>
>
> In Python we use `local[4]` for unit tests, while in Scala/Java we use
> `local[2]` and `local` for some unit tests in SQL, MLLib, and other
> components. If the operation depends on partition IDs, e.g., random number
> generator, this will lead to different result in Python and Scala/Java. It
> would be nice to use the same number in all unit tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
