[ https://issues.apache.org/jira/browse/SPARK-28735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-28735: ---------------------------------- Description: Build Spark and run PySpark UT with JDK11. The last commented `assertTrue` failed. {code} $ build/sbt -Phadoop-3.2 test:package $ python/run-tests --testnames 'pyspark.ml.tests.test_algorithms' --python-executables python ... ====================================================================== FAIL: test_raw_and_probability_prediction (pyspark.ml.tests.test_algorithms.MultilayerPerceptronClassifierTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dongjoon/APACHE/spark-master/python/pyspark/ml/tests/test_algorithms.py", line 89, in test_raw_and_probability_prediction self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1E-4)) AssertionError: False is not true {code} {code:python} class MultilayerPerceptronClassifierTest(SparkSessionTestCase): def test_raw_and_probability_prediction(self): data_path = "data/mllib/sample_multiclass_classification_data.txt" df = self.spark.read.format("libsvm").load(data_path) mlp = MultilayerPerceptronClassifier(maxIter=100, layers=[4, 5, 4, 3], blockSize=128, seed=123) model = mlp.fit(df) test = self.sc.parallelize([Row(features=Vectors.dense(0.1, 0.1, 0.25, 0.25))]).toDF() result = model.transform(test).head() expected_prediction = 2.0 expected_probability = [0.0, 0.0, 1.0] expected_rawPrediction = [-11.6081922998, -8.15827998691, 22.17757045] self.assertTrue(result.prediction, expected_prediction) self.assertTrue(np.allclose(result.probability, expected_probability, atol=1E-4)) self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1E-4)) # self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1E-4)) {code} was: Build Spark with JDK11 and run `python/run-tests --testnames 'pyspark.ml.tests.test_algorithms' --python-executables python`. The last commented `assertTrue` failed. - https://github.com/apache/spark/pull/25443/commits/593a154813880fb13e3091043d809e0c00e57bc5 {code:python} class MultilayerPerceptronClassifierTest(SparkSessionTestCase): def test_raw_and_probability_prediction(self): data_path = "data/mllib/sample_multiclass_classification_data.txt" df = self.spark.read.format("libsvm").load(data_path) mlp = MultilayerPerceptronClassifier(maxIter=100, layers=[4, 5, 4, 3], blockSize=128, seed=123) model = mlp.fit(df) test = self.sc.parallelize([Row(features=Vectors.dense(0.1, 0.1, 0.25, 0.25))]).toDF() result = model.transform(test).head() expected_prediction = 2.0 expected_probability = [0.0, 0.0, 1.0] expected_rawPrediction = [-11.6081922998, -8.15827998691, 22.17757045] self.assertTrue(result.prediction, expected_prediction) self.assertTrue(np.allclose(result.probability, expected_probability, atol=1E-4)) self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1E-4)) # self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1E-4)) {code} > MultilayerPerceptronClassifierTest.test_raw_and_probability_prediction fails > on JDK11 > ------------------------------------------------------------------------------------- > > Key: SPARK-28735 > URL: https://issues.apache.org/jira/browse/SPARK-28735 > Project: Spark > Issue Type: Sub-task > Components: PySpark > Affects Versions: 3.0.0 > Reporter: Dongjoon Hyun > Priority: Major > > Build Spark and run PySpark UT with JDK11. The last commented `assertTrue` > failed. > {code} > $ build/sbt -Phadoop-3.2 test:package > $ python/run-tests --testnames 'pyspark.ml.tests.test_algorithms' > --python-executables python > ... > ====================================================================== > FAIL: test_raw_and_probability_prediction > (pyspark.ml.tests.test_algorithms.MultilayerPerceptronClassifierTest) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Users/dongjoon/APACHE/spark-master/python/pyspark/ml/tests/test_algorithms.py", > line 89, in test_raw_and_probability_prediction > self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, > atol=1E-4)) > AssertionError: False is not true > {code} > {code:python} > class MultilayerPerceptronClassifierTest(SparkSessionTestCase): > def test_raw_and_probability_prediction(self): > data_path = "data/mllib/sample_multiclass_classification_data.txt" > df = self.spark.read.format("libsvm").load(data_path) > mlp = MultilayerPerceptronClassifier(maxIter=100, layers=[4, 5, 4, 3], > blockSize=128, seed=123) > model = mlp.fit(df) > test = self.sc.parallelize([Row(features=Vectors.dense(0.1, 0.1, > 0.25, 0.25))]).toDF() > result = model.transform(test).head() > expected_prediction = 2.0 > expected_probability = [0.0, 0.0, 1.0] > expected_rawPrediction = [-11.6081922998, -8.15827998691, > 22.17757045] > self.assertTrue(result.prediction, expected_prediction) > self.assertTrue(np.allclose(result.probability, > expected_probability, atol=1E-4)) > self.assertTrue(np.allclose(result.rawPrediction, > expected_rawPrediction, atol=1E-4)) > # self.assertTrue(np.allclose(result.rawPrediction, > expected_rawPrediction, atol=1E-4)) > {code} -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org