[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13101


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220509850
  
Merged into master and branch-2.0. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220502372
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220502374
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58929/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220502291
  
**[Test build #58929 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58929/consoleFull)**
 for PR 13101 at commit 
[`138818b`](https://github.com/apache/spark/commit/138818bc1aba76bc17738612e5bd98538856fef1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `public class JavaMultilayerPerceptronClassifierSuite extends 
SharedSparkSession `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220501406
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220501409
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58928/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220501338
  
**[Test build #58928 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58928/consoleFull)**
 for PR 13101 at commit 
[`9d7a89c`](https://github.com/apache/spark/commit/9d7a89c3e407994dde1cae89a2059f315155aff5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220500920
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220500923
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58923/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220500757
  
**[Test build #58923 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58923/consoleFull)**
 for PR 13101 at commit 
[`65307c5`](https://github.com/apache/spark/commit/65307c506eb0492daed035b52ab560c4cf2556b7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class MultivariateGaussian @Since(\"2.0.0\") (`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220497757
  
LGTM pending Jenkins


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220497096
  
**[Test build #58929 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58929/consoleFull)**
 for PR 13101 at commit 
[`138818b`](https://github.com/apache/spark/commit/138818bc1aba76bc17738612e5bd98538856fef1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[techaddict]

Github user techaddict commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220497162
  
@mengxr @andrewor14 @srowen Addressed all comments. Thanks for reviewing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220497012
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58924/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220497011
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220496921
  
**[Test build #58924 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58924/consoleFull)**
 for PR 13101 at commit 
[`56de3e4`](https://github.com/apache/spark/commit/56de3e41dfa29025f94488143e034eabf8e53b21).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220496492
  
**[Test build #58928 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58928/consoleFull)**
 for PR 13101 at commit 
[`9d7a89c`](https://github.com/apache/spark/commit/9d7a89c3e407994dde1cae89a2059f315155aff5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63978870
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public abstract class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
--- End diff --

Ditto.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63978864
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public abstract class SharedSparkSession {
+
+  public transient SparkSession spark;
--- End diff --

minor: This should be protected.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63978848
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public abstract class SharedSparkSession {
--- End diff --

Shall we include `implements Serializable` to save some code?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63978717
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public abstract class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
--- End diff --

Some tests depend on the default number of partitions. So it would be 
better to say `local[2]` instead of local.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220491675
  
**[Test build #58924 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58924/consoleFull)**
 for PR 13101 at commit 
[`56de3e4`](https://github.com/apache/spark/commit/56de3e41dfa29025f94488143e034eabf8e53b21).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[techaddict]

Github user techaddict commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63977568
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public abstract class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
--- End diff --

Done 👍 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[techaddict]

Github user techaddict commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63977203
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public abstract class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
+  .getOrCreate();
+jsc = new JavaSparkContext(spark.sparkContext());
--- End diff --

all jsc.stop will do is stop the underlying sparkContext which is anyways 
done by sparkSession.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63976971
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public abstract class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
+  .getOrCreate();
+jsc = new JavaSparkContext(spark.sparkContext());
--- End diff --

jsc needs to be stopped too? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63976948
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public abstract class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
--- End diff --

This can set the app name to the name of the test class with 
`getClass.getSimpleName`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220489852
  
**[Test build #58923 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58923/consoleFull)**
 for PR 13101 at commit 
[`65307c5`](https://github.com/apache/spark/commit/65307c506eb0492daed035b52ab560c4cf2556b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220461336
  
Great to see 900 lines were removed:)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63959779
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
+  .getOrCreate();
+jsc = new JavaSparkContext(spark.sparkContext());
+
+customSetUp();
+  }
+
+  protected void customSetUp() throws IOException {}
+
+  @After
+  public void tearDown() {
+spark.stop();
+spark = null;
+
+customTearDown();
--- End diff --

Ditto. It is more like a pattern choice. For tearing down, users might want 
to insert some operations before `spark` is gone out of scope. So let users 
call `super.tearDown()` seems more reasonable here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63959615
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
+  .getOrCreate();
+jsc = new JavaSparkContext(spark.sparkContext());
+
+customSetUp();
+  }
+
+  protected void customSetUp() throws IOException {}
--- End diff --

Shall we let the test to call `super.setUp()` instead?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220459294
  
@techaddict thanks. One more comment. The rest looks pretty good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63958762
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
+  .getOrCreate();
+jsc = new JavaSparkContext(spark.sparkContext());
+
+customSetUp();
+  }
+
+  protected void customSetUp() throws IOException {}
--- End diff --

I talked to @mengxr actually and perhaps a more common practice here is to 
just override `setUp()` and call `super.setUp()` in the first line. Same with 
`tearDown`. Then we don't need these custom things. Also please make this an 
abstract class


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[techaddict]

Github user techaddict commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220442073
  
@andrewor14 comments addressed + ran `lint-java` locally and fixed all the 
issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220441597
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58893/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220441596
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220441446
  
**[Test build #58893 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58893/consoleFull)**
 for PR 13101 at commit 
[`92ae70b`](https://github.com/apache/spark/commit/92ae70ba248af9bbed077353d51677a3dbb2b224).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  case class TaskMetricsUIData(`
  * `  case class InputMetricsUIData(bytesRead: Long, recordsRead: Long)`
  * `  case class OutputMetricsUIData(bytesWritten: Long, recordsWritten: 
Long)`
  * `  case class ShuffleReadMetricsUIData(`
  * `  case class ShuffleWriteMetricsUIData(`
  * `class DenseMatrix @Since(\"2.0.0\") (`
  * `class SparseMatrix @Since(\"2.0.0\") (`
  * `class DenseVector @Since(\"2.0.0\") ( @Since(\"2.0.0\") val values: 
Array[Double]) extends Vector `
  * `class SparseVector @Since(\"2.0.0\") (`
  * `class QueryExecutionException(CapturedException):`
  * `trait ObjectProducerExec extends SparkPlan `
  * `trait ObjectConsumerExec extends UnaryExecNode `
  * `case class DeserializeToObjectExec(`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220431087
  
**[Test build #58893 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58893/consoleFull)**
 for PR 13101 at commit 
[`92ae70b`](https://github.com/apache/spark/commit/92ae70ba248af9bbed077353d51677a3dbb2b224).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220416568
  
@techaddict can you run `dev/lint-java` locally? Jenkins doesn't run that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63933070
  
--- Diff: mllib/src/test/java/org/apache/spark/ml/JavaPipelineSuite.java ---
@@ -17,47 +17,30 @@
 
 package org.apache.spark.ml;
 
-import org.junit.After;
-import org.junit.Before;
 import org.junit.Test;
 
+import org.apache.spark.SharedSparkSession;
 import org.apache.spark.api.java.JavaRDD;
-import org.apache.spark.api.java.JavaSparkContext;
 import org.apache.spark.ml.classification.LogisticRegression;
 import static 
org.apache.spark.ml.classification.LogisticRegressionSuite.generateLogisticInputAsList;
 import org.apache.spark.ml.feature.LabeledPoint;
 import org.apache.spark.ml.feature.StandardScaler;
 import org.apache.spark.sql.Dataset;
 import org.apache.spark.sql.Row;
-import org.apache.spark.sql.SparkSession;
 
 /**
  * Test Pipeline construction and fitting in Java.
  */
-public class JavaPipelineSuite {
+public class JavaPipelineSuite extends SharedSparkSession {
 
-  private transient SparkSession spark;
-  private transient JavaSparkContext jsc;
   private transient Dataset dataset;
 
-  @Before
-  public void setUp() {
-spark = SparkSession.builder()
-  .master("local")
-  .appName("JavaPipelineSuite")
-  .getOrCreate();
-jsc = new JavaSparkContext(spark.sparkContext());
+  public void customSetUp() {
--- End diff --

Please add `@Override` everywhere we do this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63932970
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
+  .getOrCreate();
+jsc = new JavaSparkContext(spark.sparkContext());
+
+customSetUp();
+customSetUpWithException();
+  }
+
+  public void customSetUp() {}
+
+  // TODO: Remove this once we have a way to use customSetUp that Exception
+  public void customSetUpWithException() throws IOException {}
--- End diff --

this is really weird... just make the normal `doSetup` throw `IOException`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/13101#discussion_r63932881
  
--- Diff: mllib/src/test/java/org/apache/spark/SharedSparkSession.java ---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import java.io.IOException;
+
+import org.junit.After;
+import org.junit.Before;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+
+public class SharedSparkSession {
+
+  public transient SparkSession spark;
+  public transient JavaSparkContext jsc;
+
+  @Before
+  public void setUp() throws IOException {
+spark = SparkSession.builder()
+  .master("local")
+  .appName("shared-spark-session")
+  .getOrCreate();
+jsc = new JavaSparkContext(spark.sparkContext());
+
+customSetUp();
+customSetUpWithException();
+  }
+
+  public void customSetUp() {}
--- End diff --

I would call this `doSetup` and `doTearDown`. Also many of these can be 
protected so please mark them as such.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[techaddict]

Github user techaddict commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-220348713
  
ping @srowen @andrewor14 @mengxr 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219915352
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58737/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219915350
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219915297
  
**[Test build #58737 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58737/consoleFull)**
 for PR 13101 at commit 
[`f3fa1f5`](https://github.com/apache/spark/commit/f3fa1f5b84873eead74157dacd53a2765a957245).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219910742
  
**[Test build #58737 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58737/consoleFull)**
 for PR 13101 at commit 
[`f3fa1f5`](https://github.com/apache/spark/commit/f3fa1f5b84873eead74157dacd53a2765a957245).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[techaddict]

Github user techaddict commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219801934
  
ping @andrewor14 @mengxr 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[techaddict]

Github user techaddict commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219100845
  
cc: @andrewor14 @andrewor14 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219100659
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219100660
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58582/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219100517
  
**[Test build #58582 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58582/consoleFull)**
 for PR 13101 at commit 
[`12ba028`](https://github.com/apache/spark/commit/12ba02811d375d8360bbb6d71631f111a279ce67).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219091420
  
**[Test build #58582 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58582/consoleFull)**
 for PR 13101 at commit 
[`12ba028`](https://github.com/apache/spark/commit/12ba02811d375d8360bbb6d71631f111a279ce67).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219089137
  
**[Test build #58581 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58581/consoleFull)**
 for PR 13101 at commit 
[`c181fa6`](https://github.com/apache/spark/commit/c181fa6de0d9713c313437a48f5d0e134a3e66dc).
 * This patch **fails RAT tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219089147
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58581/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219089145
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13101#issuecomment-219088964
  
**[Test build #58581 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58581/consoleFull)**
 for PR 13101 at commit 
[`c181fa6`](https://github.com/apache/spark/commit/c181fa6de0d9713c313437a48f5d0e134a3e66dc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15296][MLlib] Refactor All Java Tests t...

[techaddict]

GitHub user techaddict opened a pull request:

https://github.com/apache/spark/pull/13101

[SPARK-15296][MLlib] Refactor All Java Tests that use SparkSession

## What changes were proposed in this pull request?
Refactor All Java Tests that use SparkSession, to extend SharedSparkSesion

## How was this patch tested?
Existing Tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/techaddict/spark SPARK-15296

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13101.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13101


commit a7125a7a70363d337bd5da81843deb2d38383bcd
Author: Sandeep Singh 
Date:   2016-05-13T13:10:14Z

ML Classification done

commit 4a72c57f48f8f45657871fe308f4d7bfd35e5fa8
Author: Sandeep Singh 
Date:   2016-05-13T14:16:19Z

ml clustering done

commit 151b1cfcca30f95375f89fbe808d3ec85c05f7c9
Author: Sandeep Singh 
Date:   2016-05-13T14:37:05Z

ML Feature done

commit 97574ec8ef83a043049b502a86142ff57bc01e38
Author: Sandeep Singh 
Date:   2016-05-13T14:41:06Z

ML Feature remove unused imports

commit a2348731b61afc82fac8c30c7cd5b7a07ff1e9fa
Author: Sandeep Singh 
Date:   2016-05-13T14:41:36Z

ML Param Done - Remove SparkSession, esc since not used

commit a00e7fb80ccb600ddcf1eba89aee5983475239c5
Author: Sandeep Singh 
Date:   2016-05-13T15:16:07Z

ml regression done

commit 1ccdb7d3494c4c919740303d32aeb0f6d4317fef
Author: Sandeep Singh 
Date:   2016-05-13T15:25:38Z

ml libsvm done

commit 89497326ae080c275c4c1528c28c6ee594a21b23
Author: Sandeep Singh 
Date:   2016-05-13T15:29:44Z

ml tuning, util done

commit 03d1951a427e9b7ecde67d33e3ea955cba8907be
Author: Sandeep Singh 
Date:   2016-05-13T15:42:26Z

mllib classification

commit a90cbb30b2ef669a515c846e8fe264b859791d3a
Author: Sandeep Singh 
Date:   2016-05-13T15:45:43Z

mllib clustering

commit afc9a567d983957d394882821b52686c77bef8d7
Author: Sandeep Singh 
Date:   2016-05-13T15:49:48Z

mllib evaluation and feature

commit 83ae1149cfb3901a4efdf06e0b8b2bc25f4ef343
Author: Sandeep Singh 
Date:   2016-05-13T15:55:40Z

mllib fpm

commit 4b77a74d14cd512904c66eae6bcd2fc39a39ca6e
Author: Sandeep Singh 
Date:   2016-05-13T15:56:50Z

mllib random

commit 6457f23194d1f11375170c144cbaaf05db9199b4
Author: Sandeep Singh 
Date:   2016-05-13T15:58:14Z

mllib recommendation

commit 28aebf6a06f6ac11e22d47857eeae099a5e78212
Author: Sandeep Singh 
Date:   2016-05-13T16:00:47Z

mllib regression

commit 0a8f6cbf698558000ef5ede4ea5486f727684f63
Author: Sandeep Singh 
Date:   2016-05-13T16:04:37Z

mllib tree




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org