[GitHub] [spark] AmplabJenkins removed a comment on pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

[GitBox]

AmplabJenkins removed a comment on pull request #28968:
URL: https://github.com/apache/spark/pull/28968#issuecomment-663814316







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

[GitBox]

AmplabJenkins commented on pull request #28968:
URL: https://github.com/apache/spark/pull/28968#issuecomment-663814316







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

[GitBox]

SparkQA removed a comment on pull request #28968:
URL: https://github.com/apache/spark/pull/28968#issuecomment-663812052


   **[Test build #126528 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126528/testReport)**
 for PR 28968 at commit 
[`a78fd43`](https://github.com/apache/spark/commit/a78fd4314ba39d1feb63ba1539ac9a2acf40de77).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

[GitBox]

SparkQA commented on pull request #28968:
URL: https://github.com/apache/spark/pull/28968#issuecomment-663814251


   **[Test build #126528 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126528/testReport)**
 for PR 28968 at commit 
[`a78fd43`](https://github.com/apache/spark/commit/a78fd4314ba39d1feb63ba1539ac9a2acf40de77).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `class InheritableThread(threading.Thread):`



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins removed a comment on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663812247







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663812247







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

SparkQA commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663812083


   **[Test build #126522 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126522/testReport)**
 for PR 29227 at commit 
[`5ddfdbf`](https://github.com/apache/spark/commit/5ddfdbf62d6bd891e924e9b7579c3405de04a1d8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

SparkQA removed a comment on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663799771


   **[Test build #126522 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126522/testReport)**
 for PR 29227 at commit 
[`5ddfdbf`](https://github.com/apache/spark/commit/5ddfdbf62d6bd891e924e9b7579c3405de04a1d8).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins removed a comment on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663812031







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663812031







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

[GitBox]

SparkQA commented on pull request #28968:
URL: https://github.com/apache/spark/pull/28968#issuecomment-663812052


   **[Test build #126528 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126528/testReport)**
 for PR 28968 at commit 
[`a78fd43`](https://github.com/apache/spark/commit/a78fd4314ba39d1feb63ba1539ac9a2acf40de77).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

SparkQA removed a comment on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663798988


   **[Test build #126521 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126521/testReport)**
 for PR 29227 at commit 
[`614d293`](https://github.com/apache/spark/commit/614d2934afb37af6adcbb8e5180f4976223df93b).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

SparkQA commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663811862


   **[Test build #126521 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126521/testReport)**
 for PR 29227 at commit 
[`614d293`](https://github.com/apache/spark/commit/614d2934afb37af6adcbb8e5180f4976223df93b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

[GitBox]

AmplabJenkins removed a comment on pull request #28968:
URL: https://github.com/apache/spark/pull/28968#issuecomment-663811589







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

[GitBox]

AmplabJenkins commented on pull request #28968:
URL: https://github.com/apache/spark/pull/28968#issuecomment-663811589







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28968: [SPARK-32010][PYTHON][CORE] Add InheritableThread for local properties and fixing a thread leak issue in pinned thread mode

[GitBox]

HyukjinKwon commented on pull request #28968:
URL: https://github.com/apache/spark/pull/28968#issuecomment-663811508


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29222: [SPARK-32428] [EXAMPLES] Make BinaryClassificationMetricsExample cons…

[GitBox]

HyukjinKwon commented on pull request #29222:
URL: https://github.com/apache/spark/pull/29222#issuecomment-663809518


   +1



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29014: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning

[GitBox]

AmplabJenkins commented on pull request #29014:
URL: https://github.com/apache/spark/pull/29014#issuecomment-663809218







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29014: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning

[GitBox]

AmplabJenkins removed a comment on pull request #29014:
URL: https://github.com/apache/spark/pull/29014#issuecomment-663809218







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29014: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning

[GitBox]

SparkQA commented on pull request #29014:
URL: https://github.com/apache/spark/pull/29014#issuecomment-663809146


   **[Test build #126527 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126527/testReport)**
 for PR 29014 at commit 
[`fb8f39d`](https://github.com/apache/spark/commit/fb8f39d34dabd4429390d79919256fd4244ddba6).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29215: [SPARK-32363][PYTHON][BUILD][3.0] Fix flakiness in pip package testing in Jenkins

[GitBox]

HyukjinKwon commented on pull request #29215:
URL: https://github.com/apache/spark/pull/29215#issuecomment-663809125


   Thank you @dongjoon-hyun.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins removed a comment on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663808753


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126526/
   Test FAILed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

SparkQA removed a comment on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663808556


   **[Test build #126526 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126526/testReport)**
 for PR 29226 at commit 
[`288c34e`](https://github.com/apache/spark/commit/288c34ebaaec481f0fdf0db1938cf39e3970eb6d).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] xuanyuanking commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

[GitBox]

xuanyuanking commented on a change in pull request #24173:
URL: https://github.com/apache/spark/pull/24173#discussion_r460363515



##
File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
##
@@ -509,4 +549,3 @@ object StateStore extends Logging {
 }
   }
 }
-

Review comment:
   nit for the last empty line.

##
File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
##
@@ -391,10 +399,18 @@ object StateStore extends Logging {
 require(version >= 0)
 val storeProvider = loadedProviders.synchronized {
   startMaintenanceIfNeeded()
+
+  val newProvIdSchemaCheck = 
StateStoreProviderId.withNoPartitionInformation(storeProviderId)
+  if (!schemaValidated.contains(newProvIdSchemaCheck)) {

Review comment:
   Seems we should check `stateSchemaCheckEnabled` here. Then if it's 
disabled, we will not send extra RPC requests?

##
File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
##
@@ -407,6 +423,12 @@ object StateStore extends Logging {
 loadedProviders.remove(storeProviderId).foreach(_.close())
   }
 
+  /** Unload all state store providers: unit test purpose */
+  private[sql] def unloadAll(): Unit = loadedProviders.synchronized {

Review comment:
   Ditto, if we eagerly check the `stateSchemaCheckEnabled` config, then 
this test specific function also can be removed and we can use the config to 
control the behavior in the test?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins removed a comment on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663808748


   Merged build finished. Test FAILed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

SparkQA commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663808742


   **[Test build #126526 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126526/testReport)**
 for PR 29226 at commit 
[`288c34e`](https://github.com/apache/spark/commit/288c34ebaaec481f0fdf0db1938cf39e3970eb6d).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663808748







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins removed a comment on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663808623







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663808623







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

SparkQA commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663808556


   **[Test build #126526 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126526/testReport)**
 for PR 29226 at commit 
[`288c34e`](https://github.com/apache/spark/commit/288c34ebaaec481f0fdf0db1938cf39e3970eb6d).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

[GitBox]

HyukjinKwon commented on a change in pull request #29121:
URL: https://github.com/apache/spark/pull/29121#discussion_r460362434



##
File path: python/pyspark/ml/tests/test_stat.py
##
@@ -40,7 +40,7 @@ def test_chisquaretest(self):
 
 
 if __name__ == "__main__":
-from pyspark.ml.tests.test_stat import *
+from pyspark.ml.tests.test_stat import *  # noqa: F401

Review comment:
   No, no. don't remove this. It will keep the source code path in console 
output. It is also used in coverage report as well. XML reporter is used in 
Jenkins.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

[GitBox]

HyukjinKwon commented on a change in pull request #29121:
URL: https://github.com/apache/spark/pull/29121#discussion_r460362184



##
File path: python/pyspark/heapq3.py
##
@@ -865,7 +865,7 @@ def nlargest(n, iterable, key=None):
 
 # If available, use C implementation
 try:
-from _heapq import *
+from _heapq import *  # noqa: F401

Review comment:
   This too. It was ported back from CPython 3, and is excluded in other 
linter checks. Can we exclude this file as well?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

[GitBox]

HyukjinKwon commented on a change in pull request #29121:
URL: https://github.com/apache/spark/pull/29121#discussion_r460361924



##
File path: python/pyspark/cloudpickle/cloudpickle.py
##
@@ -57,7 +57,6 @@
 from .compat import pickle
 from typing import Generic, Union, Tuple, Callable
 from pickle import _getattribute
-from importlib._bootstrap import _find_spec

Review comment:
   @Fokko, can we exclude this file in the linter checking? We're already 
skipping other lint checks (see `dev/tox.ini`) and this is usually exactly 
matched to the release of cloudpickle.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29218: [SPARK-32408][BUILD] Enable crossPaths back to prevent side effects

[GitBox]

HyukjinKwon commented on pull request #29218:
URL: https://github.com/apache/spark/pull/29218#issuecomment-663805469


   Thank you @dongjoon-hyun.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test

[GitBox]

HyukjinKwon closed pull request #29212:
URL: https://github.com/apache/spark/pull/29212


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test

[GitBox]

HyukjinKwon commented on pull request #29212:
URL: https://github.com/apache/spark/pull/29212#issuecomment-663805318


   Merged to master.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test

[GitBox]

HyukjinKwon commented on pull request #29212:
URL: https://github.com/apache/spark/pull/29212#issuecomment-663805260


   @dongjoon-hyun and @srowen, let me merge this to make GitHub Actions 
correctly tests pip packaging. At least it fixes the issue as shown above and I 
am pretty sure this fix is correct. This is also test-only and rather minor. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils

[GitBox]

HyukjinKwon edited a comment on pull request #29217:
URL: https://github.com/apache/spark/pull/29217#issuecomment-663804755


   I think it's okay. Installing Python 3.8 in GitHub Actions overwrites 
`python3`. So, we won't face the tests being skipped even when we backport as 
long as there's the explicit Python 3 installation.
   
   I wouldn't say this is only a GitHub specific issue. It was `python3.6` in 
the codes for Jenkins environment when I added. It was to explicitly pick a 
Python version but we're already using `python3` to pick Python 3 in many 
places in our script.
   
   It fixes the issue in GitHub Actions but also keep the codes consistent with 
other places. Plus, it will reduce overhead of maintenance when we drop or 
deprecate minor Python version. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils

[GitBox]

HyukjinKwon edited a comment on pull request #29217:
URL: https://github.com/apache/spark/pull/29217#issuecomment-663804755


   I think it's okay. Installing Python 3.8 in GitHub Actions overwrites 
`python3`. So, we won't face the tests being skipped even when we backport as 
long as there's the explicit Python 3 installation.
   
   I wouldn't say this is only a GitHub specific issue. It was `python3.6` in 
the codes for Jenkins environment when I added. It was to explicitly pick a 
Python version but we're already using `python3` to pick Python 3 in many 
places like our dev scripts.
   
   It fixes the issue in GitHub Actions but also keep the codes consistent with 
other places. Plus, it will reduce overhead of maintenance when we drop or 
deprecate minor Python version. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils

[GitBox]

HyukjinKwon commented on pull request #29217:
URL: https://github.com/apache/spark/pull/29217#issuecomment-663804755


   I think it's okay. Installing Python 3.8 in GitHub Actions overwrites 
`python3`. So, we won't face the tests being skipped even when we backport as 
long as the explicit Python 3 installation.
   
   I wouldn't say this is a GitHub specific issues. It was `python3.6` in the 
codes for Jenkins environment when I added. It was to explicitly pick a Python 
version but we're already using `python3` to pick Python 3 in many places in 
our script.
   
   It fixes the issue in GitHub Actions but also keep the codes consistent with 
other places. Plus, it will reduce overhead of maintenance when we drop or 
deprecate minor Python version. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils

[GitBox]

HyukjinKwon edited a comment on pull request #29217:
URL: https://github.com/apache/spark/pull/29217#issuecomment-663804755


   I think it's okay. Installing Python 3.8 in GitHub Actions overwrites 
`python3`. So, we won't face the tests being skipped even when we backport as 
long as there's the explicit Python 3 installation.
   
   I wouldn't say this is a GitHub specific issues. It was `python3.6` in the 
codes for Jenkins environment when I added. It was to explicitly pick a Python 
version but we're already using `python3` to pick Python 3 in many places in 
our script.
   
   It fixes the issue in GitHub Actions but also keep the codes consistent with 
other places. Plus, it will reduce overhead of maintenance when we drop or 
deprecate minor Python version. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins removed a comment on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663803486







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663803486







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

SparkQA removed a comment on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663788798


   **[Test build #126519 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126519/testReport)**
 for PR 29227 at commit 
[`a445d2f`](https://github.com/apache/spark/commit/a445d2ffc9d6cd5c5406fc5fc3fc2f393e34daf1).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

SparkQA commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663803331


   **[Test build #126519 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126519/testReport)**
 for PR 29227 at commit 
[`a445d2f`](https://github.com/apache/spark/commit/a445d2ffc9d6cd5c5406fc5fc3fc2f393e34daf1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

[GitBox]

AmplabJenkins removed a comment on pull request #29121:
URL: https://github.com/apache/spark/pull/29121#issuecomment-663803155







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

[GitBox]

AmplabJenkins commented on pull request #29121:
URL: https://github.com/apache/spark/pull/29121#issuecomment-663803155







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

[GitBox]

SparkQA commented on pull request #29121:
URL: https://github.com/apache/spark/pull/29121#issuecomment-663803067


   **[Test build #126525 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126525/testReport)**
 for PR 29121 at commit 
[`582a7ef`](https://github.com/apache/spark/commit/582a7ef4af4ddeabda6cc449d66787364deaeaae).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

AmplabJenkins removed a comment on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663802552







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29121: [SPARK-32319][PYSPARK] Remove unused imports

[GitBox]

dongjoon-hyun commented on pull request #29121:
URL: https://github.com/apache/spark/pull/29121#issuecomment-663802581


   Retest this please.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

AmplabJenkins commented on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663802552







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] agrawaldevesh commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

agrawaldevesh commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663802421


   Sorry it didn't work. I will make another go at it. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

SparkQA commented on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663802397


   **[Test build #126524 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126524/testReport)**
 for PR 29228 at commit 
[`acb4e80`](https://github.com/apache/spark/commit/acb4e808b0f2e2ca0c0c72a7aa6f1378d10da62f).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29002: [SPARK-32175][CORE] Fix the order between initialization for ExecutorPlugin and starting heartbeat thread

[GitBox]

AmplabJenkins removed a comment on pull request #29002:
URL: https://github.com/apache/spark/pull/29002#issuecomment-663802208


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126520/
   Test FAILed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29002: [SPARK-32175][CORE] Fix the order between initialization for ExecutorPlugin and starting heartbeat thread

[GitBox]

AmplabJenkins removed a comment on pull request #29002:
URL: https://github.com/apache/spark/pull/29002#issuecomment-663802206


   Merged build finished. Test FAILed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29002: [SPARK-32175][CORE] Fix the order between initialization for ExecutorPlugin and starting heartbeat thread

[GitBox]

AmplabJenkins commented on pull request #29002:
URL: https://github.com/apache/spark/pull/29002#issuecomment-663802206







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29002: [SPARK-32175][CORE] Fix the order between initialization for ExecutorPlugin and starting heartbeat thread

[GitBox]

SparkQA removed a comment on pull request #29002:
URL: https://github.com/apache/spark/pull/29002#issuecomment-663790544


   **[Test build #126520 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126520/testReport)**
 for PR 29002 at commit 
[`0b8da96`](https://github.com/apache/spark/commit/0b8da9620e736fc5cb485c349866593033b1fe33).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29002: [SPARK-32175][CORE] Fix the order between initialization for ExecutorPlugin and starting heartbeat thread

[GitBox]

SparkQA commented on pull request #29002:
URL: https://github.com/apache/spark/pull/29002#issuecomment-663802095


   **[Test build #126520 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126520/testReport)**
 for PR 29002 at commit 
[`0b8da96`](https://github.com/apache/spark/commit/0b8da9620e736fc5cb485c349866593033b1fe33).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] titsuki commented on pull request #29222: [SPARK-32428] [EXAMPLES] Make BinaryClassificationMetricsExample cons…

[GitBox]

titsuki commented on pull request #29222:
URL: https://github.com/apache/spark/pull/29222#issuecomment-663801010


   @huaxingao @srowen 
   Thanks for your reviews! I've also fixed the other examples.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

AmplabJenkins removed a comment on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663800607


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126523/
   Test FAILed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

AmplabJenkins removed a comment on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663800601


   Merged build finished. Test FAILed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

SparkQA removed a comment on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663800430


   **[Test build #126523 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126523/testReport)**
 for PR 29228 at commit 
[`14de6c2`](https://github.com/apache/spark/commit/14de6c264c182a084300ed24046c25509e587b83).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

AmplabJenkins commented on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663800601







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

SparkQA commented on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663800599


   **[Test build #126523 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126523/testReport)**
 for PR 29228 at commit 
[`14de6c2`](https://github.com/apache/spark/commit/14de6c264c182a084300ed24046c25509e587b83).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `trait SparkConfHelper `



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

SparkQA commented on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663800430


   **[Test build #126523 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126523/testReport)**
 for PR 29228 at commit 
[`14de6c2`](https://github.com/apache/spark/commit/14de6c264c182a084300ed24046c25509e587b83).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

AmplabJenkins removed a comment on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663799898







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins removed a comment on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663799916







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663799916







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] beliefer commented on pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

beliefer commented on pull request #28917:
URL: https://github.com/apache/spark/pull/28917#issuecomment-663799879


   Because conflicts, I will close this PR.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

AmplabJenkins commented on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663799898







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] beliefer commented on pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

beliefer commented on pull request #29228:
URL: https://github.com/apache/spark/pull/29228#issuecomment-663799816


   cc @jiangxb1987 @Ngone51



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] beliefer closed pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

beliefer closed pull request #28917:
URL: https://github.com/apache/spark/pull/28917


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] beliefer opened a new pull request #29228: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox]

beliefer opened a new pull request #29228:
URL: https://github.com/apache/spark/pull/29228


   ### What changes were proposed in this pull request?
   `DAGSchedulerSuite` exists some issue:
   `afterEach` and init are called when the `SparkConf` of the default 
`SparkContext` has no configuration that the test case must set. This causes 
the `SparkContext` initialized in `beforeEach` to be discarded without being 
used, resulting in waste. On the other hand, the flexibility to add 
configurations to `SparkConf` should be addressed by the test framework.
   
   
   ### Why are the changes needed?
   Reduce overhead about init `SparkContext`.
   Rewrite the test framework to support apply specified spark configurations.
   
   
   ### Does this PR introduce _any_ user-facing change?
   'No'.
   
   
   ### How was this patch tested?
   Jenkins test.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

SparkQA commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663799771


   **[Test build #126522 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126522/testReport)**
 for PR 29227 at commit 
[`5ddfdbf`](https://github.com/apache/spark/commit/5ddfdbf62d6bd891e924e9b7579c3405de04a1d8).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

dongjoon-hyun commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663799656


   @srowen . The order is flipped in three files. `else` statement is assigned 
to Scala 2.12 always.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins removed a comment on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663799101







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

AmplabJenkins commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663799101







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

SparkQA commented on pull request #29227:
URL: https://github.com/apache/spark/pull/29227#issuecomment-663798988


   **[Test build #126521 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126521/testReport)**
 for PR 29227 at commit 
[`614d293`](https://github.com/apache/spark/commit/614d2934afb37af6adcbb8e5180f4976223df93b).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

dongjoon-hyun commented on a change in pull request #29227:
URL: https://github.com/apache/spark/pull/29227#discussion_r460355637



##
File path: bin/load-spark-env.cmd
##
@@ -37,26 +37,24 @@ if [%SPARK_ENV_LOADED%] == [] (
 
 rem Setting SPARK_SCALA_VERSION if not already set.
 
-rem TODO: revisit for Scala 2.13 support
-set SPARK_SCALA_VERSION=2.12
-rem if [%SPARK_SCALA_VERSION%] == [] (
-rem   set SCALA_VERSION_1=2.12
-rem   set SCALA_VERSION_2=2.11
-rem
-rem   set ASSEMBLY_DIR1=%SPARK_HOME%\assembly\target\scala-%SCALA_VERSION_1%
-rem   set ASSEMBLY_DIR2=%SPARK_HOME%\assembly\target\scala-%SCALA_VERSION_2%
-rem   set 
ENV_VARIABLE_DOC=https://spark.apache.org/docs/latest/configuration.html#environment-variables
-rem   if exist %ASSEMBLY_DIR2% if exist %ASSEMBLY_DIR1% (
-rem echo "Presence of build for multiple Scala versions detected 
(%ASSEMBLY_DIR1% and %ASSEMBLY_DIR2%)."
-rem echo "Remove one of them or, set SPARK_SCALA_VERSION=%SCALA_VERSION_1% 
in %SPARK_ENV_CMD%."
-rem echo "Visit %ENV_VARIABLE_DOC% for more details about setting 
environment variables in spark-env.cmd."
-rem echo "Either clean one of them or, set SPARK_SCALA_VERSION in 
spark-env.cmd."
-rem exit 1
-rem   )
-rem   if exist %ASSEMBLY_DIR1% (
-rem set SPARK_SCALA_VERSION=%SCALA_VERSION_1%
-rem   ) else (
-rem set SPARK_SCALA_VERSION=%SCALA_VERSION_2%
-rem   )
-rem )
+if [%SPARK_SCALA_VERSION%] == [] (
+  set SCALA_VERSION_1=2.12

Review comment:
   Got it. Thanks!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins removed a comment on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663798416







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] imback82 commented on pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

[GitBox]

imback82 commented on pull request #29020:
URL: https://github.com/apache/spark/pull/29020#issuecomment-663798434


   Thanks @wypoon for the suggestion. I updated the description, and hopefully 
that clarifies things a bit more.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663798416







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

SparkQA commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663798252


   **[Test build #126518 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126518/testReport)**
 for PR 29226 at commit 
[`558702e`](https://github.com/apache/spark/commit/558702e6f55d6c17439d1942000bea08acde0b29).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

SparkQA removed a comment on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663784088


   **[Test build #126518 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126518/testReport)**
 for PR 29226 at commit 
[`558702e`](https://github.com/apache/spark/commit/558702e6f55d6c17439d1942000bea08acde0b29).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins removed a comment on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663796598







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

AmplabJenkins commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663796598







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

SparkQA removed a comment on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663780471


   **[Test build #126517 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126517/testReport)**
 for PR 29226 at commit 
[`7659d22`](https://github.com/apache/spark/commit/7659d22bcc3e2a423a1068dd15d9b11eba76c2df).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

SparkQA commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663796437


   **[Test build #126517 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126517/testReport)**
 for PR 29226 at commit 
[`7659d22`](https://github.com/apache/spark/commit/7659d22bcc3e2a423a1068dd15d9b11eba76c2df).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] holdenk commented on pull request #29226: Fix flakyness of BlockManagerDecommissionIntegrationSuite

[GitBox]

holdenk commented on pull request #29226:
URL: https://github.com/apache/spark/pull/29226#issuecomment-663795770


   Oh awesome, thanks for fixing this. I totally misdiagnosed this issue
   (thought it was around placement of the partitions).
   
   On Fri, Jul 24, 2020 at 5:36 PM Apache Spark QA 
   wrote:
   
   > *Test build #126518 has started
   > 
*
   > for PR 29226 at commit 558702e
   > 

   > .
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or
   > unsubscribe
   > 

   > .
   >
   -- 
   Cell : 425-233-8271
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] wypoon commented on a change in pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

[GitBox]

wypoon commented on a change in pull request #29020:
URL: https://github.com/apache/spark/pull/29020#discussion_r460350116



##
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##
@@ -868,13 +868,17 @@ private[spark] class AppStatusListener(
 // check if there is a new peak value for any of the executor level memory 
metrics
 // for the live UI. SparkListenerExecutorMetricsUpdate events are only 
processed
 // for the live UI.
-event.executorUpdates.foreach { case (_, peakUpdates) =>
+event.executorUpdates.foreach { case (key, peakUpdates) =>
   liveExecutors.get(event.execId).foreach { exec =>
 if (exec.peakExecutorMetrics.compareAndUpdatePeakValues(peakUpdates)) {
   maybeUpdate(exec, now)

Review comment:
   I am not so familiar with `AppStatusListener`, so please educate me. 
From what I see, for most events, `maybeUpdate` is called. I assume that we 
only want to update a live application if it hasn't been updated in the last 
configured period. The `SparkListenerExecutorMetricsUpdate` event here is 
processed for a live application. For the `SparkListenerStageExecutorMetrics` 
event, according to Edwina's comment, that is only processed when reading event 
logs, so `update` is called in that case.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] wypoon edited a comment on pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

[GitBox]

wypoon edited a comment on pull request #29020:
URL: https://github.com/apache/spark/pull/29020#issuecomment-663794380


   In the PR description,
   
   "This PR proposes to expose the peak executor metrics at the stage level via 
the REST API (/applications//stages/ or 
/applications//stages//)."
   
   there is a typo; "" should be "".
   
   It might be helpful to clarify that the peak values of executor metrics 
**for the stage** are shown for each stage in the first 
(/applications//stages/) and peak values of executor metrics 
**for each executor for the stage** are shown for the second 
/applications//stages//). In other 
words, the first shows **per-stage** peaks, and the second shows **per-executor 
per-stage** peaks (for just that stage).
   At least, this is my understanding of what is intended.
   Edwina's PR description has
   
   "Peak values for metrics are show for each stage. For executor summaries for 
each stage, the peak values per executor are also shown."
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] imback82 commented on a change in pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

[GitBox]

imback82 commented on a change in pull request #29020:
URL: https://github.com/apache/spark/pull/29020#discussion_r460351674



##
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##
@@ -868,13 +868,17 @@ private[spark] class AppStatusListener(
 // check if there is a new peak value for any of the executor level memory 
metrics
 // for the live UI. SparkListenerExecutorMetricsUpdate events are only 
processed
 // for the live UI.
-event.executorUpdates.foreach { case (_, peakUpdates) =>
+event.executorUpdates.foreach { case (key, peakUpdates) =>
   liveExecutors.get(event.execId).foreach { exec =>
 if (exec.peakExecutorMetrics.compareAndUpdatePeakValues(peakUpdates)) {
   maybeUpdate(exec, now)

Review comment:
   We need `update` here since we know that new peak values are received 
(by checking `compareAndUpdatePeakValues`). If we use `maybeUpdate`, we may 
miss recording these new peak values.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] wypoon commented on pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

[GitBox]

wypoon commented on pull request #29020:
URL: https://github.com/apache/spark/pull/29020#issuecomment-663794380


   In the PR description,
   
   "This PR proposes to expose the peak executor metrics at the stage level via 
the REST API (/applications//stages/ or 
/applications//stages//)."
   
   there is a typo; "" should be "".
   
   It might be helpful to clarify that the peak values of executor metrics 
**for the stage** are shown for each stage in the first 
(/applications//stages/) and peak values of executor metrics 
**for each executor for the stage** are shown for the second 
/applications//stages//).
   At least, this is my understanding of what is intended.
   Edwina's PR description has
   
   "Peak values for metrics are show for each stage. For executor summaries for 
each stage, the peak values per executor are also shown."
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #29227: [SPARK-32434][CORE] Support Scala 2.13 in AbstractCommandBuilder and load-spark-env scripts

[GitBox]

srowen commented on a change in pull request #29227:
URL: https://github.com/apache/spark/pull/29227#discussion_r460351305



##
File path: bin/load-spark-env.cmd
##
@@ -37,26 +37,24 @@ if [%SPARK_ENV_LOADED%] == [] (
 
 rem Setting SPARK_SCALA_VERSION if not already set.
 
-rem TODO: revisit for Scala 2.13 support
-set SPARK_SCALA_VERSION=2.12
-rem if [%SPARK_SCALA_VERSION%] == [] (
-rem   set SCALA_VERSION_1=2.12
-rem   set SCALA_VERSION_2=2.11
-rem
-rem   set ASSEMBLY_DIR1=%SPARK_HOME%\assembly\target\scala-%SCALA_VERSION_1%
-rem   set ASSEMBLY_DIR2=%SPARK_HOME%\assembly\target\scala-%SCALA_VERSION_2%
-rem   set 
ENV_VARIABLE_DOC=https://spark.apache.org/docs/latest/configuration.html#environment-variables
-rem   if exist %ASSEMBLY_DIR2% if exist %ASSEMBLY_DIR1% (
-rem echo "Presence of build for multiple Scala versions detected 
(%ASSEMBLY_DIR1% and %ASSEMBLY_DIR2%)."
-rem echo "Remove one of them or, set SPARK_SCALA_VERSION=%SCALA_VERSION_1% 
in %SPARK_ENV_CMD%."
-rem echo "Visit %ENV_VARIABLE_DOC% for more details about setting 
environment variables in spark-env.cmd."
-rem echo "Either clean one of them or, set SPARK_SCALA_VERSION in 
spark-env.cmd."
-rem exit 1
-rem   )
-rem   if exist %ASSEMBLY_DIR1% (
-rem set SPARK_SCALA_VERSION=%SCALA_VERSION_1%
-rem   ) else (
-rem set SPARK_SCALA_VERSION=%SCALA_VERSION_2%
-rem   )
-rem )
+if [%SPARK_SCALA_VERSION%] == [] (
+  set SCALA_VERSION_1=2.12

Review comment:
   We might want to flip the order here; it might matter to the logic to 
look for the alternate 2.13 build first. Same for the other files.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] wypoon commented on a change in pull request #29020: [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API

[GitBox]

wypoon commented on a change in pull request #29020:
URL: https://github.com/apache/spark/pull/29020#discussion_r460350116



##
File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
##
@@ -868,13 +868,17 @@ private[spark] class AppStatusListener(
 // check if there is a new peak value for any of the executor level memory 
metrics
 // for the live UI. SparkListenerExecutorMetricsUpdate events are only 
processed
 // for the live UI.
-event.executorUpdates.foreach { case (_, peakUpdates) =>
+event.executorUpdates.foreach { case (key, peakUpdates) =>
   liveExecutors.get(event.execId).foreach { exec =>
 if (exec.peakExecutorMetrics.compareAndUpdatePeakValues(peakUpdates)) {
   maybeUpdate(exec, now)

Review comment:
   I am not so familiar with `AppStatusListener`, so please educate me. 
From what I see, for most events, `maybeUpdate` is called. I assume that we 
only want to update a live application if it hasn't been updated in the last 
configured period. For the  `SparkListenerStageExecutorMetrics` event, 
according to Edwina's comment, this is only processed when reading event logs, 
so `update` is called in that case.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor

[GitBox]

AmplabJenkins removed a comment on pull request #29193:
URL: https://github.com/apache/spark/pull/29193#issuecomment-663790929







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

[GitBox]

AmplabJenkins commented on pull request #29193:
URL: https://github.com/apache/spark/pull/29193#issuecomment-663790929







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lo

[GitBox]

SparkQA removed a comment on pull request #29193:
URL: https://github.com/apache/spark/pull/29193#issuecomment-663758723


   **[Test build #126515 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126515/testReport)**
 for PR 29193 at commit 
[`e54f221`](https://github.com/apache/spark/commit/e54f22186a1f82ac9704f6644e057dadfc285b86).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29002: [SPARK-32175][CORE] Fix the order between initialization for ExecutorPlugin and starting heartbeat thread

[GitBox]

AmplabJenkins removed a comment on pull request #29002:
URL: https://github.com/apache/spark/pull/29002#issuecomment-663790650







This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

[GitBox]

SparkQA commented on pull request #29193:
URL: https://github.com/apache/spark/pull/29193#issuecomment-663790710


   **[Test build #126515 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126515/testReport)**
 for PR 29193 at commit 
[`e54f221`](https://github.com/apache/spark/commit/e54f22186a1f82ac9704f6644e057dadfc285b86).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org