[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195477414 **[Test build #52931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52931/consoleFull)** for PR 11615 at commit [`8422f6e`](https://github.com/apache/spark/commit/8422f6e1e5c8617dd8bffaa6c47429aadfc7bc10). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195476295 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195255151 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195255153 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52908/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195254957 **[Test build #52908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52908/consoleFull)** for PR 11615 at commit [`8422f6e`](https://github.com/apache/spark/commit/8422f6e1e5c8617dd8bffaa6c47429aadfc7bc10). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195224677 **[Test build #52908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52908/consoleFull)** for PR 11615 at commit [`8422f6e`](https://github.com/apache/spark/commit/8422f6e1e5c8617dd8bffaa6c47429aadfc7bc10). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-19502 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195153720 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52872/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195153719 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195153594 **[Test build #52872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52872/consoleFull)** for PR 11615 at commit [`8422f6e`](https://github.com/apache/spark/commit/8422f6e1e5c8617dd8bffaa6c47429aadfc7bc10). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195115654 **[Test build #52872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52872/consoleFull)** for PR 11615 at commit [`8422f6e`](https://github.com/apache/spark/commit/8422f6e1e5c8617dd8bffaa6c47429aadfc7bc10). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11615#discussion_r55772647
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -270,9 +270,25 @@ private[spark] object JettyUtils extends Logging {
gzipHandlers.foreach(collection.addHandler)
connectors.foreach(_.setHost(hostName))
+ // As each acceptor and each selector will use one thread, the
number of threads should at
+ // least be the number of acceptors and selectors plus 1. (See
SPARK-13776)
+ var minThreads = 1
+ connectors.foreach { c =>
+// Currently we only use "SelectChannelConnector"
+val connector = c.asInstanceOf[SelectChannelConnector]
+// Limit the max acceptor number to 8 so that we don't waste a lot
of threads
+connector.setAcceptors(math.min(connector.getAcceptors, 8))
+// The number of selectors always equals to the number of acceptors
--- End diff --
See
https://github.com/eclipse/jetty.project/blob/jetty-8.1.14.v20131031/jetty-server/src/main/java/org/eclipse/jetty/server/nio/SelectChannelConnector.java#L260
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195072873 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195072876 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52848/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195072574 **[Test build #52848 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52848/consoleFull)** for PR 11615 at commit [`ea99768`](https://github.com/apache/spark/commit/ea99768afe0990d32383fc7a1039a27793db678e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11615#discussion_r55743040
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -270,9 +270,19 @@ private[spark] object JettyUtils extends Logging {
gzipHandlers.foreach(collection.addHandler)
connectors.foreach(_.setHost(hostName))
+ // As each Acceptor will use one thread, the number of threads
should at least be the number
+ // of acceptors plus 1. (See SPARK-13776)
+ var minThreads = 1
+ connectors.collect { case c: AbstractConnector => c }.foreach { c =>
--- End diff --
Nit: the Scala math class is `math` vs `Math` but I wouldn't change it just
for that. I think this looks like the right kind of approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195021970 **[Test build #52848 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52848/consoleFull)** for PR 11615 at commit [`ea99768`](https://github.com/apache/spark/commit/ea99768afe0990d32383fc7a1039a27793db678e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-195021406 Updated to use a small acceptor number --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11615#discussion_r55729755
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -270,9 +270,21 @@ private[spark] object JettyUtils extends Logging {
gzipHandlers.foreach(collection.addHandler)
connectors.foreach(_.setHost(hostName))
+ // As each Acceptor will use one thread, the number of threads
should at least be the number
+ // of acceptors plus 1.
+ val minThreads =
+connectors.collect { case c: AbstractConnector => c
}.map(_.getAcceptors).sum + 1
server.setConnectors(connectors.toArray)
val pool = new QueuedThreadPool
+ var maxThreads = conf.getInt("spark.ui.threads", pool.getMaxThreads)
--- End diff --
Agree, if it's just for web UIs then I see no need for a large number of
acceptors. I think your change is still needed to make sure the pool has enough
threads in any event? though maybe the config is unnecessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11615#discussion_r55725635
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -270,9 +270,21 @@ private[spark] object JettyUtils extends Logging {
gzipHandlers.foreach(collection.addHandler)
connectors.foreach(_.setHost(hostName))
+ // As each Acceptor will use one thread, the number of threads
should at least be the number
+ // of acceptors plus 1.
+ val minThreads =
+connectors.collect { case c: AbstractConnector => c
}.map(_.getAcceptors).sum + 1
server.setConnectors(connectors.toArray)
val pool = new QueuedThreadPool
+ var maxThreads = conf.getInt("spark.ui.threads", pool.getMaxThreads)
--- End diff --
I was worried that 1 thread for processing requests may be not enough.
@srowen what do you think about `just hard-core the max number of acceptors
to a small number`? Maybe just 8?
The default configuration of Jetty seems good to a web application that
supports massive connections. However, Spark Web UI doesn't need to support so
many users, so it's a waste to create many threads.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11615#discussion_r55642314
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -270,9 +270,21 @@ private[spark] object JettyUtils extends Logging {
gzipHandlers.foreach(collection.addHandler)
connectors.foreach(_.setHost(hostName))
+ // As each Acceptor will use one thread, the number of threads
should at least be the number
+ // of acceptors plus 1.
+ val minThreads =
+connectors.collect { case c: AbstractConnector => c
}.map(_.getAcceptors).sum + 1
server.setConnectors(connectors.toArray)
val pool = new QueuedThreadPool
+ var maxThreads = conf.getInt("spark.ui.threads", pool.getMaxThreads)
--- End diff --
Do we even need config here? when would I otherwise want to control this?
It seems like you want to always make the pool have at least acceptors+1
threads.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-194588783 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-194588787 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52768/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-194588517 **[Test build #52768 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52768/consoleFull)** for PR 11615 at commit [`ce6cad3`](https://github.com/apache/spark/commit/ce6cad308c108ff0386af627c4af892466cfffa3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-194540494 Another option is just hard-core the number of acceptors to a small number. I don't think there will be a lot of connections to access the Web UI at the same time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13776][WebUI]Add spark.ui.threads to se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11615#issuecomment-194540445 **[Test build #52768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52768/consoleFull)** for PR 11615 at commit [`ce6cad3`](https://github.com/apache/spark/commit/ce6cad308c108ff0386af627c4af892466cfffa3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
