[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92621825 Why would we get wrong properties in this case? We are explicitly setting it in the thread that is launching the Spark job, so there is no question of inheriting local properties by child threads, etc. So that issue should not affect this case, isnt it? Or am I missing something? Without #5288, SparkListenerJobStart is created like `SparkListenerJobStart(..., localProperties.get)`. So the `properties` in `SparkListenerJobStart` is the same object of `localProperties.get`, which is supposed to be limited in its thread. And after `job.run`, we will clear some properties in `localProperties.get`, because `SparkListenerJobStart.properties` is the same object of `localProperties.get`, if `job.run` is very fast, we may not find the properties we need, because they have been cleared. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92647115 [Test build #30223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30223/consoleFull) for PR 5473 at commit [`7168807`](https://github.com/apache/spark/commit/7168807f46d7ae7213b9f05071f5a1d25baa5b5e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92646872 Please ignore that `Processing time` and `Total delay` are `-`. It's an issue about `JobScheduler`. I will open a JIRA for it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92660739 Please ignore that Processing time and Total delay are -. It's an issue about JobScheduler. I will open a JIRA for it. After thinking it carefully, this may not be an issue, since StreamingContext should be stopped in such case. @tdas, do you think if we still need to show Processing time and Total delay in this case? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92662703 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30220/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92662661 [Test build #30220 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30220/consoleFull) for PR 5473 at commit [`0b226f9`](https://github.com/apache/spark/commit/0b226f9fd79d88ba3505553fdda8f6e774abf47b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class BatchPage(parent: StreamingTab) extends WebUIPage(batch) ` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92646413 I set the max-width of the error column to `300px`. Here is the new screenshot:  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92682373 [Test build #30223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30223/consoleFull) for PR 5473 at commit [`7168807`](https://github.com/apache/spark/commit/7168807f46d7ae7213b9f05071f5a1d25baa5b5e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class BatchPage(parent: StreamingTab) extends WebUIPage(batch) ` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92682417 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30223/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5473#discussion_r28301652
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.ui
+
+import javax.servlet.http.HttpServletRequest
+
+import org.apache.commons.lang3.StringEscapeUtils
+import org.apache.spark.streaming.Time
+import org.apache.spark.ui.{UIUtils, WebUIPage}
+import org.apache.spark.streaming.ui.StreamingJobProgressListener.{JobId,
OutputOpId}
+import org.apache.spark.ui.jobs.UIData.JobUIData
+
+import scala.xml.{NodeSeq, Node}
+
+class BatchPage(parent: StreamingTab) extends WebUIPage(batch) {
+ private val streaminglistener = parent.listener
+ private val sparkListener = parent.ssc.sc.jobProgressListener
+
+ private def columns: Seq[Node] = {
+thOutput Op Id/th
+ thDescription/th
+ thDuration/th
+ thJob Id/th
+ thDuration/th
+ th class=sorttable_nosortStages: Succeeded/Total/th
+ th class=sorttable_nosortTasks (for all stages):
Succeeded/Total/th
+ thLast Error/th
--- End diff --
I was thinking `Last Error` of tasks. But here is Stage. I agree it should
be `Error`. Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5473#discussion_r28301670
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala
---
@@ -179,7 +181,14 @@ class JobScheduler(val ssc: StreamingContext) extends
Logging {
PairRDDFunctions.disableOutputSpecValidation.withValue(true) {
job.run()
}
+ ssc.sc.setLocalProperty(JobScheduler.BATCH_TIME_PROPERTY_KEY, null)
+ ssc.sc.setLocalProperty(JobScheduler.OUTPUT_OP_ID_PROPERTY_KEY, null)
eventActor ! JobCompleted(job)
}
}
}
+
+private[streaming] object JobScheduler {
+ private[streaming] val BATCH_TIME_PROPERTY_KEY =
spark.streaming.internal.batchTime
--- End diff --
Right. I just removed `private[streaming]`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5473#discussion_r28301657
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala
---
@@ -170,8 +170,10 @@ class JobScheduler(val ssc: StreamingContext) extends
Logging {
ssc.waiter.notifyError(e)
}
- private class JobHandler(job: Job) extends Runnable {
+ private class JobHandler(job: Job) extends Runnable with Logging {
def run() {
+ ssc.sc.setLocalProperty(JobScheduler.BATCH_TIME_PROPERTY_KEY,
job.time.milliseconds.toString)
+ ssc.sc.setLocalProperty(JobScheduler.OUTPUT_OP_ID_PROPERTY_KEY,
job.outputOpId.toString)
--- End diff --
Put them to `finally`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92618458 Why would we get wrong properties in this case? We are explicitly setting it in the thread that is launching the Spark job, so there is no question of inheriting local properties by child threads, etc. So that issue should not affect this case, isnt it? Or am I missing something? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92606473 @zsxwing Any thoughts on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92613622 There should be tests in the StreamingListenerSuite that tests the local property setting. As I explained in `https://github.com/apache/spark/pull/5288#issuecomment-92579899`, we may get wrong properties in `SparkListenerJobStart.properties`. Because the tests will be non-deterministic, I think it's better to add them after @JoshRosen fixes the issue. In the screenshots, is there any way to limit the max col size of the error? When there is error, it squishes all the other columns. Let me try some css style to fix it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92611601 [Test build #30220 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30220/consoleFull) for PR 5473 at commit [`0b226f9`](https://github.com/apache/spark/commit/0b226f9fd79d88ba3505553fdda8f6e774abf47b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-92288077 Its looking pretty good! Here are some preliminary comments. I will do a more detailed pass tomorrow during the day. 1. There should be tests in the StreamingListenerSuite that tests the local property setting. 2. In the screenshots, is there any way to limit the max col size of the error? When there is error, it squishes all the other columns. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5473#discussion_r28223951
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala
---
@@ -179,7 +181,14 @@ class JobScheduler(val ssc: StreamingContext) extends
Logging {
PairRDDFunctions.disableOutputSpecValidation.withValue(true) {
job.run()
}
+ ssc.sc.setLocalProperty(JobScheduler.BATCH_TIME_PROPERTY_KEY, null)
+ ssc.sc.setLocalProperty(JobScheduler.OUTPUT_OP_ID_PROPERTY_KEY, null)
eventActor ! JobCompleted(job)
}
}
}
+
+private[streaming] object JobScheduler {
+ private[streaming] val BATCH_TIME_PROPERTY_KEY =
spark.streaming.internal.batchTime
--- End diff --
If the object private[streaming] then all the field are already
private[streaming], isnt it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5473#discussion_r28224797
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala ---
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.ui
+
+import javax.servlet.http.HttpServletRequest
+
+import org.apache.commons.lang3.StringEscapeUtils
+import org.apache.spark.streaming.Time
+import org.apache.spark.ui.{UIUtils, WebUIPage}
+import org.apache.spark.streaming.ui.StreamingJobProgressListener.{JobId,
OutputOpId}
+import org.apache.spark.ui.jobs.UIData.JobUIData
+
+import scala.xml.{NodeSeq, Node}
+
+class BatchPage(parent: StreamingTab) extends WebUIPage(batch) {
+ private val streaminglistener = parent.listener
+ private val sparkListener = parent.ssc.sc.jobProgressListener
+
+ private def columns: Seq[Node] = {
+thOutput Op Id/th
+ thDescription/th
+ thDuration/th
+ thJob Id/th
+ thDuration/th
+ th class=sorttable_nosortStages: Succeeded/Total/th
+ th class=sorttable_nosortTasks (for all stages):
Succeeded/Total/th
+ thLast Error/th
--- End diff --
Why is this `Last Error` and not just `Error`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5473#discussion_r28223843
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/BatchInfo.scala
---
@@ -58,4 +58,11 @@ case class BatchInfo(
*/
def totalDelay: Option[Long] = schedulingDelay.zip(processingDelay)
.map(x = x._1 + x._2).headOption
+
+ /**
+ * The number of recorders received by the receivers in this batch.
+ */
+ def numRecords: Long = receivedBlockInfo.map { case (_, infos) =
--- End diff --
+1 on exposing this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5473#discussion_r28223913
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala
---
@@ -170,8 +170,10 @@ class JobScheduler(val ssc: StreamingContext) extends
Logging {
ssc.waiter.notifyError(e)
}
- private class JobHandler(job: Job) extends Runnable {
+ private class JobHandler(job: Job) extends Runnable with Logging {
def run() {
+ ssc.sc.setLocalProperty(JobScheduler.BATCH_TIME_PROPERTY_KEY,
job.time.milliseconds.toString)
+ ssc.sc.setLocalProperty(JobScheduler.OUTPUT_OP_ID_PROPERTY_KEY,
job.outputOpId.toString)
--- End diff --
This setting and clearing of property needs to be done in a
`tryfinally` so that it gets cleared in case of any exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-91907788 [Test build #30084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30084/consoleFull) for PR 5473 at commit [`0c7b2eb`](https://github.com/apache/spark/commit/0c7b2ebd0b2ce1ed615c06ef76c707538b5d6da4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class BatchPage(parent: StreamingTab) extends WebUIPage(batch) ` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-91907795 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30084/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/5473 [SPARK-6862][Streaming][WebUI][WIP] Add BatchPage to display details of a batch This is an initial commit for SPARK-6862. Once SPARK-6796 is merged, I will add the links to StreamingPage so that the user can jump to BatchPage. Screenshots:   You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark SPARK-6862 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5473.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5473 commit 0c7b2ebd0b2ce1ed615c06ef76c707538b5d6da4 Author: zsxwing zsxw...@gmail.com Date: 2015-04-09T03:56:58Z Add BatchPage to display details of a batch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-91890461 cc @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5473#issuecomment-91890386
My test codes:
```Scala
import org.apache.spark._
import org.apache.spark.streaming._
object StreamingApp {
def main(args: Array[String]): Unit = {
val conf = new
SparkConf().setMaster(local[2]).setAppName(NetworkWordCount).set(spark.streaming.concurrentJobs,
3)
val ssc = new StreamingContext(conf, Seconds(10))
val lines = ssc.socketTextStream(localhost, )
val words = lines.flatMap(_.split( ))
val pairs = words.map(word = {
Thread.sleep(1000)
(word, 1)
})
val wordCounts = pairs.reduceByKey((x: Int, y: Int) = x + y, 2)
wordCounts.foreachRDD {
rdd = rdd.foreach { v =
println(v)
}
}
wordCounts.foreachRDD { rdd =
rdd.foreach { v =
println(v)
}
rdd.foreach { v =
println(v)
//throw new RuntimeException(Oops!)
}
}
ssc.start()
ssc.awaitTermination()
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6862][Streaming][WebUI][WIP] Add BatchP...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5473#issuecomment-91890772 [Test build #30084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30084/consoleFull) for PR 5473 at commit [`0c7b2eb`](https://github.com/apache/spark/commit/0c7b2ebd0b2ce1ed615c06ef76c707538b5d6da4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
