[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-175333657 @liyezhang556520 can you remove the "WIP" from the title. cc @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170858673 **[Test build #49235 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49235/consoleFull)** for PR 7753 at commit [`5e031ce`](https://github.com/apache/spark/commit/5e031ce3cc90a6b77133488b8070288dff11ce95). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170860844 **[Test build #49235 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49235/consoleFull)** for PR 7753 at commit [`5e031ce`](https://github.com/apache/spark/commit/5e031ce3cc90a6b77133488b8070288dff11ce95). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170860861 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170860864 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49235/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170927367 **[Test build #49239 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49239/consoleFull)** for PR 7753 at commit [`87f8172`](https://github.com/apache/spark/commit/87f8172fbc1e219cea18e80996b6b0fd12b141de). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170942709 Hi @steveloughran , thank you for your attention and your comments, and your further comments on this PR are pretty appreciated. Your advice is quite correct, we need to make a regression test for history server suite. I think this is related with what @squito mentioned that we need to add json api support. I'm wondering whether we can make it done in another PR after this PR be able to merge or this feature totally accepted by the community. And of course, rest api support is supposed to be one part of this feature. @squito, what's your opinion on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170984971 **[Test build #49239 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49239/consoleFull)** for PR 7753 at commit [`87f8172`](https://github.com/apache/spark/commit/87f8172fbc1e219cea18e80996b6b0fd12b141de). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170985319 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-170985320 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49239/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-168497378 One thought here: it'd probably be nice to have a json history with the new events as part of the history server suite regression tests âthat'll catch any changes to the new events which will break backwards compatibility --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r47849328
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala
---
@@ -50,6 +53,39 @@ class NettyBlockTransferService(conf: SparkConf,
securityManager: SecurityManage
private[this] var server: TransportServer = _
private[this] var clientFactory: TransportClientFactory = _
private[this] var appId: String = _
+ private[this] var clock: Clock = new SystemClock()
+
+ /**
+ * Use a different clock for this allocation manager. This is mainly
used for testing.
+ */
+ def setClock(newClock: Clock): Unit = {
+clock = newClock
+ }
+
+ private[spark] override def getMemMetrics(executorMetrics:
ExecutorMetrics): Unit = {
+val currentTime = clock.getTimeMillis()
+val clientPooledAllocator = clientFactory.getPooledAllocator()
+val serverAllocator = server.getAllocator()
+val clientOffHeapSize: Long = sumOfMetrics(
+ clientPooledAllocator.directArenas().asScala.toList)
+val clientOnHeapSize: Long =
sumOfMetrics(clientPooledAllocator.heapArenas().asScala.toList)
+val serverOffHeapSize: Long =
sumOfMetrics(serverAllocator.directArenas().asScala.toList)
+val serverOnHeapSize: Long =
sumOfMetrics(serverAllocator.heapArenas().asScala.toList)
--- End diff --
push the `.asScala.toList` into the helper method
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r47849888
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala ---
@@ -65,8 +65,8 @@ private[spark] trait TaskScheduler {
* alive. Return true if the driver knows about the given block manager.
Otherwise, return false,
* indicating that the block manager should re-register.
*/
- def executorHeartbeatReceived(execId: String, taskMetrics: Array[(Long,
TaskMetrics)],
-blockManagerId: BlockManagerId): Boolean
+ def executorHeartbeatReceived(execId: String, executorMetrics:
ExecutorMetrics,
+ taskMetrics: Array[(Long, TaskMetrics)], blockManagerId:
BlockManagerId): Boolean
--- End diff --
I know this was wrong before, but as long as you're touching this -- with
multiline methods, each arg should be on its own line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r47849097
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.executor
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * :: DeveloperApi ::
+ * Metrics tracked during the execution of an executor.
+ *
+ * So, when adding new fields, take into consideration that the whole
object can be serialized for
+ * shipping off at any time to consumers of the SparkListener interface.
+ */
+@DeveloperApi
+class ExecutorMetrics extends Serializable {
+
+ /**
+ * Host's name the executor runs on
+ */
+ private var _hostname: String = ""
+ def hostname: String = _hostname
+ private[spark] def setHostname(value: String) = _hostname = value
+
+ /**
+ * Host's port the executor runs on
+ */
+ private var _port: Option[Int] = None
+ def port: Option[Int] = _port
+ private[spark] def setPort(value: Option[Int]) = _port = value
+
+ private[spark] def hostPort: String = {
+val hp = port match {
+ case None => hostname
+ case value => hostname + ":" + value
+}
+hp
+ }
+
+ private var _transportMetrics: TransportMetrics = new TransportMetrics
+ def transportMetrics: TransportMetrics = _transportMetrics
+ private[spark] def setTransportMetrics(value: TransportMetrics) = {
+_transportMetrics = value
+ }
+
+ // for test only
+ def metricsDetails: (String, Long, Long, Long) = {
--- End diff --
rather than putting this method here, just define it inside the test case
where you use it (though I think you probably don't even need it there)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-163157990 **[Test build #47415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47415/consoleFull)** for PR 7753 at commit [`4b3dbe4`](https://github.com/apache/spark/commit/4b3dbe4d41cb06f7ee9b48e816e147ca46a0dea3). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-163158139 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47415/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-163158136 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-163137831 **[Test build #47415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47415/consoleFull)** for PR 7753 at commit [`4b3dbe4`](https://github.com/apache/spark/commit/4b3dbe4d41cb06f7ee9b48e816e147ca46a0dea3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-162750981 **[Test build #47305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47305/consoleFull)** for PR 7753 at commit [`4123ac7`](https://github.com/apache/spark/commit/4123ac7924275c534dd7849c108f71eebdf4b4a1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-162751915 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-162751916 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47305/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-162751912 **[Test build #47305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47305/consoleFull)** for PR 7753 at commit [`4123ac7`](https://github.com/apache/spark/commit/4123ac7924275c534dd7849c108f71eebdf4b4a1). * This patch **fails Scala style tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r46152658
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
+ test("test HashMap size for MemoryListener") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+val execId2 = "exec-2"
+
+(1 to 2).foreach { i =>
+
listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(i))
+ listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(i))
+}
+// stages are all completed, no activeStages now
+assert(listener.activeStagesToMem.isEmpty)
+
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, new ExecutorMetrics))
+// ExecutorMetrics is not related with Stages directly
+assert(listener.activeStagesToMem.isEmpty)
+
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId2, new ExecutorMetrics))
+// totally 2 executors updated their metrics
+assert(listener.activeExecutorIdToMem.size === 2)
+assert(listener.activeStagesToMem.size === 1)
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+assert(listener.activeStagesToMem.isEmpty)
+assert(listener.completedStagesToMem.size === 3)
+assert(listener.activeExecutorIdToMem.size ===
listener.latestExecIdToExecMetrics.size)
+assert(listener.removedExecutorIdToMem.isEmpty)
+ }
+
+ test("test first stage with no executor metrics update") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// stage 1, no metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+
+// stage 2, with one metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics = MemoryListenerSuite.createExecutorMetrics("host-1",
0L, 20, 10)
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, execMetrics))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+
+val mapForStage1 = listener.completedStagesToMem.get((1, 0)).get
+// no metrics for stage 1 since no metrics update for stage 1
+assert(mapForStage1.get(execId1).get.transportInfo === None)
+val mapForStage2 = listener.completedStagesToMem.get((2, 0)).get
+assert(mapForStage2.size === 1)
+val memInfo = mapForStage2.get(execId1).get
+assert(memInfo.transportInfo.isDefined)
+val transMetrics = memInfo.transportInfo.get
+assert((20, 10, MemTime(20, 0), MemTime(10, 0)) ===
(transMetrics.onHeapSize,
+ transMetrics.offHeapSize, transMetrics.peakOnHeapSizeTime,
transMetrics.peakOffHeapSizeTime))
+
+listener.onExecutorRemoved(SparkListenerExecutorRemoved(0L, execId1,
""))
+ }
+
+ test("test multiple metrics updated in one stage") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// multiple metrics updated in one stage
+
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r46154647 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -85,6 +85,9 @@ private[spark] class Executor( env.blockManager.initialize(conf.getAppId) } + private val executorMetrics: ExecutorMetrics = new ExecutorMetrics + executorMetrics.setHostname(Utils.localHostName) --- End diff -- Sorry that I forgot the reason why I didn't get the port from `rpcEnv` originally, but it seems we can get the port from it directly. Let me try to add it back. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r46152192
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
--- End diff --
No, I'll remove this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r46153027
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
---
@@ -122,6 +122,105 @@ class EventLoggingListenerSuite extends SparkFunSuite
with LocalSparkContext wit
"a fine:mind$dollar{bills}.1", None, Some("lz4")))
}
+ test("test event logger logging executor metrics") {
+import org.apache.spark.scheduler.cluster._
+import org.apache.spark.ui.memory._
+val conf = EventLoggingListenerSuite.getLoggingConf(testDirPath)
+val eventLogger = new EventLoggingListener("test-memListener", None,
testDirPath.toUri(), conf)
+val execId = "exec-1"
+val hostName = "host-1"
+
+eventLogger.start()
+eventLogger.onExecutorAdded(SparkListenerExecutorAdded(
+ 0L, execId, new ExecutorInfo(hostName, 1, Map.empty)))
+
+// stage 1 and stage 2 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics1 = MemoryListenerSuite.createExecutorMetrics(hostName,
1L, 20, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics1))
+val execMetrics2 = MemoryListenerSuite.createExecutorMetrics(hostName,
2L, 30, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics2))
+// stage1 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+// stage3 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+val execMetrics3 = MemoryListenerSuite.createExecutorMetrics(hostName,
3L, 30, 30)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics3))
+val execMetrics4 = MemoryListenerSuite.createExecutorMetrics(hostName,
4L, 20, 25)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics4))
+// stage 2 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+val execMetrics5 = MemoryListenerSuite.createExecutorMetrics(hostName,
5L, 15, 15)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics5))
+val execMetrics6 = MemoryListenerSuite.createExecutorMetrics(hostName,
6L, 25, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics6))
+// stage 3 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+eventLogger.onExecutorRemoved(SparkListenerExecutorRemoved(7L, execId,
""))
+
+// Totally there are 15 logged events, including:
+// 2 events of executor Added/Removed
+// 6 events of stage Submitted/Completed
+// 7 events of executorMetrics update (3 combined metrics and 4
original metrics)
+assert(eventLogger.loggedEvents.size === 15)
+eventLogger.stop()
+
+val logData = EventLoggingListener.openEventLog(new
Path(eventLogger.logPath), fileSystem)
+val lines = readLines(logData)
+Utils.tryWithSafeFinally {
+ // totally there are 15 lines, including SparkListenerLogStart event
and 14 other events
+ assert(lines.size === 16)
+
+ // 4 executor metrics that is the latest metrics updated before
stage submit and complete
+ val jsonMetrics = JsonProtocol.sparkEventFromJson(parse(lines(5)))
--- End diff --
Good idea, I'll add an integration test, make it cleaner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r46152057
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.executor
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * :: DeveloperApi ::
+ * Metrics tracked during the execution of an executor.
+ *
+ * So, when adding new fields, take into consideration that the whole
object can be serialized for
+ * shipping off at any time to consumers of the SparkListener interface.
+ */
+@DeveloperApi
+class ExecutorMetrics extends Serializable {
+
+ /**
+ * Host's name the executor runs on
+ */
+ private var _hostname: String = _
+ def hostname: String = _hostname
+ private[spark] def setHostname(value: String) = _hostname = value
+
+ private var _transportMetrics: TransportMetrics = new TransportMetrics
+ def transportMetrics: TransportMetrics = _transportMetrics
+ private[spark] def setTransportMetrics(value: TransportMetrics) = {
+_transportMetrics = value
+ }
+
+ // for test only
+ def metricsDetails: (String, Long, Long, Long) = {
+(hostname, transportMetrics.timeStamp, transportMetrics.onHeapSize,
+ transportMetrics.offHeapSize)
+ }
+}
+
+/**
+ * :: DeveloperApi ::
+ * Metrics for network layer
+ */
+@DeveloperApi
+class TransportMetrics (
--- End diff --
Yes, that's true, thank you for pointing it out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r46152802
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
---
@@ -122,6 +122,105 @@ class EventLoggingListenerSuite extends SparkFunSuite
with LocalSparkContext wit
"a fine:mind$dollar{bills}.1", None, Some("lz4")))
}
+ test("test event logger logging executor metrics") {
+import org.apache.spark.scheduler.cluster._
+import org.apache.spark.ui.memory._
+val conf = EventLoggingListenerSuite.getLoggingConf(testDirPath)
+val eventLogger = new EventLoggingListener("test-memListener", None,
testDirPath.toUri(), conf)
+val execId = "exec-1"
+val hostName = "host-1"
+
+eventLogger.start()
+eventLogger.onExecutorAdded(SparkListenerExecutorAdded(
+ 0L, execId, new ExecutorInfo(hostName, 1, Map.empty)))
+
+// stage 1 and stage 2 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics1 = MemoryListenerSuite.createExecutorMetrics(hostName,
1L, 20, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics1))
+val execMetrics2 = MemoryListenerSuite.createExecutorMetrics(hostName,
2L, 30, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics2))
+// stage1 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+// stage3 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+val execMetrics3 = MemoryListenerSuite.createExecutorMetrics(hostName,
3L, 30, 30)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics3))
+val execMetrics4 = MemoryListenerSuite.createExecutorMetrics(hostName,
4L, 20, 25)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics4))
+// stage 2 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+val execMetrics5 = MemoryListenerSuite.createExecutorMetrics(hostName,
5L, 15, 15)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics5))
+val execMetrics6 = MemoryListenerSuite.createExecutorMetrics(hostName,
6L, 25, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics6))
+// stage 3 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+eventLogger.onExecutorRemoved(SparkListenerExecutorRemoved(7L, execId,
""))
+
+// Totally there are 15 logged events, including:
+// 2 events of executor Added/Removed
+// 6 events of stage Submitted/Completed
+// 7 events of executorMetrics update (3 combined metrics and 4
original metrics)
+assert(eventLogger.loggedEvents.size === 15)
+eventLogger.stop()
+
+val logData = EventLoggingListener.openEventLog(new
Path(eventLogger.logPath), fileSystem)
+val lines = readLines(logData)
+Utils.tryWithSafeFinally {
+ // totally there are 15 lines, including SparkListenerLogStart event
and 14 other events
--- End diff --
Yes, forget to update the comments, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r46153585
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -228,6 +261,46 @@ private[spark] class EventLoggingListener(
fileSystem.rename(new Path(logPath + IN_PROGRESS), target)
}
+ /**
+ * According to the updated event to modify the maintained event's
metrics
+ * @param executorId the executor whose metrics will be modified
+ */
+ private def updateModifiedMetrics(executorId: String): Unit = {
+val toBeModifiedEvent = executorIdToModifiedMaxMetrics.get(executorId)
+val latestEvent = executorIdToLatestMetrics.get(executorId)
+toBeModifiedEvent match {
+ case None => if (latestEvent.isDefined)
executorIdToModifiedMaxMetrics.update(
+executorId, latestEvent.get)
--- End diff --
@squito , I'm sorry I made a mistake here, you are really careful. Thank
you correct me again. You are right that `executorIdToLatestMetrics` will be
updated before we call `updateModifiedMetrics`, I'll remove the `if` statement.
Sorry to mislead you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r46150377
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.executor
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * :: DeveloperApi ::
+ * Metrics tracked during the execution of an executor.
+ *
+ * So, when adding new fields, take into consideration that the whole
object can be serialized for
+ * shipping off at any time to consumers of the SparkListener interface.
+ */
+@DeveloperApi
+class ExecutorMetrics extends Serializable {
+
+ /**
+ * Host's name the executor runs on
+ */
+ private var _hostname: String = _
+ def hostname: String = _hostname
+ private[spark] def setHostname(value: String) = _hostname = value
+
+ private var _transportMetrics: TransportMetrics = new TransportMetrics
+ def transportMetrics: TransportMetrics = _transportMetrics
+ private[spark] def setTransportMetrics(value: TransportMetrics) = {
+_transportMetrics = value
+ }
+
+ // for test only
+ def metricsDetails: (String, Long, Long, Long) = {
+(hostname, transportMetrics.timeStamp, transportMetrics.onHeapSize,
+ transportMetrics.offHeapSize)
+ }
+}
+
+/**
+ * :: DeveloperApi ::
+ * Metrics for network layer
+ */
+@DeveloperApi
+class TransportMetrics (
--- End diff --
I think this needs to be `Serializable` (in fact, there should probably be
some unit test which makes sure that ExecutorHeartbeats are serializable ...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45927657
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
---
@@ -122,6 +122,105 @@ class EventLoggingListenerSuite extends SparkFunSuite
with LocalSparkContext wit
"a fine:mind$dollar{bills}.1", None, Some("lz4")))
}
+ test("test event logger logging executor metrics") {
+import org.apache.spark.scheduler.cluster._
+import org.apache.spark.ui.memory._
+val conf = EventLoggingListenerSuite.getLoggingConf(testDirPath)
+val eventLogger = new EventLoggingListener("test-memListener", None,
testDirPath.toUri(), conf)
+val execId = "exec-1"
+val hostName = "host-1"
+
+eventLogger.start()
+eventLogger.onExecutorAdded(SparkListenerExecutorAdded(
+ 0L, execId, new ExecutorInfo(hostName, 1, Map.empty)))
+
+// stage 1 and stage 2 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics1 = MemoryListenerSuite.createExecutorMetrics(hostName,
1L, 20, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics1))
+val execMetrics2 = MemoryListenerSuite.createExecutorMetrics(hostName,
2L, 30, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics2))
+// stage1 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+// stage3 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+val execMetrics3 = MemoryListenerSuite.createExecutorMetrics(hostName,
3L, 30, 30)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics3))
+val execMetrics4 = MemoryListenerSuite.createExecutorMetrics(hostName,
4L, 20, 25)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics4))
+// stage 2 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+val execMetrics5 = MemoryListenerSuite.createExecutorMetrics(hostName,
5L, 15, 15)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics5))
+val execMetrics6 = MemoryListenerSuite.createExecutorMetrics(hostName,
6L, 25, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics6))
+// stage 3 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+eventLogger.onExecutorRemoved(SparkListenerExecutorRemoved(7L, execId,
""))
+
+// Totally there are 15 logged events, including:
+// 2 events of executor Added/Removed
+// 6 events of stage Submitted/Completed
+// 7 events of executorMetrics update (3 combined metrics and 4
original metrics)
+assert(eventLogger.loggedEvents.size === 15)
+eventLogger.stop()
+
+val logData = EventLoggingListener.openEventLog(new
Path(eventLogger.logPath), fileSystem)
+val lines = readLines(logData)
+Utils.tryWithSafeFinally {
+ // totally there are 15 lines, including SparkListenerLogStart event
and 14 other events
+ assert(lines.size === 16)
+
+ // 4 executor metrics that is the latest metrics updated before
stage submit and complete
+ val jsonMetrics = JsonProtocol.sparkEventFromJson(parse(lines(5)))
--- End diff --
I'm not a fan of pulling out very specific lines of the log here -- it
makes it harder for the reader to follow, and also makes the tests more
brittle. Could you instead have a util method like
`getLastExecutorMetricBeforeStageEnd(events: Seq[SparkListenerEvent], stageId:
Int): SparkListenerExecutorMetricsUpdate`? Then your checks would be more
clear, they'd look like
```scala
parsedLines = line.map { line =>
JsonProtocol.sparkEventFromJson(parse(line)) }
...
checkExecutorMetrics(
metrics = getLastExecutorMetricBeforeStageEnd(parsedLines, 3),
expMetrics = ...
)
```
(not quite the right args, but hopefully that conveys the idea). You'd
also need to make sure the stage end events had a completion time in there to
be able to grab the right event.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45926858
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
---
@@ -122,6 +122,105 @@ class EventLoggingListenerSuite extends SparkFunSuite
with LocalSparkContext wit
"a fine:mind$dollar{bills}.1", None, Some("lz4")))
}
+ test("test event logger logging executor metrics") {
+import org.apache.spark.scheduler.cluster._
+import org.apache.spark.ui.memory._
+val conf = EventLoggingListenerSuite.getLoggingConf(testDirPath)
+val eventLogger = new EventLoggingListener("test-memListener", None,
testDirPath.toUri(), conf)
+val execId = "exec-1"
+val hostName = "host-1"
+
+eventLogger.start()
+eventLogger.onExecutorAdded(SparkListenerExecutorAdded(
+ 0L, execId, new ExecutorInfo(hostName, 1, Map.empty)))
+
+// stage 1 and stage 2 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics1 = MemoryListenerSuite.createExecutorMetrics(hostName,
1L, 20, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics1))
+val execMetrics2 = MemoryListenerSuite.createExecutorMetrics(hostName,
2L, 30, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics2))
+// stage1 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+// stage3 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+val execMetrics3 = MemoryListenerSuite.createExecutorMetrics(hostName,
3L, 30, 30)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics3))
+val execMetrics4 = MemoryListenerSuite.createExecutorMetrics(hostName,
4L, 20, 25)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics4))
+// stage 2 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+val execMetrics5 = MemoryListenerSuite.createExecutorMetrics(hostName,
5L, 15, 15)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics5))
+val execMetrics6 = MemoryListenerSuite.createExecutorMetrics(hostName,
6L, 25, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics6))
+// stage 3 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+eventLogger.onExecutorRemoved(SparkListenerExecutorRemoved(7L, execId,
""))
+
+// Totally there are 15 logged events, including:
+// 2 events of executor Added/Removed
+// 6 events of stage Submitted/Completed
+// 7 events of executorMetrics update (3 combined metrics and 4
original metrics)
+assert(eventLogger.loggedEvents.size === 15)
+eventLogger.stop()
+
+val logData = EventLoggingListener.openEventLog(new
Path(eventLogger.logPath), fileSystem)
+val lines = readLines(logData)
+Utils.tryWithSafeFinally {
+ // totally there are 15 lines, including SparkListenerLogStart event
and 14 other events
--- End diff --
comment is wrong (off by one). maybe just make comment "one extra line for
SparkListenerLogStart"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45928969
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -228,6 +261,46 @@ private[spark] class EventLoggingListener(
fileSystem.rename(new Path(logPath + IN_PROGRESS), target)
}
+ /**
+ * According to the updated event to modify the maintained event's
metrics
+ * @param executorId the executor whose metrics will be modified
+ */
+ private def updateModifiedMetrics(executorId: String): Unit = {
+val toBeModifiedEvent = executorIdToModifiedMaxMetrics.get(executorId)
+val latestEvent = executorIdToLatestMetrics.get(executorId)
+toBeModifiedEvent match {
+ case None => if (latestEvent.isDefined)
executorIdToModifiedMaxMetrics.update(
+executorId, latestEvent.get)
--- End diff --
Bringing back a comment from an old diff -- I had mentioned that
`latestEvent` should always be defined, and you responded:
> `latestEvent` should not always be defined when
`toBeModifiedEvent.isEmpty` is true. The case is that at the beginning, the
`latestEvent` is empty before the first metrics update event received.
I don't follow. `updateModifiedMetrics` is only called from one place,
`onExecutorMetricsUpdated`, L240. And just before that, on L239,
`executorIdToLatestMetrics(execId)` is set. so I don't see how it could be
undefined. In fact, you could just change this method to take `latestEvent:
SparkListenerExecutorMetricsUpdate` and then L240 becomes
`updateModifiedMetrics(lightEvent)`, and here you get the executorId from that
event.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45927972
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
---
@@ -122,6 +122,105 @@ class EventLoggingListenerSuite extends SparkFunSuite
with LocalSparkContext wit
"a fine:mind$dollar{bills}.1", None, Some("lz4")))
}
+ test("test event logger logging executor metrics") {
+import org.apache.spark.scheduler.cluster._
+import org.apache.spark.ui.memory._
+val conf = EventLoggingListenerSuite.getLoggingConf(testDirPath)
+val eventLogger = new EventLoggingListener("test-memListener", None,
testDirPath.toUri(), conf)
+val execId = "exec-1"
+val hostName = "host-1"
+
+eventLogger.start()
+eventLogger.onExecutorAdded(SparkListenerExecutorAdded(
+ 0L, execId, new ExecutorInfo(hostName, 1, Map.empty)))
+
+// stage 1 and stage 2 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics1 = MemoryListenerSuite.createExecutorMetrics(hostName,
1L, 20, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics1))
+val execMetrics2 = MemoryListenerSuite.createExecutorMetrics(hostName,
2L, 30, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics2))
+// stage1 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+// stage3 submitted
+
eventLogger.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+val execMetrics3 = MemoryListenerSuite.createExecutorMetrics(hostName,
3L, 30, 30)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics3))
+val execMetrics4 = MemoryListenerSuite.createExecutorMetrics(hostName,
4L, 20, 25)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics4))
+// stage 2 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+val execMetrics5 = MemoryListenerSuite.createExecutorMetrics(hostName,
5L, 15, 15)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics5))
+val execMetrics6 = MemoryListenerSuite.createExecutorMetrics(hostName,
6L, 25, 10)
+
eventLogger.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId, execMetrics6))
+// stage 3 completed
+
eventLogger.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+eventLogger.onExecutorRemoved(SparkListenerExecutorRemoved(7L, execId,
""))
+
+// Totally there are 15 logged events, including:
+// 2 events of executor Added/Removed
+// 6 events of stage Submitted/Completed
+// 7 events of executorMetrics update (3 combined metrics and 4
original metrics)
+assert(eventLogger.loggedEvents.size === 15)
+eventLogger.stop()
+
+val logData = EventLoggingListener.openEventLog(new
Path(eventLogger.logPath), fileSystem)
+val lines = readLines(logData)
+Utils.tryWithSafeFinally {
+ // totally there are 15 lines, including SparkListenerLogStart event
and 14 other events
+ assert(lines.size === 16)
+
+ // 4 executor metrics that is the latest metrics updated before
stage submit and complete
+ val jsonMetrics = JsonProtocol.sparkEventFromJson(parse(lines(5)))
--- End diff --
another idea: write an integration test, which pumps the parsed events back
into the `MemoryListener`, and make sure it has the right status for each
stage. I actually think that might be a better idea for gaining confidence.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45929033
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -205,7 +231,14 @@ private[spark] class EventLoggingListener(
override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
// No-op because logging every update would be overkill
- override def onExecutorMetricsUpdate(event:
SparkListenerExecutorMetricsUpdate): Unit = { }
+ override def onExecutorMetricsUpdate(event:
SparkListenerExecutorMetricsUpdate): Unit = {
+// In order to avoid the logged event consumes too much storage size,
taskMetrics would not
+// be logged into event log file currently
+val lightEvent = SparkListenerExecutorMetricsUpdate(
+ event.execId, event.executorMetrics, Seq.empty)
+executorIdToLatestMetrics.update(lightEvent.execId, lightEvent)
--- End diff --
nit: more idiomatic scala to write `map.update(key,value)` with `map(key) =
value` (scala syntax sugar for `update` method)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45924369
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
+ test("test HashMap size for MemoryListener") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+val execId2 = "exec-2"
+
+(1 to 2).foreach { i =>
+
listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(i))
+ listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(i))
+}
+// stages are all completed, no activeStages now
+assert(listener.activeStagesToMem.isEmpty)
+
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, new ExecutorMetrics))
+// ExecutorMetrics is not related with Stages directly
+assert(listener.activeStagesToMem.isEmpty)
+
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId2, new ExecutorMetrics))
+// totally 2 executors updated their metrics
+assert(listener.activeExecutorIdToMem.size === 2)
+assert(listener.activeStagesToMem.size === 1)
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+assert(listener.activeStagesToMem.isEmpty)
+assert(listener.completedStagesToMem.size === 3)
+assert(listener.activeExecutorIdToMem.size ===
listener.latestExecIdToExecMetrics.size)
+assert(listener.removedExecutorIdToMem.isEmpty)
+ }
+
+ test("test first stage with no executor metrics update") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// stage 1, no metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+
+// stage 2, with one metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics = MemoryListenerSuite.createExecutorMetrics("host-1",
0L, 20, 10)
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, execMetrics))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+
+val mapForStage1 = listener.completedStagesToMem.get((1, 0)).get
+// no metrics for stage 1 since no metrics update for stage 1
+assert(mapForStage1.get(execId1).get.transportInfo === None)
+val mapForStage2 = listener.completedStagesToMem.get((2, 0)).get
+assert(mapForStage2.size === 1)
+val memInfo = mapForStage2.get(execId1).get
+assert(memInfo.transportInfo.isDefined)
+val transMetrics = memInfo.transportInfo.get
+assert((20, 10, MemTime(20, 0), MemTime(10, 0)) ===
(transMetrics.onHeapSize,
+ transMetrics.offHeapSize, transMetrics.peakOnHeapSizeTime,
transMetrics.peakOffHeapSizeTime))
+
+listener.onExecutorRemoved(SparkListenerExecutorRemoved(0L, execId1,
""))
+ }
+
+ test("test multiple metrics updated in one stage") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// multiple metrics updated in one stage
+
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45925385
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
+ test("test HashMap size for MemoryListener") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+val execId2 = "exec-2"
+
+(1 to 2).foreach { i =>
+
listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(i))
+ listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(i))
+}
+// stages are all completed, no activeStages now
+assert(listener.activeStagesToMem.isEmpty)
+
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, new ExecutorMetrics))
+// ExecutorMetrics is not related with Stages directly
+assert(listener.activeStagesToMem.isEmpty)
+
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId2, new ExecutorMetrics))
+// totally 2 executors updated their metrics
+assert(listener.activeExecutorIdToMem.size === 2)
+assert(listener.activeStagesToMem.size === 1)
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+assert(listener.activeStagesToMem.isEmpty)
+assert(listener.completedStagesToMem.size === 3)
+assert(listener.activeExecutorIdToMem.size ===
listener.latestExecIdToExecMetrics.size)
+assert(listener.removedExecutorIdToMem.isEmpty)
+ }
+
+ test("test first stage with no executor metrics update") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// stage 1, no metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+
+// stage 2, with one metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics = MemoryListenerSuite.createExecutorMetrics("host-1",
0L, 20, 10)
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, execMetrics))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+
+val mapForStage1 = listener.completedStagesToMem.get((1, 0)).get
+// no metrics for stage 1 since no metrics update for stage 1
+assert(mapForStage1.get(execId1).get.transportInfo === None)
+val mapForStage2 = listener.completedStagesToMem.get((2, 0)).get
+assert(mapForStage2.size === 1)
+val memInfo = mapForStage2.get(execId1).get
+assert(memInfo.transportInfo.isDefined)
+val transMetrics = memInfo.transportInfo.get
+assert((20, 10, MemTime(20, 0), MemTime(10, 0)) ===
(transMetrics.onHeapSize,
+ transMetrics.offHeapSize, transMetrics.peakOnHeapSizeTime,
transMetrics.peakOffHeapSizeTime))
--- End diff --
or since you do a similar assert a lot, use a helper method which takes the
4 expected values and a `transMetrics`, and does the 4 asserts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45925821
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
+ test("test HashMap size for MemoryListener") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+val execId2 = "exec-2"
+
+(1 to 2).foreach { i =>
+
listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(i))
+ listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(i))
+}
+// stages are all completed, no activeStages now
+assert(listener.activeStagesToMem.isEmpty)
+
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, new ExecutorMetrics))
+// ExecutorMetrics is not related with Stages directly
+assert(listener.activeStagesToMem.isEmpty)
+
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId2, new ExecutorMetrics))
+// totally 2 executors updated their metrics
+assert(listener.activeExecutorIdToMem.size === 2)
+assert(listener.activeStagesToMem.size === 1)
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+assert(listener.activeStagesToMem.isEmpty)
+assert(listener.completedStagesToMem.size === 3)
+assert(listener.activeExecutorIdToMem.size ===
listener.latestExecIdToExecMetrics.size)
+assert(listener.removedExecutorIdToMem.isEmpty)
+ }
+
+ test("test first stage with no executor metrics update") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// stage 1, no metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+
+// stage 2, with one metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics = MemoryListenerSuite.createExecutorMetrics("host-1",
0L, 20, 10)
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, execMetrics))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+
+val mapForStage1 = listener.completedStagesToMem.get((1, 0)).get
+// no metrics for stage 1 since no metrics update for stage 1
+assert(mapForStage1.get(execId1).get.transportInfo === None)
+val mapForStage2 = listener.completedStagesToMem.get((2, 0)).get
+assert(mapForStage2.size === 1)
+val memInfo = mapForStage2.get(execId1).get
+assert(memInfo.transportInfo.isDefined)
+val transMetrics = memInfo.transportInfo.get
+assert((20, 10, MemTime(20, 0), MemTime(10, 0)) ===
(transMetrics.onHeapSize,
+ transMetrics.offHeapSize, transMetrics.peakOnHeapSizeTime,
transMetrics.peakOffHeapSizeTime))
+
+listener.onExecutorRemoved(SparkListenerExecutorRemoved(0L, execId1,
""))
+ }
+
+ test("test multiple metrics updated in one stage") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// multiple metrics updated in one stage
+
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45925912
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
+ test("test HashMap size for MemoryListener") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+val execId2 = "exec-2"
+
+(1 to 2).foreach { i =>
+
listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(i))
+ listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(i))
+}
+// stages are all completed, no activeStages now
+assert(listener.activeStagesToMem.isEmpty)
+
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, new ExecutorMetrics))
+// ExecutorMetrics is not related with Stages directly
+assert(listener.activeStagesToMem.isEmpty)
+
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId2, new ExecutorMetrics))
+// totally 2 executors updated their metrics
+assert(listener.activeExecutorIdToMem.size === 2)
+assert(listener.activeStagesToMem.size === 1)
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+assert(listener.activeStagesToMem.isEmpty)
+assert(listener.completedStagesToMem.size === 3)
+assert(listener.activeExecutorIdToMem.size ===
listener.latestExecIdToExecMetrics.size)
+assert(listener.removedExecutorIdToMem.isEmpty)
+ }
+
+ test("test first stage with no executor metrics update") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// stage 1, no metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+
+// stage 2, with one metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics = MemoryListenerSuite.createExecutorMetrics("host-1",
0L, 20, 10)
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, execMetrics))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+
+val mapForStage1 = listener.completedStagesToMem.get((1, 0)).get
+// no metrics for stage 1 since no metrics update for stage 1
+assert(mapForStage1.get(execId1).get.transportInfo === None)
+val mapForStage2 = listener.completedStagesToMem.get((2, 0)).get
+assert(mapForStage2.size === 1)
+val memInfo = mapForStage2.get(execId1).get
+assert(memInfo.transportInfo.isDefined)
+val transMetrics = memInfo.transportInfo.get
+assert((20, 10, MemTime(20, 0), MemTime(10, 0)) ===
(transMetrics.onHeapSize,
+ transMetrics.offHeapSize, transMetrics.peakOnHeapSizeTime,
transMetrics.peakOffHeapSizeTime))
+
+listener.onExecutorRemoved(SparkListenerExecutorRemoved(0L, execId1,
""))
+ }
+
+ test("test multiple metrics updated in one stage") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// multiple metrics updated in one stage
+
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-159745331 `JsonProtocolSuite` should also have a backward compatability test for the new fields --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45922193
--- Diff: core/src/main/scala/org/apache/spark/ui/memory/MemoryTab.scala ---
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import scala.collection.mutable.HashMap
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.executor.{TransportMetrics, ExecutorMetrics}
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster.ExecutorInfo
+import org.apache.spark.ui.{SparkUITab, SparkUI}
+
+private[ui] class MemoryTab(parent: SparkUI) extends SparkUITab(parent,
"memory") {
+ val memoryListener = parent.memoryListener
+ val progressListener = parent.jobProgressListener
+ attachPage(new MemoryPage(this))
+ attachPage(new StageMemoryPage(this))
+}
+
+/**
+ * :: DeveloperApi ::
+ * A SparkListener that prepares information to be displayed on the
MemoryTab
+ */
+@DeveloperApi
+class MemoryListener extends SparkListener {
+ type ExecutorId = String
+ val activeExecutorIdToMem = new HashMap[ExecutorId, MemoryUIInfo]
+ val removedExecutorIdToMem = new HashMap[ExecutorId, MemoryUIInfo]
+ // latestExecIdToExecMetrics including all executors that is active and
removed.
+ // this may consume a lot of memory when executors are changing
frequently, e.g. in dynamical
+ // allocation mode.
+ val latestExecIdToExecMetrics = new HashMap[ExecutorId, ExecutorMetrics]
+ // activeStagesToMem a map maintains all executors memory information of
each stage,
+ // the Map type is [(stageId, attemptId), Seq[(executorId, MemoryUIInfo)]
+ val activeStagesToMem = new HashMap[(Int, Int), HashMap[ExecutorId,
MemoryUIInfo]]
+ val completedStagesToMem = new HashMap[(Int, Int), HashMap[ExecutorId,
MemoryUIInfo]]
+
+ override def onExecutorMetricsUpdate(event:
SparkListenerExecutorMetricsUpdate): Unit = {
+val executorId = event.execId
+val executorMetrics = event.executorMetrics
+val memoryInfo = activeExecutorIdToMem.getOrElseUpdate(executorId, new
MemoryUIInfo)
+memoryInfo.updateExecutorMetrics(executorMetrics)
+activeStagesToMem.foreach { case (_, stageMemMetrics) =>
+ if (stageMemMetrics.contains(executorId)) {
+
stageMemMetrics.get(executorId).get.updateExecutorMetrics(executorMetrics)
+ }
+}
+latestExecIdToExecMetrics.update(executorId, executorMetrics)
+ }
+
+ override def onExecutorAdded(event: SparkListenerExecutorAdded): Unit = {
+val executorId = event.executorId
+activeExecutorIdToMem.put(executorId, new
MemoryUIInfo(event.executorInfo))
+ }
+
+ override def onExecutorRemoved(event: SparkListenerExecutorRemoved):
Unit = {
+val executorId = event.executorId
+val info = activeExecutorIdToMem.remove(executorId)
+removedExecutorIdToMem.getOrElseUpdate(executorId, info.getOrElse(new
MemoryUIInfo))
+ }
+
+ override def onStageSubmitted(event: SparkListenerStageSubmitted): Unit
= {
+val stage = (event.stageInfo.stageId, event.stageInfo.attemptId)
+val memInfoMap = new HashMap[ExecutorId, MemoryUIInfo]
+activeExecutorIdToMem.foreach(idToMem => memInfoMap.update(idToMem._1,
new MemoryUIInfo))
+activeStagesToMem.update(stage, memInfoMap)
+ }
+
+ override def onStageCompleted(event: SparkListenerStageCompleted): Unit
= {
+val stage = (event.stageInfo.stageId, event.stageInfo.attemptId)
+activeStagesToMem.get(stage).map { memInfoMap =>
+ activeExecutorIdToMem.foreach { case (executorId, _) =>
+val memInfo = memInfoMap.getOrElse(executorId, new MemoryUIInfo)
+latestExecIdToExecMetrics.get(executorId).foreach {
prevExecutorMetrics =>
+ memInfo.updateExecutorMetrics(prevExecutorMetrics)
+}
+memInfoMap.update(executorId,
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45923064
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
+ test("test HashMap size for MemoryListener") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+val execId2 = "exec-2"
+
+(1 to 2).foreach { i =>
+
listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(i))
+ listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(i))
+}
+// stages are all completed, no activeStages now
+assert(listener.activeStagesToMem.isEmpty)
+
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, new ExecutorMetrics))
+// ExecutorMetrics is not related with Stages directly
+assert(listener.activeStagesToMem.isEmpty)
+
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId2, new ExecutorMetrics))
+// totally 2 executors updated their metrics
+assert(listener.activeExecutorIdToMem.size === 2)
+assert(listener.activeStagesToMem.size === 1)
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+assert(listener.activeStagesToMem.isEmpty)
+assert(listener.completedStagesToMem.size === 3)
+assert(listener.activeExecutorIdToMem.size ===
listener.latestExecIdToExecMetrics.size)
+assert(listener.removedExecutorIdToMem.isEmpty)
+ }
+
+ test("test first stage with no executor metrics update") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// stage 1, no metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+
+// stage 2, with one metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics = MemoryListenerSuite.createExecutorMetrics("host-1",
0L, 20, 10)
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, execMetrics))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+
+val mapForStage1 = listener.completedStagesToMem.get((1, 0)).get
+// no metrics for stage 1 since no metrics update for stage 1
+assert(mapForStage1.get(execId1).get.transportInfo === None)
+val mapForStage2 = listener.completedStagesToMem.get((2, 0)).get
+assert(mapForStage2.size === 1)
+val memInfo = mapForStage2.get(execId1).get
+assert(memInfo.transportInfo.isDefined)
+val transMetrics = memInfo.transportInfo.get
+assert((20, 10, MemTime(20, 0), MemTime(10, 0)) ===
(transMetrics.onHeapSize,
+ transMetrics.offHeapSize, transMetrics.peakOnHeapSizeTime,
transMetrics.peakOffHeapSizeTime))
--- End diff --
I think this is probably clearer as 4 separate asserts
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r45920423 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -85,6 +85,9 @@ private[spark] class Executor( env.blockManager.initialize(conf.getAppId) } + private val executorMetrics: ExecutorMetrics = new ExecutorMetrics + executorMetrics.setHostname(Utils.localHostName) --- End diff -- sorry I forgot about this in my last round. I don't understand the first sentence of your reply -- why can't you get the hostname & port? it looks like you can do exactly what you suggested to get the host & port from the rpc env with `executorMetrics.setHostPort(env.rpcEnv.address.hostPort)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45921896
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -205,7 +231,14 @@ private[spark] class EventLoggingListener(
override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
// No-op because logging every update would be overkill
- override def onExecutorMetricsUpdate(event:
SparkListenerExecutorMetricsUpdate): Unit = { }
+ override def onExecutorMetricsUpdate(event:
SparkListenerExecutorMetricsUpdate): Unit = {
+// In order to avoid the logged event consumes too much storage size,
taskMetrics would not
+// be logged into event log file currently
+val lightEvent = SparkListenerExecutorMetricsUpdate(
--- End diff --
the comment above the method is inaccurate (this is no longer a no-op
obviously). Can you change it something like "Track executor metrics for
logging on stage start and end".
I'd also update the inner comment to something like "We only track the
executor metrics in each stage, so we drop the task metrics as they are quite
verbose". and maybe rename "lightEvent" to "eventWithoutTaskMetrics"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45923224
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
--- End diff --
I don't think you're using `LocalSparkContext` here at all
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45919009
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala
---
@@ -47,6 +50,39 @@ class NettyBlockTransferService(conf: SparkConf,
securityManager: SecurityManage
private[this] var server: TransportServer = _
private[this] var clientFactory: TransportClientFactory = _
private[this] var appId: String = _
+ private[this] var clock: Clock = new SystemClock()
+
+ /**
+ * Use a different clock for this allocation manager. This is mainly
used for testing.
+ */
+ def setClock(newClock: Clock): Unit = {
+clock = newClock
+ }
+
+ override def getMemMetrics(executorMetrics: ExecutorMetrics): Unit = {
--- End diff --
yeah, though the super class is `private[spark]`, new methods you add here
are still public by default :(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45922769
--- Diff:
core/src/test/scala/org/apache/spark/ui/memory/MemoryListenerSuite.scala ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ui.memory
+
+import org.apache.spark._
+import org.apache.spark.executor._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster._
+
+class MemoryListenerSuite extends SparkFunSuite with LocalSparkContext {
+ test("test HashMap size for MemoryListener") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+val execId2 = "exec-2"
+
+(1 to 2).foreach { i =>
+
listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(i))
+ listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(i))
+}
+// stages are all completed, no activeStages now
+assert(listener.activeStagesToMem.isEmpty)
+
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, new ExecutorMetrics))
+// ExecutorMetrics is not related with Stages directly
+assert(listener.activeStagesToMem.isEmpty)
+
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(3))
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId2, new ExecutorMetrics))
+// totally 2 executors updated their metrics
+assert(listener.activeExecutorIdToMem.size === 2)
+assert(listener.activeStagesToMem.size === 1)
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(3))
+
+assert(listener.activeStagesToMem.isEmpty)
+assert(listener.completedStagesToMem.size === 3)
+assert(listener.activeExecutorIdToMem.size ===
listener.latestExecIdToExecMetrics.size)
+assert(listener.removedExecutorIdToMem.isEmpty)
+ }
+
+ test("test first stage with no executor metrics update") {
+val listener = new MemoryListener
+val execId1 = "exec-1"
+
+listener.onExecutorAdded(
+ SparkListenerExecutorAdded(0L, execId1, new ExecutorInfo("host1", 1,
Map.empty)))
+
+// stage 1, no metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(1))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(1))
+
+// stage 2, with one metrics update
+listener.onStageSubmitted(MemoryListenerSuite.createStageStartEvent(2))
+val execMetrics = MemoryListenerSuite.createExecutorMetrics("host-1",
0L, 20, 10)
+
listener.onExecutorMetricsUpdate(MemoryListenerSuite.createExecutorMetricsUpdateEvent(
+ execId1, execMetrics))
+listener.onStageCompleted(MemoryListenerSuite.createStageEndEvent(2))
+
+val mapForStage1 = listener.completedStagesToMem.get((1, 0)).get
--- End diff --
nit: rather than `hashmap.get(x).get` you can just do `hashmap(x)`, here
and elsewhere
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r45584487 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -20,8 +20,10 @@ package org.apache.spark.scheduler import java.io._ import java.net.URI +import org.apache.spark.executor.TransportMetrics --- End diff -- fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45584427
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala
---
@@ -47,6 +50,39 @@ class NettyBlockTransferService(conf: SparkConf,
securityManager: SecurityManage
private[this] var server: TransportServer = _
private[this] var clientFactory: TransportClientFactory = _
private[this] var appId: String = _
+ private[this] var clock: Clock = new SystemClock()
+
+ /**
+ * Use a different clock for this allocation manager. This is mainly
used for testing.
+ */
+ def setClock(newClock: Clock): Unit = {
+clock = newClock
+ }
+
+ override def getMemMetrics(executorMetrics: ExecutorMetrics): Unit = {
--- End diff --
the super class is `private[spark]`, do we still need to specify here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45585344
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -228,6 +260,40 @@ private[spark] class EventLoggingListener(
fileSystem.rename(new Path(logPath + IN_PROGRESS), target)
}
+ /**
+ * According to the updated event to modify the maintained event's
metrics
+ * @param executorId the executor whose metrics will be modified
+ */
+ private def updateModifiedMetrics(executorId: String): Unit = {
+val toBeModifiedEvent = executorIdToModifiedMaxMetrics.get(executorId)
+val latestEvent = executorIdToLatestMetrics.get(executorId)
+if (toBeModifiedEvent.isEmpty) {
+ if (latestEvent.isDefined)
executorIdToModifiedMaxMetrics.update(executorId, latestEvent.get)
+} else {
+ val toBeModifiedMetrics =
toBeModifiedEvent.get.executorMetrics.transportMetrics
+ if (toBeModifiedMetrics.isDefined) {
--- End diff --
`latestEvent` should not always be defined when `toBeModifiedEvent.isEmpty`
is true. The case is that at the beginning, the `latestEvent` is empty before
the first metrics update event received.
Thank you for you cleaner code style example, I've updated in my code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45584147
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.executor
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * :: DeveloperApi ::
+ * Metrics tracked during the execution of an executor.
+ *
+ * So, when adding new fields, take into consideration that the whole
object can be serialized for
+ * shipping off at any time to consumers of the SparkListener interface.
+ */
+@DeveloperApi
+class ExecutorMetrics extends Serializable {
+
+ /**
+ * Host's name the executor runs on
+ */
+ private var _hostname: String = _
+ def hostname: String = _hostname
+ private[spark] def setHostname(value: String) = _hostname = value
+
+ private var _transportMetrics: Option[TransportMetrics] = None
+ def transportMetrics: Option[TransportMetrics] = _transportMetrics
+ private[spark] def setTransportMetrics(value: Option[TransportMetrics])
= {
+_transportMetrics = value
+ }
+}
+
+/**
+ * :: DeveloperApi ::
+ * Metrics for network layer
+ */
+@DeveloperApi
+case class TransportMetrics(
+timeStamp: Long,
+onHeapSize: Long,
+directSize: Long)
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r45584922
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.executor
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * :: DeveloperApi ::
+ * Metrics tracked during the execution of an executor.
+ *
+ * So, when adding new fields, take into consideration that the whole
object can be serialized for
+ * shipping off at any time to consumers of the SparkListener interface.
+ */
+@DeveloperApi
+class ExecutorMetrics extends Serializable {
+
+ /**
+ * Host's name the executor runs on
+ */
+ private var _hostname: String = _
+ def hostname: String = _hostname
+ private[spark] def setHostname(value: String) = _hostname = value
+
+ private var _transportMetrics: Option[TransportMetrics] = None
--- End diff --
changed it to be always present in the updated code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158924858 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158925588 **[Test build #46533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46533/consoleFull)** for PR 7753 at commit [`1ed48c1`](https://github.com/apache/spark/commit/1ed48c11df78e07a5f6d71de5a3d3b7fecf245b9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158923546 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158923548 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46522/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158923478 **[Test build #46522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46522/consoleFull)** for PR 7753 at commit [`1ed48c1`](https://github.com/apache/spark/commit/1ed48c11df78e07a5f6d71de5a3d3b7fecf245b9). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158889734 **[Test build #46522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46522/consoleFull)** for PR 7753 at commit [`1ed48c1`](https://github.com/apache/spark/commit/1ed48c11df78e07a5f6d71de5a3d3b7fecf245b9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158890951 @squito thank you for your comments, it helped a lot, I updated some unit tests. If you got time, can you help to review? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r45584881 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -91,7 +93,12 @@ private[spark] class EventLoggingListener( private[scheduler] val loggedEvents = new ArrayBuffer[JValue] // Visible for tests only. - private[scheduler] val logPath = getLogPath(logBaseDir, appId, appAttemptId, compressionCodecName) + private[scheduler] val logPath = getLogPath( +logBaseDir, appId, appAttemptId, compressionCodecName) --- End diff -- yes, my mistake, the width is 100, it's correct. I'll change it back later, thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-159150999 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46576/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-159150934 **[Test build #46576 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46576/consoleFull)** for PR 7753 at commit [`b438077`](https://github.com/apache/spark/commit/b43807731228c4e5c672436a197e94341bf4ffd3). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-159150997 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-159132863 **[Test build #46576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46576/consoleFull)** for PR 7753 at commit [`b438077`](https://github.com/apache/spark/commit/b43807731228c4e5c672436a197e94341bf4ffd3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158964110 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158964006 **[Test build #46533 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46533/consoleFull)** for PR 7753 at commit [`1ed48c1`](https://github.com/apache/spark/commit/1ed48c11df78e07a5f6d71de5a3d3b7fecf245b9). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-158964111 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46533/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157934465 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157934461 **[Test build #46290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46290/consoleFull)** for PR 7753 at commit [`c78628e`](https://github.com/apache/spark/commit/c78628ee70dc20186c03253ec2eada430779d3bb). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157934466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46290/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157934156 **[Test build #46290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46290/consoleFull)** for PR 7753 at commit [`c78628e`](https://github.com/apache/spark/commit/c78628ee70dc20186c03253ec2eada430779d3bb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157958565 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46292/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157958517 **[Test build #46292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46292/consoleFull)** for PR 7753 at commit [`a93bd96`](https://github.com/apache/spark/commit/a93bd962b2c9e6369a2257d70c1c378c68d77500). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = System.currentTimeMillis)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157958564 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157937502 **[Test build #46292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46292/consoleFull)** for PR 7753 at commit [`a93bd96`](https://github.com/apache/spark/commit/a93bd962b2c9e6369a2257d70c1c378c68d77500). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157418537 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46091/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157418522 **[Test build #46091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46091/consoleFull)** for PR 7753 at commit [`75e63c3`](https://github.com/apache/spark/commit/75e63c366fcee3fed5cfe5c1066ab9727fd1f574). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = System.currentTimeMillis, timeStamp: Long = 0L)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157418529 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157417841 **[Test build #46091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46091/consoleFull)** for PR 7753 at commit [`75e63c3`](https://github.com/apache/spark/commit/75e63c366fcee3fed5cfe5c1066ab9727fd1f574). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157428635 **[Test build #46092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46092/consoleFull)** for PR 7753 at commit [`0c1241c`](https://github.com/apache/spark/commit/0c1241ca4512032598371a18df1ea7ac8ef98b8a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157474249 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157474157 **[Test build #46092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46092/consoleFull)** for PR 7753 at commit [`0c1241c`](https://github.com/apache/spark/commit/0c1241ca4512032598371a18df1ea7ac8ef98b8a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `class TransportMetrics (`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = System.currentTimeMillis, timeStamp: Long = 0L)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-157474253 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46092/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153299607 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153299560 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153301490 **[Test build #44912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44912/consoleFull)** for PR 7753 at commit [`c836fb9`](https://github.com/apache/spark/commit/c836fb9bc99e11f2ae5e52ac797c14be03a62f5e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153299381 jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153298000 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44890/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153297998 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153297942 **[Test build #44890 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44890/consoleFull)** for PR 7753 at commit [`c836fb9`](https://github.com/apache/spark/commit/c836fb9bc99e11f2ae5e52ac797c14be03a62f5e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `case class TransportMetrics(`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153337574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44912/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153337572 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153337475 **[Test build #44912 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44912/consoleFull)** for PR 7753 at commit [`c836fb9`](https://github.com/apache/spark/commit/c836fb9bc99e11f2ae5e52ac797c14be03a62f5e). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class ExecutorMetrics extends Serializable `\n * `case class TransportMetrics(`\n * `class MemoryListener extends SparkListener `\n * `class MemoryUIInfo `\n * `class TransportMemSize `\n * `case class MemTime(memorySize: Long = 0L, timeStamp: Long = 0L)`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r43801975
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.executor
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * :: DeveloperApi ::
+ * Metrics tracked during the execution of an executor.
+ *
+ * So, when adding new fields, take into consideration that the whole
object can be serialized for
+ * shipping off at any time to consumers of the SparkListener interface.
+ */
+@DeveloperApi
+class ExecutorMetrics extends Serializable {
+
+ /**
+ * Host's name the executor runs on
+ */
+ private var _hostname: String = _
+ def hostname: String = _hostname
+ private[spark] def setHostname(value: String) = _hostname = value
+
+ private var _transportMetrics: Option[TransportMetrics] = None
+ def transportMetrics: Option[TransportMetrics] = _transportMetrics
+ private[spark] def setTransportMetrics(value: Option[TransportMetrics])
= {
+_transportMetrics = value
+ }
+}
+
+/**
+ * :: DeveloperApi ::
+ * Metrics for network layer
+ */
+@DeveloperApi
+case class TransportMetrics(
+timeStamp: Long,
+onHeapSize: Long,
+directSize: Long)
--- End diff --
I know "direct" was my suggestion earlier, but now I see that actually we
already use "offheap" extensively in the codebase, so lets use that instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r43802608
--- Diff:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala
---
@@ -47,6 +50,39 @@ class NettyBlockTransferService(conf: SparkConf,
securityManager: SecurityManage
private[this] var server: TransportServer = _
private[this] var clientFactory: TransportClientFactory = _
private[this] var appId: String = _
+ private[this] var clock: Clock = new SystemClock()
+
+ /**
+ * Use a different clock for this allocation manager. This is mainly
used for testing.
+ */
+ def setClock(newClock: Clock): Unit = {
+clock = newClock
+ }
+
+ override def getMemMetrics(executorMetrics: ExecutorMetrics): Unit = {
--- End diff --
the class probably should have been `private[spark]`, but since it isn't,
both of these methods should be at least `private[spark]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r43805729
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -228,6 +260,40 @@ private[spark] class EventLoggingListener(
fileSystem.rename(new Path(logPath + IN_PROGRESS), target)
}
+ /**
+ * According to the updated event to modify the maintained event's
metrics
+ * @param executorId the executor whose metrics will be modified
+ */
+ private def updateModifiedMetrics(executorId: String): Unit = {
+val toBeModifiedEvent = executorIdToModifiedMaxMetrics.get(executorId)
+val latestEvent = executorIdToLatestMetrics.get(executorId)
+if (toBeModifiedEvent.isEmpty) {
+ if (latestEvent.isDefined)
executorIdToModifiedMaxMetrics.update(executorId, latestEvent.get)
+} else {
+ val toBeModifiedMetrics =
toBeModifiedEvent.get.executorMetrics.transportMetrics
+ if (toBeModifiedMetrics.isDefined) {
--- End diff --
won't `latestEvent` always be defined? In fact, at the one call site, you
could even just pass in `latestEvent` directly so you avoid another lookup. I
also think this becomes slightly cleaner w/ pattern matching:
```scala
private def updateModifiedMetrics(executorId: String, latestEvent:
SparkListenerExecutorMetricsUpdate): Unit = {
executorIdToModifiedMaxMetrics.get(executorId) match {
case None => executorIdToModifiedMaxMetrics.update(executorId,
latestEvent)
case Some(toBeModEvent) =>
val toBeModMetrics = toBeModEvent.executorMetrics.transportMetrics
...
}
}
```
and depending on whether or not we need to keep `tranportMetrics` as an
option, we may need to handle the else case here, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-153490332 Hi @liyezhang556520 thanks for updating. Can you add some unit tests, especially MemoryListener? Eg., push some events into an EventLoggingListener, then read those events back into the MemoryListener and make sure the stages have the right metrics. You can use the cases you describe in your design doc. also all the info available in the memory tab should also be available through the json api as well ... that can wait for a followup if you really need to. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r43802736 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -20,8 +20,10 @@ package org.apache.spark.scheduler import java.io._ import java.net.URI +import org.apache.spark.executor.TransportMetrics --- End diff -- nit: ordering --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7753#discussion_r43802786 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -91,7 +93,12 @@ private[spark] class EventLoggingListener( private[scheduler] val loggedEvents = new ArrayBuffer[JValue] // Visible for tests only. - private[scheduler] val logPath = getLogPath(logBaseDir, appId, appAttemptId, compressionCodecName) + private[scheduler] val logPath = getLogPath( +logBaseDir, appId, appAttemptId, compressionCodecName) --- End diff -- this shouldn't need to change, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7753#discussion_r43802303
--- Diff:
core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.executor
+
+import org.apache.spark.annotation.DeveloperApi
+
+/**
+ * :: DeveloperApi ::
+ * Metrics tracked during the execution of an executor.
+ *
+ * So, when adding new fields, take into consideration that the whole
object can be serialized for
+ * shipping off at any time to consumers of the SparkListener interface.
+ */
+@DeveloperApi
+class ExecutorMetrics extends Serializable {
+
+ /**
+ * Host's name the executor runs on
+ */
+ private var _hostname: String = _
+ def hostname: String = _hostname
+ private[spark] def setHostname(value: String) = _hostname = value
+
+ private var _transportMetrics: Option[TransportMetrics] = None
+ def transportMetrics: Option[TransportMetrics] = _transportMetrics
+ private[spark] def setTransportMetrics(value: Option[TransportMetrics])
= {
+_transportMetrics = value
+ }
+}
+
+/**
+ * :: DeveloperApi ::
+ * Metrics for network layer
+ */
+@DeveloperApi
+case class TransportMetrics(
+timeStamp: Long,
+onHeapSize: Long,
+directSize: Long)
--- End diff --
also, I think we should avoid using a case class. The problem is binary
compatibility of the apply / unapply methods when you add a field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
