[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18935 Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81420/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #81420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81420/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #81420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81420/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18935 Just realized the last build is 18 days ago. Triggered a new now. Will merge after passing tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18935 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18935 LGTM since enableVerboseMetrics is off by default. Merging to master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18935 The change itself looks good, but I'm afraid it's a bit too verbose to add all of those metrics. cc @zsxwing . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user squito commented on the issue: https://github.com/apache/spark/pull/18935 lgtm any more thoughts @jiangxb1987 @zsxwing ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 @squito can you please review again? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 @squito , as the next following step, I would expose these metrics with MetricsSystem, I'm thinking of exposing shuffle related Netty memory usage. For RPC related memory usage, I'm not fully sure about the value of exposing them, though exposing them is a quite cheap call. And furthermore, I think we can collect them in the driver side and display either through REST API or via web UI. But I think part of should be considered well on how to deliver to the end users. As for the content of netty memory metrics, by default only two major memory usage will be exposed unless we enable verbose. I was thinking of picking some detailed metrics, but I found it is hard to decide which one is more important over others, so instead I chose to expose all of them and let user to decide which one is important. Also like you mentioned they're quite cheap and off by default, so I think it might be good to also optionally expose detailed infos. > are you at all surprised that the direct & heap memory are exactly the same? Are things getting mixed up somewhere? Or maybe those are just the values from the initialization netty always does. I think it doesn't mix up anything. AFAIK it is what Netty `PooledByteBufAllocator` does at initialization, this 16MB is the default chunk size. I guess at initialization, Netty will create one chunk per arena. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user squito commented on the issue: https://github.com/apache/spark/pull/18935 thanks for the added info @jerryshao. (a) ok makes sense now about the use of this -- it is not exposed now, you plan to expose it in future changes. That is fine, however I would like to at least see a *plan* for how you want to expose this before we merge this. I think past efforts have gotten tripped on how we'd expose it in the driver; perhaps the simplest thing to do is expose it to the metric system? There just isn't any point in this unless we're going to do something with it. (b) thanks for including the example of the metrics. But one minor thing I noticed -- are you at all surprised that the direct & heap memory are exactly the same? Are things getting mixed up somewhere? Or maybe those are just the values from the initialization netty always does. I definitely think that is more metrics than most users care about, and would want verbose off by default. I am wondering if those metrics are useful at all ... but I guess they are pretty cheap, and maybe there is some scenario where a user would care about some of them, eg. `numHugeAllocations`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
@squito , here is the list of full metrics I dumped from running
application. By default, if we don't enable verbose metric, only 2 metrics
("usedHeapMemory" and "usedDirectMemory") will be tracked. If we enable verbose
metrics, then the details of each arena will be tracked, the number of detailed
metrics is 16, so the total verbose metric number will be "16 * arena number *
2", here arena number is based on the number of cores, and "2" represents
"direct arena" and "heap arena".
I agree this might be too verbose for a user to track netty memory usage,
so by default I will not enable verbose metric. Unless if developers wants to
know the detailed behaviors of network memory usage, then they could enable
this configuration, this might be useful for tuning and performance analysis.
What do you think?
```
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveBytes
value = 16777216
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveTinyAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numAllocations
value = 7
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numDeallocations
value = 7
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numHugeDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numNormalAllocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numNormalDeallocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numSmallAllocations
value = 3
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numSmallDeallocations
value = 3
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numTinyAllocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numTinyDeallocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveBytes
value = 16777216
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveTinyAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numAllocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numDeallocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numHugeDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numNormalDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numSmallDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.he
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 Hi @squito , thanks a lot for your review. This PR doesn't expose Netty metrics to MetricsSystem or collect to driver side (what did before), it just provides a way for `TransportClientFactory` and `TransportServer` to expose metrics to other components. So we can either pull this into MetricsSystem, or report to driver in the following PRs. That's why you don't find any way for user to query in this PR. This will be done it following ones. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80835/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80835/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80835/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80829/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80829/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80829/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80822/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80822/testReport)** for PR 18935 at commit [`9c2b3f0`](https://github.com/apache/spark/commit/9c2b3f0a0156525dfdc4c7da1c9b7d1d41fd05f3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80823/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80823/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80823/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80822/testReport)** for PR 18935 at commit [`9c2b3f0`](https://github.com/apache/spark/commit/9c2b3f0a0156525dfdc4c7da1c9b7d1d41fd05f3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 Thanks @zsxwing . I was thinking to expose the details of memory allocation in Netty for user to monitor and tune, user could filter out unrelated metrics. Maybe you're right, it is too verbose and too detailed to expose each arena's detail, let me change the codes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18935 Could you bump the netty version to use its new APIs rather than reflection? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18935 I'll review this tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 @jiangxb1987 @zsxwing would you please help to review this PR, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80662/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80662/testReport)** for PR 18935 at commit [`03d0d46`](https://github.com/apache/spark/commit/03d0d46a0cd57501255d9b04b0d909ce37fef57a). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class TransportClientFactory implements Closeable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80662/testReport)** for PR 18935 at commit [`03d0d46`](https://github.com/apache/spark/commit/03d0d46a0cd57501255d9b04b0d909ce37fef57a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 Thanks @dongjoon-hyun for your review, will change it soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80641/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80641/testReport)** for PR 18935 at commit [`05c1f4d`](https://github.com/apache/spark/commit/05c1f4de4f00639d5f1acf1b9c061e4894d8286d). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` public class TransportClientFactory implements Closeable ` * `public class NettyMemoryMetrics implements MetricSet ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80641/testReport)** for PR 18935 at commit [`05c1f4d`](https://github.com/apache/spark/commit/05c1f4de4f00639d5f1acf1b9c061e4894d8286d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18935 Retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18935 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80618/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80618/testReport)** for PR 18935 at commit [`05c1f4d`](https://github.com/apache/spark/commit/05c1f4de4f00639d5f1acf1b9c061e4894d8286d). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` public class TransportClientFactory implements Closeable ` * `public class NettyMemoryMetrics implements MetricSet ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 @jsoltren @squito I take the liberty to get another shot of exposing Netty memory metrics. This PR doesn't address any display related things, only expose memory usage from Netty client factory and server. The following PRs could leverage this to display shuffle, RPC as well as other component's memory usage for Netty. This can be connected to `MetricsSystem`, also can be collected back to driver as you did before. Please help to review, your comments are greatly appreciated, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18935 **[Test build #80618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80618/testReport)** for PR 18935 at commit [`05c1f4d`](https://github.com/apache/spark/commit/05c1f4de4f00639d5f1acf1b9c061e4894d8286d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
