[jira] [Commented] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats

[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745362#comment-16745362 ] Edwina Lu commented on SPARK-26329: --- [~wypoon], this sounds like a good solution. Sending the

[jira] [Updated] (SPARK-26399) Add new stage-level REST APIs and parameters

[ https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-26399: -- Description: Add the peak values for the metrics to the stages REST API. Also add a new

[jira] [Created] (SPARK-26399) Add new stage-level REST APIs and parameters

Edwina Lu created SPARK-26399: - Summary: Add new stage-level REST APIs and parameters Key: SPARK-26399 URL: https://issues.apache.org/jira/browse/SPARK-26399 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-23431) Expose the new executor memory metrics at the stage level

[ https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23431: -- Description: Collect and show the new executor memory metrics for each stage, to provide more

[jira] [Commented] (SPARK-23431) Expose the new executor memory metrics at the stage level

[ https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724472#comment-16724472 ] Edwina Lu commented on SPARK-23431: --- Splitting off the new REST APIs into a new subtask. > Expose the

[jira] [Updated] (SPARK-23432) Expose executor memory metrics in the web UI for executors

[ https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23432: -- Summary: Expose executor memory metrics in the web UI for executors (was: Expose executor memory

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717988#comment-16717988 ] Edwina Lu commented on SPARK-23206: --- [~irashid], [~rezasafi], [~wypoon], as discussed, I've split

[jira] [Updated] (SPARK-23432) Expose executor memory metrics in the web UI for executors

[ https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23432: -- Description: Add the new memory metrics (jvmUsedMemory, executionMemory, storageMemory, and

[jira] [Updated] (SPARK-23432) Expose executor memory metrics in the web UI for executors

[ https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23432: -- Description: Add the new memory metrics (jvmUsedMemory, executionMemory, storageMemory, and

[jira] [Created] (SPARK-26341) Expose executor memory metrics at the stage level, in the Stages tab

Edwina Lu created SPARK-26341: - Summary: Expose executor memory metrics at the stage level, in the Stages tab Key: SPARK-26341 URL: https://issues.apache.org/jira/browse/SPARK-26341 Project: Spark

[jira] [Commented] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats

[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715491#comment-16715491 ] Edwina Lu commented on SPARK-26329: --- More frequent polling (and more accurate metrics) sound good.

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665628#comment-16665628 ] Edwina Lu commented on SPARK-23206: --- [~irashid], yes, I am planning to work on the other tasks for

[jira] [Updated] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23429: -- Fix Version/s: 3.0.0 > Add executor memory metrics to heartbeat and expose in executors REST API >

[jira] [Commented] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492960#comment-16492960 ] Edwina Lu commented on SPARK-24356: --- Thanks! This is interesting, and could help with some of our

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472405#comment-16472405 ] Edwina Lu commented on SPARK-23206: --- The design discussion for SPARK-23206 is scheduled for Monday, May

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470789#comment-16470789 ] Edwina Lu commented on SPARK-23206: --- [~irashid], I do not have the rest of the changes for 2.3/master

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446247#comment-16446247 ] Edwina Lu edited comment on SPARK-23206 at 4/22/18 3:36 AM: CANCELLED: -The

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446247#comment-16446247 ] Edwina Lu commented on SPARK-23206: --- The design discussion is scheduled for Monday 4/23 PDT at 11am

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443025#comment-16443025 ] Edwina Lu commented on SPARK-23206: --- We are planning a design discussion for early next week. Please

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442928#comment-16442928 ] Edwina Lu commented on SPARK-23206: --- [~assia6], there is an initial PR: [[Github] Pull Request #20940

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441006#comment-16441006 ] Edwina Lu commented on SPARK-23206: --- [~assia6], could you please try the new link,

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440326#comment-16440326 ] Edwina Lu commented on SPARK-23206: --- [~smilegator], please try:

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438879#comment-16438879 ] Edwina Lu edited comment on SPARK-23206 at 4/17/18 3:15 AM: After discussion

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438879#comment-16438879 ] Edwina Lu edited comment on SPARK-23206 at 4/17/18 3:14 AM: After discussion

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: (was: SPARK-23206 Design Doc.pdf) > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: SPARK-23206 Design Doc.pdf > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: SPARK-23206 Design Doc.pdf > Additional Memory Tuning Metrics >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438879#comment-16438879 ] Edwina Lu commented on SPARK-23206: --- After discussion with [~irashid] on the PR, we've decided to move

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430922#comment-16430922 ] Edwina Lu commented on SPARK-23206: --- The doc wasn't very clear what the quantile values were for.

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430765#comment-16430765 ] Edwina Lu commented on SPARK-23206: --- [~irashid], this would be quantile values for peak executor memory

[jira] [Updated] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23429: -- Description: Add new executor level memory metrics ( jvmUsedMemory, onHeapExecutionMemory,

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409775#comment-16409775 ] Edwina Lu commented on SPARK-23429: --- [~assia6], it's being reviewed internally right now. I will post

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387083#comment-16387083 ] Edwina Lu commented on SPARK-23206: --- [~assia6], sorry for the delay. I'm planning to submit a pull

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378879#comment-16378879 ] Edwina Lu commented on SPARK-23206: --- [~assia6], I haven't submitted the PR yet – the code is currently

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364979#comment-16364979 ] Edwina Lu commented on SPARK-23206: --- After discussion with [~cltlfcjin] and the eBay Hadoop team, we

[jira] [Created] (SPARK-23432) Expose executor memory metrics in the web UI for executors and stages

Edwina Lu created SPARK-23432: - Summary: Expose executor memory metrics in the web UI for executors and stages Key: SPARK-23432 URL: https://issues.apache.org/jira/browse/SPARK-23432 Project: Spark

[jira] [Created] (SPARK-23431) Expose the new executor memory metrics at the stage level

Edwina Lu created SPARK-23431: - Summary: Expose the new executor memory metrics at the stage level Key: SPARK-23431 URL: https://issues.apache.org/jira/browse/SPARK-23431 Project: Spark Issue

[jira] [Updated] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23429: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-23206 > Add executor memory metrics to

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364967#comment-16364967 ] Edwina Lu commented on SPARK-23429: --- Subtask of SPARK-23206 > Add executor memory metrics to heartbeat

[jira] [Created] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

Edwina Lu created SPARK-23429: - Summary: Add executor memory metrics to heartbeat and expose in executors REST API Key: SPARK-23429 URL: https://issues.apache.org/jira/browse/SPARK-23429 Project: Spark

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Issue Type: Umbrella (was: Improvement) > Additional Memory Tuning Metrics >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358866#comment-16358866 ] Edwina Lu commented on SPARK-23206: --- [~irashid], total memory and off heap memory is also very useful

[jira] [Comment Edited] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357846#comment-16357846 ] Edwina Lu edited comment on SPARK-23206 at 2/9/18 2:30 AM: --- [~cltlfcjin],

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357846#comment-16357846 ] Edwina Lu commented on SPARK-23206: --- [~cltlfcjin], thanks for uploading the screenshot – these would be

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357841#comment-16357841 ] Edwina Lu commented on SPARK-23206: --- [~jerryshao], thanks for your help and advice. [~cltlfcjin], I'll

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355994#comment-16355994 ] Edwina Lu commented on SPARK-23206: --- We ([~jerryshao], [~zhz] and I) are planning a conference call via

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: StageTab.png > Additional Memory Tuning Metrics > > >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: (was: StageTab.png) > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: (was: ExecutorsTab.png) > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: ExecutorsTab.png > Additional Memory Tuning Metrics > > >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341526#comment-16341526 ] Edwina Lu commented on SPARK-23206: --- We'd like to monitor the following executor level metrics: * JVM

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: ExecutorsTab.png > Additional Memory Tuning Metrics > > >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: (was: ExecutorsTab.png) > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: ExecutorsTab.png > Additional Memory Tuning Metrics > > >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: StageTab.png > Additional Memory Tuning Metrics > > >

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340474#comment-16340474 ] Edwina Lu commented on SPARK-23206: --- [~jerryshao], yes  SPARK-9103 is similar, and this proposal

[jira] [Comment Edited] (SPARK-9103) Tracking spark's memory usage

[ https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340449#comment-16340449 ] Edwina Lu edited comment on SPARK-9103 at 1/26/18 2:08 AM: --- We (at LinkedIn) are

[jira] [Commented] (SPARK-9103) Tracking spark's memory usage

[ https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340449#comment-16340449 ] Edwina Lu commented on SPARK-9103: -- We (at LinkedIn) are interested in gathering more memory metrics as

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338559#comment-16338559 ] Edwina Lu commented on SPARK-23206: --- Thanks, [~zsxwing]. Making the new metrics available in the

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Description: At LinkedIn, we have multiple clusters, running thousands of Spark applications, and

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: MemoryTuningMetricsDesignDoc.pdf > Additional Memory Tuning Metrics >

[jira] [Created] (SPARK-23206) Additional Memory Tuning Metrics

Edwina Lu created SPARK-23206: - Summary: Additional Memory Tuning Metrics Key: SPARK-23206 URL: https://issues.apache.org/jira/browse/SPARK-23206 Project: Spark Issue Type: Improvement