[jira] [Resolved] (SPARK-13723) YARN - Change behavior of --num-executors when spark.dynamicAllocation.enabled true

[ https://issues.apache.org/jira/browse/SPARK-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13723. --- Resolution: Fixed Fix Version/s: 2.0.0 > YARN - Change behavior of --num-executors whe

[jira] [Updated] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15725: -- Fix Version/s: (was: 2.0.0) 2.0.1 > Dynamic allocation hangs YARN app wh

[jira] [Commented] (SPARK-15955) Failed Spark application returns with exitcode equals to zero

[ https://issues.apache.org/jira/browse/SPARK-15955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15348220#comment-15348220 ] Thomas Graves commented on SPARK-15955: --- there are some corner cases in spark 1.x t

[jira] [Updated] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

[ https://issues.apache.org/jira/browse/SPARK-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15990: -- Assignee: Saisai Shao > Support rolling log aggregation for Spark running on YARN > ---

[jira] [Resolved] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

[ https://issues.apache.org/jira/browse/SPARK-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15990. --- Resolution: Fixed Fix Version/s: 2.1.0 > Support rolling log aggregation for Spark run

[jira] [Commented] (SPARK-16382) YARN - Dynamic allocation with spark.executor.instances should increase max executors.

[ https://issues.apache.org/jira/browse/SPARK-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364363#comment-15364363 ] Thomas Graves commented on SPARK-16382: --- I think we should fail and complain and I

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364834#comment-15364834 ] Thomas Graves commented on SPARK-8425: -- Added some questions to the design doc > Add

[jira] [Commented] (SPARK-27005) Design sketch: Accelerator-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781728#comment-16781728 ] Thomas Graves commented on SPARK-27005: --- It seems like we are mixing gpu's as stat

[jira] [Commented] (SPARK-27024) Design executor interface to support GPU resources

[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781734#comment-16781734 ] Thomas Graves commented on SPARK-27024: --- I will be looking at this and propose a d

[jira] [Commented] (SPARK-27024) Design executor interface to support GPU resources

[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782167#comment-16782167 ] Thomas Graves commented on SPARK-27024: --- This and SPARK-27005 basically split the

[jira] [Commented] (SPARK-27005) Design sketch: Accelerator-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784555#comment-16784555 ] Thomas Graves commented on SPARK-27005: --- so we have both a google design doc and t

[jira] [Comment Edited] (SPARK-27005) Design sketch: Accelerator-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784555#comment-16784555 ] Thomas Graves edited comment on SPARK-27005 at 3/5/19 3:40 PM: ---

[jira] [Updated] (SPARK-27024) Executor interface for cluster managers to support GPU resources

[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27024: -- Description: The executor interface shall deal with the resources allocated to the executor by

[jira] [Updated] (SPARK-27024) Executor interface for cluster managers to support GPU resources

[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27024: -- Description: The executor interface shall deal with the resources allocated to the executor b

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809893#comment-16809893 ] Thomas Graves commented on SPARK-27364: --- working on this, will post a basic design

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812488#comment-16812488 ] Thomas Graves commented on SPARK-27364: --- There are 3 main user facing impacts for

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/8/19 3:01 PM: ---

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/8/19 3:10 PM: ---

[jira] [Assigned] (SPARK-27361) YARN support for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27361: - Assignee: Thomas Graves > YARN support for GPU-aware scheduling > -

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815403#comment-16815403 ] Thomas Graves commented on SPARK-27396: --- I can shephard it. > SPIP: Public APIs f

[jira] [Commented] (SPARK-27176) Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4

[ https://issues.apache.org/jira/browse/SPARK-27176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815408#comment-16815408 ] Thomas Graves commented on SPARK-27176: --- It looks like the hadoop-3.2 profile no l

[jira] [Commented] (SPARK-27176) Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4

[ https://issues.apache.org/jira/browse/SPARK-27176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815414#comment-16815414 ] Thomas Graves commented on SPARK-27176: --- looks like I see one: https://github.com/

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/11/19 1:48 PM: --

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815465#comment-16815465 ] Thomas Graves commented on SPARK-27364: --- So there is actually another one we need

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815465#comment-16815465 ] Thomas Graves edited comment on SPARK-27364 at 4/11/19 2:37 PM: --

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818053#comment-16818053 ] Thomas Graves commented on SPARK-25250: --- [~cloud_fan] can you please add details a

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819153#comment-16819153 ] Thomas Graves commented on SPARK-27396: --- Since I don't hear any strong objections

[jira] [Created] (SPARK-27488) Driver interface to support GPU resources

Thomas Graves created SPARK-27488: - Summary: Driver interface to support GPU resources Key: SPARK-27488 URL: https://issues.apache.org/jira/browse/SPARK-27488 Project: Spark Issue Type: Stor

[jira] [Commented] (SPARK-27488) Driver interface to support GPU resources

[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820119#comment-16820119 ] Thomas Graves commented on SPARK-27488: --- Note, the api design is here: https://iss

[jira] [Resolved] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27364. --- Resolution: Fixed > User-facing APIs for GPU-aware scheduling >

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820120#comment-16820120 ] Thomas Graves commented on SPARK-27364: --- based on no comments on this I'm going to

[jira] [Reopened] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-27364: --- reopening since it has a subtask > User-facing APIs for GPU-aware scheduling >

[jira] [Created] (SPARK-27489) UI updates to show executor resource information

Thomas Graves created SPARK-27489: - Summary: UI updates to show executor resource information Key: SPARK-27489 URL: https://issues.apache.org/jira/browse/SPARK-27489 Project: Spark Issue Type

[jira] [Created] (SPARK-27492) High level user documentation

Thomas Graves created SPARK-27492: - Summary: High level user documentation Key: SPARK-27492 URL: https://issues.apache.org/jira/browse/SPARK-27492 Project: Spark Issue Type: Story C

[jira] [Updated] (SPARK-27495) Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Summary: Support Stage level resource configuration and scheduling (was: Support Stage level

[jira] [Created] (SPARK-27495) Support Stage level resource scheduling

Thomas Graves created SPARK-27495: - Summary: Support Stage level resource scheduling Key: SPARK-27495 URL: https://issues.apache.org/jira/browse/SPARK-27495 Project: Spark Issue Type: Story

[jira] [Commented] (SPARK-24655) [K8S] Custom Docker Image Expectations and Documentation

[ https://issues.apache.org/jira/browse/SPARK-24655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821270#comment-16821270 ] Thomas Graves commented on SPARK-24655: --- >From the linked issues it seems the goal

[jira] [Commented] (SPARK-27492) High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821900#comment-16821900 ] Thomas Graves commented on SPARK-27492: --- Sorry, it is under the epic and didn't re

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Summary: GPU scheduling - High level user documentation (was: High level user documentation)

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark,  https://issues.apache

[jira] [Commented] (SPARK-27495) Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves commented on SPARK-27495: --- Unfortunately the link to the original de

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:28 PM: --

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:29 PM: --

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823097#comment-16823097 ] Thomas Graves commented on SPARK-27396: --- thanks for the questions and commenting,

[jira] [Commented] (SPARK-23850) We should not redact username|user|url from UI by default

[ https://issues.apache.org/jira/browse/SPARK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449054#comment-16449054 ] Thomas Graves commented on SPARK-23850: --- the url seems somewhat silly to me to, loo

[jira] [Resolved] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-22683. --- Resolution: Fixed Fix Version/s: 2.4.0 > DynamicAllocation wastes resources by allocat

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450131#comment-16450131 ] Thomas Graves commented on SPARK-22683: --- Note this added a new config spark.dynamic

[jira] [Assigned] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-22683: - Assignee: Julien Cuquemelle > DynamicAllocation wastes resources by allocating container

[jira] [Created] (SPARK-24124) Spark history server should create spark.history.store.path and set permissions properly

Thomas Graves created SPARK-24124: - Summary: Spark history server should create spark.history.store.path and set permissions properly Key: SPARK-24124 URL: https://issues.apache.org/jira/browse/SPARK-24124

[jira] [Commented] (SPARK-24124) Spark history server should create spark.history.store.path and set permissions properly

[ https://issues.apache.org/jira/browse/SPARK-24124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458835#comment-16458835 ] Thomas Graves commented on SPARK-24124: --- [~vanzin]  any objections to this? > Spar

[jira] [Commented] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467803#comment-16467803 ] Thomas Graves commented on SPARK-21033: --- [~cloud_fan] the followup PR [https://gith

[jira] [Commented] (SPARK-6235) Address various 2G limits

[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482746#comment-16482746 ] Thomas Graves commented on SPARK-6235: -- >> Still unsupported: * large task results

[jira] [Created] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and it doesn't have any other active executors

Thomas Graves created SPARK-24413: - Summary: Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and it doesn't have any other active executors Key: SPARK-24413 URL:

[jira] [Updated] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24413: -- Summary: Executor Blacklisting shouldn't immediately fail the application if dynamic allocatio

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493978#comment-16493978 ] Thomas Graves commented on SPARK-24413: --- [~imranr]  thoughts on this? > Executor

[jira] [Created] (SPARK-24414) Stages page doesn't show all task attempts when failures

Thomas Graves created SPARK-24414: - Summary: Stages page doesn't show all task attempts when failures Key: SPARK-24414 URL: https://issues.apache.org/jira/browse/SPARK-24414 Project: Spark Is

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494083#comment-16494083 ] Thomas Graves commented on SPARK-24414: --- to reproduce this simply start a shell:

[jira] [Created] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

Thomas Graves created SPARK-24415: - Summary: Stage page aggregated executor metrics wrong when failures Key: SPARK-24415 URL: https://issues.apache.org/jira/browse/SPARK-24415 Project: Spark

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Attachment: Screen Shot 2018-05-29 at 2.15.38 PM.png > Stage page aggregated executor metrics

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the ag

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494199#comment-16494199 ] Thomas Graves commented on SPARK-24414: --- looks like this was broken by SPARK-23147

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494237#comment-16494237 ] Thomas Graves commented on SPARK-24414: --- I am looking to see if we can just return

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494265#comment-16494265 ] Thomas Graves commented on SPARK-24414: --- also just an fyi I also filed SPARK-24415

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494279#comment-16494279 ] Thomas Graves commented on SPARK-24413: --- thanks for linking those we can just dup

[jira] [Resolved] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24413. --- Resolution: Duplicate > Executor Blacklisting shouldn't immediately fail the application if

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the ag

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the ag

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the ag

[jira] [Commented] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495200#comment-16495200 ] Thomas Graves commented on SPARK-24415: --- this might actually be an order of events

[jira] [Commented] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495204#comment-16495204 ] Thomas Graves commented on SPARK-24415: --- It also looks like in the history server

[jira] [Commented] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495224#comment-16495224 ] Thomas Graves commented on SPARK-24415: --- ok so the issue here is in the AppStatusL

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Priority: Critical (was: Major) > Stage page aggregated executor metrics wrong when failures

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512500#comment-16512500 ] Thomas Graves commented on SPARK-24552: --- I agree, I don't think changing the attem

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Blocker (was: Major) > Task attempt numbers are reused when stages are retried > --

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512504#comment-16512504 ] Thomas Graves commented on SPARK-24552: --- Note if this is a correctness bug and can

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Critical (was: Blocker) > Task attempt numbers are reused when stages are retried >

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512519#comment-16512519 ] Thomas Graves commented on SPARK-24552: --- sorry just realized the v2 api is still m

[jira] [Commented] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512643#comment-16512643 ] Thomas Graves commented on SPARK-24539: --- Its possible, I thought when I checked th

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512691#comment-16512691 ] Thomas Graves commented on SPARK-22148: --- ok, just update if you start working on i

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Affects Version/s: 2.2.0 2.2.1 2.3.0

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Blocker (was: Critical) > Task attempt numbers are reused when stages are retried >

[jira] [Updated] (SPARK-24519) MapStatus has 2000 hardcoded

[ https://issues.apache.org/jira/browse/SPARK-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24519: -- Description: MapStatus uses hardcoded value of 2000 partitions to determine if it should use

[jira] [Created] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

Thomas Graves created SPARK-24622: - Summary: Task attempts in other stage attempts not killed when one task attempt succeeds Key: SPARK-24622 URL: https://issues.apache.org/jira/browse/SPARK-24622 Pro

[jira] [Commented] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

[ https://issues.apache.org/jira/browse/SPARK-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519416#comment-16519416 ] Thomas Graves commented on SPARK-24622: --- Need to investigate further/test to make

[jira] [Commented] (SPARK-24611) Clean up OutputCommitCoordinator

[ https://issues.apache.org/jira/browse/SPARK-24611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519420#comment-16519420 ] Thomas Graves commented on SPARK-24611: --- [~joshrosen]  just noticed you were the l

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519430#comment-16519430 ] Thomas Graves commented on SPARK-24552: --- this is actually a problem with hadoop co

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves commented on SPARK-24552: --- more details on hadoop committer side: S

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:01 PM: --

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:02 PM: --

[jira] [Resolved] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24589. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0

[jira] [Updated] (SPARK-22897) Expose stageAttemptId in TaskContext

[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22897: -- Fix Version/s: 2.2.2 > Expose stageAttemptId in TaskContext > ---

[jira] [Resolved] (SPARK-24519) MapStatus has 2000 hardcoded

[ https://issues.apache.org/jira/browse/SPARK-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24519. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > MapStatus

[jira] [Resolved] (SPARK-24372) Create script for preparing RCs

[ https://issues.apache.org/jira/browse/SPARK-24372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24372. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0 > Create scr

[jira] [Resolved] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-23309. --- Resolution: Works for Me > Spark 2.3 cached query performance 20-30% worse then spark 2.2 >

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527865#comment-16527865 ] Thomas Graves commented on SPARK-23309: --- We tried this on a newest 2.3.1 and haven

[jira] [Commented] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528293#comment-16528293 ] Thomas Graves commented on SPARK-24615: --- maybe I"m missing it but how is this work

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529869#comment-16529869 ] Thomas Graves commented on SPARK-17181: --- that would be a question for [~marymwu]

[jira] [Updated] (SPARK-13343) speculative tasks that didn't commit shouldn't be marked as success

[ https://issues.apache.org/jira/browse/SPARK-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13343: -- Description: Currently Speculative tasks that didn't commit can show up as success  (dependin

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16537034#comment-16537034 ] Thomas Graves commented on SPARK-16534: --- If we aren't going to do this we should c

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16537035#comment-16537035 ] Thomas Graves commented on SPARK-16534: --- I agree it seems a bit of a bad user stor

<    4   5   6   7   8   9   10   11   12   13   >