[jira] [Resolved] (SPARK-30322) Add stage level scheduling docs

[ https://issues.apache.org/jira/browse/SPARK-30322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30322. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed > Add stage

[jira] [Updated] (SPARK-30322) Add stage level scheduling docs

[ https://issues.apache.org/jira/browse/SPARK-30322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-30322: -- Description: Add stage level scheduling docs. > Add stage level scheduling docs >

[jira] [Commented] (SPARK-32470) Remove task result size check for shuffle map stage

[ https://issues.apache.org/jira/browse/SPARK-32470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167233#comment-17167233 ] Thomas Graves commented on SPARK-32470: --- Please add a description to the Jira as to why > Remove

[jira] [Resolved] (SPARK-32175) Fix the order between initialization for ExecutorPlugin and starting heartbeat thread

[ https://issues.apache.org/jira/browse/SPARK-32175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-32175. --- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed > Fix the order

[jira] [Comment Edited] (SPARK-32429) Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch

[ https://issues.apache.org/jira/browse/SPARK-32429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166627#comment-17166627 ] Thomas Graves edited comment on SPARK-32429 at 7/28/20, 6:47 PM: - Yes so

[jira] [Commented] (SPARK-32429) Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch

[ https://issues.apache.org/jira/browse/SPARK-32429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166627#comment-17166627 ] Thomas Graves commented on SPARK-32429: --- Yes so for this first implementation we didn't really

[jira] [Commented] (SPARK-32429) Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch

[ https://issues.apache.org/jira/browse/SPARK-32429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165934#comment-17165934 ] Thomas Graves commented on SPARK-32429: --- So this doesn't address the task side, it addresses the

[jira] [Assigned] (SPARK-30794) Stage Level scheduling: Add ability to set off heap memory

[ https://issues.apache.org/jira/browse/SPARK-30794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-30794: - Assignee: Zhongwei Zhu > Stage Level scheduling: Add ability to set off heap memory >

[jira] [Resolved] (SPARK-30794) Stage Level scheduling: Add ability to set off heap memory

[ https://issues.apache.org/jira/browse/SPARK-30794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30794. --- Fix Version/s: 3.1.0 Resolution: Fixed > Stage Level scheduling: Add ability to set

[jira] [Updated] (SPARK-32429) Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch

[ https://issues.apache.org/jira/browse/SPARK-32429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-32429: -- Description: It would be nice if standalone mode could allow users to set

[jira] [Commented] (SPARK-32287) Flaky Test: ExecutorAllocationManagerSuite.add executors default profile

[ https://issues.apache.org/jira/browse/SPARK-32287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164512#comment-17164512 ] Thomas Graves commented on SPARK-32287: --- I haven't been able to reproduce but the best I can tell

[jira] [Created] (SPARK-32430) Allow plugins to inject rules into AQE query stage preparation

Thomas Graves created SPARK-32430: - Summary: Allow plugins to inject rules into AQE query stage preparation Key: SPARK-32430 URL: https://issues.apache.org/jira/browse/SPARK-32430 Project: Spark

[jira] [Commented] (SPARK-32429) Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch

[ https://issues.apache.org/jira/browse/SPARK-32429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164440#comment-17164440 ] Thomas Graves commented on SPARK-32429: --- [~jiangxb1987] [~Ngone51] thoughts on the above proposal?

[jira] [Created] (SPARK-32429) Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch

Thomas Graves created SPARK-32429: - Summary: Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch Key: SPARK-32429 URL: https://issues.apache.org/jira/browse/SPARK-32429 Project:

[jira] [Resolved] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-31418. --- Fix Version/s: 3.1.0 Assignee: Venkata krishnan Sowrirajan Resolution: Fixed

[jira] [Created] (SPARK-32334) Investigate commonizing Columnar and Row data transformations

Thomas Graves created SPARK-32334: - Summary: Investigate commonizing Columnar and Row data transformations Key: SPARK-32334 URL: https://issues.apache.org/jira/browse/SPARK-32334 Project: Spark

[jira] [Created] (SPARK-32333) Drop references to Master

Thomas Graves created SPARK-32333: - Summary: Drop references to Master Key: SPARK-32333 URL: https://issues.apache.org/jira/browse/SPARK-32333 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-32332) AQE doesn't adequately allow for Columnar Processing extension

Thomas Graves created SPARK-32332: - Summary: AQE doesn't adequately allow for Columnar Processing extension Key: SPARK-32332 URL: https://issues.apache.org/jira/browse/SPARK-32332 Project: Spark

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158739#comment-17158739 ] Thomas Graves commented on SPARK-32037: --- Any other opinions on what we should go with here? >

[jira] [Commented] (SPARK-32287) Flaky Test: ExecutorAllocationManagerSuite.add executors default profile

[ https://issues.apache.org/jira/browse/SPARK-32287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158521#comment-17158521 ] Thomas Graves commented on SPARK-32287: --- I'll try to reproduce and investigate locally > Flaky

[jira] [Assigned] (SPARK-32036) Remove references to "blacklist"/"whitelist" language (outside of blacklisting feature)

[ https://issues.apache.org/jira/browse/SPARK-32036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-32036: - Assignee: Erik Krogen > Remove references to "blacklist"/"whitelist" language (outside

[jira] [Resolved] (SPARK-32036) Remove references to "blacklist"/"whitelist" language (outside of blacklisting feature)

[ https://issues.apache.org/jira/browse/SPARK-32036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-32036. --- Fix Version/s: 3.1.0 Resolution: Fixed > Remove references to

[jira] [Commented] (SPARK-32120) Single GPU is allocated multiple times

[ https://issues.apache.org/jira/browse/SPARK-32120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152118#comment-17152118 ] Thomas Graves commented on SPARK-32120: --- Yes this was a known limitation when we did SPARK-30969

[jira] [Commented] (SPARK-32119) ExecutorPlugin doesn't work with Standalone Cluster

[ https://issues.apache.org/jira/browse/SPARK-32119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148746#comment-17148746 ] Thomas Graves commented on SPARK-32119: --- You can specify the jars in extraClassPath but it

[jira] [Commented] (SPARK-32135) Show Spark Driver name on Spark history web page

[ https://issues.apache.org/jira/browse/SPARK-32135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148742#comment-17148742 ] Thomas Graves commented on SPARK-32135: --- [~gaurangi]can you please clarify what you mean by

[jira] [Assigned] (SPARK-32068) Spark 3 UI task launch time show in error time zone

[ https://issues.apache.org/jira/browse/SPARK-32068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-32068: - Assignee: JinxinTang > Spark 3 UI task launch time show in error time zone >

[jira] [Resolved] (SPARK-32068) Spark 3 UI task launch time show in error time zone

[ https://issues.apache.org/jira/browse/SPARK-32068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-32068. --- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed > Spark 3 UI task

[jira] [Resolved] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24615. --- Fix Version/s: 3.0.0 Resolution: Fixed > SPIP: Accelerator-aware task scheduling for

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144194#comment-17144194 ] Thomas Graves commented on SPARK-24615: --- [~mengxr]  [~jiangxb1987]  it would be nice to mark this

[jira] [Commented] (SPARK-32037) Rename blacklisting feature to avoid language with racist connotation

[ https://issues.apache.org/jira/browse/SPARK-32037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143151#comment-17143151 ] Thomas Graves commented on SPARK-32037: --- I agree healthy/unhealty could mean other things then the

[jira] [Resolved] (SPARK-31029) Occasional class not found error in user's Future code using global ExecutionContext

[ https://issues.apache.org/jira/browse/SPARK-31029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-31029. --- Fix Version/s: 3.1.0 Assignee: shanyu zhao Resolution: Fixed > Occasional

[jira] [Commented] (ARROW-9019) [Python] hdfs fails to connect to for HDFS 3.x cluster

[ https://issues.apache.org/jira/browse/ARROW-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135940#comment-17135940 ] Thomas Graves commented on ARROW-9019: -- can you give more details on what was missing?  I used the

[jira] [Resolved] (SPARK-30845) spark-submit pyspark app on yarn uploads local pyspark archives

[ https://issues.apache.org/jira/browse/SPARK-30845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30845. --- Fix Version/s: 3.1.0 Assignee: shanyu zhao Resolution: Fixed > spark-submit

[jira] [Created] (ARROW-9019) pyarrow hdfs fails to connect to for HDFS 3.x cluster

Thomas Graves created ARROW-9019: Summary: pyarrow hdfs fails to connect to for HDFS 3.x cluster Key: ARROW-9019 URL: https://issues.apache.org/jira/browse/ARROW-9019 Project: Apache Arrow

[jira] [Created] (ARROW-9019) pyarrow hdfs fails to connect to for HDFS 3.x cluster

Thomas Graves created ARROW-9019: Summary: pyarrow hdfs fails to connect to for HDFS 3.x cluster Key: ARROW-9019 URL: https://issues.apache.org/jira/browse/ARROW-9019 Project: Apache Arrow

[jira] [Created] (SPARK-31856) Handle locality wait reset better when executors added

Thomas Graves created SPARK-31856: - Summary: Handle locality wait reset better when executors added Key: SPARK-31856 URL: https://issues.apache.org/jira/browse/SPARK-31856 Project: Spark

[jira] [Updated] (SPARK-31788) Error when creating UnionRDD of PairRDDs

[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31788: -- Priority: Blocker (was: Major) > Error when creating UnionRDD of PairRDDs >

[jira] [Commented] (SPARK-31788) Error when creating UnionRDD of PairRDDs

[ https://issues.apache.org/jira/browse/SPARK-31788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113574#comment-17113574 ] Thomas Graves commented on SPARK-31788: --- [~sanket991]the link you provided is not to public Apache

[jira] [Resolved] (SPARK-29303) UI updates for stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29303. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed > UI updates

[jira] [Commented] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112245#comment-17112245 ] Thomas Graves commented on SPARK-31437: --- Originally when I thought about this briefly I was

[jira] [Resolved] (SPARK-31621) Spark Master UI Fails to load if application is waiting for workers to launch driver

[ https://issues.apache.org/jira/browse/SPARK-31621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-31621. --- Fix Version/s: 3.1.0 3.0.0 Resolution: Fixed > Spark Master UI

[jira] [Assigned] (SPARK-31621) Spark Master UI Fails to load if application is waiting for workers to launch driver

[ https://issues.apache.org/jira/browse/SPARK-31621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-31621: - Assignee: Akshat Bordia > Spark Master UI Fails to load if application is waiting for

[jira] [Assigned] (SPARK-31235) Separates different categories of applications

[ https://issues.apache.org/jira/browse/SPARK-31235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-31235: - Assignee: wangzhun > Separates different categories of applications >

[jira] [Resolved] (SPARK-31235) Separates different categories of applications

[ https://issues.apache.org/jira/browse/SPARK-31235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-31235. --- Fix Version/s: (was: 3.0.0) 3.1.0 Resolution: Fixed >

[jira] [Created] (SPARK-31637) Stage Level Scheduling UI - add tooltips for resource profile ino

Thomas Graves created SPARK-31637: - Summary: Stage Level Scheduling UI - add tooltips for resource profile ino Key: SPARK-31637 URL: https://issues.apache.org/jira/browse/SPARK-31637 Project: Spark

[jira] [Commented] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084086#comment-17084086 ] Thomas Graves commented on SPARK-31437: --- so there are multiple reasons they are tied together for

[jira] [Created] (SPARK-31444) Pyspark memory and cores calculation doesn't account for task cpus

Thomas Graves created SPARK-31444: - Summary: Pyspark memory and cores calculation doesn't account for task cpus Key: SPARK-31444 URL: https://issues.apache.org/jira/browse/SPARK-31444 Project: Spark

[jira] [Commented] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083250#comment-17083250 ] Thomas Graves commented on SPARK-31437: --- This is something I wanted to eventually do but I have

[jira] [Updated] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31437: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Try assigning tasks to

[jira] [Updated] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31418: -- Issue Type: Improvement (was: Bug) > Blacklisting feature aborts Spark job without retrying

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078278#comment-17078278 ] Thomas Graves commented on SPARK-22148: --- so off the top of my head, I think the main issue with

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077614#comment-17077614 ] Thomas Graves commented on SPARK-22148: --- I'm not sure I follow what you are saying.  Are you just

[jira] [Updated] (SPARK-31378) stage level scheduling dynamic allocation bug with initial num executors

[ https://issues.apache.org/jira/browse/SPARK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31378: -- Affects Version/s: (was: 3.0.0) 3.1.0 > stage level scheduling

[jira] [Created] (SPARK-31378) stage level scheduling dynamic allocation bug with initial num executors

Thomas Graves created SPARK-31378: - Summary: stage level scheduling dynamic allocation bug with initial num executors Key: SPARK-31378 URL: https://issues.apache.org/jira/browse/SPARK-31378 Project:

[jira] [Commented] (SPARK-30299) Dynamic allocation with Standalone mode calculates to many executors needed

[ https://issues.apache.org/jira/browse/SPARK-30299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076525#comment-17076525 ] Thomas Graves commented on SPARK-30299: --- Note that there are other places in the code that uses

[jira] [Resolved] (SPARK-29153) ResourceProfile conflict resolution stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29153. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed >

[jira] [Resolved] (SPARK-31179) Fast fail the connection while last shuffle connection failed in the last retry IO wait

[ https://issues.apache.org/jira/browse/SPARK-31179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-31179. --- Fix Version/s: 3.1.0 Assignee: feiwang Resolution: Fixed > Fast fail the

[jira] [Created] (SPARK-31323) Stage level scheduling: dedup resource profiles on creation

Thomas Graves created SPARK-31323: - Summary: Stage level scheduling: dedup resource profiles on creation Key: SPARK-31323 URL: https://issues.apache.org/jira/browse/SPARK-31323 Project: Spark

[jira] [Commented] (SPARK-30873) Handling Node Decommissioning for Yarn cluster manger in Spark

[ https://issues.apache.org/jira/browse/SPARK-30873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071953#comment-17071953 ] Thomas Graves commented on SPARK-30873: --- so I think this is a dup of SPARK-30835. > Handling Node

[jira] [Updated] (SPARK-31314) Revert SPARK-29285 to fix shuffle regression caused by creating temporary file eagerly

[ https://issues.apache.org/jira/browse/SPARK-31314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31314: -- Issue Type: Bug (was: Improvement) > Revert SPARK-29285 to fix shuffle regression caused by

[jira] [Resolved] (SPARK-31219) YarnShuffleService doesn't close idle netty channel

[ https://issues.apache.org/jira/browse/SPARK-31219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-31219. --- Fix Version/s: 3.1.0 3.0.0 Assignee: Manu Zhang

[jira] [Updated] (SPARK-31219) YarnShuffleService doesn't close idle netty channel

[ https://issues.apache.org/jira/browse/SPARK-31219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31219: -- Issue Type: Bug (was: Improvement) > YarnShuffleService doesn't close idle netty channel >

[jira] [Commented] (SPARK-30322) Add stage level scheduling docs

[ https://issues.apache.org/jira/browse/SPARK-30322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068690#comment-17068690 ] Thomas Graves commented on SPARK-30322: --- document merge strategy (max values) > Add stage level

[jira] [Resolved] (SPARK-29154) Update Spark scheduler for stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29154. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed > Update

[jira] [Created] (SPARK-31055) Update config docs for shuffle local host reads to have dep on external shuffle service

Thomas Graves created SPARK-31055: - Summary: Update config docs for shuffle local host reads to have dep on external shuffle service Key: SPARK-31055 URL: https://issues.apache.org/jira/browse/SPARK-31055

[jira] [Commented] (SPARK-27651) Avoid the network when block manager fetches shuffle blocks from the same host

[ https://issues.apache.org/jira/browse/SPARK-27651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052251#comment-17052251 ] Thomas Graves commented on SPARK-27651: --- thanks, that makes sense. I can look in more details at

[jira] [Comment Edited] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

[ https://issues.apache.org/jira/browse/SPARK-31043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051664#comment-17051664 ] Thomas Graves edited comment on SPARK-31043 at 3/4/20, 10:18 PM: -

[jira] [Commented] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

[ https://issues.apache.org/jira/browse/SPARK-31043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051664#comment-17051664 ] Thomas Graves commented on SPARK-31043: --- rebuilt and still see the error. The full exception in

[jira] [Commented] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

[ https://issues.apache.org/jira/browse/SPARK-31043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051655#comment-17051655 ] Thomas Graves commented on SPARK-31043: --- A couple of my colleagues actually ran into this and

[jira] [Commented] (SPARK-27651) Avoid the network when block manager fetches shuffle blocks from the same host

[ https://issues.apache.org/jira/browse/SPARK-27651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051630#comment-17051630 ] Thomas Graves commented on SPARK-27651: --- It looks like this only works when using the external

[jira] [Commented] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

[ https://issues.apache.org/jira/browse/SPARK-31043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051554#comment-17051554 ] Thomas Graves commented on SPARK-31043: --- I'm working on tracing down what broke this. [~srowen]

[jira] [Created] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

Thomas Graves created SPARK-31043: - Summary: Spark 3.0 built against hadoop2.7 can't start standalone master Key: SPARK-31043 URL: https://issues.apache.org/jira/browse/SPARK-31043 Project: Spark

[jira] [Assigned] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-30049: - Assignee: Javier Fuentes > SQL fails to parse when comment contains an unmatched quote

[jira] [Resolved] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30049. --- Fix Version/s: 3.1.0 3.0.0 Resolution: Fixed > SQL fails to parse

[jira] [Assigned] (SPARK-30388) mark all running map stages of finished job as finished

[ https://issues.apache.org/jira/browse/SPARK-30388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-30388: - Assignee: Xuesen Liang > mark all running map stages of finished job as finished >

[jira] [Resolved] (SPARK-30388) mark all running map stages of finished job as finished

[ https://issues.apache.org/jira/browse/SPARK-30388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30388. --- Fix Version/s: 3.1.0 3.0.0 Resolution: Fixed > mark all running

[jira] [Resolved] (SPARK-29149) Update YARN cluster manager For Stage Level Scheduling

[ https://issues.apache.org/jira/browse/SPARK-29149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29149. --- Fix Version/s: 3.1.0 Resolution: Fixed > Update YARN cluster manager For Stage Level

[jira] [Commented] (SPARK-30987) ResourceDiscoveryPluginSuite sometimes fails

[ https://issues.apache.org/jira/browse/SPARK-30987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17047783#comment-17047783 ] Thomas Graves commented on SPARK-30987: --- all of these fail in just starting up the local-cluster.

[jira] [Created] (SPARK-30987) ResourceDiscoveryPluginSuite sometimes fails

Thomas Graves created SPARK-30987: - Summary: ResourceDiscoveryPluginSuite sometimes fails Key: SPARK-30987 URL: https://issues.apache.org/jira/browse/SPARK-30987 Project: Spark Issue Type:

[jira] [Updated] (SPARK-30977) ResourceProfile and Builder should be private in spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-30977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-30977: -- Target Version/s: 3.0.0 > ResourceProfile and Builder should be private in spark 3.0 >

[jira] [Assigned] (SPARK-30977) ResourceProfile and Builder should be private in spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-30977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-30977: - Assignee: (was: Thomas Graves) > ResourceProfile and Builder should be private in

[jira] [Commented] (SPARK-30977) ResourceProfile and Builder should be private in spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-30977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046891#comment-17046891 ] Thomas Graves commented on SPARK-30977: --- I'm working on this should have pr by end of day >

[jira] [Created] (SPARK-30977) ResourceProfile and Builder should be private in spark 3.0

Thomas Graves created SPARK-30977: - Summary: ResourceProfile and Builder should be private in spark 3.0 Key: SPARK-30977 URL: https://issues.apache.org/jira/browse/SPARK-30977 Project: Spark

[jira] [Resolved] (SPARK-30942) Fix the warning for requiring cores to be limiting resource

[ https://issues.apache.org/jira/browse/SPARK-30942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30942. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed > Fix the

[jira] [Commented] (SPARK-30322) Add stage level scheduling docs

[ https://issues.apache.org/jira/browse/SPARK-30322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044561#comment-17044561 ] Thomas Graves commented on SPARK-30322: --- Document the yarn priority behavior > Add stage level

[jira] [Updated] (SPARK-30942) Fix the warning for requiring cores to be limiting resource

[ https://issues.apache.org/jira/browse/SPARK-30942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-30942: -- Summary: Fix the warning for requiring cores to be limiting resource (was: Fix the warning

[jira] [Created] (SPARK-30942) Fix the warning for requireing cores to be limiting resource

Thomas Graves created SPARK-30942: - Summary: Fix the warning for requireing cores to be limiting resource Key: SPARK-30942 URL: https://issues.apache.org/jira/browse/SPARK-30942 Project: Spark

[jira] [Created] (SPARK-30831) Executors UI shows more active tasks then possible

Thomas Graves created SPARK-30831: - Summary: Executors UI shows more active tasks then possible Key: SPARK-30831 URL: https://issues.apache.org/jira/browse/SPARK-30831 Project: Spark Issue

[jira] [Resolved] (SPARK-29148) Modify dynamic allocation manager for stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29148. --- Fix Version/s: 3.1.0 Resolution: Fixed > Modify dynamic allocation manager for stage

[jira] [Created] (SPARK-30794) Stage Level scheduling: Add ability to set off heap memory

Thomas Graves created SPARK-30794: - Summary: Stage Level scheduling: Add ability to set off heap memory Key: SPARK-30794 URL: https://issues.apache.org/jira/browse/SPARK-30794 Project: Spark

[jira] [Commented] (SPARK-28845) Enable spark.sql.execution.sortBeforeRepartition only for retried stages

[ https://issues.apache.org/jira/browse/SPARK-28845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034755#comment-17034755 ] Thomas Graves commented on SPARK-28845: --- [~cloud_fan] [~XuanYuan] I wanted to followup on this

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034490#comment-17034490 ] Thomas Graves commented on SPARK-24615: --- This is purely a scheduling feature and Spark will assign

[jira] [Commented] (SPARK-24655) [K8S] Custom Docker Image Expectations and Documentation

[ https://issues.apache.org/jira/browse/SPARK-24655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031870#comment-17031870 ] Thomas Graves commented on SPARK-24655: --- some other discussions on this from

[jira] [Created] (SPARK-30750) stage level scheduling: Add ability to set dynamic allocation configs

Thomas Graves created SPARK-30750: - Summary: stage level scheduling: Add ability to set dynamic allocation configs Key: SPARK-30750 URL: https://issues.apache.org/jira/browse/SPARK-30750 Project:

[jira] [Updated] (SPARK-30749) stage level scheduling: Better cleanup of Resource profiles

[ https://issues.apache.org/jira/browse/SPARK-30749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-30749: -- Summary: stage level scheduling: Better cleanup of Resource profiles (was: Better cleanup of

[jira] [Created] (SPARK-30749) Better cleanup of Resource profiles

Thomas Graves created SPARK-30749: - Summary: Better cleanup of Resource profiles Key: SPARK-30749 URL: https://issues.apache.org/jira/browse/SPARK-30749 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031637#comment-17031637 ] Thomas Graves commented on SPARK-24615: --- yes it will be in 3.0, the feature is complete other then

[jira] [Created] (SPARK-30742) Resource discovery should protect against user returing empty string for address

Thomas Graves created SPARK-30742: - Summary: Resource discovery should protect against user returing empty string for address Key: SPARK-30742 URL: https://issues.apache.org/jira/browse/SPARK-30742

[jira] [Resolved] (SPARK-30689) Allow custom resource scheduling to work with YARN versions that don't support custom resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-30689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30689. --- Fix Version/s: 3.0.0 Assignee: Thomas Graves Resolution: Fixed > Allow

[jira] [Assigned] (SPARK-30511) Spark marks intentionally killed speculative tasks as pending leads to holding idle executors

[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-30511: - Assignee: Zebing Lin > Spark marks intentionally killed speculative tasks as pending

<    1   2   3   4   5   6   7   8   9   10   >